Hi Hyukjin, Thanks for putting this proposal together. I think native data source support is a great direction for Spark.
One suggestion on the interface: could we provide *idiomatic Rust and C++ SDKs* on top of the ABI proposed in #59313 <https://github.com/apache/spark/pull/59313>, and make those the primary APIs for connector authors? The SDKs could handle JNI bindings, native handles, plan serialization, error conversion, and Arrow C Stream ownership. Connector authors would implement schema discovery, scan planning, pushdown, and partition readers that return Arrow batches. This would preserve the ability to write connectors entirely in Rust or C++, while centralizing the bridge code. I put together a draft Rust SDK prototype (#59442) <https://github.com/apache/spark/pull/59442> to illustrate the idea. It uses the proposed ABI and covers batch reads, including filter/limit pushdown, column pruning, and partition planning. The example connector implements two Rust traits and uses a registration macro to generate the JNI exports. Would this be a useful direction to explore alongside the PR? Thanks, Gengliang On Sun, Oct 11, 2026 at 4:34 PM Holden Karau <[email protected]> wrote: > Oh this is good, it has some similar parts to ADBC data source here with > the possible native code. > > > Twitter: https://twitter.com/holdenkarau > Fight Health Insurance: https://www.fighthealthinsurance.com/ > <https://www.fighthealthinsurance.com/?q=hk_email> > Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau > Pronouns: she/her > > On Sun, Oct 11, 2026 at 11:38 PM Hyukjin Kwon <[email protected]> > wrote: > >> Hi all, >> >> I have an open PR (https://github.com/apache/spark/pull/59313) and I >> would like others to take a look especially the interface part. >> >> *TL;DR:* >> >> - It is JNI binding. There are example implementations of Data >> Sources in Rust and C++ in PR description. >> - It will look up the PATH with spark*datasource** native library >> (like how we do in Python Data Source), and register it automatically. >> Otherwise, users can manually add it via spark.addArtifact >> - The performance, from my benchmark, can improve: read side by >> around 2x: write side 10%. >> - Downstream Data Sources would be implemented in pure C++ or pure >> Rust, etc. >> >> *There might be some other aspects to consider e.g.,:* >> >> - There could be some design options, e.g., the downstream Data >> Sources could be mixed with Java. In that case, we could provide a better >> interface, or, we wouldn’t even need this PR at all because downstream >> Data >> Sources could already have JNI binding itself on their own. >> - Using FFM instead of JNI could be an option too. For now, 1. it is >> using JNI to avoid having a header file in diff languages and 2. to >> support >> lower JDK. I plan to release version 2 with FFM in Spark 5. >> >> Please review this PR - I plan to open this awhile. >> >
