Oh this is good, it has some similar parts to ADBC data source here with the possible native code.
Twitter: https://twitter.com/holdenkarau Fight Health Insurance: https://www.fighthealthinsurance.com/ <https://www.fighthealthinsurance.com/?q=hk_email> Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> YouTube Live Streams: https://www.youtube.com/user/holdenkarau Pronouns: she/her On Sun, Oct 11, 2026 at 11:38 PM Hyukjin Kwon <[email protected]> wrote: > Hi all, > > I have an open PR (https://github.com/apache/spark/pull/59313) and I > would like others to take a look especially the interface part. > > *TL;DR:* > > - It is JNI binding. There are example implementations of Data Sources > in Rust and C++ in PR description. > - It will look up the PATH with spark*datasource** native library > (like how we do in Python Data Source), and register it automatically. > Otherwise, users can manually add it via spark.addArtifact > - The performance, from my benchmark, can improve: read side by around > 2x: write side 10%. > - Downstream Data Sources would be implemented in pure C++ or pure > Rust, etc. > > *There might be some other aspects to consider e.g.,:* > > - There could be some design options, e.g., the downstream Data > Sources could be mixed with Java. In that case, we could provide a better > interface, or, we wouldn’t even need this PR at all because downstream Data > Sources could already have JNI binding itself on their own. > - Using FFM instead of JNI could be an option too. For now, 1. it is > using JNI to avoid having a header file in diff languages and 2. to support > lower JDK. I plan to release version 2 with FFM in Spark 5. > > Please review this PR - I plan to open this awhile. >
