gengliangwang opened a new pull request, #59442:
URL: https://github.com/apache/spark/pull/59442
### What changes were proposed in this pull request?
Add a standalone Rust SDK prototype for **batch reads** under
`examples/src/main/rust/native-data-source`.
Connector authors implement `BatchSource` and `ScanBuilder`, return ordinary
Arrow
`RecordBatchReader` instances, and register the provider with
`export_data_source!(RangeSource)`. The SDK handles JNI exports, native
handles,
typed plan serialization, errors, and Arrow C Stream ownership. Rust traits
stay
inside the plugin; they are not a cross-library Rust ABI.
The prototype includes:
- Filter pushdown with `Exact`, `PruningOnly`, and `Unsupported` results;
global
versus per-partition limit support; and top-level column projection. The
bridge
declines limit pushdown when filters remain residual.
- A Range connector with lazy bounded batches, disjoint partitions, integer
filter pushdown, global limits, and empty/reordered projections.
- A `.sparkpkg` packaging helper, a Spark smoke script, and build/API
documentation.
**Runtime dependency:** the adapter targets the batch JNI ABI from #59313 at
`ff7b1ae0150fbcb3b6a7c67bea8accf6aa4ad9bd`. This PR does not include that
framework.
Running the connector in Spark requires a distribution containing it. The
standalone Rust and JNI tests do not require building Spark.
This draft is intended to evaluate a connector-facing SDK on top of the
proposed
native protocol. It does not establish a stable SDK or modify Spark's
Maven/SBT
build. Writes, Structured Streaming, C++ bindings, cancellation, and Spark
memory
accounting are outside this demo's scope. The README documents the pinned
framework's Arrow layout restrictions and the remaining integration work.
### Why are the changes needed?
Implementing native connectors directly against JNI requires each connector
to
repeat handle management, plan serialization, error conversion, and Arrow
lifecycle code. A Rust SDK lets connector authors focus on planning and
reading
while making pushdown semantics explicit. This working prototype gives
reviewers
a concrete API and connector to evaluate alongside #59313.
### Does this PR introduce _any_ user-facing change?
It adds an opt-in example and experimental Rust API. There is no change to
Spark's
built-in data sources or default build, and no published crate or
compatibility
commitment. With the required framework, the packaged example can be loaded
via
`spark.addArtifact(...)` and `spark.read.format("rust_sdk_range").load()`.
### How was this patch tested?
From `examples/src/main/rust/native-data-source`, using Rust 1.89.0 and JDK
17:
```bash
cargo build --workspace --locked
cargo test --workspace --locked
cargo clippy --workspace --all-targets --locked -- -D warnings
cargo fmt --all -- --check
python3 package.py \
--library target/debug/libspark_datasource_rust_sdk_range.so \
--output target/rust_sdk_range.sparkpkg
```
All 19 tests passed: 18 Rust unit tests and one integration test that
compiles a
minimal Java fixture, starts a real JVM with `-Xcheck:jni`, loads the
connector
shared library, and consumes its Arrow C Stream. Coverage includes serialized
plans, exact/residual filters, limits, empty projections, integer boundaries,
retained batch ownership after stream release, and provider errors/panics.
Also checked the package manifest and embedded library bytes, Python script
syntax, license headers, formatting, and whitespace.
The Spark smoke script is provided but **was not run**: the current worktree
is
based on master and does not contain #59313. The JNI fixture tests are not
full
Spark integration tests. Full Spark builds and Apache RAT were not run.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
