HyukjinKwon opened a new pull request, #59931: URL: https://github.com/apache/spark/pull/59931
### What changes were proposed in this pull request? Follow-ups to SPARK-60101 (#59424) that change existing functions of version 1 of `spark_datasource.h`, or the meaning of a flag, so they cannot be added later within version 1. Version 1 first ships in 4.4.0. They come from @gengliangwang's review of the C ABI from the Rust SDK side (#59520), in https://github.com/apache/spark/pull/59424#issuecomment-6480975990: 1. **The executors get the schema, and whether a read is batch or streaming.** `read` gets a size-prefixed `SparkDsReadInfo` with: - the kind of the read, `SPARK_DS_READ_BATCH` or `SPARK_DS_READ_MICRO_BATCH`; - the schema to read; - the state of the reader and the partition. `create_data_writer` gets a `SparkDsDataWriterInfo` with the schema to write, the state of the writer and the epoch ID. `write` then gets only the array of each batch. 2. **`SPARK_DS_TRUNCATE` is split.** `SPARK_DS_BATCH_TRUNCATE` covers the save mode `overwrite`, and `SPARK_DS_STREAMING_TRUNCATE` the output mode `complete`. A library can then support one without the other. 3. **The planning calls return the state.** `partitions`, `stream_partitions` and `create_writer` return the state of the reader or the writer through a `set_state` sink. This replaces `serialize_reader`, `serialize_stream_reader` and `serialize_writer`, which Spark always called right after them. Spark rejects a state that is set more than once. 4. **The stream reader calls take size-prefixed inputs.** `create_stream_reader` gets a `SparkDsStreamReaderInfo` with the checkpoint location of the source. `latest_offset` gets a `SparkDsLatestOffsetInfo` with the offset that the next micro-batch starts after. Admission control can then be added later as a field and a capability. On the Spark side, the micro-batch stream implements `SupportsAdmissionControl.latestOffset(start, limit)` and `reportLatestOffset`. The header, the FFM bindings in `NativeLibrary`, the C++ test library, the docs and the Rust and C++ examples are updated accordingly. ### Why are the changes needed? Each change removes code from every SDK and every library: - schemas carried in the state of readers and writers; - a separate serialize call per handle; - tracking the previous offset on the library side. Since they change signatures or the meaning of a flag, they can only land before version 1 ships. Afterwards, a change like this needs a new version of the interface. ### Does this PR introduce _any_ user-facing change? No for released versions: native data sources are new in the unreleased 4.4.0. A library built against the header of master has to be rebuilt against the new header. ### How was this patch tested? New tests, and existing tests updated: - `NativeDataSourceSuite`: - "pass the checkpoint location and the start of a micro-batch to the stream reader" also restarts the query from its checkpoint. - "reject a state returned more than once" covers `partitions`, `stream_partitions` and `create_writer`. - The truncate checks of "operations that a data source does not support" now cover each flag separately. - The test library checks that `read` and `create_data_writer` get the kind, the schema and the size of their structs. - `ColumnarDataSourceSuite`: "the stream reader gets its checkpoint location and the start of each micro-batch". The Rust (arrow-rs 58.4.1) and C++ (Arrow C++ 26) examples of the docs were built and run against the new header. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
