buraksenn opened a new pull request, #24224: URL: https://github.com/apache/datafusion/pull/24224
## Which issue does this PR close? - Closes #24196. ## Rationale for this change `ArrowSource` distinguishes between Arrow IPC file and stream formats, but this format was not serialized. As a result, a stream scan round-tripped through protobuf as a file scan, selecting the wrong opener and potentially allowing unsupported range-based repartitioning. ## What changes are included in this PR? - Add an `ArrowIpcFormat` discriminator to `ArrowScanExecNode`. - Serialize and restore the `ArrowSource` IPC format. - Decode payloads without the new field as file format, preserving the previous behavior. - Regenerate the prost and pbjson models. - Add regression coverage for file, stream, and older payloads without the format field. ## Are these changes tested? Yes: - `cargo test -p datafusion-proto --test proto_integration roundtrip_arrow` - `cargo test -p datafusion-proto --test proto_integration arrow_scan_without_format_field_decodes_as_file_format` - `cargo fmt --all` - Targeted all-feature clippy for `datafusion-datasource-arrow` and `datafusion-proto` ## Are there any user-facing changes? Arrow IPC stream scans now preserve their format across protobuf round trips. The binary protobuf change is additive. Older payloads continue to decode as file format. Stream preservation requires both producer and consumer to include this change because older versions do not carry or read the discriminator. The generated Rust `ArrowScanExecNode` struct gains a `format` field. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
