pingzh opened a new pull request, #5481: URL: https://github.com/apache/datafusion-comet/pull/5481
## Which issue does this PR close? Part of #5352. This is the third foundational PR and does not close the issue. Previous PRs: - https://github.com/apache/datafusion-comet/pull/5473 - https://github.com/apache/datafusion-comet/pull/5476 ## Rationale for this change Previous PRs introduced the RSS partition writer and destination-aware shuffle plans, but native shuffle execution still supports only local files. This PR enables the existing shuffle execution path to write to either local files or a remote shuffle service. ## What changes are included in this PR? - Introduce `ShuffleWriterDestination` with local and RSS destinations. - Add a destination-aware `ShuffleWriterExec` constructor while preserving the existing local constructor. - Reuse existing partitioners for both local and remote shuffle output. - Preserve typed callback errors and enforce RSS frame-size limits. - Update spill logging and metrics documentation for remote destinations. - Add end-to-end coverage for compression codecs, partitioning strategies, empty schemas, remote spilling, callback failures, and local compatibility. Binding RSS destinations to task-owned JVM callbacks will be addressed in a subsequent PR. ## How are these changes tested? - Added 10 end-to-end RSS execution tests. - Passed 98 Rust tests across shuffle, JNI bridge, protobuf, and planner suites. - Passed all 31 tests in `CometNativeShuffleSuite`. - Passed workspace-wide Clippy, Rust formatting checks, and the full Maven reactor build. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
