pingzh opened a new pull request, #5476: URL: https://github.com/apache/datafusion-comet/pull/5476
## Which issue does this PR close? Part of #5352. This is the second foundational PR and does not close the issue. Previous PR: - https://github.com/apache/datafusion-comet/pull/5473 ## Rationale for this change Comet’s native shuffle plans currently assume all partition data is written to local files. Supporting remote shuffle services requires plans to explicitly describe their output destination while preserving compatibility with existing local shuffle behavior. ## What changes are included in this PR? - Add a protobuf partition-writer descriptor supporting local and RSS destinations. - Serialize local shuffle plans with both the new descriptor and existing output-file fields for backward compatibility. - Update native planning to: - Support legacy plans without a partition-writer descriptor. - Resolve local output paths from the new descriptor. - Reject missing destinations, invalid paths, and conflicting legacy fields. - Reject RSS destinations until remote shuffle execution is introduced in a subsequent PR. - Preserve existing shuffle execution behavior. ## How are these changes tested? - Four protobuf tests verify local/RSS serialization and compatibility between legacy and updated plans. - Nine native planner tests cover valid destinations, legacy fallback, malformed plans, conflicting paths, and unsupported RSS destinations. - Three new JVM tests verify local, RSS, and legacy plan serialization. - All 31 native-shuffle JVM tests, 50 native shuffle tests, and 25 JNI tests pass. - Full-workspace Clippy, Rust formatting, ScalaStyle, Spotless, and the complete Maven reactor all pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
