pingzh opened a new pull request, #5501: URL: https://github.com/apache/datafusion-comet/pull/5501
## Which issue does this PR close? Part of #5352. This is the fifth foundational PR and does not close the issue. Previous PRs: - https://github.com/apache/datafusion-comet/pull/5473 - https://github.com/apache/datafusion-comet/pull/5476 - https://github.com/apache/datafusion-comet/pull/5481 - https://github.com/apache/datafusion-comet/pull/5491 ## Rationale for this change Previous PRs introduced backend-neutral RSS partition writers, destination-aware shuffle plans, native shuffle execution, and task-owned JVM callbacks. Integrating Apache Celeborn additionally requires a Celeborn-specific partition pusher and a shuffle manager that preserves existing Celeborn behavior for ordinary Spark shuffles. ## What changes are included in this PR? - Add `CelebornShufflePartitionPusher`, which: - Sends complete native shuffle frames through Celeborn’s existing client. - Captures shuffle, mapper, attempt, and partition metadata. - Validates frame boundaries, partition identifiers, and transport byte counts. - Preserves original client exceptions across reflective calls. - Avoids introducing a compile-time Celeborn dependency. - Add `CometCelebornShuffleManager`, which: - Loads the existing Celeborn shuffle manager reflectively. - Delegates ordinary Spark shuffle registration, writing, reading, cleanup, and shutdown. - Rejects unsupported Comet shuffle paths until subsequent integration PRs. - Fails clearly when the optional Celeborn client is unavailable. - Register the new test suites in Linux and macOS CI workflows. Map-side lifecycle management, Celeborn shuffle reading, and native-only planning will be addressed in subsequent PRs. ## How are these changes tested? - Added 11 partition-pusher tests covering frame validation, task metadata, transport accounting, client failures, and worker-thread execution. - Added 12 shuffle-manager tests covering delegation, reader ranges, lifecycle handling, unsupported shuffle paths, and missing Celeborn dependencies. - Passed all 35 existing native shuffle tests. - Passed all 58 tests across the three relevant suites. - Passed ScalaStyle, Spotless, Apache license-header verification, and CI suite-registration checks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
