pingzh opened a new pull request, #5513: URL: https://github.com/apache/datafusion-comet/pull/5513
## Which issue does this PR close? Part of #5352. This is the sixth foundational PR and does not close the issue. Previous PRs: - https://github.com/apache/datafusion-comet/pull/5473 - https://github.com/apache/datafusion-comet/pull/5476 - https://github.com/apache/datafusion-comet/pull/5481 - https://github.com/apache/datafusion-comet/pull/5491 - https://github.com/apache/datafusion-comet/pull/5501 ## Rationale for this change The existing Celeborn integration provides the shuffle manager and partition-pusher foundation but does not yet complete the native map-side shuffle lifecycle. Production use requires bounded frames, executor-wide backpressure, asynchronous completion tracking, cancellation handling, speculative-attempt coordination, retry-safe commits, and support for nested Arrow data. ## What changes are included in this PR? - Wire native Comet shuffle writers into Celeborn without creating local shuffle files. - Add executor-wide byte admission covering native encoding, JNI copies, and Celeborn transport. - Track asynchronous push completion and preserve admission until requests actually complete. - Handle task cancellation, mapper completion, cleanup, and asynchronous transport failures. - Coordinate shuffle generations, speculative attempts, retries, and Spark output commits. - Handle stale commit ownership on Spark 3.4. - Support Celeborn 0.6 and 0.7 mapper-completion APIs. - Preserve existing CRC accounting, encrypted payload handling, and generic JNI callback compatibility. - Split oversized batches into complete row-aligned frames and compact nested Arrow dictionaries, lists, maps, and structs. - Add configurable limits: - `spark.comet.shuffle.rss.maxFrameBytes` - `spark.comet.shuffle.rss.maxInFlightBytes` - Register the new native shuffle writer suite in Linux and macOS CI. ## How are these changes tested? - 101 JVM tests across: - `CelebornShufflePartitionPusherSuite` - `CometCelebornShuffleManagerSuite` - `CometCelebornNativeShuffleWriterSuite` - `CometConfSuite` - `CometNativeShuffleSuite` - 67 Rust shuffle tests, including bounded frames, nested Arrow compaction, backpressure, failed writers, and legacy callbacks. - 17 native planner tests covering local and remote shuffle destinations. - 3 JNI bridge tests. - Scala formatting, Rust formatting, Scala style checks, and CI suite-registration checks all pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
