andygrove commented on code in PR #6018: URL: https://github.com/apache/datafusion-comet/pull/6018#discussion_r4050388416
########## docs/source/contributor-guide/native_shuffle.md: ########## @@ -172,6 +194,63 @@ Native shuffle (`CometExchange`) is selected when all of the following condition 4. Arrow FFI transfers the `RecordBatch` to JVM as a `ColumnarBatch`. +## Direct Read (ShuffleScan) + +Direct read lets a native operator consume shuffle output without the batch ever being decoded in +the JVM or crossing Arrow FFI. It is controlled by `spark.comet.shuffle.directRead.enabled`, which +defaults to `true` and requires `spark.comet.shuffle.enabled`. It applies to both native shuffle and +JVM columnar shuffle, because both write the same Arrow IPC block format. + +### How the path is selected + +`CometExchangeSink.shouldUseShuffleScan` (`spark/src/main/scala/org/apache/comet/serde/operator/CometSink.scala`) +decides during plan serialization. When the sink's input is a Comet shuffle exchange, it emits a +`ShuffleScan` operator in place of the usual `Scan`. If any output type fails +`supportedSinkDataType`, it records the fallback reason `Unsupported data type for shuffle direct read` +and the slot serializes as a regular `Scan` instead. Review Comment: You're right, I had the boundary wrong. `convertToShuffleScan` returns `None` and `convert` never falls through to `super.convert`, and even if it did, the base `CometSink.convert` gates on the same `supportedSinkDataType`, so there is nothing to fall back to. The stage just stays as Spark's, via the `getOrElse(s)` in `CometExecRule`. Rewrote the paragraph in 67e734aad to say that, and moved the regular `Scan` description up to the case it actually describes, which is `shouldUseShuffleScan` returning false. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
