andygrove opened a new issue, #6913: URL: https://github.com/apache/datafusion-comet/issues/6913
### What is the problem the feature request solves? When a native Iceberg write follows a Comet shuffle, `CometIcebergWriteExec.doExecuteColumnar` takes `child.executeColumnar()` and hands it to the native plan (IcebergWrite, Projection, Scan) as an Arrow C stream. So the shuffle blocks are read by `CometShuffleBlockIterator` in the JVM, decoded through `decode_shuffle_block_jni`, exported to the JVM over FFI, imported again by the native `ScanExec`, and string columns are validated as UTF-8 a second time (`decode_string_arrays`). Shuffle direct read (`spark.comet.shuffle.directRead.enabled`, on by default) exists to avoid that round trip, but it is wired only through `CometExchangeSink` for plans built by `CometNativeExec`, so a native Iceberg write never uses it. In `CometIcebergEndToEndBenchmark` (#6910), the JVM side of the round trip, excluding the decode itself, is 1.7% of all CPU for nested types and 2.3% for primitives, plus about 0.3% for the import and the UTF-8 check. ### Describe the potential solution Give the native write plan a `ShuffleScan` input when the writer's child is a Comet shuffle stage, as `CometExchangeSink.convertToShuffleScan` does, so shuffle blocks are decoded natively and fed straight into the writer. ### Additional context `spark/src/main/scala/org/apache/spark/sql/comet/CometIcebergWriteExec.scala` (`doExecuteColumnar`) and `spark/src/main/scala/org/apache/comet/serde/operator/CometSink.scala` (`shouldUseShuffleScan`, `convertToShuffleScan`). Measured on the fully native configuration of `CometIcebergEndToEndBenchmark`. Part of #6911. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
