andygrove opened a new issue, #6911: URL: https://github.com/apache/datafusion-comet/issues/6911
### What is the problem the feature request solves? #6910 adds `CometIcebergEndToEndBenchmark`, which times an Iceberg read, a `REPARTITION(10, id)` hash shuffle and an Iceberg write over five groups of column types. With the native scan, shuffle and writer, Comet runs it 2.0x (primitives) to 2.9x (nested types) faster than Spark. Profiling that pipeline shows where the remaining time goes. This epic collects the findings so they can be worked and tracked in one place. Share of all sampled CPU per iteration, with the native scan, shuffle and writer: | Component | Nested types | Primitives | | --- | --- | --- | | Native writer: Parquet encoding (parquet-rs) | 41% | 25% | | Native Iceberg scan: Parquet decoding | 14% | 11% | | Shuffle write: LZ4 compression and IPC encoding | 13% | 14% | | Shuffle write: interleave gather | 9% | 8% | | Native writer: dictionary pre-pass, flush, other | 4% | 6% | | Shuffle read decode | 2% | 3% | Comet is already cheaper than Spark in every stage. For nested types, in CPU-seconds per iteration: scan 0.8 against 2.9, shuffle write 1.3 against 4.2, shuffle read 0.1 against 1.5, Parquet write 2.5 against 5.9. Each statement also pays about 95 ms of fixed driver time, about 66 ms of it the iceberg-java commit, which every configuration pays. Turning native writes on by default is tracked in #5644 (PR #6664) and is not part of this epic. ### Describe the potential solution Sub-issues are linked below as they are filed. ### Additional context Measured on `main` at 907c0e074 with #6910 applied: Spark 4.1, Iceberg 1.11, `local[5]` on an Apple M3 Ultra, 2M rows, snappy-compressed tables, async-profiler 4.3 CPU sampling at 2 ms plus a wall-clock pass over the driver thread. The percentages include some JIT compilation (5% to 10% of CPU). Each sub-issue gives the measurement it rests on, and `CometIcebergEndToEndBenchmark` is the A/B harness for any fix. Related: #5649. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
