andygrove commented on PR #6937: URL: https://github.com/apache/datafusion-comet/pull/6937#issuecomment-6175840728
@viirya here are end-to-end numbers. I ran `CometIcebergEndToEndBenchmark`'s corpus, session and statement with release builds of `main` and of this PR, two alternating runs of each on an M3 Max. The PR has no JVM changes, so only libcomet differed. Besides the benchmark's unpartitioned target, I added a fanout case that writes a target partitioned by `bucket(16, id)` with the fanout writer. Each task's 210,000 or so rows split into 16 partitions of about 13,000, so every row goes through the pre-pass: 160 pre-passes per statement, against 10 for the unpartitioned target. Whole-statement times were noisy on a shared machine. The iceberg-java writer, which doesn't run the chooser, moved by up to 7% between runs. So I also logged the CPU time of every pre-pass inside the pipeline, summed it per statement, and took the median of 14 statements per build. The pre-pass's share of a statement's CPU time: | Types | Unpartitioned, `main` | This PR | Fanout, `main` | This PR | | ------------ | --------------------- | ------- | -------------- | ------- | | `primitives` | 2.1% | 0.6% | 13.8% | 5.6% | | `structs` | 2.8% | 1.1% | 17.8% | 9.1% | | `arrays` | 2.6% | 1.0% | 19.0% | 9.0% | | `maps` | 2.3% | 1.4% | 17.2% | 11.1% | | `nested` | 2.2% | 1.3% | 17.8% | 11.0% | For the stock, unpartitioned benchmark that saves 0.9% to 1.8% of the CPU. That matches the estimate, and it's too small for wall time to resolve here. For the fanout write it saves 7.5% to 11.4%, and median statement CPU fell 6% to 16% and wall time 1% to 8%. So the change matters mostly for fanout writes into many partitions, where every row is sampled. I've put these numbers in the description in place of the estimate. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
