HarshilMital commented on issue #7072: URL: https://github.com/apache/datafusion-comet/issues/7072#issuecomment-6233870403
Hi @andygrove Benchmarked on a GCP `c4a-standard-8` (Axion ARM, dedicated vCPUs), `main` vs. this change (single commit on top of `main`), using: ``` cargo bench -p datafusion-comet-shuffle --bench shuffle_writer ``` | Benchmark | Change | p | |---|---|---| | end to end, Hash(16) | −0.03% | 0.92 | | end to end, Range(16) | −0.01% | 0.95 | | high partition, 200 | −0.05% | 0.49 | | high partition, 2000 | +0.31% | 0.34 | | high partition, 8000 | −1.1% | within noise threshold | No measurable difference on the paths that exercise `buffer_partitioned_batch_may_spill`. The other cases moved about ±1% in both directions. `write encoded (compression=None)` showed +14.5%, but it doesn't go through this code path, so I read it as code-layout noise from recompiling. Per the issue, I'm not opening a PR. Happy to if you'd still like the change for readability. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
