Shekharrajak commented on PR #3903: URL: https://github.com/apache/datafusion-comet/pull/3903#issuecomment-4196364000
After fix the benchmark table : 11.9% overall improvement: the cached file handle optimization -- the finalize path opens each spill file once (S opens) instead of once per partition (N*S opens). Output sizes matching. **Configuration:** 5M rows, 10 columns, 200 partitions, zstd(1), 50 MB memory limit, 10 spill events, 3 iterations + 1 warmup | Metric | main (baseline) | This PR (optimized) | Delta | |--------|----------------|---------------------|-------| | **avg write time** | 1.354s | 1.193s | **-11.9%** | | **min write time** | 1.323s | 1.186s | **-10.4%** | | **max write time** | 1.387s | 1.196s | **-13.8%** | | **throughput** | 3,693,244 rows/s | 4,192,690 rows/s | **+13.5%** | | write time (I/O) | 0.111s (8.2%) | 0.114s (9.6%) | +2.7% (noise) | | encode time | 0.747s (55.2%) | 0.730s (61.3%) | -2.3% (noise) | | repart time | 0.040s (3.0%) | 0.042s (3.5%) | +5% (noise) | | output size | 226.16 MiB | 226.21 MiB | identical | | spill count | 10 | 10 | identical | | spilled bytes | -- | 221.74 MiB | -- | | data size | 440.36 MiB | 440.36 MiB | identical | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
