zhuqi-lucas commented on PR #23690: URL: https://github.com/apache/datafusion/pull/23690#issuecomment-5011497590
Closing this draft as premature. The core motivation (SIGMOD 2025 paper's claim that DataFusion's Full Compaction is suboptimal) is theoretically sound, and the microbenchmark I added shows the mechanism (smaller bypass threshold → less memcpy on medium-sized batches). But when I A/B tested against DataFusion's actual join benchmarks (`sort_merge_join`), the results came in within noise (±5%), because those benches use a single-level join and don't reproduce the deep chunk-reducing pipelines where the paper reports its wins. Without a DataFusion-native benchmark that reflects the paper's target workload (deep chained hash joins, similar to Join Order Benchmark), there is no way to demonstrate an end-to-end win to justify the API change here. The right sequence is probably: 1. Add a JOB-style deep-join benchmark to `datafusion-benchmarks`. 2. Baseline it. If the paper's prediction holds, the numbers will show it. 3. Then revisit this configurability change with concrete evidence. Happy to help with any of the above later. For now, closing to avoid clutter. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
