Dandandan commented on PR #2338: URL: https://github.com/apache/datafusion-ballista/pull/2338#issuecomment-5318674267
A/B for the rule as it now stands, TPC-H SF10, one scheduler + two executors x 4 vcores, AQE with `collect_statistics=true` and `max_partitions_per_task=0`. Three rounds, variants alternated per round so both share any machine drift, three iterations each — 9 samples per variant per query, medians below. | Query | base | this PR | ratio | | ----: | ---: | ------: | ----: | | 2 | 0.449 | 0.420 | 0.94 | | 7 | 2.016 | 1.921 | 0.95 | | 8 | 0.614 | 0.623 | 1.01 | | 12 | 0.899 | 0.843 | 0.94 | | 19 | 0.698 | 0.651 | 0.93 | | 20 | 1.463 | 1.455 | 0.99 | | **sum of medians** | **6.14s** | **5.91s** | **0.963** | Read this as "no regressions" rather than as a 4% win. q12 and q19 keep their stage boundaries under this rule, so their plans are byte-identical between the two variants — the ~5% they move is the noise floor of this setup, which means q7 and q2 are inside it too. A single machine at SF10 is the wrong instrument for the thing being removed: the shuffle write lands in the page cache and the barrier costs almost nothing when every task is milliseconds from its input. What is not noise is the earlier revision that inlined filtered scans as well. On the same harness it measured q19 at **1.14x** — inlining a `part` scan filtered to a fraction of a percent made every consuming task rebuild what one stage had materialized. That is what the `Precision::Exact` guard is for, and q19 is now back at parity. Plan effect, unchanged from the description: q7 goes from 8 stages to 6, and q19's filtered `part` build side keeps its boundary. Row counts match the pre-change run on every query in the table. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
