adriangb commented on PR #25752: URL: https://github.com/apache/datafusion/pull/25752#issuecomment-5834857652
## Benchmark summary Branch head 641d75866a against its merge-base on `main`. Times are the min of 5 iterations, in ms. Ratio = branch / main. **Bold** = a change of more than 7%. ClickBench query numbers are the bot numbers (from 0). | Run | Trigger | main | branch | |---|---|---|---| | 1 | [pushdown on](https://github.com/apache/datafusion/pull/25752#issuecomment-5834580872) | defaults (`pushdown_filters = false`) | `pushdown_filters = true` | | 2 | [defaults](https://github.com/apache/datafusion/pull/25752#issuecomment-5834621883) | defaults | defaults | ### Run 1: branch with pushdown on vs. main with pushdown off | Suite | main | branch | branch / main | faster / slower > 7% | slowest query | Criterion (no query slower) | |---|---|---|---|---|---|---| | [TPC-H SF1](https://github.com/apache/datafusion/pull/25752#issuecomment-5834736488) | 730 | 738 | 1.01 | 3 / 10 | Q9 1.32 | Fails (10 slower) | | [TPC-H SF10](https://github.com/apache/datafusion/pull/25752#issuecomment-5834755561) | 6,617 | 6,438 | 0.97 | 4 / 8 | Q4 1.17 | Fails (8 slower) | | [TPC-DS SF1](https://github.com/apache/datafusion/pull/25752#issuecomment-5834754976) | 8,807 | 6,741 | **0.77** | 38 / 24 | Q87 1.28 | Fails (24 slower) | | [ClickBench partitioned](https://github.com/apache/datafusion/pull/25752#issuecomment-5834807050) | 19,523 | 16,123 | **0.83** | 7 / 1 | Q25 1.26 | Fails (1 slower) | | Suite | Faster > 7% | Slower > 7% | |---|---|---| | TPC-H SF1 | Q17 0.61, Q18 0.88, Q19 0.81 | Q2 1.11, Q3 1.18, Q4 1.17, Q5 1.15, Q7 1.15, Q9 1.32, Q11 1.17, Q14 1.09, Q15 1.10, Q22 1.13 | | TPC-H SF10 | Q6 0.91, Q12 0.91, Q18 0.46, Q20 0.91 | Q2 1.13, Q3 1.15, Q4 1.17, Q5 1.14, Q9 1.13, Q10 1.09, Q14 1.09, Q22 1.09 | | TPC-DS SF1 | Q1 0.83, Q2 0.52, Q3 0.90, Q4 0.78, Q5 0.92, Q6 0.85, Q7 0.80, Q8 0.40, Q11 0.70, Q15 0.39, Q16 0.92, Q18 0.88, Q20 0.87, Q24 0.89, Q26 0.88, Q40 0.82, Q45 0.70, Q49 0.72, Q54 0.52, Q56 0.89, Q57 0.80, Q58 0.66, Q59 0.67, Q62 0.86, Q64 0.81, Q65 0.69, Q66 0.83, Q71 0.92, Q72 0.06, Q74 0.88, Q78 0.73, Q80 0.81, Q83 0.89, Q91 0.79, Q94 0.92, Q95 0.67, Q98 0.90, Q99 0.82 | Q10 1.17, Q13 1.10, Q17 1.17, Q21 1.13, Q23 1.14, Q29 1.11, Q31 1.12, Q35 1.16, Q37 1.26, Q38 1.26, Q48 1.19, Q50 1.25, Q51 1.08, Q69 1.26, Q70 1.10, Q75 1.12, Q76 1.12, Q82 1.12, Q85 1.18, Q86 1.17, Q87 1.28, Q89 1.22, Q90 1.10, Q96 1.10 | | ClickBench partitioned | Q22 0.85, Q23 0.08, Q24 0.68, Q32 0.88, Q33 0.93, Q34 0.92, Q42 0.92 | Q25 1.26 | Conclusion: the totals are the same or better in all suites (TPC-DS 0.77, ClickBench 0.83), but the criterion fails in all suites: 43 queries are more than 7% slower than main with pushdown off (worst 1.32). ### Run 2: branch vs. main, both with defaults (pushdown off) | Suite | main | branch | branch / main | faster / slower > 7% | slowest query | Criterion (no query slower) | |---|---|---|---|---|---|---| | [TPC-H SF1](https://github.com/apache/datafusion/pull/25752#issuecomment-5834737264) | 739 | 742 | 1.00 | 4 / 0 | Q15 1.05 | Holds | | [TPC-H SF10](https://github.com/apache/datafusion/pull/25752#issuecomment-5834755124) | 6,307 | 6,326 | 1.00 | 3 / 0 | Q15 1.05 | Holds | | [TPC-DS SF1](https://github.com/apache/datafusion/pull/25752#issuecomment-5834758745) | 8,983 | 9,035 | 1.01 | 4 / 9 | Q37 1.25 | Fails (9 slower) | | [ClickBench partitioned](https://github.com/apache/datafusion/pull/25752#issuecomment-5834816001) | 19,730 | 17,185 | **0.87** | 5 / 6 | Q15 1.22 | Fails (6 slower) | | Suite | Faster > 7% | Slower > 7% | |---|---|---| | TPC-H SF1 | Q6 0.90, Q12 0.91, Q16 0.92, Q19 0.91 | - | | TPC-H SF10 | Q6 0.88, Q12 0.84, Q19 0.88 | - | | TPC-DS SF1 | Q1 0.90, Q43 0.93, Q61 0.92, Q65 0.53 | Q12 1.10, Q21 1.07, Q32 1.10, Q37 1.25, Q38 1.11, Q53 1.07, Q73 1.12, Q82 1.14, Q87 1.09 | | ClickBench partitioned | Q0 0.81, Q23 0.22, Q24 0.93, Q26 0.93, Q32 0.91 | Q10 1.07, Q13 1.13, Q14 1.17, Q15 1.22, Q36 1.08, Q39 1.09 | Conclusion: at the default configuration, the stack does not change the TPC-H and TPC-DS totals, and ClickBench is 13% faster. The criterion holds on TPC-H. TPC-DS and ClickBench have 15 small regressions (worst 1.25). Some of them are in the bot noise for short queries. ### Key queries | Query | main, run 1 | branch, pushdown on | ratio | main, run 2 | branch, defaults | ratio | |---|---|---|---|---|---|---| | TPC-H SF1 Q6 | 16.1 | 15.3 | 0.95 | 16.3 | 14.7 | **0.90** | | TPC-H SF1 Q12 | 21.2 | 20.7 | 0.97 | 20.9 | 19.0 | **0.91** | | TPC-H SF1 Q4 | 16.9 | 19.8 | **1.17** | 17.3 | 18.0 | 1.04 | | TPC-H SF1 Q17 | 72.4 | 43.9 | **0.61** | 73.6 | 69.1 | 0.94 | | TPC-H SF1 Q18 | 59.7 | 52.7 | **0.88** | 59.7 | 62.5 | 1.05 | | TPC-H SF10 Q6 | 127 | 116 | **0.91** | 124 | 109 | **0.88** | | TPC-H SF10 Q12 | 151 | 138 | **0.91** | 148 | 124 | **0.84** | | TPC-H SF10 Q4 | 115 | 135 | **1.17** | 111 | 115 | 1.03 | | TPC-H SF10 Q17 | 623 | 623 | 1.00 | 575 | 573 | 1.00 | | TPC-H SF10 Q18 | 756 | 350 | **0.46** | 704 | 700 | 0.99 | | TPC-DS SF1 Q9 | 53.9 | 56.8 | 1.05 | 53.3 | 54.4 | 1.02 | | TPC-DS SF1 Q17 | 76.3 | 89.3 | **1.17** | 78.2 | 80.3 | 1.03 | | TPC-DS SF1 Q50 | 56.5 | 70.7 | **1.25** | 57.4 | 57.8 | 1.01 | | TPC-DS SF1 Q64 | 351 | 284 | **0.81** | 359 | 361 | 1.01 | | TPC-DS SF1 Q65 | 128 | 88.2 | **0.69** | 131 | 69.7 | **0.53** | | TPC-DS SF1 Q72 | 1,700 | 104 | **0.06** | 1,771 | 1,770 | 1.00 | | TPC-DS SF1 Q82 | 15.8 | 17.7 | **1.12** | 16.2 | 18.4 | **1.14** | | ClickBench Q10 | 63.5 | 63.6 | 1.00 | 63.9 | 68.4 | **1.07** | | ClickBench Q23 | 3,157 | 260 | **0.08** | 3,150 | 690 | **0.22** | | ClickBench Q24 | 40.2 | 27.3 | **0.68** | 42.8 | 39.8 | **0.93** | | ClickBench Q25 | 105 | 133 | **1.26** | 114 | 107 | 0.94 | | ClickBench Q26 | 41.0 | 38.8 | 0.95 | 42.8 | 39.8 | **0.93** | Conclusion: with pushdown on, TPC-H Q6 and Q12 are no longer regressions (0.91 to 0.97). TPC-H Q4, TPC-DS Q17, Q50 and Q82, and ClickBench Q25 are still slower than main with pushdown off. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
