Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
Dandandan commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3905053370 > [#20160 (comment)](https://github.com/apache/datafusion/pull/20160#issuecomment-3902329306) > > This is the main improvement. Ok - yes I see some improvements here and there but it is still largely regressing main with ~30s (TPCDS runs in ~50s without and ~80s with filter pushdown). See e.g. this run https://github.com/apache/datafusion/pull/20318#issuecomment-3902690761 against main without dynamic filter pushdown. ``` │ QQuery 64 │ 1194.15 ms │ 31181.42 ms │ 26.11x slower │ ``` This ~26x regression (and many others) is still unchanged in this PR: ( ``` │ QQuery 64 │ 28583.66 ms │ 28523.14 ms │ no change │ ``` As we're running with ```DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS=true DATAFUSION_EXECUTION_PARQUET_REORDER_FILTERS=true``` also the main branch is showing the regressions - so we're comparing both "slow" versions. I think I now have an understanding why the current approaches adaptiveness isn't helping _that much_ yet. As we're only checking the filters on `open` it is only sorted / considered / discarded when the query consists of many files i.e. more files than threads. In other cases, it will evaluate / scan the columns regardless of the tracking (as it will open the files directly at the start of the query / query phase when the selectivity is yet unknown). I think for it to work effectively, it needs to integrate more with the parquet reader to remove or add a filter based on the adaptiveness _during_ the scan. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
adriangb commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3902629184 https://github.com/apache/datafusion/pull/20160#issuecomment-3902329306 This is the main improvement. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
Dandandan commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3902568680 > 🤖: Benchmark completed > > Details > > ``` > Comparing HEAD and dynamic-dynamic-filters > > Benchmark tpch_sf1.json > > ┏━━━┳━━━┳━┳━━━┓ > ┃ Query ┃ HEAD ┃ dynamic-dynamic-filters ┃Change ┃ > ┡━━━╇━━━╇━╇━━━┩ > │ QQuery 1 │ 190.43 ms │ 187.12 ms │ no change │ > │ QQuery 2 │ 99.64 ms │ 101.74 ms │ no change │ > │ QQuery 3 │ 177.58 ms │ 177.63 ms │ no change │ > │ QQuery 4 │ 139.07 ms │ 139.99 ms │ no change │ > │ QQuery 5 │ 305.90 ms │ 295.46 ms │ no change │ > │ QQuery 6 │ 206.58 ms │ 199.63 ms │ no change │ > │ QQuery 7 │ 276.96 ms │ 270.85 ms │ no change │ > │ QQuery 8 │ 323.63 ms │ 325.31 ms │ no change │ > │ QQuery 9 │ 428.27 ms │ 328.44 ms │ +1.30x faster │ > │ QQuery 10 │ 279.12 ms │ 281.80 ms │ no change │ > │ QQuery 11 │ 80.22 ms │78.44 ms │ no change │ > │ QQuery 12 │ 259.66 ms │ 268.57 ms │ no change │ > │ QQuery 13 │ 217.43 ms │ 213.38 ms │ no change │ > │ QQuery 14 │ 110.72 ms │ 107.96 ms │ no change │ > │ QQuery 15 │ 197.84 ms │ 198.16 ms │ no change │ > │ QQuery 16 │ 77.90 ms │78.08 ms │ no change │ > │ QQuery 17 │ 225.25 ms │ 227.57 ms │ no change │ > │ QQuery 18 │ 497.24 ms │ 332.51 ms │ +1.50x faster │ > │ QQuery 19 │ 153.84 ms │ 154.09 ms │ no change │ > │ QQuery 20 │ 158.26 ms │ 158.02 ms │ no change │ > │ QQuery 21 │ 357.71 ms │ 362.74 ms │ no change │ > │ QQuery 22 │ 65.88 ms │63.47 ms │ no change │ > └───┴───┴─┴───┘ > ┏┳━━━┓ > ┃ Benchmark Summary ┃ ┃ > ┡╇━━━┩ > │ Total Time (HEAD) │ 4829.13ms │ > │ Total Time (dynamic-dynamic-filters) │ 4550.95ms │ > │ Average Time (HEAD)│ 219.51ms │ > │ Average Time (dynamic-dynamic-filters) │ 206.86ms │ > │ Queries Faster │ 2 │ > │ Queries Slower │ 0 │ > │ Queries with No Change │20 │ > │ Queries with Failure │ 0 │ > └┴───┘ > ``` So an improvement, but still most queries are slower than without pushdown. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
Dandandan commented on code in PR #20160:
URL: https://github.com/apache/datafusion/pull/20160#discussion_r2807953987
##
datafusion/common/src/config.rs:
##
@@ -1115,6 +1115,34 @@ config_namespace! {
/// See:
pub hash_join_inlist_pushdown_max_distinct_values: usize, default = 150
+/// Minimum number of rows to process before making a selectivity
decision
+/// for adaptive filtering of join dynamic filters.
+///
+/// The filter will remain in a tracking state until this many rows
have been
+/// processed. This ensures statistical stability before making the
disable decision.
+/// Only used when `enable_adaptive_filter_selectivity_tracking` is
true.
+pub adaptive_filter_min_rows_for_selectivity: usize, default = 100_000
+
+/// Selectivity threshold for adaptive disabling of join dynamic
filters.
Review Comment:
Nvm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
alamb-ghbot commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3902554995 🤖: Benchmark completed Details ``` Comparing HEAD and dynamic-dynamic-filters Benchmark tpch_sf1.json ┏━━━┳━━━┳━┳━━━┓ ┃ Query ┃ HEAD ┃ dynamic-dynamic-filters ┃Change ┃ ┡━━━╇━━━╇━╇━━━┩ │ QQuery 1 │ 190.43 ms │ 187.12 ms │ no change │ │ QQuery 2 │ 99.64 ms │ 101.74 ms │ no change │ │ QQuery 3 │ 177.58 ms │ 177.63 ms │ no change │ │ QQuery 4 │ 139.07 ms │ 139.99 ms │ no change │ │ QQuery 5 │ 305.90 ms │ 295.46 ms │ no change │ │ QQuery 6 │ 206.58 ms │ 199.63 ms │ no change │ │ QQuery 7 │ 276.96 ms │ 270.85 ms │ no change │ │ QQuery 8 │ 323.63 ms │ 325.31 ms │ no change │ │ QQuery 9 │ 428.27 ms │ 328.44 ms │ +1.30x faster │ │ QQuery 10 │ 279.12 ms │ 281.80 ms │ no change │ │ QQuery 11 │ 80.22 ms │78.44 ms │ no change │ │ QQuery 12 │ 259.66 ms │ 268.57 ms │ no change │ │ QQuery 13 │ 217.43 ms │ 213.38 ms │ no change │ │ QQuery 14 │ 110.72 ms │ 107.96 ms │ no change │ │ QQuery 15 │ 197.84 ms │ 198.16 ms │ no change │ │ QQuery 16 │ 77.90 ms │78.08 ms │ no change │ │ QQuery 17 │ 225.25 ms │ 227.57 ms │ no change │ │ QQuery 18 │ 497.24 ms │ 332.51 ms │ +1.50x faster │ │ QQuery 19 │ 153.84 ms │ 154.09 ms │ no change │ │ QQuery 20 │ 158.26 ms │ 158.02 ms │ no change │ │ QQuery 21 │ 357.71 ms │ 362.74 ms │ no change │ │ QQuery 22 │ 65.88 ms │63.47 ms │ no change │ └───┴───┴─┴───┘ ┏┳━━━┓ ┃ Benchmark Summary ┃ ┃ ┡╇━━━┩ │ Total Time (HEAD) │ 4829.13ms │ │ Total Time (dynamic-dynamic-filters) │ 4550.95ms │ │ Average Time (HEAD)│ 219.51ms │ │ Average Time (dynamic-dynamic-filters) │ 206.86ms │ │ Queries Faster │ 2 │ │ Queries Slower │ 0 │ │ Queries with No Change │20 │ │ Queries with Failure │ 0 │ └┴───┘ ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
alamb-ghbot commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3902553598 🤖 `./gh_compare_branch.sh` [gh_compare_branch.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_branch.sh) Running Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Comparing dynamic-dynamic-filters (6b80e85272c6c020de4b0377d043418d27ca120c) to 7d217b1ec3bd4d447961e84bc3fe7f7708546b2d [diff](https://github.com/apache/datafusion/compare/7d217b1ec3bd4d447961e84bc3fe7f7708546b2d..6b80e85272c6c020de4b0377d043418d27ca120c) using: tpch Results will be posted here when complete -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
alamb-ghbot commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3902553546 🤖: Benchmark completed Details ``` Comparing HEAD and dynamic-dynamic-filters Benchmark clickbench_extended.json ┏━━┳━┳━┳━━━┓ ┃ Query┃HEAD ┃ dynamic-dynamic-filters ┃Change ┃ ┡━━╇━╇━╇━━━┩ │ QQuery 0 │ 2333.88 ms │ 2282.15 ms │ no change │ │ QQuery 1 │ 902.72 ms │ 901.98 ms │ no change │ │ QQuery 2 │ 1770.23 ms │ 1837.92 ms │ no change │ │ QQuery 3 │ 1026.75 ms │ 1035.71 ms │ no change │ │ QQuery 4 │ 2236.95 ms │ 2218.72 ms │ no change │ │ QQuery 5 │ 28820.03 ms │ 28276.07 ms │ no change │ │ QQuery 6 │ 106.63 ms │ 108.90 ms │ no change │ │ QQuery 7 │ 2607.37 ms │ 2589.24 ms │ no change │ └──┴─┴─┴───┘ ┏┳┓ ┃ Benchmark Summary ┃┃ ┡╇┩ │ Total Time (HEAD) │ 39804.56ms │ │ Total Time (dynamic-dynamic-filters) │ 39250.68ms │ │ Average Time (HEAD)│ 4975.57ms │ │ Average Time (dynamic-dynamic-filters) │ 4906.33ms │ │ Queries Faster │ 0 │ │ Queries Slower │ 0 │ │ Queries with No Change │ 8 │ │ Queries with Failure │ 0 │ └┴┘ Benchmark clickbench_partitioned.json ┏━━━┳━┳━┳━━━┓ ┃ Query ┃HEAD ┃ dynamic-dynamic-filters ┃Change ┃ ┡━━━╇━╇━╇━━━┩ │ QQuery 0 │ 2.81 ms │ 2.73 ms │ no change │ │ QQuery 1 │53.31 ms │53.74 ms │ no change │ │ QQuery 2 │ 135.72 ms │ 136.45 ms │ no change │ │ QQuery 3 │ 153.92 ms │ 158.02 ms │ no change │ │ QQuery 4 │ 1029.46 ms │ 1041.85 ms │ no change │ │ QQuery 5 │ 1281.12 ms │ 1317.28 ms │ no change │ │ QQuery 6 │18.04 ms │18.68 ms │ no change │ │ QQuery 7 │66.99 ms │69.21 ms │ no change │ │ QQuery 8 │ 1392.35 ms │ 1411.79 ms │ no change │ │ QQuery 9 │ 1765.44 ms │ 1855.18 ms │ 1.05x slower │ │ QQuery 10 │ 463.48 ms │ 493.12 ms │ 1.06x slower │ │ QQuery 11 │ 518.06 ms │ 532.45 ms │ no change │ │ QQuery 12 │ 1395.31 ms │ 1411.39 ms │ no change │ │ QQuery 13 │ 2032.24 ms │ 2064.42 ms │ no change │ │ QQuery 14 │ 1393.44 ms │ 1424.02 ms │ no change │ │ QQuery 15 │ 1149.14 ms │ 1190.38 ms │ no change │ │ QQuery 16 │ 2496.69 ms │ 2425.39 ms │ no change │ │ QQuery 17 │ 2466.44 ms │ 2422.12 ms │ no change │ │ QQuery 18 │ 5236.18 ms │ 4891.87 ms │ +1.07x faster │ │ QQuery 19 │ 139.14 ms │ 143.88 ms │ no change │ │ QQuery 20 │ 1934.65 ms │ 1894.61 ms │ no change │ │ QQuery 21 │ 2373.05 ms │ 2290.86 ms │ no change │ │ QQuery 22 │ 4024.68 ms │ 3934.15 ms │ no change │ │ QQuery 23 │ 1117.58 ms │ 1077.16 ms │ no change │ │ QQuery 24 │ 260.22 ms │ 249.19 ms │ no change │ │ QQuery 25 │ 643.51 ms │ 622.44 ms │ no change │ │ QQuery 26 │ 349.97 ms │ 341.18 ms │ no change │ │ QQuery 27 │ 3022.65 ms │ 2982.49 ms │ no change │ │ QQuery 28 │ 25368.11 ms │ 22303.63 ms │ +1.14x faster │ │ QQuery 29 │ 980.85 ms │ 980.14 ms │ no change │ │ QQuery 30 │ 1279.17 ms │ 1277.87 ms │ no change │ │ QQuery 31 │ 1344.29 ms │ 1355.01 ms │ no change │ │ QQuery 32 │ 4438.48 ms │ 4094.77 ms │ +1.08x faster │ │ QQuery 33 │ 5575.74 ms │ 5178.24 ms │ +1.08x faster │ │ QQuery 34 │ 6252.24 ms │ 5849.34 ms │ +1.07x faster │ │ QQuery 35 │ 1839.81 ms │ 1851.88 ms │ no change │ │ QQuery 36 │ 186.39 ms │ 184.70 ms │ no change │ │ QQuery 37 │89.73 ms │88.12 ms │ no change │ │ QQuery 38 │93.20 ms │92.75 ms │ no change │ │ QQuery
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
alamb-ghbot commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3902522414 🤖: Benchmark completed Details ``` Comparing HEAD and dynamic-dynamic-filters Benchmark clickbench_extended.json ┏━━┳━┳━┳━━┓ ┃ Query┃HEAD ┃ dynamic-dynamic-filters ┃ Change ┃ ┡━━╇━╇━╇━━┩ │ QQuery 0 │ 2319.91 ms │ 2266.04 ms │no change │ │ QQuery 1 │ 909.83 ms │ 941.73 ms │no change │ │ QQuery 2 │ 1825.57 ms │ 1832.62 ms │no change │ │ QQuery 3 │ 1065.79 ms │ 1049.66 ms │no change │ │ QQuery 4 │ 2172.21 ms │ 2147.47 ms │no change │ │ QQuery 5 │ 28486.93 ms │ 28532.89 ms │no change │ │ QQuery 6 │ 3869.11 ms │ 3839.34 ms │no change │ │ QQuery 7 │ 2498.59 ms │ 2691.88 ms │ 1.08x slower │ └──┴─┴─┴──┘ ┏┳┓ ┃ Benchmark Summary ┃┃ ┡╇┩ │ Total Time (HEAD) │ 43147.95ms │ │ Total Time (dynamic-dynamic-filters) │ 43301.63ms │ │ Average Time (HEAD)│ 5393.49ms │ │ Average Time (dynamic-dynamic-filters) │ 5412.70ms │ │ Queries Faster │ 0 │ │ Queries Slower │ 1 │ │ Queries with No Change │ 7 │ │ Queries with Failure │ 0 │ └┴┘ Benchmark clickbench_partitioned.json ┏━━━┳━┳━┳━━━┓ ┃ Query ┃HEAD ┃ dynamic-dynamic-filters ┃Change ┃ ┡━━━╇━╇━╇━━━┩ │ QQuery 0 │ 2.69 ms │ 2.68 ms │ no change │ │ QQuery 1 │51.04 ms │50.35 ms │ no change │ │ QQuery 2 │ 132.51 ms │ 138.59 ms │ no change │ │ QQuery 3 │ 155.42 ms │ 156.72 ms │ no change │ │ QQuery 4 │ 1062.37 ms │ 1050.79 ms │ no change │ │ QQuery 5 │ 1317.23 ms │ 1273.76 ms │ no change │ │ QQuery 6 │ 6.52 ms │ 7.37 ms │ 1.13x slower │ │ QQuery 7 │55.76 ms │56.52 ms │ no change │ │ QQuery 8 │ 1417.82 ms │ 1410.99 ms │ no change │ │ QQuery 9 │ 1805.78 ms │ 1745.79 ms │ no change │ │ QQuery 10 │ 342.04 ms │ 354.76 ms │ no change │ │ QQuery 11 │ 400.38 ms │ 405.22 ms │ no change │ │ QQuery 12 │ 1248.00 ms │ 1228.55 ms │ no change │ │ QQuery 13 │ 1951.63 ms │ 1883.18 ms │ no change │ │ QQuery 14 │ 1273.02 ms │ 1219.80 ms │ no change │ │ QQuery 15 │ 1167.93 ms │ 1200.64 ms │ no change │ │ QQuery 16 │ 2528.40 ms │ 2484.32 ms │ no change │ │ QQuery 17 │ 2493.98 ms │ 2469.47 ms │ no change │ │ QQuery 18 │ 4772.26 ms │ 4685.91 ms │ no change │ │ QQuery 19 │ 121.34 ms │ 121.07 ms │ no change │ │ QQuery 20 │ 1891.21 ms │ 1885.37 ms │ no change │ │ QQuery 21 │ 2224.21 ms │ 2166.49 ms │ no change │ │ QQuery 22 │ 3790.93 ms │ 3668.75 ms │ no change │ │ QQuery 23 │ 12308.05 ms │ 12051.88 ms │ no change │ │ QQuery 24 │ 208.13 ms │ 211.40 ms │ no change │ │ QQuery 25 │ 478.79 ms │ 446.01 ms │ +1.07x faster │ │ QQuery 26 │ 206.15 ms │ 203.48 ms │ no change │ │ QQuery 27 │ 2698.06 ms │ 2621.15 ms │ no change │ │ QQuery 28 │ 24317.58 ms │ 21590.02 ms │ +1.13x faster │ │ QQuery 29 │ 963.72 ms │ 979.53 ms │ no change │ │ QQuery 30 │ 1271.71 ms │ 1238.31 ms │ no change │ │ QQuery 31 │ 1384.12 ms │ 1287.83 ms │ +1.07x faster │ │ QQuery 32 │ 4153.09 ms │ 3911.84 ms │ +1.06x faster │ │ QQuery 33 │ 5317.23 ms │ 5318.61 ms │ no change │ │ QQuery 34 │ 6035.12 ms │ 5586.14 ms │ +1.08x faster │ │ QQuery 35 │ 1924.51 ms │ 1853.20 ms │ no change │ │ QQuery 36 │ 193.17 ms │ 188.44 ms │ no change │ │ QQuery 37 │75.29 ms │75.97 ms │ no change │ │ QQuery 38 │ 117.46 ms │ 114.
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
alamb-ghbot commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3902522458 🤖 `./gh_compare_branch.sh` [gh_compare_branch.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_branch.sh) Running Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Comparing dynamic-dynamic-filters (6b80e85272c6c020de4b0377d043418d27ca120c) to 7d217b1ec3bd4d447961e84bc3fe7f7708546b2d [diff](https://github.com/apache/datafusion/compare/7d217b1ec3bd4d447961e84bc3fe7f7708546b2d..6b80e85272c6c020de4b0377d043418d27ca120c) using: tpch_mem clickbench_partitioned clickbench_extended Results will be posted here when complete -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
alamb-ghbot commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3902487561 🤖 `./gh_compare_branch.sh` [gh_compare_branch.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_branch.sh) Running Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Comparing dynamic-dynamic-filters (6b80e85272c6c020de4b0377d043418d27ca120c) to 7d217b1ec3bd4d447961e84bc3fe7f7708546b2d [diff](https://github.com/apache/datafusion/compare/7d217b1ec3bd4d447961e84bc3fe7f7708546b2d..6b80e85272c6c020de4b0377d043418d27ca120c) using: tpch_mem clickbench_partitioned clickbench_extended Results will be posted here when complete -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
Dandandan commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3902482536 run benchmark tpch DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS=true DATAFUSION_EXECUTION_PARQUET_REORDER_FILTERS=true -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
Dandandan commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3902466861 run benchmarks DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS=true DATAFUSION_EXECUTION_PARQUET_REORDER_FILTERS=true -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
Dandandan commented on code in PR #20160:
URL: https://github.com/apache/datafusion/pull/20160#discussion_r2807877494
##
datafusion/common/src/config.rs:
##
@@ -1115,6 +1115,34 @@ config_namespace! {
/// See:
pub hash_join_inlist_pushdown_max_distinct_values: usize, default = 150
+/// Minimum number of rows to process before making a selectivity
decision
+/// for adaptive filtering of join dynamic filters.
+///
+/// The filter will remain in a tracking state until this many rows
have been
+/// processed. This ensures statistical stability before making the
disable decision.
+/// Only used when `enable_adaptive_filter_selectivity_tracking` is
true.
+pub adaptive_filter_min_rows_for_selectivity: usize, default = 100_000
+
+/// Selectivity threshold for adaptive disabling of join dynamic
filters.
Review Comment:
This looks now for any filter, no?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
Dandandan commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3902446436 run benchmarks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
alamb-ghbot commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3902440450 🤖: Benchmark completed Details ``` Comparing HEAD and dynamic-dynamic-filters Benchmark clickbench_partitioned.json ┏━━━┳━┳━┳━━━┓ ┃ Query ┃HEAD ┃ dynamic-dynamic-filters ┃Change ┃ ┡━━━╇━╇━╇━━━┩ │ QQuery 0 │ 2.67 ms │ 2.60 ms │ no change │ │ QQuery 1 │52.82 ms │54.76 ms │ no change │ │ QQuery 2 │ 139.47 ms │ 134.80 ms │ no change │ │ QQuery 3 │ 150.76 ms │ 153.93 ms │ no change │ │ QQuery 4 │ 1095.03 ms │ 1045.74 ms │ no change │ │ QQuery 5 │ 1336.42 ms │ 1251.55 ms │ +1.07x faster │ │ QQuery 6 │16.90 ms │18.71 ms │ 1.11x slower │ │ QQuery 7 │71.14 ms │70.91 ms │ no change │ │ QQuery 8 │ 1432.85 ms │ 1411.87 ms │ no change │ │ QQuery 9 │ 1808.43 ms │ 1830.26 ms │ no change │ │ QQuery 10 │ 482.40 ms │ 484.62 ms │ no change │ │ QQuery 11 │ 532.13 ms │ 527.29 ms │ no change │ │ QQuery 12 │ 1428.29 ms │ 1394.67 ms │ no change │ │ QQuery 13 │ 2073.17 ms │ 2095.31 ms │ no change │ │ QQuery 14 │ 1424.87 ms │ 1415.96 ms │ no change │ │ QQuery 15 │ 1220.43 ms │ 1201.90 ms │ no change │ │ QQuery 16 │ 2555.10 ms │ 2476.60 ms │ no change │ │ QQuery 17 │ 2548.77 ms │ 2457.11 ms │ no change │ │ QQuery 18 │ 4822.58 ms │ 4990.47 ms │ no change │ │ QQuery 19 │ 139.57 ms │ 141.92 ms │ no change │ │ QQuery 20 │ 1856.36 ms │ 1873.17 ms │ no change │ │ QQuery 21 │ 2328.75 ms │ 2315.48 ms │ no change │ │ QQuery 22 │ 4019.35 ms │ 3962.20 ms │ no change │ │ QQuery 23 │ 1113.63 ms │ 1066.36 ms │ no change │ │ QQuery 24 │ 252.80 ms │ 235.38 ms │ +1.07x faster │ │ QQuery 25 │ 646.85 ms │ 627.24 ms │ no change │ │ QQuery 26 │ 345.06 ms │ 341.83 ms │ no change │ │ QQuery 27 │ 2978.75 ms │ 2921.99 ms │ no change │ │ QQuery 28 │ 24986.39 ms │ 22252.33 ms │ +1.12x faster │ │ QQuery 29 │ 945.32 ms │ 948.70 ms │ no change │ │ QQuery 30 │ 1315.90 ms │ 1296.59 ms │ no change │ │ QQuery 31 │ 1363.44 ms │ 1340.15 ms │ no change │ │ QQuery 32 │ 3966.35 ms │ 4485.00 ms │ 1.13x slower │ │ QQuery 33 │ 5172.24 ms │ 5724.71 ms │ 1.11x slower │ │ QQuery 34 │ 5650.50 ms │ 5698.06 ms │ no change │ │ QQuery 35 │ 1891.09 ms │ 1840.89 ms │ no change │ │ QQuery 36 │ 180.74 ms │ 175.47 ms │ no change │ │ QQuery 37 │90.54 ms │91.29 ms │ no change │ │ QQuery 38 │91.51 ms │90.31 ms │ no change │ │ QQuery 39 │ 304.83 ms │ 295.45 ms │ no change │ │ QQuery 40 │58.05 ms │58.62 ms │ no change │ │ QQuery 41 │50.72 ms │50.52 ms │ no change │ │ QQuery 42 │36.96 ms │38.31 ms │ no change │ └───┴─┴─┴───┘ ┏┳┓ ┃ Benchmark Summary ┃┃ ┡╇┩ │ Total Time (HEAD) │ 82979.94ms │ │ Total Time (dynamic-dynamic-filters) │ 80891.03ms │ │ Average Time (HEAD)│ 1929.77ms │ │ Average Time (dynamic-dynamic-filters) │ 1881.19ms │ │ Queries Faster │ 3 │ │ Queries Slower │ 3 │ │ Queries with No Change │ 37 │ │ Queries with Failure │ 0 │ └┴┘ ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
alamb-ghbot commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3902401279 🤖 `./gh_compare_branch.sh` [gh_compare_branch.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_branch.sh) Running Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Comparing dynamic-dynamic-filters (6b80e85272c6c020de4b0377d043418d27ca120c) to 7d217b1ec3bd4d447961e84bc3fe7f7708546b2d [diff](https://github.com/apache/datafusion/compare/7d217b1ec3bd4d447961e84bc3fe7f7708546b2d..6b80e85272c6c020de4b0377d043418d27ca120c) using: clickbench_partitioned Results will be posted here when complete -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
adriangb commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3902367269 run benchmark clickbench_partitioned DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS=true DATAFUSION_EXECUTION_PARQUET_REORDER_FILTERS=true -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
adriangb commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3902367486 @Dandandan numbers looking pretty good here! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
alamb-ghbot commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3902329306 🤖: Benchmark completed Details ``` Comparing HEAD and dynamic-dynamic-filters Benchmark tpcds_sf1.json ┏━━━┳━┳━┳━━━┓ ┃ Query ┃HEAD ┃ dynamic-dynamic-filters ┃Change ┃ ┡━━━╇━╇━╇━━━┩ │ QQuery 1 │85.69 ms │82.02 ms │ no change │ │ QQuery 2 │ 249.17 ms │ 222.14 ms │ +1.12x faster │ │ QQuery 3 │ 154.98 ms │ 153.11 ms │ no change │ │ QQuery 4 │ 1650.07 ms │ 1640.11 ms │ no change │ │ QQuery 5 │ 308.00 ms │ 258.81 ms │ +1.19x faster │ │ QQuery 6 │ 421.38 ms │ 341.54 ms │ +1.23x faster │ │ QQuery 7 │ 527.29 ms │ 525.40 ms │ no change │ │ QQuery 8 │ 241.77 ms │ 191.28 ms │ +1.26x faster │ │ QQuery 9 │ 874.56 ms │ 854.18 ms │ no change │ │ QQuery 10 │ 167.78 ms │ 174.40 ms │ no change │ │ QQuery 11 │ 1059.58 ms │ 1034.59 ms │ no change │ │ QQuery 12 │54.49 ms │55.99 ms │ no change │ │ QQuery 13 │ 857.59 ms │ 815.89 ms │ no change │ │ QQuery 14 │ 2504.87 ms │ 2015.41 ms │ +1.24x faster │ │ QQuery 15 │33.25 ms │32.01 ms │ no change │ │ QQuery 16 │65.87 ms │65.27 ms │ no change │ │ QQuery 17 │ 304.06 ms │ 254.01 ms │ +1.20x faster │ │ QQuery 18 │ 480.20 ms │ 480.66 ms │ no change │ │ QQuery 19 │ 190.09 ms │ 188.50 ms │ no change │ │ QQuery 20 │26.47 ms │27.84 ms │ 1.05x slower │ │ QQuery 21 │47.52 ms │47.70 ms │ no change │ │ QQuery 22 │ 771.23 ms │ 725.47 ms │ +1.06x faster │ │ QQuery 23 │ 2496.35 ms │ 1847.84 ms │ +1.35x faster │ │ QQuery 24 │ 251.08 ms │ 256.18 ms │ no change │ │ QQuery 25 │ 452.80 ms │ 408.52 ms │ +1.11x faster │ │ QQuery 26 │ 272.39 ms │ 277.37 ms │ no change │ │ QQuery 27 │ 532.02 ms │ 520.59 ms │ no change │ │ QQuery 28 │ 471.83 ms │ 514.00 ms │ 1.09x slower │ │ QQuery 29 │ 384.85 ms │ 343.12 ms │ +1.12x faster │ │ QQuery 30 │92.58 ms │92.69 ms │ no change │ │ QQuery 31 │ 298.92 ms │ 289.02 ms │ no change │ │ QQuery 32 │77.58 ms │78.38 ms │ no change │ │ QQuery 33 │ 198.53 ms │ 193.49 ms │ no change │ │ QQuery 34 │ 194.26 ms │ 199.79 ms │ no change │ │ QQuery 35 │ 251.81 ms │ 249.91 ms │ no change │ │ QQuery 36 │ 344.85 ms │ 353.36 ms │ no change │ │ QQuery 37 │ 1005.95 ms │ 970.58 ms │ no change │ │ QQuery 38 │ 183.52 ms │ 177.89 ms │ no change │ │ QQuery 39 │ 220.75 ms │ 222.64 ms │ no change │ │ QQuery 40 │ 194.66 ms │ 195.82 ms │ no change │ │ QQuery 41 │27.07 ms │27.77 ms │ no change │ │ QQuery 42 │ 147.03 ms │ 144.72 ms │ no change │ │ QQuery 43 │ 167.19 ms │ 166.40 ms │ no change │ │ QQuery 44 │28.97 ms │29.22 ms │ no change │ │ QQuery 45 │70.55 ms │71.46 ms │ no change │ │ QQuery 46 │ 324.61 ms │ 316.43 ms │ no change │ │ QQuery 47 │ 1230.42 ms │ 1121.14 ms │ +1.10x faster │ │ QQuery 48 │ 732.29 ms │ 677.80 ms │ +1.08x faster │ │ QQuery 49 │ 552.59 ms │ 545.51 ms │ no change │ │ QQuery 50 │ 750.98 ms │ 695.66 ms │ +1.08x faster │ │ QQuery 51 │ 355.45 ms │ 369.98 ms │ no change │ │ QQuery 52 │ 146.77 ms │ 147.34 ms │ no change │ │ QQuery 53 │ 212.87 ms │ 174.59 ms │ +1.22x faster │ │ QQuery 54 │ 186.30 ms │ 185.48 ms │ no change │ │ QQuery 55 │ 144.74 ms │ 145.77 ms │ no change │ │ QQuery 56 │ 190.16 ms │ 194.21 ms │ no change │ │ QQuery 57 │ 320.25 ms │ 309.05 ms │ no change │ │ QQuery 58 │ 606.34 ms │ 488.52 ms │ +1.24x faster │ │ QQuery 59 │ 365.60 ms │ 307.62 ms │ +1.19x faster │ │ QQuery 60 │ 206.15 ms │ 203.47 ms │ no change │ │ QQuery 61 │ 271.08 ms │ 267.23 ms │ no change │ │ QQuery 62 │ 1321.41 ms
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
alamb-ghbot commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3902280642 🤖 `./gh_compare_branch.sh` [gh_compare_branch.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_branch.sh) Running Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Comparing dynamic-dynamic-filters (6b80e85272c6c020de4b0377d043418d27ca120c) to 7d217b1ec3bd4d447961e84bc3fe7f7708546b2d [diff](https://github.com/apache/datafusion/compare/7d217b1ec3bd4d447961e84bc3fe7f7708546b2d..6b80e85272c6c020de4b0377d043418d27ca120c) using: tpcds Results will be posted here when complete -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
adriangb commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3902168811 run benchmark tpcds DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS=true DATAFUSION_EXECUTION_PARQUET_REORDER_FILTERS=true -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
adriangb commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3864495397 @Dandandan mind giving this a look and maybe running benchmarks locally to see if you can repro the difference with CI? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
adriangb commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3863017620 Welp this is what I get locally: ``` Benchmark tpcds_sf1.json ┏━━━┳┳━┳━━━┓ ┃ Query ┃ main ┃ dynamic-dynamic-filters ┃Change ┃ ┡━━━╇╇━╇━━━┩ │ QQuery 1 │ 32.89 ms │32.21 ms │ no change │ │ QQuery 2 │ 99.13 ms │97.92 ms │ no change │ │ QQuery 3 │ 88.50 ms │86.12 ms │ no change │ │ QQuery 4 │ 888.10 ms │ 834.87 ms │ +1.06x faster │ │ QQuery 5 │ 128.92 ms │ 129.69 ms │ no change │ │ QQuery 6 │ 540.93 ms │ 505.51 ms │ +1.07x faster │ │ QQuery 7 │ 247.75 ms │ 241.79 ms │ no change │ │ QQuery 8 │ 87.37 ms │84.65 ms │ no change │ │ QQuery 9 │ 97.68 ms │89.40 ms │ +1.09x faster │ │ QQuery 10 │ 77.78 ms │75.94 ms │ no change │ │ QQuery 11 │ 583.81 ms │ 530.29 ms │ +1.10x faster │ │ QQuery 12 │ 32.09 ms │30.40 ms │ +1.06x faster │ │ QQuery 13 │ 292.75 ms │ 282.44 ms │ no change │ │ QQuery 14 │ 667.82 ms │ 647.10 ms │ no change │ │ QQuery 15 │ 10.95 ms │10.13 ms │ +1.08x faster │ │ QQuery 16 │ 27.97 ms │27.17 ms │ no change │ │ QQuery 17 │ 157.08 ms │ 150.89 ms │ no change │ │ QQuery 18 │ 84.72 ms │82.09 ms │ no change │ │ QQuery 19 │ 116.94 ms │ 113.83 ms │ no change │ │ QQuery 20 │9.26 ms │ 8.64 ms │ +1.07x faster │ │ QQuery 21 │ 12.29 ms │11.47 ms │ +1.07x faster │ │ QQuery 22 │ 254.95 ms │ 249.47 ms │ no change │ │ QQuery 23 │ 705.17 ms │ 690.92 ms │ no change │ │ QQuery 24 │ 273.67 ms │ 272.43 ms │ no change │ │ QQuery 25 │ 234.96 ms │ 233.40 ms │ no change │ │ QQuery 26 │ 58.54 ms │57.34 ms │ no change │ │ QQuery 27 │ 244.24 ms │ 235.81 ms │ no change │ │ QQuery 28 │ 116.74 ms │ 114.42 ms │ no change │ │ QQuery 29 │ 189.89 ms │ 186.40 ms │ no change │ │ QQuery 30 │ 30.83 ms │29.74 ms │ no change │ │ QQuery 31 │ 126.17 ms │ 124.62 ms │ no change │ │ QQuery 32 │ 43.52 ms │42.18 ms │ no change │ │ QQuery 33 │ 103.03 ms │ 101.38 ms │ no change │ │ QQuery 34 │ 77.78 ms │75.19 ms │ no change │ │ QQuery 35 │ 75.19 ms │74.32 ms │ no change │ │ QQuery 36 │ 151.70 ms │ 146.84 ms │ no change │ │ QQuery 37 │ 140.87 ms │ 134.92 ms │ no change │ │ QQuery 38 │ 59.36 ms │56.85 ms │ no change │ │ QQuery 39 │ 63.28 ms │62.26 ms │ no change │ │ QQuery 40 │ 71.61 ms │70.52 ms │ no change │ │ QQuery 41 │8.90 ms │ 8.31 ms │ +1.07x faster │ │ QQuery 42 │ 83.98 ms │80.72 ms │ no change │ │ QQuery 43 │ 63.67 ms │62.03 ms │ no change │ │ QQuery 44 │6.77 ms │ 6.17 ms │ +1.10x faster │ │ QQuery 45 │ 34.73 ms │34.06 ms │ no change │ │ QQuery 46 │ 163.21 ms │ 158.32 ms │ no change │ │ QQuery 47 │ 440.58 ms │ 418.90 ms │ no change │ │ QQuery 48 │ 198.77 ms │ 194.12 ms │ no change │ │ QQuery 49 │ 179.43 ms │ 177.62 ms │ no change │ │ QQuery 50 │ 136.37 ms │ 132.76 ms │ no change │ │ QQuery 51 │ 120.78 ms │ 120.80 ms │ no change │ │ QQuery 52 │ 82.34 ms │81.40 ms │ no change │ │ QQuery 53 │ 78.53 ms │77.03 ms │ no change │ │ QQuery 54 │ 107.32 ms │ 105.03 ms │ no change │ │ QQuery 55 │ 81.56 ms │80.31 ms │ no change │ │ QQuery 56 │ 105.01 ms │ 101.97 ms │ no change │ │ QQuery 57 │ 109.67 ms │ 107.57 ms │ no change │ │ QQuery 58 │ 197.32 ms │ 197.19 ms │ no change │ │ QQuery 59 │ 123.94 ms │ 119.86 ms │ no change │ │ QQuery 60 │ 104.55 ms │ 101.52 ms │ no change │ │ QQuery 61 │ 130.55 ms │ 127.62 ms │ no change │ │ QQuery 62 │ 374.16 ms │ 370.54 ms │ no change │ │ QQuery 63 │ 78.68 ms │76.60 ms │ no change │ │ QQuery 64 │ 495.05 ms │
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
adriangb commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3863014064 run benchmark tpch10 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
alamb-ghbot commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3863014454 🤖: Benchmark completed Details ``` Comparing HEAD and dynamic-dynamic-filters Benchmark tpch_sf10.json ┏━━━┳━━┳━┳━━┓ ┃ Query ┃ HEAD ┃ dynamic-dynamic-filters ┃ Change ┃ ┡━━━╇━━╇━╇━━┩ │ QQuery 1 │ FAIL │FAIL │ incomparable │ │ QQuery 2 │ FAIL │FAIL │ incomparable │ │ QQuery 3 │ FAIL │FAIL │ incomparable │ │ QQuery 4 │ FAIL │FAIL │ incomparable │ │ QQuery 5 │ FAIL │FAIL │ incomparable │ │ QQuery 6 │ FAIL │FAIL │ incomparable │ │ QQuery 7 │ FAIL │FAIL │ incomparable │ │ QQuery 8 │ FAIL │FAIL │ incomparable │ │ QQuery 9 │ FAIL │FAIL │ incomparable │ │ QQuery 10 │ FAIL │FAIL │ incomparable │ │ QQuery 11 │ FAIL │FAIL │ incomparable │ │ QQuery 12 │ FAIL │FAIL │ incomparable │ │ QQuery 13 │ FAIL │FAIL │ incomparable │ │ QQuery 14 │ FAIL │FAIL │ incomparable │ │ QQuery 15 │ FAIL │FAIL │ incomparable │ │ QQuery 16 │ FAIL │FAIL │ incomparable │ │ QQuery 17 │ FAIL │FAIL │ incomparable │ │ QQuery 18 │ FAIL │FAIL │ incomparable │ │ QQuery 19 │ FAIL │FAIL │ incomparable │ │ QQuery 20 │ FAIL │FAIL │ incomparable │ │ QQuery 21 │ FAIL │FAIL │ incomparable │ │ QQuery 22 │ FAIL │FAIL │ incomparable │ └───┴──┴─┴──┘ ┏┳┓ ┃ Benchmark Summary ┃┃ ┡╇┩ │ Total Time (HEAD) │ 0.00ms │ │ Total Time (dynamic-dynamic-filters) │ 0.00ms │ │ Average Time (HEAD)│ 0.00ms │ │ Average Time (dynamic-dynamic-filters) │ 0.00ms │ │ Queries Faster │ 0 │ │ Queries Slower │ 0 │ │ Queries with No Change │ 0 │ │ Queries with Failure │ 22 │ └┴┘ ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
alamb-ghbot commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3863014405 🤖 `./gh_compare_branch.sh` [gh_compare_branch.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_branch.sh) Running Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Comparing dynamic-dynamic-filters (bc6cfac62242884747b0b3c33d7b8822892da560) to ee01acf607b721d8d6366eeeb4507cf4c3952b2e [diff](https://github.com/apache/datafusion/compare/ee01acf607b721d8d6366eeeb4507cf4c3952b2e..bc6cfac62242884747b0b3c33d7b8822892da560) using: tpch10 Results will be posted here when complete -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
alamb-ghbot commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3863014692 🤖 Hi @adriangb, thanks for the request (https://github.com/apache/datafusion/pull/20160#issuecomment-3863014515). [`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks. - Standard: clickbench_1, clickbench_extended, clickbench_partitioned, clickbench_pushdown, external_aggr, tpcds, tpch, tpch10, tpch_mem, tpch_mem10 - Criterion: aggregate_query_sql, aggregate_vectorized, case_when, character_length, in_list, left, range_and_generate_series, reset_plan_states, sort, sql_planner, strpos, substr_index, with_hashes Please choose one or more of these with `run benchmark ` or `run benchmark ...` Unsupported benchmarks: tpcds10. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
adriangb commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3863014515 run benchmark tpcds10 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
alamb-ghbot commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3863008167 🤖: Benchmark completed Details ``` Comparing HEAD and dynamic-dynamic-filters Benchmark tpcds_sf1.json ┏━━━┳━┳━┳━━━┓ ┃ Query ┃HEAD ┃ dynamic-dynamic-filters ┃Change ┃ ┡━━━╇━╇━╇━━━┩ │ QQuery 1 │72.58 ms │73.00 ms │ no change │ │ QQuery 2 │ 214.95 ms │ 209.41 ms │ no change │ │ QQuery 3 │ 155.34 ms │ 158.28 ms │ no change │ │ QQuery 4 │ 1871.30 ms │ 1920.29 ms │ no change │ │ QQuery 5 │ 260.11 ms │ 260.47 ms │ no change │ │ QQuery 6 │ 1431.92 ms │ 1430.68 ms │ no change │ │ QQuery 7 │ 513.76 ms │ 519.85 ms │ no change │ │ QQuery 8 │ 168.58 ms │ 173.66 ms │ no change │ │ QQuery 9 │ 291.61 ms │ 300.71 ms │ no change │ │ QQuery 10 │ 175.19 ms │ 173.39 ms │ no change │ │ QQuery 11 │ 1289.81 ms │ 1294.05 ms │ no change │ │ QQuery 12 │71.59 ms │70.25 ms │ no change │ │ QQuery 13 │ 539.71 ms │ 549.65 ms │ no change │ │ QQuery 14 │ 1900.57 ms │ 1890.35 ms │ no change │ │ QQuery 15 │28.49 ms │27.12 ms │ no change │ │ QQuery 16 │63.11 ms │66.58 ms │ 1.06x slower │ │ QQuery 17 │ 357.15 ms │ 362.40 ms │ no change │ │ QQuery 18 │ 194.84 ms │ 192.45 ms │ no change │ │ QQuery 19 │ 218.74 ms │ 219.17 ms │ no change │ │ QQuery 20 │26.98 ms │23.06 ms │ +1.17x faster │ │ QQuery 21 │36.77 ms │36.47 ms │ no change │ │ QQuery 22 │ 699.56 ms │ 717.66 ms │ no change │ │ QQuery 23 │ 1774.36 ms │ 1787.33 ms │ no change │ │ QQuery 24 │ 677.01 ms │ 695.07 ms │ no change │ │ QQuery 25 │ 529.69 ms │ 523.56 ms │ no change │ │ QQuery 26 │ 125.65 ms │ 128.17 ms │ no change │ │ QQuery 27 │ 500.06 ms │ 521.17 ms │ no change │ │ QQuery 28 │ 283.99 ms │ 300.12 ms │ 1.06x slower │ │ QQuery 29 │ 451.37 ms │ 445.34 ms │ no change │ │ QQuery 30 │75.41 ms │74.72 ms │ no change │ │ QQuery 31 │ 301.99 ms │ 307.69 ms │ no change │ │ QQuery 32 │78.72 ms │81.55 ms │ no change │ │ QQuery 33 │ 207.15 ms │ 205.40 ms │ no change │ │ QQuery 34 │ 160.75 ms │ 159.59 ms │ no change │ │ QQuery 35 │ 176.44 ms │ 175.62 ms │ no change │ │ QQuery 36 │ 282.51 ms │ 285.22 ms │ no change │ │ QQuery 37 │ 259.68 ms │ 261.73 ms │ no change │ │ QQuery 38 │ 153.23 ms │ 158.87 ms │ no change │ │ QQuery 39 │ 195.67 ms │ 196.86 ms │ no change │ │ QQuery 40 │ 180.15 ms │ 177.45 ms │ no change │ │ QQuery 41 │26.30 ms │25.78 ms │ no change │ │ QQuery 42 │ 144.30 ms │ 145.95 ms │ no change │ │ QQuery 43 │ 126.71 ms │ 130.97 ms │ no change │ │ QQuery 44 │28.33 ms │29.07 ms │ no change │ │ QQuery 45 │85.13 ms │87.47 ms │ no change │ │ QQuery 46 │ 321.63 ms │ 326.76 ms │ no change │ │ QQuery 47 │ 1024.15 ms │ 1044.01 ms │ no change │ │ QQuery 48 │ 402.24 ms │ 412.54 ms │ no change │ │ QQuery 49 │ 374.95 ms │ 384.82 ms │ no change │ │ QQuery 50 │ 335.36 ms │ 332.22 ms │ no change │ │ QQuery 51 │ 303.33 ms │ 313.86 ms │ no change │ │ QQuery 52 │ 147.02 ms │ 150.33 ms │ no change │ │ QQuery 53 │ 144.73 ms │ 147.67 ms │ no change │ │ QQuery 54 │ 204.06 ms │ 205.93 ms │ no change │ │ QQuery 55 │ 142.18 ms │ 144.60 ms │ no change │ │ QQuery 56 │ 205.71 ms │ 207.88 ms │ no change │ │ QQuery 57 │ 292.42 ms │ 293.22 ms │ no change │ │ QQuery 58 │ 501.79 ms │ 501.03 ms │ no change │ │ QQuery 59 │ 291.79 ms │ 286.11 ms │ no change │ │ QQuery 60 │ 211.26 ms │ 210.43 ms │ no change │ │ QQuery 61 │ 249.97 ms │ 244.21 ms │ no change │ │ QQuery 62 │ 1292.35 ms
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
alamb-ghbot commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3862955240 🤖 `./gh_compare_branch.sh` [gh_compare_branch.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_branch.sh) Running Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Comparing dynamic-dynamic-filters (bc6cfac62242884747b0b3c33d7b8822892da560) to ee01acf607b721d8d6366eeeb4507cf4c3952b2e [diff](https://github.com/apache/datafusion/compare/ee01acf607b721d8d6366eeeb4507cf4c3952b2e..bc6cfac62242884747b0b3c33d7b8822892da560) using: tpcds Results will be posted here when complete -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
adriangb commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3862954810 run benchmark tpcds -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
adriangb commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3856925669 Hmm I am seeing much larger (1.20x faster) changes locally. Will have to run again and see what’s going on. I do think it’s worth pondering what the units should be here. In particular I think we should incorporate filter evaluation time, something like (rows_filtered)/(compute_time) or (bytes_filtered)/(compute_time) (GB/s, higher is better). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
alamb-ghbot commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3856893798 🤖: Benchmark completed Details ``` Comparing HEAD and dynamic-dynamic-filters Benchmark tpcds_sf1.json ┏━━━┳━┳━┳━━━┓ ┃ Query ┃HEAD ┃ dynamic-dynamic-filters ┃Change ┃ ┡━━━╇━╇━╇━━━┩ │ QQuery 1 │73.50 ms │71.74 ms │ no change │ │ QQuery 2 │ 215.54 ms │ 208.18 ms │ no change │ │ QQuery 3 │ 155.75 ms │ 152.95 ms │ no change │ │ QQuery 4 │ 1889.95 ms │ 1914.54 ms │ no change │ │ QQuery 5 │ 268.60 ms │ 258.15 ms │ no change │ │ QQuery 6 │ 1420.68 ms │ 1432.08 ms │ no change │ │ QQuery 7 │ 545.31 ms │ 509.12 ms │ +1.07x faster │ │ QQuery 8 │ 169.15 ms │ 165.64 ms │ no change │ │ QQuery 9 │ 295.98 ms │ 291.15 ms │ no change │ │ QQuery 10 │ 172.03 ms │ 173.29 ms │ no change │ │ QQuery 11 │ 1263.10 ms │ 1268.09 ms │ no change │ │ QQuery 12 │67.21 ms │67.40 ms │ no change │ │ QQuery 13 │ 541.98 ms │ 536.28 ms │ no change │ │ QQuery 14 │ 1853.25 ms │ 1836.15 ms │ no change │ │ QQuery 15 │28.00 ms │27.65 ms │ no change │ │ QQuery 16 │64.23 ms │63.18 ms │ no change │ │ QQuery 17 │ 355.16 ms │ 358.79 ms │ no change │ │ QQuery 18 │ 191.50 ms │ 190.06 ms │ no change │ │ QQuery 19 │ 214.33 ms │ 210.89 ms │ no change │ │ QQuery 20 │24.81 ms │24.04 ms │ no change │ │ QQuery 21 │36.82 ms │35.30 ms │ no change │ │ QQuery 22 │ 746.76 ms │ 712.64 ms │ no change │ │ QQuery 23 │ 1756.38 ms │ 1749.21 ms │ no change │ │ QQuery 24 │ 680.99 ms │ 680.40 ms │ no change │ │ QQuery 25 │ 523.28 ms │ 524.11 ms │ no change │ │ QQuery 26 │ 127.14 ms │ 127.88 ms │ no change │ │ QQuery 27 │ 510.77 ms │ 512.42 ms │ no change │ │ QQuery 28 │ 296.98 ms │ 297.98 ms │ no change │ │ QQuery 29 │ 452.76 ms │ 453.09 ms │ no change │ │ QQuery 30 │75.12 ms │75.19 ms │ no change │ │ QQuery 31 │ 312.61 ms │ 299.83 ms │ no change │ │ QQuery 32 │82.14 ms │80.06 ms │ no change │ │ QQuery 33 │ 206.41 ms │ 203.71 ms │ no change │ │ QQuery 34 │ 157.71 ms │ 157.70 ms │ no change │ │ QQuery 35 │ 174.13 ms │ 178.41 ms │ no change │ │ QQuery 36 │ 286.13 ms │ 286.66 ms │ no change │ │ QQuery 37 │ 258.62 ms │ 256.19 ms │ no change │ │ QQuery 38 │ 153.60 ms │ 147.67 ms │ no change │ │ QQuery 39 │ 199.69 ms │ 193.37 ms │ no change │ │ QQuery 40 │ 179.92 ms │ 180.98 ms │ no change │ │ QQuery 41 │25.20 ms │25.10 ms │ no change │ │ QQuery 42 │ 144.79 ms │ 139.86 ms │ no change │ │ QQuery 43 │ 127.12 ms │ 125.07 ms │ no change │ │ QQuery 44 │27.86 ms │28.11 ms │ no change │ │ QQuery 45 │85.90 ms │83.57 ms │ no change │ │ QQuery 46 │ 321.52 ms │ 316.25 ms │ no change │ │ QQuery 47 │ 1022.81 ms │ 1006.83 ms │ no change │ │ QQuery 48 │ 406.56 ms │ 401.27 ms │ no change │ │ QQuery 49 │ 382.96 ms │ 376.56 ms │ no change │ │ QQuery 50 │ 327.49 ms │ 333.33 ms │ no change │ │ QQuery 51 │ 302.24 ms │ 306.77 ms │ no change │ │ QQuery 52 │ 145.09 ms │ 142.72 ms │ no change │ │ QQuery 53 │ 146.60 ms │ 143.77 ms │ no change │ │ QQuery 54 │ 201.58 ms │ 197.47 ms │ no change │ │ QQuery 55 │ 144.77 ms │ 139.61 ms │ no change │ │ QQuery 56 │ 205.57 ms │ 202.46 ms │ no change │ │ QQuery 57 │ 289.26 ms │ 284.24 ms │ no change │ │ QQuery 58 │ 492.54 ms │ 496.96 ms │ no change │ │ QQuery 59 │ 286.83 ms │ 285.11 ms │ no change │ │ QQuery 60 │ 212.64 ms │ 208.20 ms │ no change │ │ QQuery 61 │ 244.74 ms │ 247.41 ms │ no change │ │ QQuery 62 │ 1239.99 ms
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
adriangb commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3856792441 run benchmark tpcds -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
alamb-ghbot commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3856793842 🤖 `./gh_compare_branch.sh` [gh_compare_branch.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_branch.sh) Running Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Comparing dynamic-dynamic-filters (9910b18539e46ff449852cc81c2a039425e8d704) to c92ba2962c0fb09255c08172f05430b7aa150f5d [diff](https://github.com/apache/datafusion/compare/c92ba2962c0fb09255c08172f05430b7aa150f5d..9910b18539e46ff449852cc81c2a039425e8d704) using: tpcds Results will be posted here when complete -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
alamb-ghbot commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3856147441 🤖: Benchmark completed Details ``` Comparing HEAD and dynamic-dynamic-filters Benchmark tpcds_sf1.json ┏━━━┳━┳━┳━━━┓ ┃ Query ┃HEAD ┃ dynamic-dynamic-filters ┃Change ┃ ┡━━━╇━╇━╇━━━┩ │ QQuery 1 │73.94 ms │71.11 ms │ no change │ │ QQuery 2 │ 221.38 ms │ 206.92 ms │ +1.07x faster │ │ QQuery 3 │ 156.47 ms │ 153.44 ms │ no change │ │ QQuery 4 │ 1883.91 ms │ 1866.82 ms │ no change │ │ QQuery 5 │ 255.67 ms │ 258.62 ms │ no change │ │ QQuery 6 │ 1457.72 ms │ 1441.12 ms │ no change │ │ QQuery 7 │ 524.45 ms │ 513.67 ms │ no change │ │ QQuery 8 │ 169.97 ms │ 166.72 ms │ no change │ │ QQuery 9 │ 307.91 ms │ 289.94 ms │ +1.06x faster │ │ QQuery 10 │ 171.01 ms │ 172.08 ms │ no change │ │ QQuery 11 │ 1288.10 ms │ 1261.80 ms │ no change │ │ QQuery 12 │66.99 ms │67.77 ms │ no change │ │ QQuery 13 │ 547.06 ms │ 537.90 ms │ no change │ │ QQuery 14 │ 1833.65 ms │ 1836.76 ms │ no change │ │ QQuery 15 │28.29 ms │27.00 ms │ no change │ │ QQuery 16 │64.66 ms │62.37 ms │ no change │ │ QQuery 17 │ 356.31 ms │ 354.29 ms │ no change │ │ QQuery 18 │ 194.29 ms │ 185.69 ms │ no change │ │ QQuery 19 │ 215.73 ms │ 210.89 ms │ no change │ │ QQuery 20 │25.73 ms │24.20 ms │ +1.06x faster │ │ QQuery 21 │35.70 ms │35.58 ms │ no change │ │ QQuery 22 │ 748.67 ms │ 713.48 ms │ no change │ │ QQuery 23 │ 1783.92 ms │ 1761.40 ms │ no change │ │ QQuery 24 │ 690.55 ms │ 688.44 ms │ no change │ │ QQuery 25 │ 526.20 ms │ 518.62 ms │ no change │ │ QQuery 26 │ 127.65 ms │ 125.06 ms │ no change │ │ QQuery 27 │ 508.12 ms │ 510.89 ms │ no change │ │ QQuery 28 │ 300.10 ms │ 302.95 ms │ no change │ │ QQuery 29 │ 449.38 ms │ 440.72 ms │ no change │ │ QQuery 30 │76.79 ms │74.65 ms │ no change │ │ QQuery 31 │ 306.34 ms │ 297.01 ms │ no change │ │ QQuery 32 │82.35 ms │80.08 ms │ no change │ │ QQuery 33 │ 206.30 ms │ 202.92 ms │ no change │ │ QQuery 34 │ 158.84 ms │ 158.01 ms │ no change │ │ QQuery 35 │ 170.27 ms │ 171.52 ms │ no change │ │ QQuery 36 │ 286.03 ms │ 278.26 ms │ no change │ │ QQuery 37 │ 260.75 ms │ 257.24 ms │ no change │ │ QQuery 38 │ 153.30 ms │ 153.01 ms │ no change │ │ QQuery 39 │ 200.55 ms │ 198.61 ms │ no change │ │ QQuery 40 │ 177.77 ms │ 179.41 ms │ no change │ │ QQuery 41 │25.18 ms │25.10 ms │ no change │ │ QQuery 42 │ 146.68 ms │ 141.77 ms │ no change │ │ QQuery 43 │ 126.82 ms │ 123.50 ms │ no change │ │ QQuery 44 │27.89 ms │27.75 ms │ no change │ │ QQuery 45 │86.30 ms │83.93 ms │ no change │ │ QQuery 46 │ 325.71 ms │ 317.84 ms │ no change │ │ QQuery 47 │ 1016.88 ms │ 1012.74 ms │ no change │ │ QQuery 48 │ 400.72 ms │ 402.88 ms │ no change │ │ QQuery 49 │ 383.60 ms │ 386.92 ms │ no change │ │ QQuery 50 │ 337.08 ms │ 330.03 ms │ no change │ │ QQuery 51 │ 304.34 ms │ 301.13 ms │ no change │ │ QQuery 52 │ 144.74 ms │ 142.63 ms │ no change │ │ QQuery 53 │ 144.55 ms │ 144.89 ms │ no change │ │ QQuery 54 │ 201.33 ms │ 201.04 ms │ no change │ │ QQuery 55 │ 145.61 ms │ 144.06 ms │ no change │ │ QQuery 56 │ 204.03 ms │ 203.68 ms │ no change │ │ QQuery 57 │ 289.90 ms │ 284.65 ms │ no change │ │ QQuery 58 │ 496.64 ms │ 494.03 ms │ no change │ │ QQuery 59 │ 290.20 ms │ 284.52 ms │ no change │ │ QQuery 60 │ 215.40 ms │ 208.56 ms │ no change │ │ QQuery 61 │ 241.77 ms │ 243.37 ms │ no change │ │ QQuery 62 │ 1262.03 ms
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
alamb-ghbot commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3856041104 🤖 `./gh_compare_branch.sh` [gh_compare_branch.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_branch.sh) Running Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Comparing dynamic-dynamic-filters (9a90c3782363725d7318942a2bc06a6682e00f64) to c92ba2962c0fb09255c08172f05430b7aa150f5d [diff](https://github.com/apache/datafusion/compare/c92ba2962c0fb09255c08172f05430b7aa150f5d..9a90c3782363725d7318942a2bc06a6682e00f64) using: tpcds Results will be posted here when complete -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
adriangb commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3856040400 run benchmark tpcds -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
alamb-ghbot commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3855998235 🤖: Benchmark completed Details ``` Comparing HEAD and dynamic-dynamic-filters Benchmark tpcds_sf1.json ┏━━━┳━┳━┳━━━┓ ┃ Query ┃HEAD ┃ dynamic-dynamic-filters ┃Change ┃ ┡━━━╇━╇━╇━━━┩ │ QQuery 1 │71.79 ms │72.86 ms │ no change │ │ QQuery 2 │ 216.81 ms │ 207.13 ms │ no change │ │ QQuery 3 │ 156.35 ms │ 151.73 ms │ no change │ │ QQuery 4 │ 1878.67 ms │ 1849.63 ms │ no change │ │ QQuery 5 │ 262.88 ms │ 256.31 ms │ no change │ │ QQuery 6 │ 1460.47 ms │ 1399.43 ms │ no change │ │ QQuery 7 │ 514.21 ms │ 515.29 ms │ no change │ │ QQuery 8 │ 170.56 ms │ 165.39 ms │ no change │ │ QQuery 9 │ 296.97 ms │ 294.02 ms │ no change │ │ QQuery 10 │ 173.24 ms │ 166.89 ms │ no change │ │ QQuery 11 │ 1291.34 ms │ 1256.21 ms │ no change │ │ QQuery 12 │68.76 ms │64.58 ms │ +1.06x faster │ │ QQuery 13 │ 547.00 ms │ 540.21 ms │ no change │ │ QQuery 14 │ 1851.00 ms │ 1847.62 ms │ no change │ │ QQuery 15 │28.17 ms │26.28 ms │ +1.07x faster │ │ QQuery 16 │64.63 ms │61.65 ms │ no change │ │ QQuery 17 │ 355.09 ms │ 352.96 ms │ no change │ │ QQuery 18 │ 191.95 ms │ 189.92 ms │ no change │ │ QQuery 19 │ 215.23 ms │ 210.00 ms │ no change │ │ QQuery 20 │24.23 ms │24.29 ms │ no change │ │ QQuery 21 │35.38 ms │34.87 ms │ no change │ │ QQuery 22 │ 752.09 ms │ 734.63 ms │ no change │ │ QQuery 23 │ 1742.56 ms │ 1772.29 ms │ no change │ │ QQuery 24 │ 685.98 ms │ 678.13 ms │ no change │ │ QQuery 25 │ 521.26 ms │ 518.93 ms │ no change │ │ QQuery 26 │ 125.14 ms │ 122.95 ms │ no change │ │ QQuery 27 │ 510.44 ms │ 507.89 ms │ no change │ │ QQuery 28 │ 298.01 ms │ 298.01 ms │ no change │ │ QQuery 29 │ 446.44 ms │ 439.43 ms │ no change │ │ QQuery 30 │72.94 ms │77.04 ms │ 1.06x slower │ │ QQuery 31 │ 312.36 ms │ 299.47 ms │ no change │ │ QQuery 32 │80.95 ms │79.55 ms │ no change │ │ QQuery 33 │ 206.36 ms │ 201.21 ms │ no change │ │ QQuery 34 │ 158.15 ms │ 153.93 ms │ no change │ │ QQuery 35 │ 174.10 ms │ 168.99 ms │ no change │ │ QQuery 36 │ 287.90 ms │ 281.33 ms │ no change │ │ QQuery 37 │ 260.08 ms │ 254.26 ms │ no change │ │ QQuery 38 │ 151.90 ms │ 148.18 ms │ no change │ │ QQuery 39 │ 196.88 ms │ 195.05 ms │ no change │ │ QQuery 40 │ 176.29 ms │ 178.10 ms │ no change │ │ QQuery 41 │25.57 ms │25.17 ms │ no change │ │ QQuery 42 │ 143.99 ms │ 141.85 ms │ no change │ │ QQuery 43 │ 128.49 ms │ 123.85 ms │ no change │ │ QQuery 44 │28.71 ms │27.65 ms │ no change │ │ QQuery 45 │84.71 ms │85.60 ms │ no change │ │ QQuery 46 │ 316.62 ms │ 311.19 ms │ no change │ │ QQuery 47 │ 1035.67 ms │ 1001.11 ms │ no change │ │ QQuery 48 │ 399.39 ms │ 405.25 ms │ no change │ │ QQuery 49 │ 385.97 ms │ 376.47 ms │ no change │ │ QQuery 50 │ 326.57 ms │ 329.22 ms │ no change │ │ QQuery 51 │ 304.25 ms │ 296.87 ms │ no change │ │ QQuery 52 │ 146.03 ms │ 142.18 ms │ no change │ │ QQuery 53 │ 146.78 ms │ 143.43 ms │ no change │ │ QQuery 54 │ 202.66 ms │ 198.11 ms │ no change │ │ QQuery 55 │ 144.65 ms │ 140.58 ms │ no change │ │ QQuery 56 │ 204.22 ms │ 202.49 ms │ no change │ │ QQuery 57 │ 288.73 ms │ 283.31 ms │ no change │ │ QQuery 58 │ 487.16 ms │ 504.95 ms │ no change │ │ QQuery 59 │ 290.13 ms │ 285.92 ms │ no change │ │ QQuery 60 │ 212.18 ms │ 207.90 ms │ no change │ │ QQuery 61 │ 246.24 ms │ 244.90 ms │ no change │ │ QQuery 62 │ 1240.27 ms
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
alamb-ghbot commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3855911916 🤖 `./gh_compare_branch.sh` [gh_compare_branch.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_branch.sh) Running Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Comparing dynamic-dynamic-filters (0835dd2e6aaf69e9759c0b01043bf26b966d162f) to c92ba2962c0fb09255c08172f05430b7aa150f5d [diff](https://github.com/apache/datafusion/compare/c92ba2962c0fb09255c08172f05430b7aa150f5d..0835dd2e6aaf69e9759c0b01043bf26b966d162f) using: tpcds Results will be posted here when complete -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
alamb-ghbot commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3855911545 🤖: Benchmark completed Details ``` Comparing HEAD and dynamic-dynamic-filters Benchmark tpch_sf1.json ┏━━━┳━━━┳━┳━━━┓ ┃ Query ┃ HEAD ┃ dynamic-dynamic-filters ┃Change ┃ ┡━━━╇━━━╇━╇━━━┩ │ QQuery 1 │ 181.44 ms │ 177.08 ms │ no change │ │ QQuery 2 │ 85.46 ms │86.28 ms │ no change │ │ QQuery 3 │ 119.33 ms │ 121.16 ms │ no change │ │ QQuery 4 │ 76.31 ms │77.53 ms │ no change │ │ QQuery 5 │ 168.76 ms │ 167.91 ms │ no change │ │ QQuery 6 │ 66.18 ms │65.67 ms │ no change │ │ QQuery 7 │ 204.94 ms │ 207.06 ms │ no change │ │ QQuery 8 │ 166.49 ms │ 165.28 ms │ no change │ │ QQuery 9 │ 222.99 ms │ 220.56 ms │ no change │ │ QQuery 10 │ 181.34 ms │ 181.70 ms │ no change │ │ QQuery 11 │ 62.22 ms │60.04 ms │ no change │ │ QQuery 12 │ 116.31 ms │ 116.68 ms │ no change │ │ QQuery 13 │ 217.24 ms │ 225.71 ms │ no change │ │ QQuery 14 │ 91.51 ms │92.05 ms │ no change │ │ QQuery 15 │ 124.54 ms │ 123.26 ms │ no change │ │ QQuery 16 │ 58.34 ms │58.92 ms │ no change │ │ QQuery 17 │ 250.47 ms │ 252.93 ms │ no change │ │ QQuery 18 │ 306.19 ms │ 306.54 ms │ no change │ │ QQuery 19 │ 132.26 ms │ 133.61 ms │ no change │ │ QQuery 20 │ 127.25 ms │ 123.44 ms │ no change │ │ QQuery 21 │ 254.26 ms │ 252.49 ms │ no change │ │ QQuery 22 │ 40.45 ms │39.53 ms │ no change │ └───┴───┴─┴───┘ ┏┳━━━┓ ┃ Benchmark Summary ┃ ┃ ┡╇━━━┩ │ Total Time (HEAD) │ 3254.29ms │ │ Total Time (dynamic-dynamic-filters) │ 3255.41ms │ │ Average Time (HEAD)│ 147.92ms │ │ Average Time (dynamic-dynamic-filters) │ 147.97ms │ │ Queries Faster │ 0 │ │ Queries Slower │ 0 │ │ Queries with No Change │22 │ │ Queries with Failure │ 0 │ └┴───┘ ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
adriangb commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3855851837 run benchmark tpcds -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
alamb-ghbot commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3855851411 🤖 `./gh_compare_branch.sh` [gh_compare_branch.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_branch.sh) Running Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Comparing dynamic-dynamic-filters (214d54d043fe4e9e799aedf1154420a2cfc273d8) to c92ba2962c0fb09255c08172f05430b7aa150f5d [diff](https://github.com/apache/datafusion/compare/c92ba2962c0fb09255c08172f05430b7aa150f5d..214d54d043fe4e9e799aedf1154420a2cfc273d8) using: tpch Results will be posted here when complete -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
adriangb commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3855851167 run benchmark tpch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
adriangb commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3855815837 I expect benchmarks to look bad - there's overhead this wrapper introduces right now even in no-op mode. I'm going to push a fix and run benches again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
alamb-ghbot commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3855643096 🤖: Benchmark completed Details ``` Comparing HEAD and dynamic-dynamic-filters Benchmark tpch_sf1.json ┏━━━┳━━━┳━┳━━━┓ ┃ Query ┃ HEAD ┃ dynamic-dynamic-filters ┃Change ┃ ┡━━━╇━━━╇━╇━━━┩ │ QQuery 1 │ 183.59 ms │ 182.05 ms │ no change │ │ QQuery 2 │ 86.16 ms │86.25 ms │ no change │ │ QQuery 3 │ 123.52 ms │ 123.63 ms │ no change │ │ QQuery 4 │ 77.82 ms │76.82 ms │ no change │ │ QQuery 5 │ 171.51 ms │ 166.34 ms │ no change │ │ QQuery 6 │ 66.28 ms │64.86 ms │ no change │ │ QQuery 7 │ 204.87 ms │ 203.42 ms │ no change │ │ QQuery 8 │ 166.10 ms │ 162.66 ms │ no change │ │ QQuery 9 │ 227.99 ms │ 223.17 ms │ no change │ │ QQuery 10 │ 180.73 ms │ 180.58 ms │ no change │ │ QQuery 11 │ 61.14 ms │60.76 ms │ no change │ │ QQuery 12 │ 117.17 ms │ 117.25 ms │ no change │ │ QQuery 13 │ 217.17 ms │ 222.40 ms │ no change │ │ QQuery 14 │ 91.27 ms │88.63 ms │ no change │ │ QQuery 15 │ 123.28 ms │ 122.67 ms │ no change │ │ QQuery 16 │ 59.59 ms │57.53 ms │ no change │ │ QQuery 17 │ 251.18 ms │ 252.79 ms │ no change │ │ QQuery 18 │ 308.74 ms │ 309.84 ms │ no change │ │ QQuery 19 │ 133.37 ms │ 131.86 ms │ no change │ │ QQuery 20 │ 125.73 ms │ 126.30 ms │ no change │ │ QQuery 21 │ 257.56 ms │ 250.46 ms │ no change │ │ QQuery 22 │ 41.77 ms │43.60 ms │ no change │ └───┴───┴─┴───┘ ┏┳━━━┓ ┃ Benchmark Summary ┃ ┃ ┡╇━━━┩ │ Total Time (HEAD) │ 3276.53ms │ │ Total Time (dynamic-dynamic-filters) │ 3253.83ms │ │ Average Time (HEAD)│ 148.93ms │ │ Average Time (dynamic-dynamic-filters) │ 147.90ms │ │ Queries Faster │ 0 │ │ Queries Slower │ 0 │ │ Queries with No Change │22 │ │ Queries with Failure │ 0 │ └┴───┘ ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
alamb-ghbot commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3855639054 🤖 `./gh_compare_branch.sh` [gh_compare_branch.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_branch.sh) Running Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Comparing dynamic-dynamic-filters (bd9591269c2a9c740c3a29ff27e35adb9c842f44) to c92ba2962c0fb09255c08172f05430b7aa150f5d [diff](https://github.com/apache/datafusion/compare/c92ba2962c0fb09255c08172f05430b7aa150f5d..bd9591269c2a9c740c3a29ff27e35adb9c842f44) using: tpch Results will be posted here when complete -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
alamb-ghbot commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3855638858 🤖: Benchmark completed Details ``` Comparing HEAD and dynamic-dynamic-filters Benchmark tpch_sf1.json ┏━━━┳━━━┳━┳━━━┓ ┃ Query ┃ HEAD ┃ dynamic-dynamic-filters ┃Change ┃ ┡━━━╇━━━╇━╇━━━┩ │ QQuery 1 │ 179.73 ms │ 179.63 ms │ no change │ │ QQuery 2 │ 86.42 ms │85.23 ms │ no change │ │ QQuery 3 │ 124.40 ms │ 122.71 ms │ no change │ │ QQuery 4 │ 78.85 ms │77.12 ms │ no change │ │ QQuery 5 │ 169.89 ms │ 168.71 ms │ no change │ │ QQuery 6 │ 65.54 ms │67.78 ms │ no change │ │ QQuery 7 │ 205.35 ms │ 211.90 ms │ no change │ │ QQuery 8 │ 160.97 ms │ 162.85 ms │ no change │ │ QQuery 9 │ 224.78 ms │ 216.23 ms │ no change │ │ QQuery 10 │ 180.84 ms │ 178.02 ms │ no change │ │ QQuery 11 │ 62.98 ms │59.62 ms │ +1.06x faster │ │ QQuery 12 │ 117.48 ms │ 117.01 ms │ no change │ │ QQuery 13 │ 219.16 ms │ 221.10 ms │ no change │ │ QQuery 14 │ 89.96 ms │88.88 ms │ no change │ │ QQuery 15 │ 125.22 ms │ 121.54 ms │ no change │ │ QQuery 16 │ 61.87 ms │58.97 ms │ no change │ │ QQuery 17 │ 257.79 ms │ 252.09 ms │ no change │ │ QQuery 18 │ 310.85 ms │ 304.87 ms │ no change │ │ QQuery 19 │ 138.14 ms │ 131.65 ms │ no change │ │ QQuery 20 │ 125.64 ms │ 125.06 ms │ no change │ │ QQuery 21 │ 255.07 ms │ 250.18 ms │ no change │ │ QQuery 22 │ 40.20 ms │40.10 ms │ no change │ └───┴───┴─┴───┘ ┏┳━━━┓ ┃ Benchmark Summary ┃ ┃ ┡╇━━━┩ │ Total Time (HEAD) │ 3281.12ms │ │ Total Time (dynamic-dynamic-filters) │ 3241.24ms │ │ Average Time (HEAD)│ 149.14ms │ │ Average Time (dynamic-dynamic-filters) │ 147.33ms │ │ Queries Faster │ 1 │ │ Queries Slower │ 0 │ │ Queries with No Change │21 │ │ Queries with Failure │ 0 │ └┴───┘ ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
alamb-ghbot commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3855634671 🤖 `./gh_compare_branch.sh` [gh_compare_branch.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_branch.sh) Running Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Comparing dynamic-dynamic-filters (bd9591269c2a9c740c3a29ff27e35adb9c842f44) to c92ba2962c0fb09255c08172f05430b7aa150f5d [diff](https://github.com/apache/datafusion/compare/c92ba2962c0fb09255c08172f05430b7aa150f5d..bd9591269c2a9c740c3a29ff27e35adb9c842f44) using: tpch Results will be posted here when complete -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
alamb-ghbot commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3855634209 🤖: Benchmark completed Details ``` Comparing HEAD and dynamic-dynamic-filters Benchmark tpcds_sf1.json ┏━━━┳━┳━┳━━━┓ ┃ Query ┃HEAD ┃ dynamic-dynamic-filters ┃Change ┃ ┡━━━╇━╇━╇━━━┩ │ QQuery 1 │73.03 ms │72.19 ms │ no change │ │ QQuery 2 │ 216.42 ms │ 213.00 ms │ no change │ │ QQuery 3 │ 157.55 ms │ 154.02 ms │ no change │ │ QQuery 4 │ 1921.19 ms │ 2116.78 ms │ 1.10x slower │ │ QQuery 5 │ 259.22 ms │ 261.63 ms │ no change │ │ QQuery 6 │ 1457.05 ms │ 1224.10 ms │ +1.19x faster │ │ QQuery 7 │ 517.50 ms │ 500.07 ms │ no change │ │ QQuery 8 │ 167.00 ms │ 164.80 ms │ no change │ │ QQuery 9 │ 303.83 ms │ 295.25 ms │ no change │ │ QQuery 10 │ 174.40 ms │ 169.75 ms │ no change │ │ QQuery 11 │ 1286.14 ms │ 1279.81 ms │ no change │ │ QQuery 12 │67.86 ms │66.96 ms │ no change │ │ QQuery 13 │ 542.12 ms │ 539.32 ms │ no change │ │ QQuery 14 │ 1845.08 ms │ 1901.43 ms │ no change │ │ QQuery 15 │28.55 ms │ 122.86 ms │ 4.30x slower │ │ QQuery 16 │64.94 ms │ 180.33 ms │ 2.78x slower │ │ QQuery 17 │ 355.34 ms │ 438.65 ms │ 1.23x slower │ │ QQuery 18 │ 192.54 ms │ 387.14 ms │ 2.01x slower │ │ QQuery 19 │ 215.34 ms │ 207.90 ms │ no change │ │ QQuery 20 │24.59 ms │ 108.44 ms │ 4.41x slower │ │ QQuery 21 │35.94 ms │ 380.63 ms │ 10.59x slower │ │ QQuery 22 │ 747.69 ms │ 848.25 ms │ 1.13x slower │ │ QQuery 23 │ 1759.12 ms │ 1848.79 ms │ 1.05x slower │ │ QQuery 24 │ 681.16 ms │ 662.29 ms │ no change │ │ QQuery 25 │ 517.65 ms │ 515.14 ms │ no change │ │ QQuery 26 │ 125.63 ms │ 285.69 ms │ 2.27x slower │ │ QQuery 27 │ 507.48 ms │ 501.38 ms │ no change │ │ QQuery 28 │ 299.95 ms │ 302.43 ms │ no change │ │ QQuery 29 │ 445.55 ms │ 439.76 ms │ no change │ │ QQuery 30 │74.52 ms │73.65 ms │ no change │ │ QQuery 31 │ 301.40 ms │ 297.91 ms │ no change │ │ QQuery 32 │82.75 ms │ 132.59 ms │ 1.60x slower │ │ QQuery 33 │ 207.60 ms │ 202.12 ms │ no change │ │ QQuery 34 │ 158.78 ms │ 155.24 ms │ no change │ │ QQuery 35 │ 173.36 ms │ 187.88 ms │ 1.08x slower │ │ QQuery 36 │ 294.53 ms │ 283.60 ms │ no change │ │ QQuery 37 │ 259.33 ms │ 258.64 ms │ no change │ │ QQuery 38 │ 152.78 ms │ 172.27 ms │ 1.13x slower │ │ QQuery 39 │ 197.20 ms │ 1660.68 ms │ 8.42x slower │ │ QQuery 40 │ 176.28 ms │ 173.94 ms │ no change │ │ QQuery 41 │24.35 ms │24.54 ms │ no change │ │ QQuery 42 │ 144.75 ms │ 142.10 ms │ no change │ │ QQuery 43 │ 126.54 ms │ 124.73 ms │ no change │ │ QQuery 44 │28.47 ms │28.65 ms │ no change │ │ QQuery 45 │82.92 ms │82.77 ms │ no change │ │ QQuery 46 │ 317.91 ms │ 318.33 ms │ no change │ │ QQuery 47 │ 1026.32 ms │ 1019.10 ms │ no change │ │ QQuery 48 │ 395.30 ms │ 398.92 ms │ no change │ │ QQuery 49 │ 382.94 ms │ 375.77 ms │ no change │ │ QQuery 50 │ 329.82 ms │ 332.32 ms │ no change │ │ QQuery 51 │ 301.95 ms │ 302.24 ms │ no change │ │ QQuery 52 │ 144.66 ms │ 143.23 ms │ no change │ │ QQuery 53 │ 146.79 ms │ 141.98 ms │ no change │ │ QQuery 54 │ 203.08 ms │ 224.04 ms │ 1.10x slower │ │ QQuery 55 │ 144.49 ms │ 142.24 ms │ no change │ │ QQuery 56 │ 206.41 ms │ 203.46 ms │ no change │ │ QQuery 57 │ 289.28 ms │ 439.70 ms │ 1.52x slower │ │ QQuery 58 │ 484.32 ms │ 488.51 ms │ no change │ │ QQuery 59 │ 290.83 ms │ 289.39 ms │ no change │ │ QQuery 60 │ 210.70 ms │ 206.66 ms │ no change │ │ QQuery 61 │ 245.28 ms │ 249.00 ms │ no change │ │ QQuery 62 │ 1299.75 ms
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
alamb-ghbot commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3855489181 🤖 `./gh_compare_branch.sh` [gh_compare_branch.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_branch.sh) Running Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Comparing dynamic-dynamic-filters (bd9591269c2a9c740c3a29ff27e35adb9c842f44) to c92ba2962c0fb09255c08172f05430b7aa150f5d [diff](https://github.com/apache/datafusion/compare/c92ba2962c0fb09255c08172f05430b7aa150f5d..bd9591269c2a9c740c3a29ff27e35adb9c842f44) using: tpcds Results will be posted here when complete -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
Dandandan commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3855450408 run benchmark tpch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
Dandandan commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3855447811 run benchmark tpcds tpch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
adriangb commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3854029859 show benchmark queue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
adriangb commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3853830119 run benchmarks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
adriangb commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-3853782375 run benchmark tpchds tpch tpch10 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
adriangb commented on PR #20160: URL: https://github.com/apache/datafusion/pull/20160#issuecomment-385373 run benchmark tpchds tpch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: Add selectivity-tracking wrapper for dynamic filters [datafusion]
adriangb commented on code in PR #20160:
URL: https://github.com/apache/datafusion/pull/20160#discussion_r2768613927
##
datafusion/physical-expr/src/expressions/adaptive_selectivity_filter.rs:
##
@@ -0,0 +1,486 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! A wrapper [`PhysicalExpr`] that tracks filter selectivity at runtime and
+//! automatically disables filters that aren't pruning enough rows.
+
+use std::any::Any;
+use std::fmt::Display;
+use std::hash::Hash;
+use std::sync::Arc;
+use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
+
+use arrow::array::{ArrayRef, BooleanArray};
+use arrow::datatypes::{DataType, Schema};
+use arrow::record_batch::RecordBatch;
+use datafusion_common::Result;
+use datafusion_expr::ColumnarValue;
+use datafusion_physical_expr_common::physical_expr::DynHash;
+use parking_lot::RwLock;
+
+use crate::PhysicalExpr;
+
+/// Configuration for selectivity-based filter disabling.
+#[derive(Debug, Clone)]
+pub struct SelectivityConfig {
+/// Threshold above which the filter is disabled (e.g., 0.95 = 95%
selectivity).
+/// If the filter passes this fraction or more of rows, it will be
disabled.
+pub threshold: f64,
Review Comment:
Could be in GB/s? Rows/s?
##
datafusion/physical-expr/src/expressions/adaptive_selectivity_filter.rs:
##
@@ -0,0 +1,486 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! A wrapper [`PhysicalExpr`] that tracks filter selectivity at runtime and
+//! automatically disables filters that aren't pruning enough rows.
+
+use std::any::Any;
+use std::fmt::Display;
+use std::hash::Hash;
+use std::sync::Arc;
+use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
+
+use arrow::array::{ArrayRef, BooleanArray};
+use arrow::datatypes::{DataType, Schema};
+use arrow::record_batch::RecordBatch;
+use datafusion_common::Result;
+use datafusion_expr::ColumnarValue;
+use datafusion_physical_expr_common::physical_expr::DynHash;
+use parking_lot::RwLock;
+
+use crate::PhysicalExpr;
+
+/// Configuration for selectivity-based filter disabling.
+#[derive(Debug, Clone)]
+pub struct SelectivityConfig {
+/// Threshold above which the filter is disabled (e.g., 0.95 = 95%
selectivity).
+/// If the filter passes this fraction or more of rows, it will be
disabled.
+pub threshold: f64,
+/// Minimum rows to process before making a selectivity decision.
+pub min_rows: usize,
Review Comment:
Could be in GB? Number of batches? Time? (or all of the above?)
##
datafusion/common/src/config.rs:
##
@@ -1115,6 +1115,34 @@ config_namespace! {
/// See:
pub hash_join_inlist_pushdown_max_distinct_values: usize, default = 150
+/// Enable selectivity-based disabling of dynamic filters from joins.
+///
+/// When enabled, join dynamic filters that pass most rows (above the
threshold)
+/// will be automatically disabled to avoid evaluation overhead. This
is useful
+/// when the build side of a join covers most of the probe side
values, making
+/// the filter expensive to evaluate for little benefit.
+///
+/// The selectivity tracking resets when the dynamic filter is updated
(e.g., when
+/// the hash table is built), allowing the filter to be re-evaluated
with new data.
+
