zhuqi-lucas commented on PR #23701: URL: https://github.com/apache/datafusion/pull/23701#issuecomment-5020916373
Coupling refinement: **union-bounds fast path is now gated on `pushdown_filters=false`** (commit bf00cbfa2). @zhuqi-lucas noted correctly that relaxing the filter for `pushdown_filters=true` might not be desirable — arrow-rs `RowFilter` amortizes CASE per-row cost via lazy decode of the remaining columns for filtered rows, so the tighter per-partition selectivity is worth paying for there. **New effect matrix**: | `pushdown_filters` | `enable_membership_filter` | Partitioned filter shape | |---|---|---| | **false** (default) | any | **union bounds** (fast path) | | true | false | CASE hash-routed, bounds only | | true | true | CASE hash-routed + membership (historical) | **Effect on the three benchmarks already reported**: unchanged. TPC-H / TPC-DS / ClickBench partitioned all run `pushdown_filters=false` (the default), so they still hit the union-bounds fast path. **New behavior to validate**: `pushdown_filters=true` scenarios. Triggering `clickbench_pushdown` (which sets `--pushdown` → runtime `pushdown_filters=true`) to confirm no regression on that path. @github-actions run benchmarks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
