devdattatalele commented on issue #23817: URL: https://github.com/apache/datafusion/issues/23817#issuecomment-5652485526
I'd like to help drive this, and I can bring a concrete consumer to the table. I work on ParadeDB's pg_search, where right now every partitioned-mode join dynamic filter contributes zero pruning: our index pushdown only recurses AND chains, so a CASE root extracts no InLists, and our fast-field prefilter rejects the conjunct because CaseExpr is not in its allowlist. The sharp edge is that our range co-partitioned join rewrite is itself what flips eligible joins into Partitioned mode, so our own optimization currently disables our own pruning. With the structured form, a range-partitioned scan that already knows its target partition can select cases[partition] and push exactly that partition's InList and bounds into the index, which is better than what even the CollectLeft encoding offers. On the total vs partial concern: I think it is structurally defused by keeping current() lowered to the same total CASE the join builds today, and making per-partition evaluation strictly opt-in through current_partitioned(partitioning), returning Some only when the consumer proves equivalence: same variant, equal partition count, partition expressions equivalent after remapping into the consumer schema, identical hash random state, and excluding preserve_file_partitions-derived Hash metadata, since #20195 is precisely the case where dynamic filtering is disabled today because the declared partitioning is not proof that partition indexes follow the hash router. Anything that cannot prove equivalence uses the total lowering, with debug asserts on both sides. Concretely I would slice it as: (1) LiveFilterExpr plus PartitionedFilterExpr with the shape agreed above, partition-scoped update, and current() lowering that reproduces build_partitioned_filter's fast paths, zero behavior change; (2) producer migration in shared_bounds; (3) consumer opt-in where DataSourceExec::execute knows its partition; (4) proto following the #24618 destructure convention; with benchmarks per #24095. I would defer the distributed union() semantics to a follow-up since RFC #553 is still open, and #24235's InList-collapse idea is worth folding into the lowering rather than competing with it. @jayshrivastava @peterxcli do either of you intend to implement this? If not, I will take it in those slices, and if you have already started I can pick up whichever pieces are free. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
