zhuqi-lucas commented on issue #21207: URL: https://github.com/apache/datafusion/issues/21207#issuecomment-5029901073
Thanks for the ping @adriangb — indeed I've been exploring this exact trade-off in #23701. Our approach there is a simpler "union bounds" shape for Partitioned mode: `col >= global_min AND col <= global_max` (equivalent to what CollectLeft mode already emits). It sacrifices per-partition selectivity for O(1) per-row cost regardless of N. The trade-off matrix as I see it: | | Per-row cost | Selectivity | Membership | |------------------------|--------------|--------------------|------------| | CASE hash-routed | O(1) + hash | Exact per-partition| ✓ | | MultiMapLookup (#21931)| O(N) probes | Exact | ✓ | | Union bounds (#23701) | O(1) | Loose (superset) | ✗ | Union bounds relies on the downstream HashJoin's hash lookup for exact membership — semantics preserving, and cheap per-row at any N. But it loses per-partition selectivity, so for skewed / low-density joins membership-based filters would prune more. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
