leborchuk commented on PR #1942: URL: https://github.com/apache/cloudberry/pull/1942#issuecomment-5498156724
> what's difference with [6c41d27](https://github.com/apache/cloudberry/commit/6c41d270a2ec968eadf011401ba1e5d12a5ab71f) impl? Runtime filter pushdown is intra-slice only. The RuntimeFilter executor node reaches the HashJoin's in-memory hash table via a plain executor pointer (node->hjstate->hj_HashTable, nodeRuntimeFilter.c:83-86); the Hash variant hands bloom scankeys to a registered SeqScan/DynamicSeqScanState in the same process. Nothing ever crosses a Motion or the network. Use the same hash function. Anser is cross-slice, cross-segment. Per-segment blooms are unioned on the coordinator into a global filter; any consumer anywhere can use it. That's the general MPP case from the paper — non-colocated joins, producer and consumer in different slices, even different joins sharing an equivalence-class condition_key. Could use different hash function. The built-in structurally cannot do any of that. But let's return to the example. For the proposed example you are right, existing approach is better. Explain with `gp_enable_runtime_filter_pushdown TO on`: ``` Gather Motion 3:1 (slice1; segments: 3) (cost=0.00..1689.57 rows=199266 width=386) (actual time=24.519..254.783 rows=100000 loops=1) -> Hash Right Join (cost=0.00..1402.93 rows=66422 width=386) (actual time=24.110..198.894 rows=33850 loops=1) Hash Cond: (a.id = aef.id) Extra Text: (seg1) Hash chain length 10.1 avg, 20 max, using 3365 of 262144 buckets. -> Seq Scan on applications a (cost=0.00..499.02 rows=333334 width=353) (actual time=1.823..154.653 rows=3386 loops=1) Rows Removed by Pushdown Runtime Filter: 329405 -> Hash (cost=437.42..437.42 rows=33334 width=33) (actual time=21.867..21.868 rows=33850 loops=1) Buckets: 262144 Batches: 1 Memory Usage: 4233kB -> Redistribute Motion 3:3 (slice2; segments: 3) (cost=0.00..437.42 rows=33334 width=33) (actual time=1.316..17.193 rows=33850 loops=1) Hash Key: aef.id -> Seq Scan on applications_extra_fields aef (cost=0.00..431.94 rows=33334 width=33) (actual time=0.299..6.707 rows=50000 loops=1) Planning Time: 6.577 ms Optimizer: GPORCA Execution Time: 258.942 ms ``` Explain with `gp_anser_runtime_filter=on`: ``` Gather Motion 3:1 (slice1; segments: 3) (cost=0.00..1689.57 rows=199266 width=386) (actual time=88.531..358.959 rows=100000 loops=1) -> Hash Right Join (cost=0.00..1402.93 rows=66422 width=386) (actual time=88.154..310.096 rows=33850 loops=1) Hash Cond: (a.id = aef.id) Extra Text: (seg1) Hash chain length 10.1 avg, 20 max, using 3365 of 262144 buckets. -> Custom Scan (Anser Bloom Consumer) (cost=0.00..499.02 rows=333334 width=353) (actual time=21.028..224.871 rows=3385 loops=1) Bloom Filter Size: 1048576 bytes Bloom Filter Stats: memory=1024kB checked=332791 rejected=329406 Rows Removed by Bloom Filter: 329406 -> Seq Scan on applications a (cost=0.00..499.02 rows=333334 width=353) (actual time=1.252..158.640 rows=334042 loops=1) -> Hash (cost=437.42..437.42 rows=33334 width=33) (actual time=66.655..66.657 rows=33850 loops=1) Buckets: 262144 Batches: 1 Memory Usage: 4233kB -> Redistribute Motion 3:3 (slice2; segments: 3) (cost=0.00..437.42 rows=33334 width=33) (actual time=1.271..63.106 rows=33850 loops=1) Hash Key: aef.id -> Custom Scan (Anser Bloom Producer) (cost=0.00..431.94 rows=33334 width=33) (actual time=0.357..49.022 rows=50000 loops=1) Bloom Filter Size: 1048576 bytes Bloom Filter Stats: memory=1024kB -> Seq Scan on applications_extra_fields aef (cost=0.00..431.94 rows=33334 width=33) (actual time=0.355..8.658 rows=50000 loops =1) Planning Time: 5.853 ms Optimizer: GPORCA Execution Time: 369.783 ms ``` The interesting detail is that `Rows Removed by` is practically for both optimizations. It means that we could cross-check new approach with existing one. I did it and fixed a couple of bugs ) Another idea is why use separate step for filter out rows? We could push down all filters close to AM-level. And use it, for example in PAX or in the future iceberg approach. I want to implement it in the future but since we are talking about it here, add push-down to seq scan. The true meaning this PR is to add Anser, bloomfilters here just the tool for check how whole system works. I'm going to address all issues in other PR's, where I could just use working system. Here we have for about 8500 lines of the new code ... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
