leborchuk opened a new pull request, #1942: URL: https://github.com/apache/cloudberry/pull/1942
That's the MVP for the Anser https://vldb.org/pvldb/vol16/p3636-wu.pdf Here we covered only scenario with adding bloomfilters to the query execution plan. ### The main idea is as a follow - we have a distributed plan with motions and hash join, when one table is joined with another one, and only a small subset of rows meets the join criteria. So it'd be useful to filter out rows from big one table in advance. - we could use bloom filters to filter out rows. To do so we should gather it on segments, combine in one on master, redistribute to the segments and filter out rows before hash join (after seq scan) The overall execution plan should looks like ``` postgres=# explain analyze select aef.name ,aef.value ,aef.id ,a.* from applications_extra_fields as aef left join applications as a on aef.id = a.id; QUERY PLAN ------------------------------------------------------------------------------------------------------------------------------------------------------------ ---- Gather Motion 3:1 (slice1; segments: 3) (cost=0.00..1689.57 rows=199266 width=386) (actual time=79.799..416.211 rows=100000 loops=1) -> Hash Right Join (cost=0.00..1402.93 rows=66422 width=386) (actual time=78.964..299.241 rows=33850 loops=1) Hash Cond: (a.id = aef.id) Extra Text: (seg1) Hash chain length 10.1 avg, 20 max, using 3365 of 262144 buckets. -> Custom Scan (Anser Bloom Consumer) (cost=0.00..499.02 rows=333334 width=353) (actual time=21.440..224.105 rows=3385 loops=1) Bloom Filter Size: 1048576 bytes Bloom Filter Stats: memory=1024kB checked=332791 rejected=329406 Rows Removed by Bloom Filter: 329406 -> Seq Scan on applications a (cost=0.00..499.02 rows=333334 width=353) (actual time=2.663..193.493 rows=334042 loops=1) -> Hash (cost=437.42..437.42 rows=33334 width=33) (actual time=56.086..56.088 rows=33850 loops=1) Buckets: 262144 Batches: 1 Memory Usage: 4233kB -> Redistribute Motion 3:3 (slice2; segments: 3) (cost=0.00..437.42 rows=33334 width=33) (actual time=0.176..50.171 rows=33850 loops=1) Hash Key: aef.id -> Custom Scan (Anser Bloom Producer) (cost=0.00..431.94 rows=33334 width=33) (actual time=0.361..41.874 rows=50000 loops=1) Bloom Filter Size: 1048576 bytes Bloom Filter Stats: memory=1024kB -> Seq Scan on applications_extra_fields aef (cost=0.00..431.94 rows=33334 width=33) (actual time=0.360..6.863 rows=50000 loops =1) Planning Time: 15.377 ms (slice0) Executor memory: 1143K bytes. (slice1) Executor memory: 8605K bytes avg x 3x(0) workers, 8637K bytes max (seg0). Work_mem: 4233K bytes max. (slice2) Executor memory: 2370K bytes avg x 3x(0) workers, 2370K bytes max (seg2). Memory used: 128000kB Optimizer: GPORCA Execution Time: 427.266 ms ``` ### The main architecture overview ### Open questions 1. We use Custom Nodes, do not create our own Anser nodes. The main reason here is to make PG rebase process easier. Is it OK? 2. We use libpq protocol to send/get data to/from master. Since we added new functions to work with Anser those functions were registered in a catalog with oid 8195-8197, and catversion was increased. It it Ok or we should create our own protocol? 3. Anser is runtime instrumentation to improve execution time without changes in current planning flow. So `PlannedStmt * planner` was modified and special hook `AnserApplyRuntimeFilters` was added. We generate execution plan and after that add to it special steps for gathering/redistribution runtime data. Is it Ok or we should fix the planner too? 4. Anser consumer does not have timeout. So when step executes it just opens connection an wait for data. Do not use timeout, we cannot use it here, just open connection and wait data. If something goes wrong consumer will be wait forever. It'd be better to limit waits somehow, but I cannot understand how to do it. ### How to enable && test it ``` SET gp_anser_enable=on SET gp_anser_runtime_filter=on; ``` ``` make -C src/test/modules/anser make -C src/test/modules/anser install make -C src/test/modules/anser installcheck ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
