Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20366 )
Change subject: IMPALA-12357: Skip scheduling bloom filter from full-build scan ...................................................................... Patch Set 4: (3 comments) http://gerrit.cloudera.org:8080/#/c/20366/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20366/4//COMMIT_MSG@22 PS4, Line 22: e any predicate filter > One filter I am confused about is null checking - won't we add a null check You mean planner add null checking predicate on join key column? I don't think I see that optimization yet. http://gerrit.cloudera.org:8080/#/c/20366/2/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java File fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java: http://gerrit.cloudera.org:8080/#/c/20366/2/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@858 PS2, Line 858: > I still don't get it - don't we check rule 2 with hasScanConjuncts() and in Yes. Basically the reason we have hasEliminatedFilter flag and loop one more time upon filter removal on previous loop is that some remaining filter might have scan node that turn into not having incomingFilters at all. If the filter iteration is sorted on filter ID, there is a chance that we can do all necessary removal in single pass. But since filter ID assignment is based on pre-order traversal of plan tree, it is possible that removal of filter in the deep right hand can cause different filter in the deep left hand to be eligible for the next removal (its scan node is not targeted by any incoming filter anymore). TPC-DS Q64 is an example of query with two deep left and right subtree of join node near plan root. http://gerrit.cloudera.org:8080/#/c/20366/2/testdata/workloads/functional-planner/queries/PlannerTest/bloom-filter-assignment.test File testdata/workloads/functional-planner/queries/PlannerTest/bloom-filter-assignment.test: http://gerrit.cloudera.org:8080/#/c/20366/2/testdata/workloads/functional-planner/queries/PlannerTest/bloom-filter-assignment.test@543 PS2, Line 543: # IMPALA-12357: filter size 512KB can achieve fpp lower than 0.9. > FK/PK joins the runtime filter is not useful if there are no > predicates/runtime filters on the build side, regardless of FPP. This is tricky because it only hold if the PK-FK constraint is truly enforced (all FK column values is inclusive towards all PK column values). Another case is if PK has no NULL, but FK has many NULL. In that case, filter is still beneficial to remove rows with NULL, regardless FPP. Hence this patch chose to only remove filter with high FPP. -- To view, visit http://gerrit.cloudera.org:8080/20366 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I494533bc06da84e606cbd1ae1619083333089a5e Gerrit-Change-Number: 20366 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto <[email protected]> Gerrit-Reviewer: Aman Sinha <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Comment-Date: Wed, 23 Aug 2023 15:26:45 +0000 Gerrit-HasComments: Yes
