Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20366 )
Change subject: IMPALA-12357: Skip scheduling bloom filter from full-build scan ...................................................................... Patch Set 4: (3 comments) http://gerrit.cloudera.org:8080/#/c/20366/2/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java File fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java: http://gerrit.cloudera.org:8080/#/c/20366/2/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@858 PS2, Line 858: > Why do we continue only if the last filter was eliminated? It is because of rule 2: "The build scan does not have any predicate filter nor consume any runtime filter". If highFppFilters is not empty and hasEliminatedFilter == false, then the ScanNodes producing the remaining high fpp filters may have either of predicate filter or consuming some other runtime filter. Thus, even though the fpp is high, the resulting bloom filter might be more selective because that ScanNode may not be a full scan after all. Changed the test to have 3 cascading filters. http://gerrit.cloudera.org:8080/#/c/20366/2/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@863 PS2, Line 863: if (incomingFilters == null || incomingFilters.isEmpty()) { : if (LOG.isTraceEnabled()) { : > Is the loop needed after filtering to rank <= 1? Yes, still needed. Rank is obtained from PlanFragment traversal. This one, however, is PlanNode traversal. This is needed to really drill down to the bottom of the fragment (ScanNode), in case the scan fragment has other PlanNode colocated with it. http://gerrit.cloudera.org:8080/#/c/20366/2/testdata/workloads/functional-planner/queries/PlannerTest/bloom-filter-assignment.test File testdata/workloads/functional-planner/queries/PlannerTest/bloom-filter-assignment.test: http://gerrit.cloudera.org:8080/#/c/20366/2/testdata/workloads/functional-planner/queries/PlannerTest/bloom-filter-assignment.test@543 PS2, Line 543: # IMPALA-12357: filter size 512KB can achieve fpp lower than 0.9. > I am a bit surprised that no other test changes were needed - I thought tha I think this is because our test workload is relatively small while the default RUNTIME_FILTER_MAX_SIZE is 16MB. Thus, it is hard to hit high fpp. -- To view, visit http://gerrit.cloudera.org:8080/20366 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I494533bc06da84e606cbd1ae1619083333089a5e Gerrit-Change-Number: 20366 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto <[email protected]> Gerrit-Reviewer: Aman Sinha <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Comment-Date: Tue, 22 Aug 2023 17:56:24 +0000 Gerrit-HasComments: Yes
