Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20366 )
Change subject: IMPALA-12357: Skip scheduling bloom filter from full-build scan ...................................................................... Patch Set 9: (3 comments) http://gerrit.cloudera.org:8080/#/c/20366/7//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20366/7//COMMIT_MSG@26 PS7, Line 26: max_filter_error_rate_from_full_scan flag (default to 0.9). > One more thing that comes to mind is having stats on the table - what will In computeNdvEstimate(), filter NDV is first estimated as cardinality of the build side, then overridden by column stats if exist and smaller. In the event where build side cardinality is also unknown, est_fpp_ will stay 0 because of early return in calculateFilterSize() (RuntimeFilterGenerator.java:726 in this patch). Ofcourse, there is also an issue about possible stale stats (some row deletion has happen). But given high enough RUNTIME_FILTER_MAX_SIZE, I think it should accommodate sufficiently high filter NDV estimate before hitting fpp > 0.9. http://gerrit.cloudera.org:8080/#/c/20366/7//COMMIT_MSG@27 PS7, Line 27: : The fourth criteria is an additional control to eliminate based on fpp : threshold because low > I think we could also try to detect likely PK-FK joins here as an additiona Added the PK-FK estimation check. http://gerrit.cloudera.org:8080/#/c/20366/7/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java File fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java: http://gerrit.cloudera.org:8080/#/c/20366/7/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@838 PS7, Line 838: il > nit: "there exists" Done -- To view, visit http://gerrit.cloudera.org:8080/20366 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I494533bc06da84e606cbd1ae1619083333089a5e Gerrit-Change-Number: 20366 Gerrit-PatchSet: 9 Gerrit-Owner: Riza Suminto <[email protected]> Gerrit-Reviewer: Aman Sinha <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: David Rorke <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Comment-Date: Thu, 31 Aug 2023 18:22:42 +0000 Gerrit-HasComments: Yes
