Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20498 )
Change subject: IMPALA-12018: Consider runtime filter for cardinality reduction ...................................................................... Patch Set 16: (2 comments) http://gerrit.cloudera.org:8080/#/c/20498/16//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20498/16//COMMIT_MSG@12 PS16, Line 12: that nit: than http://gerrit.cloudera.org:8080/#/c/20498/16/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java File fe/src/main/java/org/apache/impala/planner/ExchangeNode.java: http://gerrit.cloudera.org:8080/#/c/20498/16/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java@306 PS16, Line 306: getCardinality() I found a potential issue in resource estimation of EXCHANGE node. If getCardinality() is abnormally low here (and in estimateDeferredRPCQueueSize()) after runtime filter reduction, estimatedTotalQueueByteSize may be severely underestimate. SCAN and JOIN node is not impacted. SCAN node estimate is based on scan range count before runtime filter reduction. JOIN node estimate use cardinality, but of the build side. This patch operate on the probe pipeline, so no impact there for JOIN memory estimate. I think this patch should be modified to focus on using reduced cardinality for ProcessingCost only. Resource estimation should keep using original cardinality before runtime filter so that memory estimate stay conservative. -- To view, visit http://gerrit.cloudera.org:8080/20498 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I033789c9b63a8188484e3afde8e646563918b3e1 Gerrit-Change-Number: 20498 Gerrit-PatchSet: 16 Gerrit-Owner: Riza Suminto <[email protected]> Gerrit-Reviewer: Abhishek Rawat <[email protected]> Gerrit-Reviewer: Aman Sinha <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Daniel Becker <[email protected]> Gerrit-Reviewer: David Rorke <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Comment-Date: Thu, 14 Dec 2023 23:14:45 +0000 Gerrit-HasComments: Yes
