liuyao has posted comments on this change. ( http://gerrit.cloudera.org:8080/16842 )
Change subject: IMPALA-10377: Improve the accuracy of resource estimation PlanNode does not consider some factors when estimating memory, this will cause a large error rate ...................................................................... Patch Set 12: (6 comments) http://gerrit.cloudera.org:8080/#/c/16842/10//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/16842/10//COMMIT_MSG@7 PS10, Line 7: IMPALA-10377: Improve the accuracy of resource estimation > Few general suggestions on the commit message: Done http://gerrit.cloudera.org:8080/#/c/16842/10//COMMIT_MSG@25 PS10, Line 25: Buckets= roundUpToPowerOf2(RightTableCardinality) * > Is this describing the original issue or the nature of the fix itself ? I a Done http://gerrit.cloudera.org:8080/#/c/16842/10/fe/src/main/java/org/apache/impala/planner/AggregationNode.java File fe/src/main/java/org/apache/impala/planner/AggregationNode.java: http://gerrit.cloudera.org:8080/#/c/16842/10/fe/src/main/java/org/apache/impala/planner/AggregationNode.java@66 PS10, Line 66: // Peak memory is at least 16k, which is an empirical value > It seems if we are fixing this estimate for plain aggregates (i.e with no g Done http://gerrit.cloudera.org:8080/#/c/16842/10/fe/src/main/java/org/apache/impala/planner/AggregationNode.java@538 PS10, Line 538: if (useStreamingPreagg_) { > nit: you need not mention the 1.5 value here because it is already a static Done http://gerrit.cloudera.org:8080/#/c/16842/10/fe/src/main/java/org/apache/impala/planner/KuduScanNode.java File fe/src/main/java/org/apache/impala/planner/KuduScanNode.java: http://gerrit.cloudera.org:8080/#/c/16842/10/fe/src/main/java/org/apache/impala/planner/KuduScanNode.java@324 PS10, Line 324: kudu_scanner_thread_estimated_bytes_per_column; > You could add a new method numMaterializedSlots() to TupleDescriptor becaus Done http://gerrit.cloudera.org:8080/#/c/16842/3/testdata/workloads/functional-planner/queries/PlannerTest/topn-bytes-limit.test File testdata/workloads/functional-planner/queries/PlannerTest/topn-bytes-limit.test: http://gerrit.cloudera.org:8080/#/c/16842/3/testdata/workloads/functional-planner/queries/PlannerTest/topn-bytes-limit.test@11 PS3, Line 11: partitions=4/4 files=4 size=460B > Thanks for your patch. Since there was no meaningful change in resource est Done -- To view, visit http://gerrit.cloudera.org:8080/16842 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic01db168ff2c6d6de33ee553a8175599f035d7a1 Gerrit-Change-Number: 16842 Gerrit-PatchSet: 12 Gerrit-Owner: liuyao <[email protected]> Gerrit-Reviewer: Aman Sinha <[email protected]> Gerrit-Reviewer: Bikramjeet Vig <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]> Gerrit-Reviewer: liuyao <[email protected]> Gerrit-Comment-Date: Wed, 06 Jan 2021 08:14:11 +0000 Gerrit-HasComments: Yes
