Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11854 )
Change subject: IMPALA-7791: Compute AggregationNode's estimated rows using # instances ...................................................................... IMPALA-7791: Compute AggregationNode's estimated rows using # instances Previously, the AggregationNode calculated the estimated number of rows based on input cardinality without accounting for the division of input data across multiple fragment instances. This bloated up the memory estimates for the node. After this change, the AggregationNode accounts for the number of fragment instances while estimating the number of rows per instance. A skew factor of 1.5 was added to account for data skew among multiple fragment instances. This number was derived using empirical analysis of real-world and benchmark (tpch, tpcds) queries. Testing: Tested queries with changed estimates to avoid cases of significant underestimation of memory. Ran front-end and end-to-end tests affected by this change. Change-Id: I2cb9746fafa3e5952e28caa952837e285bcc22ac Reviewed-on: http://gerrit.cloudera.org:8080/11854 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M fe/src/main/java/org/apache/impala/planner/AggregationNode.java M testdata/workloads/functional-planner/queries/PlannerTest/max-row-size.test M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/spillable-buffer-sizing.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test 7 files changed, 97 insertions(+), 82 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11854 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I2cb9746fafa3e5952e28caa952837e285bcc22ac Gerrit-Change-Number: 11854 Gerrit-PatchSet: 4 Gerrit-Owner: Pooja Nilangekar <[email protected]> Gerrit-Reviewer: Bikramjeet Vig <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Pooja Nilangekar <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]>
