Pooja Nilangekar has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/11854 )
Change subject: IMPALA-7791: Compute AggregationNode's estimated rows using # instances ...................................................................... IMPALA-7791: Compute AggregationNode's estimated rows using # instances Previously, the AggregationNode calculated the estimated number of rows based on input cardinality without accounting for the division of input data across multiple fragment instances. This bloated up the memory estimates for the node. After this change, the AggregationNode accounts for the number of fragment instances while estimating the number of rows per instance. A skew factor of 1.5 was added to account for data skew among multiple fragment instances. This number was derived using empirical analysis of real-world and benchmark (tpch, tpcds) queries. Testing: Tested queries with changed estimates to avoid cases of significant underestimation of memory. Ran front-end and end-to-end tests affected by this change. Change-Id: I2cb9746fafa3e5952e28caa952837e285bcc22ac --- M fe/src/main/java/org/apache/impala/planner/AggregationNode.java M testdata/workloads/functional-planner/queries/PlannerTest/max-row-size.test M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/spillable-buffer-sizing.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test 7 files changed, 97 insertions(+), 82 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/11854/2 -- To view, visit http://gerrit.cloudera.org:8080/11854 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2cb9746fafa3e5952e28caa952837e285bcc22ac Gerrit-Change-Number: 11854 Gerrit-PatchSet: 2 Gerrit-Owner: Pooja Nilangekar <pooja.nilange...@cloudera.com> Gerrit-Reviewer: Bikramjeet Vig <bikramjeet....@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Pooja Nilangekar <pooja.nilange...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>
