[
https://issues.apache.org/jira/browse/IMPALA-13333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17936931#comment-17936931
]
ASF subversion and git services commented on IMPALA-13333:
----------------------------------------------------------
Commit ad124d1dba77a223ea7360e9be06d9e0a775fb05 in impala's branch
refs/heads/master from Riza Suminto
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=ad124d1db ]
IMPALA-13872: Deflake test_query_cpu_count_divisor_default assertion
IMPALA-13333 adds test that assert the value of "Cluster Memory
Admitted" counter. However, this counter can have slightly different
value depending of target filesystem (HDFS, Ozone, S3). This cause
flakiness in test_query_cpu_count_divisor_default.
This patch remove such assertion from
test_query_cpu_count_divisor_default. The remaining assertion is
sufficient to ensure correctness of system under test.
Testing:
- Run and pass test_query_cpu_count_divisor_default.
Change-Id: I676ee31728de2886acc72d11b8ece14f0238814b
Reviewed-on: http://gerrit.cloudera.org:8080/22636
Reviewed-by: Daniel Becker <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Curb memory estimation for SORT node
> ------------------------------------
>
> Key: IMPALA-13333
> URL: https://issues.apache.org/jira/browse/IMPALA-13333
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
> Reporter: Riza Suminto
> Assignee: Riza Suminto
> Priority: Major
> Fix For: Impala 5.0.0
>
>
> High cardinality overestimation can lead to severe memory overestimation for
> SORT node, even in Parallel Plan. TPC-DS Q31 and Q51 plan against synthetic
> 3TB scale workload shows such huge overestimation:
> [https://github.com/apache/impala/blob/ae6a3b9ec058dfea4b4f93d4828761f792f0b55e/testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-q31.test#L1319-L1323]
> [https://github.com/apache/impala/blob/ae6a3b9ec058dfea4b4f93d4828761f792f0b55e/testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cpu_cost/tpcds-q51.test#L511-L515]
> Planner should be aware to not estimate terabytes/petabytes of memory for
> SORT node, knowing that SORT node has ability to spill-to-disk under memory
> pressure. Planner can also take account for SORT_RUN_BYTES_LIMIT or
> MAX_SORT_RUN_SIZE option value to come up with lower memory estimate.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
