Hello Kurt Deschler, Wenzhe Zhou, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20475
to look at the new patch set (#3).
Change subject: IMPALA-12444: Fix minimum parallelism bug in scan fragment
......................................................................
IMPALA-12444: Fix minimum parallelism bug in scan fragment
Scan fragment did not follow PROCESSING_COST_MIN_THREADS set by user
even if total scan ranges allow to do so. This patch fix the issue by
exposing ScanNode.maxScannerThreads_ to
PlanFragment.adjustToMaxParallelism(). By using
ScanNode.maxScannerThreads_ as an upper bound, ScanNode does not need to
artificially lower ProcessingCost if maxScannerThreads_ is lower than
minimum parallelism dictated by the original ProcessingCost. Thus, the
synthetic ProcessingCost logic in ScanNode class is revised to only
apply if input cardinality is unknown (-1).
This patch also does the following adjustments:
- Remove some dead codes in Frontend.java and PlanFragment.java.
- Add sanity check such that PROCESSING_COST_MIN_THREADS <=
MAX_FRAGMENT_INSTANCES_PER_NODE.
- Tidy up test_query_cpu_count_divisor_default to reduce number of
SET query.
Testing:
- Update test_query_cpu_count_divisor_default to ensure that
PROCESSING_COST_MIN_THREADS is respected by scan fragment and error
is returned if PROCESSING_COST_MIN_THREADS is greater than
MAX_FRAGMENT_INSTANCES_PER_NODE.
- Pass test_executor_groups.py.
Change-Id: I69e5a80146d4ac41de5ef406fc2bdceffe3ec394
---
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/planner/CostingSegment.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/main/java/org/apache/impala/planner/ScanNode.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M tests/custom_cluster/test_executor_groups.py
6 files changed, 62 insertions(+), 86 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/20475/3
--
To view, visit http://gerrit.cloudera.org:8080/20475
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I69e5a80146d4ac41de5ef406fc2bdceffe3ec394
Gerrit-Change-Number: 20475
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Kurt Deschler <[email protected]>
Gerrit-Reviewer: Riza Suminto <[email protected]>
Gerrit-Reviewer: Wenzhe Zhou <[email protected]>
