Riza Suminto has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20475 )
Change subject: IMPALA-12444: Fix minimum parallelism bug in scan fragment ...................................................................... IMPALA-12444: Fix minimum parallelism bug in scan fragment Scan fragment did not follow PROCESSING_COST_MIN_THREADS set by user even if total scan ranges allow to do so. This patch fix the issue by exposing ScanNode.maxScannerThreads_ to PlanFragment.adjustToMaxParallelism(). By using ScanNode.maxScannerThreads_ as an upper bound, ScanNode does not need to artificially lower ProcessingCost if maxScannerThreads_ is lower than minimum parallelism dictated by the original ProcessingCost. Thus, the synthetic ProcessingCost logic in ScanNode class is revised to only apply if input cardinality is unknown (-1). This patch also does the following adjustments: - Remove some dead codes in Frontend.java and PlanFragment.java. - Add sanity check such that PROCESSING_COST_MIN_THREADS <= MAX_FRAGMENT_INSTANCES_PER_NODE. - Tidy up test_query_cpu_count_divisor_default to reduce number of SET query. Testing: - Update test_query_cpu_count_divisor_default to ensure that PROCESSING_COST_MIN_THREADS is respected by scan fragment and error is returned if PROCESSING_COST_MIN_THREADS is greater than MAX_FRAGMENT_INSTANCES_PER_NODE. - Pass test_executor_groups.py. Change-Id: I69e5a80146d4ac41de5ef406fc2bdceffe3ec394 Reviewed-on: http://gerrit.cloudera.org:8080/20475 Reviewed-by: Kurt Deschler <[email protected]> Reviewed-by: Wenzhe Zhou <[email protected]> Tested-by: Riza Suminto <[email protected]> --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/planner/CostingSegment.java M fe/src/main/java/org/apache/impala/planner/PlanFragment.java M fe/src/main/java/org/apache/impala/planner/ScanNode.java M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/custom_cluster/test_executor_groups.py 6 files changed, 62 insertions(+), 86 deletions(-) Approvals: Kurt Deschler: Looks good to me, but someone else must approve Wenzhe Zhou: Looks good to me, approved Riza Suminto: Verified -- To view, visit http://gerrit.cloudera.org:8080/20475 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I69e5a80146d4ac41de5ef406fc2bdceffe3ec394 Gerrit-Change-Number: 20475 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Kurt Deschler <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Reviewer: Wenzhe Zhou <[email protected]>
