[
https://issues.apache.org/jira/browse/IMPALA-12444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17771217#comment-17771217
]
ASF subversion and git services commented on IMPALA-12444:
----------------------------------------------------------
Commit 89a48b80a2689115d6bd90bf4312a452ae11477a in impala's branch
refs/heads/master from Riza Suminto
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=89a48b80a ]
IMPALA-12444: Fix minimum parallelism bug in scan fragment
Scan fragment did not follow PROCESSING_COST_MIN_THREADS set by user
even if total scan ranges allow to do so. This patch fix the issue by
exposing ScanNode.maxScannerThreads_ to
PlanFragment.adjustToMaxParallelism(). By using
ScanNode.maxScannerThreads_ as an upper bound, ScanNode does not need to
artificially lower ProcessingCost if maxScannerThreads_ is lower than
minimum parallelism dictated by the original ProcessingCost. Thus, the
synthetic ProcessingCost logic in ScanNode class is revised to only
apply if input cardinality is unknown (-1).
This patch also does the following adjustments:
- Remove some dead codes in Frontend.java and PlanFragment.java.
- Add sanity check such that PROCESSING_COST_MIN_THREADS <=
MAX_FRAGMENT_INSTANCES_PER_NODE.
- Tidy up test_query_cpu_count_divisor_default to reduce number of
SET query.
Testing:
- Update test_query_cpu_count_divisor_default to ensure that
PROCESSING_COST_MIN_THREADS is respected by scan fragment and error
is returned if PROCESSING_COST_MIN_THREADS is greater than
MAX_FRAGMENT_INSTANCES_PER_NODE.
- Pass test_executor_groups.py.
Change-Id: I69e5a80146d4ac41de5ef406fc2bdceffe3ec394
Reviewed-on: http://gerrit.cloudera.org:8080/20475
Reviewed-by: Kurt Deschler <[email protected]>
Reviewed-by: Wenzhe Zhou <[email protected]>
Tested-by: Riza Suminto <[email protected]>
> PROCESSING_COST_MIN_THREADS can get ignored by scan fragment.
> -------------------------------------------------------------
>
> Key: IMPALA-12444
> URL: https://issues.apache.org/jira/browse/IMPALA-12444
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Affects Versions: Impala 4.2.0
> Reporter: Riza Suminto
> Assignee: Riza Suminto
> Priority: Major
>
> There is a bug in PlanFragment.java where scan fragment might not follow
> PROCESSING_COST_MIN_THREADS set by user even if total scan ranges allow to do
> so.
> Frontend planner also need to sanity check such that
> PROCESSING_COST_MIN_THREADS <= MAX_FRAGMENT_INSTANCES_PER_NODE.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
