Qifan Chen has uploaded a new patch set (#10). (
http://gerrit.cloudera.org:8080/19033 )
Change subject: [WIP] IMPALA-11604 Planner changes for CPU usage
......................................................................
[WIP] IMPALA-11604 Planner changes for CPU usage
This patch augments IMPALA-10992 by allowing the weighted amount
of data processed to be used as a new factor in the definition
and selection of an executor group.
The weighted amount of data processed is the sum of that in every
fragment, which is the sum of that in every node in the fragment.
For each node, the weighted amount of data processed is computed
with a general formula as follows.
D = I * C * W / N
where D is the weighted amount of data processed
I is input cardinality
C is expression evaluation cost per row
W is average row size
N is number of instances
A description of the computation for each kind of plan node is
given below.
1. Aggregation node:
C and W are the sum of the costs and partial row widths for each
AggregateInfo object.
2. AnalyticEval node:
C is sum of the evaluation costs for analytic functions, partition
by equal and order by equal predicate;
3. CardinalityCheck node:
Both C and I are 1;
4. DataSource scan node:
C is computed from a subset of the selection predicates excluding
data source accepted predicates;
5. EmptySet node:
I is 0;
6. Exchange node:
A modification of the general formula when in broadcast mode:
D = (I * C * W / N) * number of receivers;
7. Hash join node:
C is sum of the evaluation cost for equi-join predicate and for
other join predicate, for both probe and build side;
8. Hbase scan node:
N is 1
9. Hdfs and Kudu scan node:
N is mt_dop when query option mt_dop >= 1, otherwise
N is number of nodes * max scan threads;
10. Nested loop join node:
When the right child is not a SingularRowSrc node, C is sum of
the evaluation cost for equi-join predicate and for other join
predicate, for both probe and build side.
When the right child is a SingularRowSrc node, the cost for build
side is multiplied by the cardinality from the probe side;
11. Select node:
Use the general formula;
12. SingularRowSrc node:
I is 1. Since the node is involved once per input in nested loop
join, the total cost of this node is computed in nested loop join.
13. Sort node:
14. Subplan node:
15. Union node:
16. Unnest node:
Change-Id: If32dc770dfffcdd0be2b5555a789a7720952c68a
---
M common/thrift/Frontend.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/AggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java
M fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java
M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M fe/src/main/java/org/apache/impala/planner/EmptySetNode.java
M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java
M fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/planner/ResourceProfile.java
M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java
M fe/src/main/java/org/apache/impala/planner/ScanNode.java
M fe/src/main/java/org/apache/impala/planner/SelectNode.java
M fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java
M fe/src/main/java/org/apache/impala/planner/SortNode.java
M fe/src/main/java/org/apache/impala/planner/SubplanNode.java
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/main/java/org/apache/impala/planner/UnnestNode.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/ExprUtil.java
29 files changed, 336 insertions(+), 56 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/19033/10
--
To view, visit http://gerrit.cloudera.org:8080/19033
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If32dc770dfffcdd0be2b5555a789a7720952c68a
Gerrit-Change-Number: 19033
Gerrit-PatchSet: 10
Gerrit-Owner: Qifan Chen <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Kurt Deschler <[email protected]>
Gerrit-Reviewer: Qifan Chen <[email protected]>
Gerrit-Reviewer: Wenzhe Zhou <[email protected]>
