Qifan Chen has uploaded a new patch set (#17). (
http://gerrit.cloudera.org:8080/19033 )
Change subject: IMPALA-11604 Planner changes for CPU usage
......................................................................
IMPALA-11604 Planner changes for CPU usage
This patch augments IMPALA-10992 by establishing an infrastructure
to allow weighted total amount of data to process to be used as a
new factor in the definition and selection of an executor group. In
this patch the weight component is set to 1.
Two flavors of the weighted amount of data processed are enabled in
this patch: the minimal amount and the max amount. The former is the
sum of that of every fragment in the query reflecting the worst level
of parallel execution among fragments. The latter is the minimal of
that of every fragment reflecting the best level of parallel
execution. The geometric mean of the best and the worst is used as
the estimated processing cost of the query.
A fragment's total amount of data processed is the sum of that of
every node in the fragment. The weighted amount of data processed
is computed with a general formula as follows.
D = (I * C * W) / N
where D is the weighted amount of data processed
I is input cardinality
C is expression evaluation cost per row, set to 1
W is average row size
N is number of instances
A description of the computation for each kind of plan node is
given below.
1. Aggregation node:
C and W are the sum of the costs and partial row widths for each
AggregateInfo object.
2. AnalyticEval node:
C is sum of the evaluation costs for analytic functions, partition
by equal and order by equal predicate;
3. CardinalityCheck node:
Both C and I are 1;
4. DataSource scan node:
C is computed from a subset of the selection predicates excluding
data source accepted predicates;
5. EmptySet node:
I is 0;
6. Exchange node:
A modification of the general formula when in broadcast mode:
D = D * number of receivers;
7. Hash join node:
probe side = (I0 * C(equi-join predicate) * W) / N +
(output cardinality * C(other join predicate) * W) / N
build side = (I1 * C(equi-join predicate) * W) / N
8. Hbase scan node:
N is 1
9. Hdfs and Kudu scan node:
N is mt_dop when query option mt_dop >= 1, otherwise
N is number of nodes * max scan threads;
10. Nested loop join node:
When the right child is not a SingularRowSrc node:
probe side = (I0 * C(equi-join predicate) * W) / N +
(output cardinality * C(other join predicate) * W) / N
build side = (I1 * C(equi-join predicate) * W) / N
When the right child is a SingularRowSrc node:
probe side = (I0 * W) / N
build side = I0 * ((I1 * W) / N)
11. Select node:
Use the general formula;
12. SingularRowSrc node:
Since the node is involved once per input in nested loop join, the
contribution of this node is computed in nested loop join;
13. Sort node:
C is the evalation cost for the sort expression and W is the width
of the intermediate tuple being sorted;
14. Subplan node:
C is 1. I is the multiplication of the cardinality of the left and
the right child;
15. Union node:
C is the cost of materializing rows from all non pass-through
children. W is the width of all non pass-through children;
16. Unnest node:
I is the cardinality of the containing subplan node and C is 1.
This patch also assumes that the number of instances of execution
overlaps each other when there is a discrepancy of instances among
nodes in a single fragment. For example when there are 6 scan thread
and 3 aggregation threads in a single fragment, 3 threads are used
for both scan and aggregate, and 3 other threads are used for the
aggregation.
Testing:
1. Unit testing by examining the best and the worst processing cost
computed for all plan nodes in all fragments for a small set of
queries;
2. Core tests.
Change-Id: If32dc770dfffcdd0be2b5555a789a7720952c68a
---
M common/thrift/Frontend.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/AggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/SortInfo.java
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java
M fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java
M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M fe/src/main/java/org/apache/impala/planner/EmptySetNode.java
M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java
M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java
M fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
A fe/src/main/java/org/apache/impala/planner/ProcessingCost.java
M fe/src/main/java/org/apache/impala/planner/ResourceProfile.java
M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java
M fe/src/main/java/org/apache/impala/planner/ScanNode.java
M fe/src/main/java/org/apache/impala/planner/SelectNode.java
M fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java
M fe/src/main/java/org/apache/impala/planner/SortNode.java
M fe/src/main/java/org/apache/impala/planner/SubplanNode.java
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/main/java/org/apache/impala/planner/UnnestNode.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
A fe/src/main/java/org/apache/impala/util/AutoScaleUtil.java
M fe/src/main/java/org/apache/impala/util/ExprUtil.java
M fe/src/main/java/org/apache/impala/util/MathUtil.java
33 files changed, 638 insertions(+), 53 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/19033/17
--
To view, visit http://gerrit.cloudera.org:8080/19033
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If32dc770dfffcdd0be2b5555a789a7720952c68a
Gerrit-Change-Number: 19033
Gerrit-PatchSet: 17
Gerrit-Owner: Qifan Chen <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Kurt Deschler <[email protected]>
Gerrit-Reviewer: Qifan Chen <[email protected]>
Gerrit-Reviewer: Wenzhe Zhou <[email protected]>
