This is an automated email from the ASF dual-hosted git repository.
dkuzmenko pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 3b562f7fefb HIVE-28832: Disable SharedWorkOptimization in
hybridgrace_hashjoin_2.q (Seonggon Namgung, reviewed by Denys Kuzmenko)
3b562f7fefb is described below
commit 3b562f7fefbb34ce6c191d23325d9f0c882271b2
Author: seonggon <[email protected]>
AuthorDate: Fri Mar 21 20:44:57 2025 +0900
HIVE-28832: Disable SharedWorkOptimization in hybridgrace_hashjoin_2.q
(Seonggon Namgung, reviewed by Denys Kuzmenko)
Closes #5706
---
.../clientpositive/hybridgrace_hashjoin_2.q | 2 +
.../llap/hybridgrace_hashjoin_2.q.out | 444 ++++++++++-----------
.../tez/hybridgrace_hashjoin_2.q.out | 434 ++++++++++----------
3 files changed, 427 insertions(+), 453 deletions(-)
diff --git a/ql/src/test/queries/clientpositive/hybridgrace_hashjoin_2.q
b/ql/src/test/queries/clientpositive/hybridgrace_hashjoin_2.q
index 6ed771ba685..44c92e55611 100644
--- a/ql/src/test/queries/clientpositive/hybridgrace_hashjoin_2.q
+++ b/ql/src/test/queries/clientpositive/hybridgrace_hashjoin_2.q
@@ -11,6 +11,8 @@ set hive.auto.convert.join.noconditionaltask=true;
set hive.auto.convert.join.noconditionaltask.size=10000000;
set hive.cbo.enable=false;
+-- Disable shared work optimization due to HIVE-26986
+set hive.optimize.shared.work=false;
SELECT '3-way mapjoin (1 big table, 2 small tables)';
diff --git
a/ql/src/test/results/clientpositive/llap/hybridgrace_hashjoin_2.q.out
b/ql/src/test/results/clientpositive/llap/hybridgrace_hashjoin_2.q.out
index 243a6a3b192..f1b75b1ad10 100644
--- a/ql/src/test/results/clientpositive/llap/hybridgrace_hashjoin_2.q.out
+++ b/ql/src/test/results/clientpositive/llap/hybridgrace_hashjoin_2.q.out
@@ -983,19 +983,18 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Map 3 <- Map 1 (BROADCAST_EDGE), Map 8 (BROADCAST_EDGE), Reducer 2
(BROADCAST_EDGE), Reducer 9 (BROADCAST_EDGE)
- Reducer 2 <- Map 1 (SIMPLE_EDGE)
- Reducer 4 <- Map 3 (CUSTOM_SIMPLE_EDGE), Union 5 (CONTAINS)
- Reducer 6 <- Union 5 (SIMPLE_EDGE)
- Reducer 7 <- Map 3 (CUSTOM_SIMPLE_EDGE), Union 5 (CONTAINS)
- Reducer 9 <- Map 8 (SIMPLE_EDGE)
+ Map 2 <- Map 1 (BROADCAST_EDGE), Map 6 (BROADCAST_EDGE)
+ Map 8 <- Map 10 (BROADCAST_EDGE), Map 7 (BROADCAST_EDGE)
+ Reducer 3 <- Map 2 (CUSTOM_SIMPLE_EDGE), Union 4 (CONTAINS)
+ Reducer 5 <- Union 4 (SIMPLE_EDGE)
+ Reducer 9 <- Map 8 (CUSTOM_SIMPLE_EDGE), Union 4 (CONTAINS)
#### A masked pattern was here ####
Vertices:
Map 1
Map Operator Tree:
TableScan
alias: x
- filterExpr: (key is not null or value is not null) (type:
boolean)
+ filterExpr: key is not null (type: boolean)
Statistics: Num rows: 25 Data size: 2150 Basic stats:
COMPLETE Column stats: COMPLETE
Filter Operator
predicate: key is not null (type: boolean)
@@ -1006,56 +1005,32 @@ STAGE PLANS:
sort order: +
Map-reduce partition columns: key (type: string)
Statistics: Num rows: 25 Data size: 2150 Basic stats:
COMPLETE Column stats: COMPLETE
+ Execution mode: vectorized, llap
+ LLAP IO: all inputs
+ Map 10
+ Map Operator Tree:
+ TableScan
+ alias: y
+ filterExpr: value is not null (type: boolean)
+ Statistics: Num rows: 500 Data size: 45500 Basic stats:
COMPLETE Column stats: COMPLETE
Filter Operator
predicate: value is not null (type: boolean)
- Statistics: Num rows: 25 Data size: 2225 Basic stats:
COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 500 Data size: 45500 Basic stats:
COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: value (type: string)
null sort order: z
sort order: +
Map-reduce partition columns: value (type: string)
- Statistics: Num rows: 25 Data size: 2225 Basic stats:
COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 500 Data size: 45500 Basic stats:
COMPLETE Column stats: COMPLETE
Execution mode: vectorized, llap
LLAP IO: all inputs
- Map 3
+ Map 2
Map Operator Tree:
TableScan
alias: z
- filterExpr: (value is not null or key is not null) (type:
boolean)
- Statistics: Num rows: 2000 Data size: 182000 Basic stats:
COMPLETE Column stats: COMPLETE
- Filter Operator
- predicate: value is not null (type: boolean)
- Statistics: Num rows: 2000 Data size: 182000 Basic stats:
COMPLETE Column stats: COMPLETE
- Map Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 value (type: string)
- 1 value (type: string)
- outputColumnNames: _col1
- input vertices:
- 0 Reducer 2
- Statistics: Num rows: 162 Data size: 14418 Basic stats:
COMPLETE Column stats: COMPLETE
- Map Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: string)
- 1 value (type: string)
- input vertices:
- 1 Map 8
- Statistics: Num rows: 263 Data size: 2104 Basic stats:
COMPLETE Column stats: COMPLETE
- Group By Operator
- aggregations: count()
- minReductionHashAggr: 0.99
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 8 Basic stats:
COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- null sort order:
- sort order:
- Statistics: Num rows: 1 Data size: 8 Basic stats:
COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: bigint)
+ filterExpr: key is not null (type: boolean)
+ probeDecodeDetails: cacheKey:HASH_MAP_MAPJOIN_106_container,
bigKeyColName:key, smallTablePos:0, keyRatio:0.5
+ Statistics: Num rows: 2000 Data size: 174000 Basic stats:
COMPLETE Column stats: COMPLETE
Filter Operator
predicate: key is not null (type: boolean)
Statistics: Num rows: 2000 Data size: 174000 Basic stats:
COMPLETE Column stats: COMPLETE
@@ -1076,7 +1051,7 @@ STAGE PLANS:
0 _col0 (type: string)
1 key (type: string)
input vertices:
- 1 Reducer 9
+ 1 Map 6
Statistics: Num rows: 250 Data size: 2000 Basic stats:
COMPLETE Column stats: COMPLETE
Group By Operator
aggregations: count()
@@ -1091,21 +1066,12 @@ STAGE PLANS:
value expressions: _col0 (type: bigint)
Execution mode: vectorized, llap
LLAP IO: all inputs
- Map 8
+ Map 6
Map Operator Tree:
TableScan
alias: y
- filterExpr: (value is not null or key is not null) (type:
boolean)
- Statistics: Num rows: 500 Data size: 45500 Basic stats:
COMPLETE Column stats: COMPLETE
- Filter Operator
- predicate: value is not null (type: boolean)
- Statistics: Num rows: 500 Data size: 45500 Basic stats:
COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: value (type: string)
- null sort order: z
- sort order: +
- Map-reduce partition columns: value (type: string)
- Statistics: Num rows: 500 Data size: 45500 Basic stats:
COMPLETE Column stats: COMPLETE
+ filterExpr: key is not null (type: boolean)
+ Statistics: Num rows: 500 Data size: 43500 Basic stats:
COMPLETE Column stats: COMPLETE
Filter Operator
predicate: key is not null (type: boolean)
Statistics: Num rows: 500 Data size: 43500 Basic stats:
COMPLETE Column stats: COMPLETE
@@ -1117,19 +1083,66 @@ STAGE PLANS:
Statistics: Num rows: 500 Data size: 43500 Basic stats:
COMPLETE Column stats: COMPLETE
Execution mode: vectorized, llap
LLAP IO: all inputs
- Reducer 2
- Execution mode: vectorized, llap
- Reduce Operator Tree:
- Select Operator
- expressions: KEY.reducesinkkey0 (type: string)
- outputColumnNames: value
- Reduce Output Operator
- key expressions: value (type: string)
- null sort order: z
- sort order: +
- Map-reduce partition columns: value (type: string)
+ Map 7
+ Map Operator Tree:
+ TableScan
+ alias: x
+ filterExpr: value is not null (type: boolean)
Statistics: Num rows: 25 Data size: 2225 Basic stats:
COMPLETE Column stats: COMPLETE
- Reducer 4
+ Filter Operator
+ predicate: value is not null (type: boolean)
+ Statistics: Num rows: 25 Data size: 2225 Basic stats:
COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: value (type: string)
+ null sort order: z
+ sort order: +
+ Map-reduce partition columns: value (type: string)
+ Statistics: Num rows: 25 Data size: 2225 Basic stats:
COMPLETE Column stats: COMPLETE
+ Execution mode: vectorized, llap
+ LLAP IO: all inputs
+ Map 8
+ Map Operator Tree:
+ TableScan
+ alias: z
+ filterExpr: value is not null (type: boolean)
+ probeDecodeDetails: cacheKey:HASH_MAP_MAPJOIN_108_container,
bigKeyColName:value, smallTablePos:0, keyRatio:0.5276872964169381
+ Statistics: Num rows: 2000 Data size: 182000 Basic stats:
COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: value is not null (type: boolean)
+ Statistics: Num rows: 2000 Data size: 182000 Basic stats:
COMPLETE Column stats: COMPLETE
+ Map Join Operator
+ condition map:
+ Inner Join 0 to 1
+ keys:
+ 0 value (type: string)
+ 1 value (type: string)
+ outputColumnNames: _col1
+ input vertices:
+ 0 Map 7
+ Statistics: Num rows: 162 Data size: 14418 Basic stats:
COMPLETE Column stats: COMPLETE
+ Map Join Operator
+ condition map:
+ Inner Join 0 to 1
+ keys:
+ 0 _col1 (type: string)
+ 1 value (type: string)
+ input vertices:
+ 1 Map 10
+ Statistics: Num rows: 263 Data size: 2104 Basic stats:
COMPLETE Column stats: COMPLETE
+ Group By Operator
+ aggregations: count()
+ minReductionHashAggr: 0.99
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats:
COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ null sort order:
+ sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats:
COMPLETE Column stats: COMPLETE
+ value expressions: _col0 (type: bigint)
+ Execution mode: vectorized, llap
+ LLAP IO: all inputs
+ Reducer 3
Execution mode: vectorized, llap
Reduce Operator Tree:
Group By Operator
@@ -1149,7 +1162,7 @@ STAGE PLANS:
sort order: +
Map-reduce partition columns: _col0 (type: bigint)
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE
Column stats: COMPLETE
- Reducer 6
+ Reducer 5
Execution mode: vectorized, llap
Reduce Operator Tree:
Group By Operator
@@ -1164,7 +1177,7 @@ STAGE PLANS:
input format:
org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 7
+ Reducer 9
Execution mode: vectorized, llap
Reduce Operator Tree:
Group By Operator
@@ -1184,20 +1197,8 @@ STAGE PLANS:
sort order: +
Map-reduce partition columns: _col0 (type: bigint)
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE
Column stats: COMPLETE
- Reducer 9
- Execution mode: vectorized, llap
- Reduce Operator Tree:
- Select Operator
- expressions: KEY.reducesinkkey0 (type: string)
- outputColumnNames: key
- Reduce Output Operator
- key expressions: key (type: string)
- null sort order: z
- sort order: +
- Map-reduce partition columns: key (type: string)
- Statistics: Num rows: 500 Data size: 43500 Basic stats:
COMPLETE Column stats: COMPLETE
- Union 5
- Vertex: Union 5
+ Union 4
+ Vertex: Union 4
Stage: Stage-0
Fetch Operator
@@ -1282,19 +1283,18 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Map 3 <- Map 1 (BROADCAST_EDGE), Map 8 (BROADCAST_EDGE), Reducer 2
(BROADCAST_EDGE), Reducer 9 (BROADCAST_EDGE)
- Reducer 2 <- Map 1 (SIMPLE_EDGE)
- Reducer 4 <- Map 3 (CUSTOM_SIMPLE_EDGE), Union 5 (CONTAINS)
- Reducer 6 <- Union 5 (SIMPLE_EDGE)
- Reducer 7 <- Map 3 (CUSTOM_SIMPLE_EDGE), Union 5 (CONTAINS)
- Reducer 9 <- Map 8 (SIMPLE_EDGE)
+ Map 2 <- Map 1 (BROADCAST_EDGE), Map 6 (BROADCAST_EDGE)
+ Map 8 <- Map 10 (BROADCAST_EDGE), Map 7 (BROADCAST_EDGE)
+ Reducer 3 <- Map 2 (CUSTOM_SIMPLE_EDGE), Union 4 (CONTAINS)
+ Reducer 5 <- Union 4 (SIMPLE_EDGE)
+ Reducer 9 <- Map 8 (CUSTOM_SIMPLE_EDGE), Union 4 (CONTAINS)
#### A masked pattern was here ####
Vertices:
Map 1
Map Operator Tree:
TableScan
alias: x
- filterExpr: (key is not null or value is not null) (type:
boolean)
+ filterExpr: key is not null (type: boolean)
Statistics: Num rows: 25 Data size: 2150 Basic stats:
COMPLETE Column stats: COMPLETE
Filter Operator
predicate: key is not null (type: boolean)
@@ -1305,56 +1305,32 @@ STAGE PLANS:
sort order: +
Map-reduce partition columns: key (type: string)
Statistics: Num rows: 25 Data size: 2150 Basic stats:
COMPLETE Column stats: COMPLETE
+ Execution mode: vectorized, llap
+ LLAP IO: all inputs
+ Map 10
+ Map Operator Tree:
+ TableScan
+ alias: y
+ filterExpr: value is not null (type: boolean)
+ Statistics: Num rows: 500 Data size: 45500 Basic stats:
COMPLETE Column stats: COMPLETE
Filter Operator
predicate: value is not null (type: boolean)
- Statistics: Num rows: 25 Data size: 2225 Basic stats:
COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 500 Data size: 45500 Basic stats:
COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: value (type: string)
null sort order: z
sort order: +
Map-reduce partition columns: value (type: string)
- Statistics: Num rows: 25 Data size: 2225 Basic stats:
COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 500 Data size: 45500 Basic stats:
COMPLETE Column stats: COMPLETE
Execution mode: vectorized, llap
LLAP IO: all inputs
- Map 3
+ Map 2
Map Operator Tree:
TableScan
alias: z
- filterExpr: (value is not null or key is not null) (type:
boolean)
- Statistics: Num rows: 2000 Data size: 182000 Basic stats:
COMPLETE Column stats: COMPLETE
- Filter Operator
- predicate: value is not null (type: boolean)
- Statistics: Num rows: 2000 Data size: 182000 Basic stats:
COMPLETE Column stats: COMPLETE
- Map Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 value (type: string)
- 1 value (type: string)
- outputColumnNames: _col1
- input vertices:
- 0 Reducer 2
- Statistics: Num rows: 162 Data size: 14418 Basic stats:
COMPLETE Column stats: COMPLETE
- Map Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: string)
- 1 value (type: string)
- input vertices:
- 1 Map 8
- Statistics: Num rows: 263 Data size: 2104 Basic stats:
COMPLETE Column stats: COMPLETE
- Group By Operator
- aggregations: count()
- minReductionHashAggr: 0.99
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 8 Basic stats:
COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- null sort order:
- sort order:
- Statistics: Num rows: 1 Data size: 8 Basic stats:
COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: bigint)
+ filterExpr: key is not null (type: boolean)
+ probeDecodeDetails: cacheKey:HASH_MAP_MAPJOIN_106_container,
bigKeyColName:key, smallTablePos:0, keyRatio:0.5
+ Statistics: Num rows: 2000 Data size: 174000 Basic stats:
COMPLETE Column stats: COMPLETE
Filter Operator
predicate: key is not null (type: boolean)
Statistics: Num rows: 2000 Data size: 174000 Basic stats:
COMPLETE Column stats: COMPLETE
@@ -1375,7 +1351,7 @@ STAGE PLANS:
0 _col0 (type: string)
1 key (type: string)
input vertices:
- 1 Reducer 9
+ 1 Map 6
Statistics: Num rows: 250 Data size: 2000 Basic stats:
COMPLETE Column stats: COMPLETE
Group By Operator
aggregations: count()
@@ -1390,21 +1366,12 @@ STAGE PLANS:
value expressions: _col0 (type: bigint)
Execution mode: vectorized, llap
LLAP IO: all inputs
- Map 8
+ Map 6
Map Operator Tree:
TableScan
alias: y
- filterExpr: (value is not null or key is not null) (type:
boolean)
- Statistics: Num rows: 500 Data size: 45500 Basic stats:
COMPLETE Column stats: COMPLETE
- Filter Operator
- predicate: value is not null (type: boolean)
- Statistics: Num rows: 500 Data size: 45500 Basic stats:
COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: value (type: string)
- null sort order: z
- sort order: +
- Map-reduce partition columns: value (type: string)
- Statistics: Num rows: 500 Data size: 45500 Basic stats:
COMPLETE Column stats: COMPLETE
+ filterExpr: key is not null (type: boolean)
+ Statistics: Num rows: 500 Data size: 43500 Basic stats:
COMPLETE Column stats: COMPLETE
Filter Operator
predicate: key is not null (type: boolean)
Statistics: Num rows: 500 Data size: 43500 Basic stats:
COMPLETE Column stats: COMPLETE
@@ -1416,19 +1383,66 @@ STAGE PLANS:
Statistics: Num rows: 500 Data size: 43500 Basic stats:
COMPLETE Column stats: COMPLETE
Execution mode: vectorized, llap
LLAP IO: all inputs
- Reducer 2
- Execution mode: vectorized, llap
- Reduce Operator Tree:
- Select Operator
- expressions: KEY.reducesinkkey0 (type: string)
- outputColumnNames: value
- Reduce Output Operator
- key expressions: value (type: string)
- null sort order: z
- sort order: +
- Map-reduce partition columns: value (type: string)
+ Map 7
+ Map Operator Tree:
+ TableScan
+ alias: x
+ filterExpr: value is not null (type: boolean)
Statistics: Num rows: 25 Data size: 2225 Basic stats:
COMPLETE Column stats: COMPLETE
- Reducer 4
+ Filter Operator
+ predicate: value is not null (type: boolean)
+ Statistics: Num rows: 25 Data size: 2225 Basic stats:
COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: value (type: string)
+ null sort order: z
+ sort order: +
+ Map-reduce partition columns: value (type: string)
+ Statistics: Num rows: 25 Data size: 2225 Basic stats:
COMPLETE Column stats: COMPLETE
+ Execution mode: vectorized, llap
+ LLAP IO: all inputs
+ Map 8
+ Map Operator Tree:
+ TableScan
+ alias: z
+ filterExpr: value is not null (type: boolean)
+ probeDecodeDetails: cacheKey:HASH_MAP_MAPJOIN_108_container,
bigKeyColName:value, smallTablePos:0, keyRatio:0.5276872964169381
+ Statistics: Num rows: 2000 Data size: 182000 Basic stats:
COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: value is not null (type: boolean)
+ Statistics: Num rows: 2000 Data size: 182000 Basic stats:
COMPLETE Column stats: COMPLETE
+ Map Join Operator
+ condition map:
+ Inner Join 0 to 1
+ keys:
+ 0 value (type: string)
+ 1 value (type: string)
+ outputColumnNames: _col1
+ input vertices:
+ 0 Map 7
+ Statistics: Num rows: 162 Data size: 14418 Basic stats:
COMPLETE Column stats: COMPLETE
+ Map Join Operator
+ condition map:
+ Inner Join 0 to 1
+ keys:
+ 0 _col1 (type: string)
+ 1 value (type: string)
+ input vertices:
+ 1 Map 10
+ Statistics: Num rows: 263 Data size: 2104 Basic stats:
COMPLETE Column stats: COMPLETE
+ Group By Operator
+ aggregations: count()
+ minReductionHashAggr: 0.99
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats:
COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ null sort order:
+ sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats:
COMPLETE Column stats: COMPLETE
+ value expressions: _col0 (type: bigint)
+ Execution mode: vectorized, llap
+ LLAP IO: all inputs
+ Reducer 3
Execution mode: vectorized, llap
Reduce Operator Tree:
Group By Operator
@@ -1448,7 +1462,7 @@ STAGE PLANS:
sort order: +
Map-reduce partition columns: _col0 (type: bigint)
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE
Column stats: COMPLETE
- Reducer 6
+ Reducer 5
Execution mode: vectorized, llap
Reduce Operator Tree:
Group By Operator
@@ -1463,7 +1477,7 @@ STAGE PLANS:
input format:
org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 7
+ Reducer 9
Execution mode: vectorized, llap
Reduce Operator Tree:
Group By Operator
@@ -1483,20 +1497,8 @@ STAGE PLANS:
sort order: +
Map-reduce partition columns: _col0 (type: bigint)
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE
Column stats: COMPLETE
- Reducer 9
- Execution mode: vectorized, llap
- Reduce Operator Tree:
- Select Operator
- expressions: KEY.reducesinkkey0 (type: string)
- outputColumnNames: key
- Reduce Output Operator
- key expressions: key (type: string)
- null sort order: z
- sort order: +
- Map-reduce partition columns: key (type: string)
- Statistics: Num rows: 500 Data size: 43500 Basic stats:
COMPLETE Column stats: COMPLETE
- Union 5
- Vertex: Union 5
+ Union 4
+ Vertex: Union 4
Stage: Stage-0
Fetch Operator
@@ -1592,9 +1594,8 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Map 2 <- Map 1 (BROADCAST_EDGE), Map 4 (BROADCAST_EDGE), Map 6
(BROADCAST_EDGE), Reducer 5 (BROADCAST_EDGE)
+ Map 2 <- Map 1 (BROADCAST_EDGE), Map 4 (BROADCAST_EDGE), Map 5
(BROADCAST_EDGE), Map 6 (BROADCAST_EDGE)
Reducer 3 <- Map 2 (CUSTOM_SIMPLE_EDGE)
- Reducer 5 <- Map 4 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -1643,7 +1644,7 @@ STAGE PLANS:
1 key (type: string)
outputColumnNames: _col1
input vertices:
- 1 Reducer 5
+ 1 Map 4
Statistics: Num rows: 18 Data size: 1602 Basic stats:
COMPLETE Column stats: COMPLETE
Map Join Operator
condition map:
@@ -1653,7 +1654,7 @@ STAGE PLANS:
1 value (type: string)
outputColumnNames: _col1
input vertices:
- 1 Map 6
+ 1 Map 5
Statistics: Num rows: 114 Data size: 10146 Basic
stats: COMPLETE Column stats: COMPLETE
Map Join Operator
condition map:
@@ -1662,7 +1663,7 @@ STAGE PLANS:
0 _col1 (type: string)
1 value (type: string)
input vertices:
- 1 Map 4
+ 1 Map 6
Statistics: Num rows: 185 Data size: 1480 Basic
stats: COMPLETE Column stats: COMPLETE
Group By Operator
aggregations: count()
@@ -1681,7 +1682,7 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: y1
- filterExpr: (((value < 'zzzzzzzz') and (key < 'zzzzzzzz'))
or (value < 'zzzzzzzzzz')) (type: boolean)
+ filterExpr: ((value < 'zzzzzzzz') and (key < 'zzzzzzzz'))
(type: boolean)
Statistics: Num rows: 500 Data size: 89000 Basic stats:
COMPLETE Column stats: COMPLETE
Filter Operator
predicate: ((value < 'zzzzzzzz') and (key < 'zzzzzzzz'))
(type: boolean)
@@ -1692,32 +1693,40 @@ STAGE PLANS:
sort order: +
Map-reduce partition columns: key (type: string)
Statistics: Num rows: 55 Data size: 9790 Basic stats:
COMPLETE Column stats: COMPLETE
+ Execution mode: vectorized, llap
+ LLAP IO: all inputs
+ Map 5
+ Map Operator Tree:
+ TableScan
+ alias: z2
+ filterExpr: ((key < 'zzzzzzzzzz') and (value <
'zzzzzzzzzz')) (type: boolean)
+ Statistics: Num rows: 2000 Data size: 356000 Basic stats:
COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: (value < 'zzzzzzzzzz') (type: boolean)
- Statistics: Num rows: 166 Data size: 15106 Basic stats:
COMPLETE Column stats: COMPLETE
+ predicate: ((key < 'zzzzzzzzzz') and (value <
'zzzzzzzzzz')) (type: boolean)
+ Statistics: Num rows: 222 Data size: 39516 Basic stats:
COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: value (type: string)
null sort order: z
sort order: +
Map-reduce partition columns: value (type: string)
- Statistics: Num rows: 166 Data size: 15106 Basic stats:
COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 222 Data size: 39516 Basic stats:
COMPLETE Column stats: COMPLETE
Execution mode: vectorized, llap
LLAP IO: all inputs
Map 6
Map Operator Tree:
TableScan
- alias: z2
- filterExpr: ((key < 'zzzzzzzzzz') and (value <
'zzzzzzzzzz')) (type: boolean)
- Statistics: Num rows: 2000 Data size: 356000 Basic stats:
COMPLETE Column stats: COMPLETE
+ alias: y2
+ filterExpr: (value < 'zzzzzzzzzz') (type: boolean)
+ Statistics: Num rows: 500 Data size: 45500 Basic stats:
COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((key < 'zzzzzzzzzz') and (value <
'zzzzzzzzzz')) (type: boolean)
- Statistics: Num rows: 222 Data size: 39516 Basic stats:
COMPLETE Column stats: COMPLETE
+ predicate: (value < 'zzzzzzzzzz') (type: boolean)
+ Statistics: Num rows: 166 Data size: 15106 Basic stats:
COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: value (type: string)
null sort order: z
sort order: +
Map-reduce partition columns: value (type: string)
- Statistics: Num rows: 222 Data size: 39516 Basic stats:
COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 166 Data size: 15106 Basic stats:
COMPLETE Column stats: COMPLETE
Execution mode: vectorized, llap
LLAP IO: all inputs
Reducer 3
@@ -1735,18 +1744,6 @@ STAGE PLANS:
input format:
org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 5
- Execution mode: vectorized, llap
- Reduce Operator Tree:
- Select Operator
- expressions: KEY.reducesinkkey0 (type: string)
- outputColumnNames: key
- Reduce Output Operator
- key expressions: key (type: string)
- null sort order: z
- sort order: +
- Map-reduce partition columns: key (type: string)
- Statistics: Num rows: 55 Data size: 9790 Basic stats:
COMPLETE Column stats: COMPLETE
Stage: Stage-0
Fetch Operator
@@ -1834,9 +1831,8 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Map 2 <- Map 1 (BROADCAST_EDGE), Map 4 (BROADCAST_EDGE), Map 6
(BROADCAST_EDGE), Reducer 5 (BROADCAST_EDGE)
+ Map 2 <- Map 1 (BROADCAST_EDGE), Map 4 (BROADCAST_EDGE), Map 5
(BROADCAST_EDGE), Map 6 (BROADCAST_EDGE)
Reducer 3 <- Map 2 (CUSTOM_SIMPLE_EDGE)
- Reducer 5 <- Map 4 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -1885,7 +1881,7 @@ STAGE PLANS:
1 key (type: string)
outputColumnNames: _col1
input vertices:
- 1 Reducer 5
+ 1 Map 4
Statistics: Num rows: 18 Data size: 1602 Basic stats:
COMPLETE Column stats: COMPLETE
Map Join Operator
condition map:
@@ -1895,7 +1891,7 @@ STAGE PLANS:
1 value (type: string)
outputColumnNames: _col1
input vertices:
- 1 Map 6
+ 1 Map 5
Statistics: Num rows: 114 Data size: 10146 Basic
stats: COMPLETE Column stats: COMPLETE
Map Join Operator
condition map:
@@ -1904,7 +1900,7 @@ STAGE PLANS:
0 _col1 (type: string)
1 value (type: string)
input vertices:
- 1 Map 4
+ 1 Map 6
Statistics: Num rows: 185 Data size: 1480 Basic
stats: COMPLETE Column stats: COMPLETE
Group By Operator
aggregations: count()
@@ -1923,7 +1919,7 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: y1
- filterExpr: (((value < 'zzzzzzzz') and (key < 'zzzzzzzz'))
or (value < 'zzzzzzzzzz')) (type: boolean)
+ filterExpr: ((value < 'zzzzzzzz') and (key < 'zzzzzzzz'))
(type: boolean)
Statistics: Num rows: 500 Data size: 89000 Basic stats:
COMPLETE Column stats: COMPLETE
Filter Operator
predicate: ((value < 'zzzzzzzz') and (key < 'zzzzzzzz'))
(type: boolean)
@@ -1934,32 +1930,40 @@ STAGE PLANS:
sort order: +
Map-reduce partition columns: key (type: string)
Statistics: Num rows: 55 Data size: 9790 Basic stats:
COMPLETE Column stats: COMPLETE
+ Execution mode: vectorized, llap
+ LLAP IO: all inputs
+ Map 5
+ Map Operator Tree:
+ TableScan
+ alias: z2
+ filterExpr: ((key < 'zzzzzzzzzz') and (value <
'zzzzzzzzzz')) (type: boolean)
+ Statistics: Num rows: 2000 Data size: 356000 Basic stats:
COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: (value < 'zzzzzzzzzz') (type: boolean)
- Statistics: Num rows: 166 Data size: 15106 Basic stats:
COMPLETE Column stats: COMPLETE
+ predicate: ((key < 'zzzzzzzzzz') and (value <
'zzzzzzzzzz')) (type: boolean)
+ Statistics: Num rows: 222 Data size: 39516 Basic stats:
COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: value (type: string)
null sort order: z
sort order: +
Map-reduce partition columns: value (type: string)
- Statistics: Num rows: 166 Data size: 15106 Basic stats:
COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 222 Data size: 39516 Basic stats:
COMPLETE Column stats: COMPLETE
Execution mode: vectorized, llap
LLAP IO: all inputs
Map 6
Map Operator Tree:
TableScan
- alias: z2
- filterExpr: ((key < 'zzzzzzzzzz') and (value <
'zzzzzzzzzz')) (type: boolean)
- Statistics: Num rows: 2000 Data size: 356000 Basic stats:
COMPLETE Column stats: COMPLETE
+ alias: y2
+ filterExpr: (value < 'zzzzzzzzzz') (type: boolean)
+ Statistics: Num rows: 500 Data size: 45500 Basic stats:
COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((key < 'zzzzzzzzzz') and (value <
'zzzzzzzzzz')) (type: boolean)
- Statistics: Num rows: 222 Data size: 39516 Basic stats:
COMPLETE Column stats: COMPLETE
+ predicate: (value < 'zzzzzzzzzz') (type: boolean)
+ Statistics: Num rows: 166 Data size: 15106 Basic stats:
COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: value (type: string)
null sort order: z
sort order: +
Map-reduce partition columns: value (type: string)
- Statistics: Num rows: 222 Data size: 39516 Basic stats:
COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 166 Data size: 15106 Basic stats:
COMPLETE Column stats: COMPLETE
Execution mode: vectorized, llap
LLAP IO: all inputs
Reducer 3
@@ -1977,18 +1981,6 @@ STAGE PLANS:
input format:
org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 5
- Execution mode: vectorized, llap
- Reduce Operator Tree:
- Select Operator
- expressions: KEY.reducesinkkey0 (type: string)
- outputColumnNames: key
- Reduce Output Operator
- key expressions: key (type: string)
- null sort order: z
- sort order: +
- Map-reduce partition columns: key (type: string)
- Statistics: Num rows: 55 Data size: 9790 Basic stats:
COMPLETE Column stats: COMPLETE
Stage: Stage-0
Fetch Operator
diff --git
a/ql/src/test/results/clientpositive/tez/hybridgrace_hashjoin_2.q.out
b/ql/src/test/results/clientpositive/tez/hybridgrace_hashjoin_2.q.out
index 487411e67dc..92f96a6d136 100644
--- a/ql/src/test/results/clientpositive/tez/hybridgrace_hashjoin_2.q.out
+++ b/ql/src/test/results/clientpositive/tez/hybridgrace_hashjoin_2.q.out
@@ -965,19 +965,18 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Map 3 <- Map 1 (BROADCAST_EDGE), Map 8 (BROADCAST_EDGE), Reducer 2
(BROADCAST_EDGE), Reducer 9 (BROADCAST_EDGE)
- Reducer 2 <- Map 1 (SIMPLE_EDGE)
- Reducer 4 <- Map 3 (CUSTOM_SIMPLE_EDGE), Union 5 (CONTAINS)
- Reducer 6 <- Union 5 (SIMPLE_EDGE)
- Reducer 7 <- Map 3 (CUSTOM_SIMPLE_EDGE), Union 5 (CONTAINS)
- Reducer 9 <- Map 8 (SIMPLE_EDGE)
+ Map 2 <- Map 1 (BROADCAST_EDGE), Map 6 (BROADCAST_EDGE)
+ Map 8 <- Map 10 (BROADCAST_EDGE), Map 7 (BROADCAST_EDGE)
+ Reducer 3 <- Map 2 (CUSTOM_SIMPLE_EDGE), Union 4 (CONTAINS)
+ Reducer 5 <- Union 4 (SIMPLE_EDGE)
+ Reducer 9 <- Map 8 (CUSTOM_SIMPLE_EDGE), Union 4 (CONTAINS)
#### A masked pattern was here ####
Vertices:
Map 1
Map Operator Tree:
TableScan
alias: x
- filterExpr: (key is not null or value is not null) (type:
boolean)
+ filterExpr: key is not null (type: boolean)
Statistics: Num rows: 25 Data size: 2150 Basic stats:
COMPLETE Column stats: COMPLETE
Filter Operator
predicate: key is not null (type: boolean)
@@ -988,55 +987,29 @@ STAGE PLANS:
sort order: +
Map-reduce partition columns: key (type: string)
Statistics: Num rows: 25 Data size: 2150 Basic stats:
COMPLETE Column stats: COMPLETE
+ Execution mode: vectorized
+ Map 10
+ Map Operator Tree:
+ TableScan
+ alias: y
+ filterExpr: value is not null (type: boolean)
+ Statistics: Num rows: 500 Data size: 45500 Basic stats:
COMPLETE Column stats: COMPLETE
Filter Operator
predicate: value is not null (type: boolean)
- Statistics: Num rows: 25 Data size: 2225 Basic stats:
COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 500 Data size: 45500 Basic stats:
COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: value (type: string)
null sort order: z
sort order: +
Map-reduce partition columns: value (type: string)
- Statistics: Num rows: 25 Data size: 2225 Basic stats:
COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 500 Data size: 45500 Basic stats:
COMPLETE Column stats: COMPLETE
Execution mode: vectorized
- Map 3
+ Map 2
Map Operator Tree:
TableScan
alias: z
- filterExpr: (value is not null or key is not null) (type:
boolean)
- Statistics: Num rows: 2000 Data size: 182000 Basic stats:
COMPLETE Column stats: COMPLETE
- Filter Operator
- predicate: value is not null (type: boolean)
- Statistics: Num rows: 2000 Data size: 182000 Basic stats:
COMPLETE Column stats: COMPLETE
- Map Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 value (type: string)
- 1 value (type: string)
- outputColumnNames: _col1
- input vertices:
- 0 Reducer 2
- Statistics: Num rows: 162 Data size: 14418 Basic stats:
COMPLETE Column stats: COMPLETE
- Map Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: string)
- 1 value (type: string)
- input vertices:
- 1 Map 8
- Statistics: Num rows: 263 Data size: 2104 Basic stats:
COMPLETE Column stats: COMPLETE
- Group By Operator
- aggregations: count()
- minReductionHashAggr: 0.99
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 8 Basic stats:
COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- null sort order:
- sort order:
- Statistics: Num rows: 1 Data size: 8 Basic stats:
COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: bigint)
+ filterExpr: key is not null (type: boolean)
+ Statistics: Num rows: 2000 Data size: 174000 Basic stats:
COMPLETE Column stats: COMPLETE
Filter Operator
predicate: key is not null (type: boolean)
Statistics: Num rows: 2000 Data size: 174000 Basic stats:
COMPLETE Column stats: COMPLETE
@@ -1057,7 +1030,7 @@ STAGE PLANS:
0 _col0 (type: string)
1 key (type: string)
input vertices:
- 1 Reducer 9
+ 1 Map 6
Statistics: Num rows: 250 Data size: 2000 Basic stats:
COMPLETE Column stats: COMPLETE
Group By Operator
aggregations: count()
@@ -1071,21 +1044,12 @@ STAGE PLANS:
Statistics: Num rows: 1 Data size: 8 Basic stats:
COMPLETE Column stats: COMPLETE
value expressions: _col0 (type: bigint)
Execution mode: vectorized
- Map 8
+ Map 6
Map Operator Tree:
TableScan
alias: y
- filterExpr: (value is not null or key is not null) (type:
boolean)
- Statistics: Num rows: 500 Data size: 45500 Basic stats:
COMPLETE Column stats: COMPLETE
- Filter Operator
- predicate: value is not null (type: boolean)
- Statistics: Num rows: 500 Data size: 45500 Basic stats:
COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: value (type: string)
- null sort order: z
- sort order: +
- Map-reduce partition columns: value (type: string)
- Statistics: Num rows: 500 Data size: 45500 Basic stats:
COMPLETE Column stats: COMPLETE
+ filterExpr: key is not null (type: boolean)
+ Statistics: Num rows: 500 Data size: 43500 Basic stats:
COMPLETE Column stats: COMPLETE
Filter Operator
predicate: key is not null (type: boolean)
Statistics: Num rows: 500 Data size: 43500 Basic stats:
COMPLETE Column stats: COMPLETE
@@ -1096,19 +1060,63 @@ STAGE PLANS:
Map-reduce partition columns: key (type: string)
Statistics: Num rows: 500 Data size: 43500 Basic stats:
COMPLETE Column stats: COMPLETE
Execution mode: vectorized
- Reducer 2
- Execution mode: vectorized
- Reduce Operator Tree:
- Select Operator
- expressions: KEY.reducesinkkey0 (type: string)
- outputColumnNames: value
- Reduce Output Operator
- key expressions: value (type: string)
- null sort order: z
- sort order: +
- Map-reduce partition columns: value (type: string)
+ Map 7
+ Map Operator Tree:
+ TableScan
+ alias: x
+ filterExpr: value is not null (type: boolean)
Statistics: Num rows: 25 Data size: 2225 Basic stats:
COMPLETE Column stats: COMPLETE
- Reducer 4
+ Filter Operator
+ predicate: value is not null (type: boolean)
+ Statistics: Num rows: 25 Data size: 2225 Basic stats:
COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: value (type: string)
+ null sort order: z
+ sort order: +
+ Map-reduce partition columns: value (type: string)
+ Statistics: Num rows: 25 Data size: 2225 Basic stats:
COMPLETE Column stats: COMPLETE
+ Execution mode: vectorized
+ Map 8
+ Map Operator Tree:
+ TableScan
+ alias: z
+ filterExpr: value is not null (type: boolean)
+ Statistics: Num rows: 2000 Data size: 182000 Basic stats:
COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: value is not null (type: boolean)
+ Statistics: Num rows: 2000 Data size: 182000 Basic stats:
COMPLETE Column stats: COMPLETE
+ Map Join Operator
+ condition map:
+ Inner Join 0 to 1
+ keys:
+ 0 value (type: string)
+ 1 value (type: string)
+ outputColumnNames: _col1
+ input vertices:
+ 0 Map 7
+ Statistics: Num rows: 162 Data size: 14418 Basic stats:
COMPLETE Column stats: COMPLETE
+ Map Join Operator
+ condition map:
+ Inner Join 0 to 1
+ keys:
+ 0 _col1 (type: string)
+ 1 value (type: string)
+ input vertices:
+ 1 Map 10
+ Statistics: Num rows: 263 Data size: 2104 Basic stats:
COMPLETE Column stats: COMPLETE
+ Group By Operator
+ aggregations: count()
+ minReductionHashAggr: 0.99
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats:
COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ null sort order:
+ sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats:
COMPLETE Column stats: COMPLETE
+ value expressions: _col0 (type: bigint)
+ Execution mode: vectorized
+ Reducer 3
Execution mode: vectorized
Reduce Operator Tree:
Group By Operator
@@ -1128,7 +1136,7 @@ STAGE PLANS:
sort order: +
Map-reduce partition columns: _col0 (type: bigint)
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE
Column stats: COMPLETE
- Reducer 6
+ Reducer 5
Execution mode: vectorized
Reduce Operator Tree:
Group By Operator
@@ -1143,7 +1151,7 @@ STAGE PLANS:
input format:
org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 7
+ Reducer 9
Execution mode: vectorized
Reduce Operator Tree:
Group By Operator
@@ -1163,20 +1171,8 @@ STAGE PLANS:
sort order: +
Map-reduce partition columns: _col0 (type: bigint)
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE
Column stats: COMPLETE
- Reducer 9
- Execution mode: vectorized
- Reduce Operator Tree:
- Select Operator
- expressions: KEY.reducesinkkey0 (type: string)
- outputColumnNames: key
- Reduce Output Operator
- key expressions: key (type: string)
- null sort order: z
- sort order: +
- Map-reduce partition columns: key (type: string)
- Statistics: Num rows: 500 Data size: 43500 Basic stats:
COMPLETE Column stats: COMPLETE
- Union 5
- Vertex: Union 5
+ Union 4
+ Vertex: Union 4
Stage: Stage-0
Fetch Operator
@@ -1261,19 +1257,18 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Map 3 <- Map 1 (BROADCAST_EDGE), Map 8 (BROADCAST_EDGE), Reducer 2
(BROADCAST_EDGE), Reducer 9 (BROADCAST_EDGE)
- Reducer 2 <- Map 1 (SIMPLE_EDGE)
- Reducer 4 <- Map 3 (CUSTOM_SIMPLE_EDGE), Union 5 (CONTAINS)
- Reducer 6 <- Union 5 (SIMPLE_EDGE)
- Reducer 7 <- Map 3 (CUSTOM_SIMPLE_EDGE), Union 5 (CONTAINS)
- Reducer 9 <- Map 8 (SIMPLE_EDGE)
+ Map 2 <- Map 1 (BROADCAST_EDGE), Map 6 (BROADCAST_EDGE)
+ Map 8 <- Map 10 (BROADCAST_EDGE), Map 7 (BROADCAST_EDGE)
+ Reducer 3 <- Map 2 (CUSTOM_SIMPLE_EDGE), Union 4 (CONTAINS)
+ Reducer 5 <- Union 4 (SIMPLE_EDGE)
+ Reducer 9 <- Map 8 (CUSTOM_SIMPLE_EDGE), Union 4 (CONTAINS)
#### A masked pattern was here ####
Vertices:
Map 1
Map Operator Tree:
TableScan
alias: x
- filterExpr: (key is not null or value is not null) (type:
boolean)
+ filterExpr: key is not null (type: boolean)
Statistics: Num rows: 25 Data size: 2150 Basic stats:
COMPLETE Column stats: COMPLETE
Filter Operator
predicate: key is not null (type: boolean)
@@ -1284,45 +1279,52 @@ STAGE PLANS:
sort order: +
Map-reduce partition columns: key (type: string)
Statistics: Num rows: 25 Data size: 2150 Basic stats:
COMPLETE Column stats: COMPLETE
+ Execution mode: vectorized
+ Map 10
+ Map Operator Tree:
+ TableScan
+ alias: y
+ filterExpr: value is not null (type: boolean)
+ Statistics: Num rows: 500 Data size: 45500 Basic stats:
COMPLETE Column stats: COMPLETE
Filter Operator
predicate: value is not null (type: boolean)
- Statistics: Num rows: 25 Data size: 2225 Basic stats:
COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 500 Data size: 45500 Basic stats:
COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: value (type: string)
null sort order: z
sort order: +
Map-reduce partition columns: value (type: string)
- Statistics: Num rows: 25 Data size: 2225 Basic stats:
COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 500 Data size: 45500 Basic stats:
COMPLETE Column stats: COMPLETE
Execution mode: vectorized
- Map 3
+ Map 2
Map Operator Tree:
TableScan
alias: z
- filterExpr: (value is not null or key is not null) (type:
boolean)
- Statistics: Num rows: 2000 Data size: 182000 Basic stats:
COMPLETE Column stats: COMPLETE
+ filterExpr: key is not null (type: boolean)
+ Statistics: Num rows: 2000 Data size: 174000 Basic stats:
COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: value is not null (type: boolean)
- Statistics: Num rows: 2000 Data size: 182000 Basic stats:
COMPLETE Column stats: COMPLETE
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 2000 Data size: 174000 Basic stats:
COMPLETE Column stats: COMPLETE
Map Join Operator
condition map:
Inner Join 0 to 1
keys:
- 0 value (type: string)
- 1 value (type: string)
- outputColumnNames: _col1
+ 0 key (type: string)
+ 1 key (type: string)
+ outputColumnNames: _col0
input vertices:
- 0 Reducer 2
- Statistics: Num rows: 162 Data size: 14418 Basic stats:
COMPLETE Column stats: COMPLETE
+ 0 Map 1
+ Statistics: Num rows: 158 Data size: 13588 Basic stats:
COMPLETE Column stats: COMPLETE
HybridGraceHashJoin: true
Map Join Operator
condition map:
Inner Join 0 to 1
keys:
- 0 _col1 (type: string)
- 1 value (type: string)
+ 0 _col0 (type: string)
+ 1 key (type: string)
input vertices:
- 1 Map 8
- Statistics: Num rows: 263 Data size: 2104 Basic stats:
COMPLETE Column stats: COMPLETE
+ 1 Map 6
+ Statistics: Num rows: 250 Data size: 2000 Basic stats:
COMPLETE Column stats: COMPLETE
HybridGraceHashJoin: true
Group By Operator
aggregations: count()
@@ -1335,29 +1337,68 @@ STAGE PLANS:
sort order:
Statistics: Num rows: 1 Data size: 8 Basic stats:
COMPLETE Column stats: COMPLETE
value expressions: _col0 (type: bigint)
+ Execution mode: vectorized
+ Map 6
+ Map Operator Tree:
+ TableScan
+ alias: y
+ filterExpr: key is not null (type: boolean)
+ Statistics: Num rows: 500 Data size: 43500 Basic stats:
COMPLETE Column stats: COMPLETE
Filter Operator
predicate: key is not null (type: boolean)
- Statistics: Num rows: 2000 Data size: 174000 Basic stats:
COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 500 Data size: 43500 Basic stats:
COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: key (type: string)
+ null sort order: z
+ sort order: +
+ Map-reduce partition columns: key (type: string)
+ Statistics: Num rows: 500 Data size: 43500 Basic stats:
COMPLETE Column stats: COMPLETE
+ Execution mode: vectorized
+ Map 7
+ Map Operator Tree:
+ TableScan
+ alias: x
+ filterExpr: value is not null (type: boolean)
+ Statistics: Num rows: 25 Data size: 2225 Basic stats:
COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: value is not null (type: boolean)
+ Statistics: Num rows: 25 Data size: 2225 Basic stats:
COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: value (type: string)
+ null sort order: z
+ sort order: +
+ Map-reduce partition columns: value (type: string)
+ Statistics: Num rows: 25 Data size: 2225 Basic stats:
COMPLETE Column stats: COMPLETE
+ Execution mode: vectorized
+ Map 8
+ Map Operator Tree:
+ TableScan
+ alias: z
+ filterExpr: value is not null (type: boolean)
+ Statistics: Num rows: 2000 Data size: 182000 Basic stats:
COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: value is not null (type: boolean)
+ Statistics: Num rows: 2000 Data size: 182000 Basic stats:
COMPLETE Column stats: COMPLETE
Map Join Operator
condition map:
Inner Join 0 to 1
keys:
- 0 key (type: string)
- 1 key (type: string)
- outputColumnNames: _col0
+ 0 value (type: string)
+ 1 value (type: string)
+ outputColumnNames: _col1
input vertices:
- 0 Map 1
- Statistics: Num rows: 158 Data size: 13588 Basic stats:
COMPLETE Column stats: COMPLETE
+ 0 Map 7
+ Statistics: Num rows: 162 Data size: 14418 Basic stats:
COMPLETE Column stats: COMPLETE
HybridGraceHashJoin: true
Map Join Operator
condition map:
Inner Join 0 to 1
keys:
- 0 _col0 (type: string)
- 1 key (type: string)
+ 0 _col1 (type: string)
+ 1 value (type: string)
input vertices:
- 1 Reducer 9
- Statistics: Num rows: 250 Data size: 2000 Basic stats:
COMPLETE Column stats: COMPLETE
+ 1 Map 10
+ Statistics: Num rows: 263 Data size: 2104 Basic stats:
COMPLETE Column stats: COMPLETE
HybridGraceHashJoin: true
Group By Operator
aggregations: count()
@@ -1371,44 +1412,7 @@ STAGE PLANS:
Statistics: Num rows: 1 Data size: 8 Basic stats:
COMPLETE Column stats: COMPLETE
value expressions: _col0 (type: bigint)
Execution mode: vectorized
- Map 8
- Map Operator Tree:
- TableScan
- alias: y
- filterExpr: (value is not null or key is not null) (type:
boolean)
- Statistics: Num rows: 500 Data size: 45500 Basic stats:
COMPLETE Column stats: COMPLETE
- Filter Operator
- predicate: value is not null (type: boolean)
- Statistics: Num rows: 500 Data size: 45500 Basic stats:
COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: value (type: string)
- null sort order: z
- sort order: +
- Map-reduce partition columns: value (type: string)
- Statistics: Num rows: 500 Data size: 45500 Basic stats:
COMPLETE Column stats: COMPLETE
- Filter Operator
- predicate: key is not null (type: boolean)
- Statistics: Num rows: 500 Data size: 43500 Basic stats:
COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: key (type: string)
- null sort order: z
- sort order: +
- Map-reduce partition columns: key (type: string)
- Statistics: Num rows: 500 Data size: 43500 Basic stats:
COMPLETE Column stats: COMPLETE
- Execution mode: vectorized
- Reducer 2
- Execution mode: vectorized
- Reduce Operator Tree:
- Select Operator
- expressions: KEY.reducesinkkey0 (type: string)
- outputColumnNames: value
- Reduce Output Operator
- key expressions: value (type: string)
- null sort order: z
- sort order: +
- Map-reduce partition columns: value (type: string)
- Statistics: Num rows: 25 Data size: 2225 Basic stats:
COMPLETE Column stats: COMPLETE
- Reducer 4
+ Reducer 3
Execution mode: vectorized
Reduce Operator Tree:
Group By Operator
@@ -1428,7 +1432,7 @@ STAGE PLANS:
sort order: +
Map-reduce partition columns: _col0 (type: bigint)
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE
Column stats: COMPLETE
- Reducer 6
+ Reducer 5
Execution mode: vectorized
Reduce Operator Tree:
Group By Operator
@@ -1443,7 +1447,7 @@ STAGE PLANS:
input format:
org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 7
+ Reducer 9
Execution mode: vectorized
Reduce Operator Tree:
Group By Operator
@@ -1463,20 +1467,8 @@ STAGE PLANS:
sort order: +
Map-reduce partition columns: _col0 (type: bigint)
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE
Column stats: COMPLETE
- Reducer 9
- Execution mode: vectorized
- Reduce Operator Tree:
- Select Operator
- expressions: KEY.reducesinkkey0 (type: string)
- outputColumnNames: key
- Reduce Output Operator
- key expressions: key (type: string)
- null sort order: z
- sort order: +
- Map-reduce partition columns: key (type: string)
- Statistics: Num rows: 500 Data size: 43500 Basic stats:
COMPLETE Column stats: COMPLETE
- Union 5
- Vertex: Union 5
+ Union 4
+ Vertex: Union 4
Stage: Stage-0
Fetch Operator
@@ -1572,9 +1564,8 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Map 2 <- Map 1 (BROADCAST_EDGE), Map 4 (BROADCAST_EDGE), Map 6
(BROADCAST_EDGE), Reducer 5 (BROADCAST_EDGE)
+ Map 2 <- Map 1 (BROADCAST_EDGE), Map 4 (BROADCAST_EDGE), Map 5
(BROADCAST_EDGE), Map 6 (BROADCAST_EDGE)
Reducer 3 <- Map 2 (CUSTOM_SIMPLE_EDGE)
- Reducer 5 <- Map 4 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -1621,7 +1612,7 @@ STAGE PLANS:
1 key (type: string)
outputColumnNames: _col1
input vertices:
- 1 Reducer 5
+ 1 Map 4
Statistics: Num rows: 18 Data size: 1602 Basic stats:
COMPLETE Column stats: COMPLETE
Map Join Operator
condition map:
@@ -1631,7 +1622,7 @@ STAGE PLANS:
1 value (type: string)
outputColumnNames: _col1
input vertices:
- 1 Map 6
+ 1 Map 5
Statistics: Num rows: 114 Data size: 10146 Basic
stats: COMPLETE Column stats: COMPLETE
Map Join Operator
condition map:
@@ -1640,7 +1631,7 @@ STAGE PLANS:
0 _col1 (type: string)
1 value (type: string)
input vertices:
- 1 Map 4
+ 1 Map 6
Statistics: Num rows: 185 Data size: 1480 Basic
stats: COMPLETE Column stats: COMPLETE
Group By Operator
aggregations: count()
@@ -1658,7 +1649,7 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: y1
- filterExpr: (((value < 'zzzzzzzz') and (key < 'zzzzzzzz'))
or (value < 'zzzzzzzzzz')) (type: boolean)
+ filterExpr: ((value < 'zzzzzzzz') and (key < 'zzzzzzzz'))
(type: boolean)
Statistics: Num rows: 500 Data size: 89000 Basic stats:
COMPLETE Column stats: COMPLETE
Filter Operator
predicate: ((value < 'zzzzzzzz') and (key < 'zzzzzzzz'))
(type: boolean)
@@ -1669,31 +1660,38 @@ STAGE PLANS:
sort order: +
Map-reduce partition columns: key (type: string)
Statistics: Num rows: 55 Data size: 9790 Basic stats:
COMPLETE Column stats: COMPLETE
+ Execution mode: vectorized
+ Map 5
+ Map Operator Tree:
+ TableScan
+ alias: z2
+ filterExpr: ((key < 'zzzzzzzzzz') and (value <
'zzzzzzzzzz')) (type: boolean)
+ Statistics: Num rows: 2000 Data size: 356000 Basic stats:
COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: (value < 'zzzzzzzzzz') (type: boolean)
- Statistics: Num rows: 166 Data size: 15106 Basic stats:
COMPLETE Column stats: COMPLETE
+ predicate: ((key < 'zzzzzzzzzz') and (value <
'zzzzzzzzzz')) (type: boolean)
+ Statistics: Num rows: 222 Data size: 39516 Basic stats:
COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: value (type: string)
null sort order: z
sort order: +
Map-reduce partition columns: value (type: string)
- Statistics: Num rows: 166 Data size: 15106 Basic stats:
COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 222 Data size: 39516 Basic stats:
COMPLETE Column stats: COMPLETE
Execution mode: vectorized
Map 6
Map Operator Tree:
TableScan
- alias: z2
- filterExpr: ((key < 'zzzzzzzzzz') and (value <
'zzzzzzzzzz')) (type: boolean)
- Statistics: Num rows: 2000 Data size: 356000 Basic stats:
COMPLETE Column stats: COMPLETE
+ alias: y2
+ filterExpr: (value < 'zzzzzzzzzz') (type: boolean)
+ Statistics: Num rows: 500 Data size: 45500 Basic stats:
COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((key < 'zzzzzzzzzz') and (value <
'zzzzzzzzzz')) (type: boolean)
- Statistics: Num rows: 222 Data size: 39516 Basic stats:
COMPLETE Column stats: COMPLETE
+ predicate: (value < 'zzzzzzzzzz') (type: boolean)
+ Statistics: Num rows: 166 Data size: 15106 Basic stats:
COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: value (type: string)
null sort order: z
sort order: +
Map-reduce partition columns: value (type: string)
- Statistics: Num rows: 222 Data size: 39516 Basic stats:
COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 166 Data size: 15106 Basic stats:
COMPLETE Column stats: COMPLETE
Execution mode: vectorized
Reducer 3
Execution mode: vectorized
@@ -1710,18 +1708,6 @@ STAGE PLANS:
input format:
org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 5
- Execution mode: vectorized
- Reduce Operator Tree:
- Select Operator
- expressions: KEY.reducesinkkey0 (type: string)
- outputColumnNames: key
- Reduce Output Operator
- key expressions: key (type: string)
- null sort order: z
- sort order: +
- Map-reduce partition columns: key (type: string)
- Statistics: Num rows: 55 Data size: 9790 Basic stats:
COMPLETE Column stats: COMPLETE
Stage: Stage-0
Fetch Operator
@@ -1809,9 +1795,8 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Map 2 <- Map 1 (BROADCAST_EDGE), Map 4 (BROADCAST_EDGE), Map 6
(BROADCAST_EDGE), Reducer 5 (BROADCAST_EDGE)
+ Map 2 <- Map 1 (BROADCAST_EDGE), Map 4 (BROADCAST_EDGE), Map 5
(BROADCAST_EDGE), Map 6 (BROADCAST_EDGE)
Reducer 3 <- Map 2 (CUSTOM_SIMPLE_EDGE)
- Reducer 5 <- Map 4 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -1859,7 +1844,7 @@ STAGE PLANS:
1 key (type: string)
outputColumnNames: _col1
input vertices:
- 1 Reducer 5
+ 1 Map 4
Statistics: Num rows: 18 Data size: 1602 Basic stats:
COMPLETE Column stats: COMPLETE
HybridGraceHashJoin: true
Map Join Operator
@@ -1870,7 +1855,7 @@ STAGE PLANS:
1 value (type: string)
outputColumnNames: _col1
input vertices:
- 1 Map 6
+ 1 Map 5
Statistics: Num rows: 114 Data size: 10146 Basic
stats: COMPLETE Column stats: COMPLETE
HybridGraceHashJoin: true
Map Join Operator
@@ -1880,7 +1865,7 @@ STAGE PLANS:
0 _col1 (type: string)
1 value (type: string)
input vertices:
- 1 Map 4
+ 1 Map 6
Statistics: Num rows: 185 Data size: 1480 Basic
stats: COMPLETE Column stats: COMPLETE
HybridGraceHashJoin: true
Group By Operator
@@ -1899,7 +1884,7 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: y1
- filterExpr: (((value < 'zzzzzzzz') and (key < 'zzzzzzzz'))
or (value < 'zzzzzzzzzz')) (type: boolean)
+ filterExpr: ((value < 'zzzzzzzz') and (key < 'zzzzzzzz'))
(type: boolean)
Statistics: Num rows: 500 Data size: 89000 Basic stats:
COMPLETE Column stats: COMPLETE
Filter Operator
predicate: ((value < 'zzzzzzzz') and (key < 'zzzzzzzz'))
(type: boolean)
@@ -1910,31 +1895,38 @@ STAGE PLANS:
sort order: +
Map-reduce partition columns: key (type: string)
Statistics: Num rows: 55 Data size: 9790 Basic stats:
COMPLETE Column stats: COMPLETE
+ Execution mode: vectorized
+ Map 5
+ Map Operator Tree:
+ TableScan
+ alias: z2
+ filterExpr: ((key < 'zzzzzzzzzz') and (value <
'zzzzzzzzzz')) (type: boolean)
+ Statistics: Num rows: 2000 Data size: 356000 Basic stats:
COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: (value < 'zzzzzzzzzz') (type: boolean)
- Statistics: Num rows: 166 Data size: 15106 Basic stats:
COMPLETE Column stats: COMPLETE
+ predicate: ((key < 'zzzzzzzzzz') and (value <
'zzzzzzzzzz')) (type: boolean)
+ Statistics: Num rows: 222 Data size: 39516 Basic stats:
COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: value (type: string)
null sort order: z
sort order: +
Map-reduce partition columns: value (type: string)
- Statistics: Num rows: 166 Data size: 15106 Basic stats:
COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 222 Data size: 39516 Basic stats:
COMPLETE Column stats: COMPLETE
Execution mode: vectorized
Map 6
Map Operator Tree:
TableScan
- alias: z2
- filterExpr: ((key < 'zzzzzzzzzz') and (value <
'zzzzzzzzzz')) (type: boolean)
- Statistics: Num rows: 2000 Data size: 356000 Basic stats:
COMPLETE Column stats: COMPLETE
+ alias: y2
+ filterExpr: (value < 'zzzzzzzzzz') (type: boolean)
+ Statistics: Num rows: 500 Data size: 45500 Basic stats:
COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: ((key < 'zzzzzzzzzz') and (value <
'zzzzzzzzzz')) (type: boolean)
- Statistics: Num rows: 222 Data size: 39516 Basic stats:
COMPLETE Column stats: COMPLETE
+ predicate: (value < 'zzzzzzzzzz') (type: boolean)
+ Statistics: Num rows: 166 Data size: 15106 Basic stats:
COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: value (type: string)
null sort order: z
sort order: +
Map-reduce partition columns: value (type: string)
- Statistics: Num rows: 222 Data size: 39516 Basic stats:
COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 166 Data size: 15106 Basic stats:
COMPLETE Column stats: COMPLETE
Execution mode: vectorized
Reducer 3
Execution mode: vectorized
@@ -1951,18 +1943,6 @@ STAGE PLANS:
input format:
org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 5
- Execution mode: vectorized
- Reduce Operator Tree:
- Select Operator
- expressions: KEY.reducesinkkey0 (type: string)
- outputColumnNames: key
- Reduce Output Operator
- key expressions: key (type: string)
- null sort order: z
- sort order: +
- Map-reduce partition columns: key (type: string)
- Statistics: Num rows: 55 Data size: 9790 Basic stats:
COMPLETE Column stats: COMPLETE
Stage: Stage-0
Fetch Operator
