Repository: hive Updated Branches: refs/heads/master 3573549bc -> 642acdf76
http://git-wip-us.apache.org/repos/asf/hive/blob/642acdf7/ql/src/test/results/clientpositive/spark/spark_dynamic_partition_pruning_3.q.out ---------------------------------------------------------------------- diff --git a/ql/src/test/results/clientpositive/spark/spark_dynamic_partition_pruning_3.q.out b/ql/src/test/results/clientpositive/spark/spark_dynamic_partition_pruning_3.q.out index 82e746a..3ae922f 100644 --- a/ql/src/test/results/clientpositive/spark/spark_dynamic_partition_pruning_3.q.out +++ b/ql/src/test/results/clientpositive/spark/spark_dynamic_partition_pruning_3.q.out @@ -1,65 +1,1307 @@ -PREHOOK: query: create table partitioned_table1 (col int) partitioned by (part_col int) +PREHOOK: query: CREATE TABLE partitioned_table1 (col int) PARTITIONED BY (part_col int) PREHOOK: type: CREATETABLE PREHOOK: Output: database:default PREHOOK: Output: default@partitioned_table1 -POSTHOOK: query: create table partitioned_table1 (col int) partitioned by (part_col int) +POSTHOOK: query: CREATE TABLE partitioned_table1 (col int) PARTITIONED BY (part_col int) POSTHOOK: type: CREATETABLE POSTHOOK: Output: database:default POSTHOOK: Output: default@partitioned_table1 -PREHOOK: query: create table partitioned_table2 (col int) partitioned by (part_col int) +PREHOOK: query: CREATE TABLE partitioned_table2 (col int) PARTITIONED BY (part_col int) PREHOOK: type: CREATETABLE PREHOOK: Output: database:default PREHOOK: Output: default@partitioned_table2 -POSTHOOK: query: create table partitioned_table2 (col int) partitioned by (part_col int) +POSTHOOK: query: CREATE TABLE partitioned_table2 (col int) PARTITIONED BY (part_col int) POSTHOOK: type: CREATETABLE POSTHOOK: Output: database:default POSTHOOK: Output: default@partitioned_table2 -PREHOOK: query: create table regular_table (col int) +PREHOOK: query: CREATE TABLE partitioned_table3 (col int) PARTITIONED BY (part_col int) PREHOOK: type: CREATETABLE PREHOOK: Output: database:default -PREHOOK: Output: default@regular_table -POSTHOOK: query: create table regular_table (col int) +PREHOOK: Output: default@partitioned_table3 +POSTHOOK: query: CREATE TABLE partitioned_table3 (col int) PARTITIONED BY (part_col int) POSTHOOK: type: CREATETABLE POSTHOOK: Output: database:default -POSTHOOK: Output: default@regular_table -PREHOOK: query: insert into table regular_table values (1) -PREHOOK: type: QUERY -PREHOOK: Output: default@regular_table -POSTHOOK: query: insert into table regular_table values (1) -POSTHOOK: type: QUERY -POSTHOOK: Output: default@regular_table -POSTHOOK: Lineage: regular_table.col EXPRESSION [(values__tmp__table__1)values__tmp__table__1.FieldSchema(name:tmp_values_col1, type:string, comment:), ] -PREHOOK: query: alter table partitioned_table1 add partition (part_col = 1) +POSTHOOK: Output: default@partitioned_table3 +PREHOOK: query: CREATE TABLE partitioned_table4 (col int) PARTITIONED BY (part_col1 int, part_col2 int) +PREHOOK: type: CREATETABLE +PREHOOK: Output: database:default +PREHOOK: Output: default@partitioned_table4 +POSTHOOK: query: CREATE TABLE partitioned_table4 (col int) PARTITIONED BY (part_col1 int, part_col2 int) +POSTHOOK: type: CREATETABLE +POSTHOOK: Output: database:default +POSTHOOK: Output: default@partitioned_table4 +PREHOOK: query: CREATE TABLE partitioned_table5 (col int) PARTITIONED BY (part_col1 int, part_col2 int) +PREHOOK: type: CREATETABLE +PREHOOK: Output: database:default +PREHOOK: Output: default@partitioned_table5 +POSTHOOK: query: CREATE TABLE partitioned_table5 (col int) PARTITIONED BY (part_col1 int, part_col2 int) +POSTHOOK: type: CREATETABLE +POSTHOOK: Output: database:default +POSTHOOK: Output: default@partitioned_table5 +PREHOOK: query: CREATE TABLE regular_table1 (col1 int, col2 int) +PREHOOK: type: CREATETABLE +PREHOOK: Output: database:default +PREHOOK: Output: default@regular_table1 +POSTHOOK: query: CREATE TABLE regular_table1 (col1 int, col2 int) +POSTHOOK: type: CREATETABLE +POSTHOOK: Output: database:default +POSTHOOK: Output: default@regular_table1 +PREHOOK: query: CREATE TABLE regular_table2 (col1 int, col2 int) +PREHOOK: type: CREATETABLE +PREHOOK: Output: database:default +PREHOOK: Output: default@regular_table2 +POSTHOOK: query: CREATE TABLE regular_table2 (col1 int, col2 int) +POSTHOOK: type: CREATETABLE +POSTHOOK: Output: database:default +POSTHOOK: Output: default@regular_table2 +PREHOOK: query: ALTER TABLE partitioned_table1 ADD PARTITION (part_col = 1) PREHOOK: type: ALTERTABLE_ADDPARTS PREHOOK: Output: default@partitioned_table1 -POSTHOOK: query: alter table partitioned_table1 add partition (part_col = 1) +POSTHOOK: query: ALTER TABLE partitioned_table1 ADD PARTITION (part_col = 1) POSTHOOK: type: ALTERTABLE_ADDPARTS POSTHOOK: Output: default@partitioned_table1 POSTHOOK: Output: default@partitioned_table1@part_col=1 -PREHOOK: query: insert into table partitioned_table1 partition (part_col = 1) values (1), (2), (3), (4), (5), (6), (7), (8), (9), (10) -PREHOOK: type: QUERY -PREHOOK: Output: default@partitioned_table1@part_col=1 -POSTHOOK: query: insert into table partitioned_table1 partition (part_col = 1) values (1), (2), (3), (4), (5), (6), (7), (8), (9), (10) -POSTHOOK: type: QUERY -POSTHOOK: Output: default@partitioned_table1@part_col=1 -POSTHOOK: Lineage: partitioned_table1 PARTITION(part_col=1).col EXPRESSION [(values__tmp__table__2)values__tmp__table__2.FieldSchema(name:tmp_values_col1, type:string, comment:), ] -PREHOOK: query: alter table partitioned_table2 add partition (part_col = 1) +PREHOOK: query: ALTER TABLE partitioned_table1 ADD PARTITION (part_col = 2) +PREHOOK: type: ALTERTABLE_ADDPARTS +PREHOOK: Output: default@partitioned_table1 +POSTHOOK: query: ALTER TABLE partitioned_table1 ADD PARTITION (part_col = 2) +POSTHOOK: type: ALTERTABLE_ADDPARTS +POSTHOOK: Output: default@partitioned_table1 +POSTHOOK: Output: default@partitioned_table1@part_col=2 +PREHOOK: query: ALTER TABLE partitioned_table1 ADD PARTITION (part_col = 3) +PREHOOK: type: ALTERTABLE_ADDPARTS +PREHOOK: Output: default@partitioned_table1 +POSTHOOK: query: ALTER TABLE partitioned_table1 ADD PARTITION (part_col = 3) +POSTHOOK: type: ALTERTABLE_ADDPARTS +POSTHOOK: Output: default@partitioned_table1 +POSTHOOK: Output: default@partitioned_table1@part_col=3 +PREHOOK: query: ALTER TABLE partitioned_table2 ADD PARTITION (part_col = 1) PREHOOK: type: ALTERTABLE_ADDPARTS PREHOOK: Output: default@partitioned_table2 -POSTHOOK: query: alter table partitioned_table2 add partition (part_col = 1) +POSTHOOK: query: ALTER TABLE partitioned_table2 ADD PARTITION (part_col = 1) POSTHOOK: type: ALTERTABLE_ADDPARTS POSTHOOK: Output: default@partitioned_table2 POSTHOOK: Output: default@partitioned_table2@part_col=1 -PREHOOK: query: insert into table partitioned_table2 partition (part_col = 1) values (1), (2), (3), (4), (5), (6), (7), (8), (9), (10) +PREHOOK: query: ALTER TABLE partitioned_table2 ADD PARTITION (part_col = 2) +PREHOOK: type: ALTERTABLE_ADDPARTS +PREHOOK: Output: default@partitioned_table2 +POSTHOOK: query: ALTER TABLE partitioned_table2 ADD PARTITION (part_col = 2) +POSTHOOK: type: ALTERTABLE_ADDPARTS +POSTHOOK: Output: default@partitioned_table2 +POSTHOOK: Output: default@partitioned_table2@part_col=2 +PREHOOK: query: ALTER TABLE partitioned_table2 ADD PARTITION (part_col = 3) +PREHOOK: type: ALTERTABLE_ADDPARTS +PREHOOK: Output: default@partitioned_table2 +POSTHOOK: query: ALTER TABLE partitioned_table2 ADD PARTITION (part_col = 3) +POSTHOOK: type: ALTERTABLE_ADDPARTS +POSTHOOK: Output: default@partitioned_table2 +POSTHOOK: Output: default@partitioned_table2@part_col=3 +PREHOOK: query: ALTER TABLE partitioned_table3 ADD PARTITION (part_col = 1) +PREHOOK: type: ALTERTABLE_ADDPARTS +PREHOOK: Output: default@partitioned_table3 +POSTHOOK: query: ALTER TABLE partitioned_table3 ADD PARTITION (part_col = 1) +POSTHOOK: type: ALTERTABLE_ADDPARTS +POSTHOOK: Output: default@partitioned_table3 +POSTHOOK: Output: default@partitioned_table3@part_col=1 +PREHOOK: query: ALTER TABLE partitioned_table3 ADD PARTITION (part_col = 2) +PREHOOK: type: ALTERTABLE_ADDPARTS +PREHOOK: Output: default@partitioned_table3 +POSTHOOK: query: ALTER TABLE partitioned_table3 ADD PARTITION (part_col = 2) +POSTHOOK: type: ALTERTABLE_ADDPARTS +POSTHOOK: Output: default@partitioned_table3 +POSTHOOK: Output: default@partitioned_table3@part_col=2 +PREHOOK: query: ALTER TABLE partitioned_table3 ADD PARTITION (part_col = 3) +PREHOOK: type: ALTERTABLE_ADDPARTS +PREHOOK: Output: default@partitioned_table3 +POSTHOOK: query: ALTER TABLE partitioned_table3 ADD PARTITION (part_col = 3) +POSTHOOK: type: ALTERTABLE_ADDPARTS +POSTHOOK: Output: default@partitioned_table3 +POSTHOOK: Output: default@partitioned_table3@part_col=3 +PREHOOK: query: ALTER TABLE partitioned_table4 ADD PARTITION (part_col1 = 1, part_col2 = 1) +PREHOOK: type: ALTERTABLE_ADDPARTS +PREHOOK: Output: default@partitioned_table4 +POSTHOOK: query: ALTER TABLE partitioned_table4 ADD PARTITION (part_col1 = 1, part_col2 = 1) +POSTHOOK: type: ALTERTABLE_ADDPARTS +POSTHOOK: Output: default@partitioned_table4 +POSTHOOK: Output: default@partitioned_table4@part_col1=1/part_col2=1 +PREHOOK: query: ALTER TABLE partitioned_table4 ADD PARTITION (part_col1 = 2, part_col2 = 2) +PREHOOK: type: ALTERTABLE_ADDPARTS +PREHOOK: Output: default@partitioned_table4 +POSTHOOK: query: ALTER TABLE partitioned_table4 ADD PARTITION (part_col1 = 2, part_col2 = 2) +POSTHOOK: type: ALTERTABLE_ADDPARTS +POSTHOOK: Output: default@partitioned_table4 +POSTHOOK: Output: default@partitioned_table4@part_col1=2/part_col2=2 +PREHOOK: query: ALTER TABLE partitioned_table4 ADD PARTITION (part_col1 = 3, part_col2 = 3) +PREHOOK: type: ALTERTABLE_ADDPARTS +PREHOOK: Output: default@partitioned_table4 +POSTHOOK: query: ALTER TABLE partitioned_table4 ADD PARTITION (part_col1 = 3, part_col2 = 3) +POSTHOOK: type: ALTERTABLE_ADDPARTS +POSTHOOK: Output: default@partitioned_table4 +POSTHOOK: Output: default@partitioned_table4@part_col1=3/part_col2=3 +PREHOOK: query: ALTER TABLE partitioned_table5 ADD PARTITION (part_col1 = 1, part_col2 = 1) +PREHOOK: type: ALTERTABLE_ADDPARTS +PREHOOK: Output: default@partitioned_table5 +POSTHOOK: query: ALTER TABLE partitioned_table5 ADD PARTITION (part_col1 = 1, part_col2 = 1) +POSTHOOK: type: ALTERTABLE_ADDPARTS +POSTHOOK: Output: default@partitioned_table5 +POSTHOOK: Output: default@partitioned_table5@part_col1=1/part_col2=1 +PREHOOK: query: ALTER TABLE partitioned_table5 ADD PARTITION (part_col1 = 2, part_col2 = 2) +PREHOOK: type: ALTERTABLE_ADDPARTS +PREHOOK: Output: default@partitioned_table5 +POSTHOOK: query: ALTER TABLE partitioned_table5 ADD PARTITION (part_col1 = 2, part_col2 = 2) +POSTHOOK: type: ALTERTABLE_ADDPARTS +POSTHOOK: Output: default@partitioned_table5 +POSTHOOK: Output: default@partitioned_table5@part_col1=2/part_col2=2 +PREHOOK: query: ALTER TABLE partitioned_table5 ADD PARTITION (part_col1 = 3, part_col2 = 3) +PREHOOK: type: ALTERTABLE_ADDPARTS +PREHOOK: Output: default@partitioned_table5 +POSTHOOK: query: ALTER TABLE partitioned_table5 ADD PARTITION (part_col1 = 3, part_col2 = 3) +POSTHOOK: type: ALTERTABLE_ADDPARTS +POSTHOOK: Output: default@partitioned_table5 +POSTHOOK: Output: default@partitioned_table5@part_col1=3/part_col2=3 +PREHOOK: query: INSERT INTO TABLE regular_table1 VALUES (0, 0), (1, 1), (2, 2) +PREHOOK: type: QUERY +PREHOOK: Output: default@regular_table1 +POSTHOOK: query: INSERT INTO TABLE regular_table1 VALUES (0, 0), (1, 1), (2, 2) +POSTHOOK: type: QUERY +POSTHOOK: Output: default@regular_table1 +POSTHOOK: Lineage: regular_table1.col1 EXPRESSION [(values__tmp__table__1)values__tmp__table__1.FieldSchema(name:tmp_values_col1, type:string, comment:), ] +POSTHOOK: Lineage: regular_table1.col2 EXPRESSION [(values__tmp__table__1)values__tmp__table__1.FieldSchema(name:tmp_values_col2, type:string, comment:), ] +PREHOOK: query: INSERT INTO TABLE regular_table2 VALUES (0, 0), (1, 1), (2, 2) +PREHOOK: type: QUERY +PREHOOK: Output: default@regular_table2 +POSTHOOK: query: INSERT INTO TABLE regular_table2 VALUES (0, 0), (1, 1), (2, 2) +POSTHOOK: type: QUERY +POSTHOOK: Output: default@regular_table2 +POSTHOOK: Lineage: regular_table2.col1 EXPRESSION [(values__tmp__table__2)values__tmp__table__2.FieldSchema(name:tmp_values_col1, type:string, comment:), ] +POSTHOOK: Lineage: regular_table2.col2 EXPRESSION [(values__tmp__table__2)values__tmp__table__2.FieldSchema(name:tmp_values_col2, type:string, comment:), ] +PREHOOK: query: INSERT INTO TABLE partitioned_table1 PARTITION (part_col = 1) VALUES (1), (2), (3), (4), (5), (6) +PREHOOK: type: QUERY +PREHOOK: Output: default@partitioned_table1@part_col=1 +POSTHOOK: query: INSERT INTO TABLE partitioned_table1 PARTITION (part_col = 1) VALUES (1), (2), (3), (4), (5), (6) +POSTHOOK: type: QUERY +POSTHOOK: Output: default@partitioned_table1@part_col=1 +POSTHOOK: Lineage: partitioned_table1 PARTITION(part_col=1).col EXPRESSION [(values__tmp__table__3)values__tmp__table__3.FieldSchema(name:tmp_values_col1, type:string, comment:), ] +PREHOOK: query: INSERT INTO TABLE partitioned_table1 PARTITION (part_col = 2) VALUES (1), (2), (3), (4), (5), (6) +PREHOOK: type: QUERY +PREHOOK: Output: default@partitioned_table1@part_col=2 +POSTHOOK: query: INSERT INTO TABLE partitioned_table1 PARTITION (part_col = 2) VALUES (1), (2), (3), (4), (5), (6) +POSTHOOK: type: QUERY +POSTHOOK: Output: default@partitioned_table1@part_col=2 +POSTHOOK: Lineage: partitioned_table1 PARTITION(part_col=2).col EXPRESSION [(values__tmp__table__4)values__tmp__table__4.FieldSchema(name:tmp_values_col1, type:string, comment:), ] +PREHOOK: query: INSERT INTO TABLE partitioned_table1 PARTITION (part_col = 3) VALUES (1), (2), (3), (4), (5), (6) +PREHOOK: type: QUERY +PREHOOK: Output: default@partitioned_table1@part_col=3 +POSTHOOK: query: INSERT INTO TABLE partitioned_table1 PARTITION (part_col = 3) VALUES (1), (2), (3), (4), (5), (6) +POSTHOOK: type: QUERY +POSTHOOK: Output: default@partitioned_table1@part_col=3 +POSTHOOK: Lineage: partitioned_table1 PARTITION(part_col=3).col EXPRESSION [(values__tmp__table__5)values__tmp__table__5.FieldSchema(name:tmp_values_col1, type:string, comment:), ] +PREHOOK: query: INSERT INTO TABLE partitioned_table2 PARTITION (part_col = 1) VALUES (1), (2), (3), (4), (5), (6) +PREHOOK: type: QUERY +PREHOOK: Output: default@partitioned_table2@part_col=1 +POSTHOOK: query: INSERT INTO TABLE partitioned_table2 PARTITION (part_col = 1) VALUES (1), (2), (3), (4), (5), (6) +POSTHOOK: type: QUERY +POSTHOOK: Output: default@partitioned_table2@part_col=1 +POSTHOOK: Lineage: partitioned_table2 PARTITION(part_col=1).col EXPRESSION [(values__tmp__table__6)values__tmp__table__6.FieldSchema(name:tmp_values_col1, type:string, comment:), ] +PREHOOK: query: INSERT INTO TABLE partitioned_table2 PARTITION (part_col = 2) VALUES (1), (2), (3), (4), (5), (6) +PREHOOK: type: QUERY +PREHOOK: Output: default@partitioned_table2@part_col=2 +POSTHOOK: query: INSERT INTO TABLE partitioned_table2 PARTITION (part_col = 2) VALUES (1), (2), (3), (4), (5), (6) +POSTHOOK: type: QUERY +POSTHOOK: Output: default@partitioned_table2@part_col=2 +POSTHOOK: Lineage: partitioned_table2 PARTITION(part_col=2).col EXPRESSION [(values__tmp__table__7)values__tmp__table__7.FieldSchema(name:tmp_values_col1, type:string, comment:), ] +PREHOOK: query: INSERT INTO TABLE partitioned_table2 PARTITION (part_col = 3) VALUES (1), (2), (3), (4), (5), (6) +PREHOOK: type: QUERY +PREHOOK: Output: default@partitioned_table2@part_col=3 +POSTHOOK: query: INSERT INTO TABLE partitioned_table2 PARTITION (part_col = 3) VALUES (1), (2), (3), (4), (5), (6) +POSTHOOK: type: QUERY +POSTHOOK: Output: default@partitioned_table2@part_col=3 +POSTHOOK: Lineage: partitioned_table2 PARTITION(part_col=3).col EXPRESSION [(values__tmp__table__8)values__tmp__table__8.FieldSchema(name:tmp_values_col1, type:string, comment:), ] +PREHOOK: query: INSERT INTO TABLE partitioned_table3 PARTITION (part_col = 1) VALUES (1), (2), (3), (4), (5), (6) +PREHOOK: type: QUERY +PREHOOK: Output: default@partitioned_table3@part_col=1 +POSTHOOK: query: INSERT INTO TABLE partitioned_table3 PARTITION (part_col = 1) VALUES (1), (2), (3), (4), (5), (6) +POSTHOOK: type: QUERY +POSTHOOK: Output: default@partitioned_table3@part_col=1 +POSTHOOK: Lineage: partitioned_table3 PARTITION(part_col=1).col EXPRESSION [(values__tmp__table__9)values__tmp__table__9.FieldSchema(name:tmp_values_col1, type:string, comment:), ] +PREHOOK: query: INSERT INTO TABLE partitioned_table3 PARTITION (part_col = 2) VALUES (1), (2), (3), (4), (5), (6) +PREHOOK: type: QUERY +PREHOOK: Output: default@partitioned_table3@part_col=2 +POSTHOOK: query: INSERT INTO TABLE partitioned_table3 PARTITION (part_col = 2) VALUES (1), (2), (3), (4), (5), (6) +POSTHOOK: type: QUERY +POSTHOOK: Output: default@partitioned_table3@part_col=2 +POSTHOOK: Lineage: partitioned_table3 PARTITION(part_col=2).col EXPRESSION [(values__tmp__table__10)values__tmp__table__10.FieldSchema(name:tmp_values_col1, type:string, comment:), ] +PREHOOK: query: INSERT INTO TABLE partitioned_table3 PARTITION (part_col = 3) VALUES (1), (2), (3), (4), (5), (6) +PREHOOK: type: QUERY +PREHOOK: Output: default@partitioned_table3@part_col=3 +POSTHOOK: query: INSERT INTO TABLE partitioned_table3 PARTITION (part_col = 3) VALUES (1), (2), (3), (4), (5), (6) +POSTHOOK: type: QUERY +POSTHOOK: Output: default@partitioned_table3@part_col=3 +POSTHOOK: Lineage: partitioned_table3 PARTITION(part_col=3).col EXPRESSION [(values__tmp__table__11)values__tmp__table__11.FieldSchema(name:tmp_values_col1, type:string, comment:), ] +PREHOOK: query: INSERT INTO TABLE partitioned_table4 PARTITION (part_col1 = 1, part_col2 = 1) VALUES (1), (2), (3), (4), (5), (6) +PREHOOK: type: QUERY +PREHOOK: Output: default@partitioned_table4@part_col1=1/part_col2=1 +POSTHOOK: query: INSERT INTO TABLE partitioned_table4 PARTITION (part_col1 = 1, part_col2 = 1) VALUES (1), (2), (3), (4), (5), (6) +POSTHOOK: type: QUERY +POSTHOOK: Output: default@partitioned_table4@part_col1=1/part_col2=1 +POSTHOOK: Lineage: partitioned_table4 PARTITION(part_col1=1,part_col2=1).col EXPRESSION [(values__tmp__table__12)values__tmp__table__12.FieldSchema(name:tmp_values_col1, type:string, comment:), ] +PREHOOK: query: INSERT INTO TABLE partitioned_table4 PARTITION (part_col1 = 2, part_col2 = 2) VALUES (1), (2), (3), (4), (5), (6) +PREHOOK: type: QUERY +PREHOOK: Output: default@partitioned_table4@part_col1=2/part_col2=2 +POSTHOOK: query: INSERT INTO TABLE partitioned_table4 PARTITION (part_col1 = 2, part_col2 = 2) VALUES (1), (2), (3), (4), (5), (6) +POSTHOOK: type: QUERY +POSTHOOK: Output: default@partitioned_table4@part_col1=2/part_col2=2 +POSTHOOK: Lineage: partitioned_table4 PARTITION(part_col1=2,part_col2=2).col EXPRESSION [(values__tmp__table__13)values__tmp__table__13.FieldSchema(name:tmp_values_col1, type:string, comment:), ] +PREHOOK: query: INSERT INTO TABLE partitioned_table4 PARTITION (part_col1 = 3, part_col2 = 3) VALUES (1), (2), (3), (4), (5), (6) +PREHOOK: type: QUERY +PREHOOK: Output: default@partitioned_table4@part_col1=3/part_col2=3 +POSTHOOK: query: INSERT INTO TABLE partitioned_table4 PARTITION (part_col1 = 3, part_col2 = 3) VALUES (1), (2), (3), (4), (5), (6) +POSTHOOK: type: QUERY +POSTHOOK: Output: default@partitioned_table4@part_col1=3/part_col2=3 +POSTHOOK: Lineage: partitioned_table4 PARTITION(part_col1=3,part_col2=3).col EXPRESSION [(values__tmp__table__14)values__tmp__table__14.FieldSchema(name:tmp_values_col1, type:string, comment:), ] +PREHOOK: query: INSERT INTO TABLE partitioned_table5 PARTITION (part_col1 = 1, part_col2 = 1) VALUES (1), (2), (3), (4), (5), (6) +PREHOOK: type: QUERY +PREHOOK: Output: default@partitioned_table5@part_col1=1/part_col2=1 +POSTHOOK: query: INSERT INTO TABLE partitioned_table5 PARTITION (part_col1 = 1, part_col2 = 1) VALUES (1), (2), (3), (4), (5), (6) +POSTHOOK: type: QUERY +POSTHOOK: Output: default@partitioned_table5@part_col1=1/part_col2=1 +POSTHOOK: Lineage: partitioned_table5 PARTITION(part_col1=1,part_col2=1).col EXPRESSION [(values__tmp__table__15)values__tmp__table__15.FieldSchema(name:tmp_values_col1, type:string, comment:), ] +PREHOOK: query: INSERT INTO TABLE partitioned_table5 PARTITION (part_col1 = 2, part_col2 = 2) VALUES (1), (2), (3), (4), (5), (6) +PREHOOK: type: QUERY +PREHOOK: Output: default@partitioned_table5@part_col1=2/part_col2=2 +POSTHOOK: query: INSERT INTO TABLE partitioned_table5 PARTITION (part_col1 = 2, part_col2 = 2) VALUES (1), (2), (3), (4), (5), (6) +POSTHOOK: type: QUERY +POSTHOOK: Output: default@partitioned_table5@part_col1=2/part_col2=2 +POSTHOOK: Lineage: partitioned_table5 PARTITION(part_col1=2,part_col2=2).col EXPRESSION [(values__tmp__table__16)values__tmp__table__16.FieldSchema(name:tmp_values_col1, type:string, comment:), ] +PREHOOK: query: INSERT INTO TABLE partitioned_table5 PARTITION (part_col1 = 3, part_col2 = 3) VALUES (1), (2), (3), (4), (5), (6) +PREHOOK: type: QUERY +PREHOOK: Output: default@partitioned_table5@part_col1=3/part_col2=3 +POSTHOOK: query: INSERT INTO TABLE partitioned_table5 PARTITION (part_col1 = 3, part_col2 = 3) VALUES (1), (2), (3), (4), (5), (6) +POSTHOOK: type: QUERY +POSTHOOK: Output: default@partitioned_table5@part_col1=3/part_col2=3 +POSTHOOK: Lineage: partitioned_table5 PARTITION(part_col1=3,part_col2=3).col EXPRESSION [(values__tmp__table__17)values__tmp__table__17.FieldSchema(name:tmp_values_col1, type:string, comment:), ] +PREHOOK: query: -- ensure the partitioned tables are treated as big tables + + +EXPLAIN SELECT count(*) FROM partitioned_table1 WHERE partitioned_table1.part_col IN ( +SELECT regular_table1.col1 FROM regular_table1 JOIN partitioned_table2 ON +regular_table1.col1 = partitioned_table2.part_col AND partitioned_table2.col > 3 AND regular_table1.col1 > 1) +PREHOOK: type: QUERY +POSTHOOK: query: -- ensure the partitioned tables are treated as big tables + + +EXPLAIN SELECT count(*) FROM partitioned_table1 WHERE partitioned_table1.part_col IN ( +SELECT regular_table1.col1 FROM regular_table1 JOIN partitioned_table2 ON +regular_table1.col1 = partitioned_table2.part_col AND partitioned_table2.col > 3 AND regular_table1.col1 > 1) +POSTHOOK: type: QUERY +STAGE DEPENDENCIES: + Stage-3 is a root stage + Stage-2 depends on stages: Stage-3 + Stage-1 depends on stages: Stage-2 + Stage-0 depends on stages: Stage-1 + +STAGE PLANS: + Stage: Stage-3 + Spark +#### A masked pattern was here #### + Vertices: + Map 3 + Map Operator Tree: + TableScan + alias: regular_table1 + Statistics: Num rows: 3 Data size: 9 Basic stats: COMPLETE Column stats: NONE + Filter Operator + predicate: (col1 > 1) (type: boolean) + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + Select Operator + expressions: col1 (type: int) + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + Spark HashTable Sink Operator + keys: + 0 _col0 (type: int) + 1 _col1 (type: int) + Select Operator + expressions: _col0 (type: int) + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + Group By Operator + keys: _col0 (type: int) + mode: hash + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + Spark Partition Pruning Sink Operator + Target column: part_col (int) + partition key expr: part_col + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + target work: Map 4 + Local Work: + Map Reduce Local Work + + Stage: Stage-2 + Spark +#### A masked pattern was here #### + Vertices: + Map 4 + Map Operator Tree: + TableScan + alias: partitioned_table2 + Statistics: Num rows: 12 Data size: 12 Basic stats: COMPLETE Column stats: NONE + Filter Operator + predicate: (col > 3) (type: boolean) + Statistics: Num rows: 4 Data size: 4 Basic stats: COMPLETE Column stats: NONE + Select Operator + expressions: part_col (type: int) + outputColumnNames: _col1 + Statistics: Num rows: 4 Data size: 4 Basic stats: COMPLETE Column stats: NONE + Map Join Operator + condition map: + Inner Join 0 to 1 + keys: + 0 _col0 (type: int) + 1 _col1 (type: int) + outputColumnNames: _col0 + input vertices: + 0 Map 3 + Statistics: Num rows: 4 Data size: 4 Basic stats: COMPLETE Column stats: NONE + Group By Operator + keys: _col0 (type: int) + mode: hash + outputColumnNames: _col0 + Statistics: Num rows: 4 Data size: 4 Basic stats: COMPLETE Column stats: NONE + Spark HashTable Sink Operator + keys: + 0 _col0 (type: int) + 1 _col0 (type: int) + Select Operator + expressions: _col0 (type: int) + outputColumnNames: _col0 + Statistics: Num rows: 4 Data size: 4 Basic stats: COMPLETE Column stats: NONE + Group By Operator + keys: _col0 (type: int) + mode: hash + outputColumnNames: _col0 + Statistics: Num rows: 4 Data size: 4 Basic stats: COMPLETE Column stats: NONE + Spark Partition Pruning Sink Operator + Target column: part_col (int) + partition key expr: part_col + Statistics: Num rows: 4 Data size: 4 Basic stats: COMPLETE Column stats: NONE + target work: Map 1 + Local Work: + Map Reduce Local Work + + Stage: Stage-1 + Spark + Edges: + Reducer 2 <- Map 1 (GROUP, 1) +#### A masked pattern was here #### + Vertices: + Map 1 + Map Operator Tree: + TableScan + alias: partitioned_table1 + Statistics: Num rows: 12 Data size: 12 Basic stats: COMPLETE Column stats: NONE + Select Operator + expressions: part_col (type: int) + outputColumnNames: _col0 + Statistics: Num rows: 12 Data size: 12 Basic stats: COMPLETE Column stats: NONE + Map Join Operator + condition map: + Left Semi Join 0 to 1 + keys: + 0 _col0 (type: int) + 1 _col0 (type: int) + input vertices: + 1 Map 4 + Statistics: Num rows: 13 Data size: 13 Basic stats: COMPLETE Column stats: NONE + Group By Operator + aggregations: count() + mode: hash + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE + Reduce Output Operator + sort order: + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE + value expressions: _col0 (type: bigint) + Local Work: + Map Reduce Local Work + Reducer 2 + Reduce Operator Tree: + Group By Operator + aggregations: count(VALUE._col0) + mode: mergepartial + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE + File Output Operator + compressed: false + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE + table: + input format: org.apache.hadoop.mapred.SequenceFileInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat + serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe + + Stage: Stage-0 + Fetch Operator + limit: -1 + Processor Tree: + ListSink + +PREHOOK: query: SELECT count(*) +FROM partitioned_table1 +WHERE partitioned_table1.part_col IN (SELECT regular_table1.col1 + FROM regular_table1 + JOIN partitioned_table2 + ON + regular_table1.col1 = partitioned_table2.part_col AND partitioned_table2.col > 3 AND regular_table1.col1 > 1) +PREHOOK: type: QUERY +PREHOOK: Input: default@partitioned_table1 +PREHOOK: Input: default@partitioned_table1@part_col=2 +PREHOOK: Input: default@partitioned_table1@part_col=3 +PREHOOK: Input: default@partitioned_table2 +PREHOOK: Input: default@partitioned_table2@part_col=2 +PREHOOK: Input: default@partitioned_table2@part_col=3 +PREHOOK: Input: default@regular_table1 +#### A masked pattern was here #### +POSTHOOK: query: SELECT count(*) +FROM partitioned_table1 +WHERE partitioned_table1.part_col IN (SELECT regular_table1.col1 + FROM regular_table1 + JOIN partitioned_table2 + ON + regular_table1.col1 = partitioned_table2.part_col AND partitioned_table2.col > 3 AND regular_table1.col1 > 1) +POSTHOOK: type: QUERY +POSTHOOK: Input: default@partitioned_table1 +POSTHOOK: Input: default@partitioned_table1@part_col=2 +POSTHOOK: Input: default@partitioned_table1@part_col=3 +POSTHOOK: Input: default@partitioned_table2 +POSTHOOK: Input: default@partitioned_table2@part_col=2 +POSTHOOK: Input: default@partitioned_table2@part_col=3 +POSTHOOK: Input: default@regular_table1 +#### A masked pattern was here #### +6 +PREHOOK: query: EXPLAIN SELECT count(*) FROM partitioned_table1, regular_table1 rt1, +regular_table1 rt2 WHERE rt1.col1 = partitioned_table1.part_col AND rt2.col1 = +partitioned_table1.part_col AND rt1.col2 > 0 AND rt2.col2 > 1 +PREHOOK: type: QUERY +POSTHOOK: query: EXPLAIN SELECT count(*) FROM partitioned_table1, regular_table1 rt1, +regular_table1 rt2 WHERE rt1.col1 = partitioned_table1.part_col AND rt2.col1 = +partitioned_table1.part_col AND rt1.col2 > 0 AND rt2.col2 > 1 +POSTHOOK: type: QUERY +STAGE DEPENDENCIES: + Stage-2 is a root stage + Stage-1 depends on stages: Stage-2 + Stage-0 depends on stages: Stage-1 + +STAGE PLANS: + Stage: Stage-2 + Spark +#### A masked pattern was here #### + Vertices: + Map 3 + Map Operator Tree: + TableScan + alias: rt1 + Statistics: Num rows: 3 Data size: 9 Basic stats: COMPLETE Column stats: NONE + Filter Operator + predicate: ((col2 > 0) and col1 is not null) (type: boolean) + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + Select Operator + expressions: col1 (type: int) + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + Spark HashTable Sink Operator + keys: + 0 _col0 (type: int) + 1 _col0 (type: int) + 2 _col0 (type: int) + Select Operator + expressions: _col0 (type: int) + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + Group By Operator + keys: _col0 (type: int) + mode: hash + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + Spark Partition Pruning Sink Operator + Target column: part_col (int) + partition key expr: part_col + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + target work: Map 1 + Local Work: + Map Reduce Local Work + Map 4 + Map Operator Tree: + TableScan + alias: rt2 + Statistics: Num rows: 3 Data size: 9 Basic stats: COMPLETE Column stats: NONE + Filter Operator + predicate: ((col2 > 1) and col1 is not null) (type: boolean) + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + Select Operator + expressions: col1 (type: int) + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + Spark HashTable Sink Operator + keys: + 0 _col0 (type: int) + 1 _col0 (type: int) + 2 _col0 (type: int) + Select Operator + expressions: _col0 (type: int) + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + Group By Operator + keys: _col0 (type: int) + mode: hash + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + Spark Partition Pruning Sink Operator + Target column: part_col (int) + partition key expr: part_col + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + target work: Map 1 + Local Work: + Map Reduce Local Work + + Stage: Stage-1 + Spark + Edges: + Reducer 2 <- Map 1 (GROUP, 1) +#### A masked pattern was here #### + Vertices: + Map 1 + Map Operator Tree: + TableScan + alias: partitioned_table1 + Statistics: Num rows: 18 Data size: 18 Basic stats: COMPLETE Column stats: NONE + Select Operator + expressions: part_col (type: int) + outputColumnNames: _col0 + Statistics: Num rows: 18 Data size: 18 Basic stats: COMPLETE Column stats: NONE + Map Join Operator + condition map: + Inner Join 0 to 1 + Inner Join 0 to 2 + keys: + 0 _col0 (type: int) + 1 _col0 (type: int) + 2 _col0 (type: int) + input vertices: + 1 Map 3 + 2 Map 4 + Statistics: Num rows: 39 Data size: 39 Basic stats: COMPLETE Column stats: NONE + Group By Operator + aggregations: count() + mode: hash + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE + Reduce Output Operator + sort order: + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE + value expressions: _col0 (type: bigint) + Local Work: + Map Reduce Local Work + Reducer 2 + Reduce Operator Tree: + Group By Operator + aggregations: count(VALUE._col0) + mode: mergepartial + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE + File Output Operator + compressed: false + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE + table: + input format: org.apache.hadoop.mapred.SequenceFileInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat + serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe + + Stage: Stage-0 + Fetch Operator + limit: -1 + Processor Tree: + ListSink + +PREHOOK: query: SELECT count(*) +FROM partitioned_table1, + regular_table1 rt1, + regular_table1 rt2 +WHERE rt1.col1 = partitioned_table1.part_col + AND rt2.col1 = partitioned_table1.part_col + AND rt1.col2 > 0 + AND rt2.col2 > 1 +PREHOOK: type: QUERY +PREHOOK: Input: default@partitioned_table1 +PREHOOK: Input: default@partitioned_table1@part_col=1 +PREHOOK: Input: default@partitioned_table1@part_col=2 +PREHOOK: Input: default@partitioned_table1@part_col=3 +PREHOOK: Input: default@regular_table1 +#### A masked pattern was here #### +POSTHOOK: query: SELECT count(*) +FROM partitioned_table1, + regular_table1 rt1, + regular_table1 rt2 +WHERE rt1.col1 = partitioned_table1.part_col + AND rt2.col1 = partitioned_table1.part_col + AND rt1.col2 > 0 + AND rt2.col2 > 1 +POSTHOOK: type: QUERY +POSTHOOK: Input: default@partitioned_table1 +POSTHOOK: Input: default@partitioned_table1@part_col=1 +POSTHOOK: Input: default@partitioned_table1@part_col=2 +POSTHOOK: Input: default@partitioned_table1@part_col=3 +POSTHOOK: Input: default@regular_table1 +#### A masked pattern was here #### +6 +Warning: Shuffle Join JOIN[14][tables = [$hdt$_0, $hdt$_1]] in Work 'Reducer 2' is a cross product +PREHOOK: query: EXPLAIN SELECT count(*) FROM partitioned_table1, partitioned_table2, regular_table1 rt1 +, regular_table1 rt2 WHERE rt1.col1 = partitioned_table1.part_col AND rt2.col1 = +partitioned_table2.part_col AND rt1.col2 > 0 AND rt2.col2 > 1 +PREHOOK: type: QUERY +POSTHOOK: query: EXPLAIN SELECT count(*) FROM partitioned_table1, partitioned_table2, regular_table1 rt1 +, regular_table1 rt2 WHERE rt1.col1 = partitioned_table1.part_col AND rt2.col1 = +partitioned_table2.part_col AND rt1.col2 > 0 AND rt2.col2 > 1 +POSTHOOK: type: QUERY +STAGE DEPENDENCIES: + Stage-2 is a root stage + Stage-1 depends on stages: Stage-2 + Stage-0 depends on stages: Stage-1 + +STAGE PLANS: + Stage: Stage-2 + Spark +#### A masked pattern was here #### + Vertices: + Map 5 + Map Operator Tree: + TableScan + alias: rt1 + Statistics: Num rows: 3 Data size: 9 Basic stats: COMPLETE Column stats: NONE + Filter Operator + predicate: ((col2 > 0) and col1 is not null) (type: boolean) + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + Select Operator + expressions: col1 (type: int) + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + Spark HashTable Sink Operator + keys: + 0 _col0 (type: int) + 1 _col0 (type: int) + Select Operator + expressions: _col0 (type: int) + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + Group By Operator + keys: _col0 (type: int) + mode: hash + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + Spark Partition Pruning Sink Operator + Target column: part_col (int) + partition key expr: part_col + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + target work: Map 1 + Local Work: + Map Reduce Local Work + Map 6 + Map Operator Tree: + TableScan + alias: rt2 + Statistics: Num rows: 3 Data size: 9 Basic stats: COMPLETE Column stats: NONE + Filter Operator + predicate: ((col2 > 1) and col1 is not null) (type: boolean) + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + Select Operator + expressions: col1 (type: int) + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + Spark HashTable Sink Operator + keys: + 0 _col1 (type: int) + 1 _col0 (type: int) + Select Operator + expressions: _col0 (type: int) + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + Group By Operator + keys: _col0 (type: int) + mode: hash + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + Spark Partition Pruning Sink Operator + Target column: part_col (int) + partition key expr: part_col + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + target work: Map 4 + Local Work: + Map Reduce Local Work + + Stage: Stage-1 + Spark + Edges: + Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 1), Map 4 (PARTITION-LEVEL SORT, 1) + Reducer 3 <- Reducer 2 (GROUP, 1) +#### A masked pattern was here #### + Vertices: + Map 1 + Map Operator Tree: + TableScan + alias: partitioned_table1 + Statistics: Num rows: 18 Data size: 18 Basic stats: COMPLETE Column stats: NONE + Select Operator + expressions: part_col (type: int) + outputColumnNames: _col0 + Statistics: Num rows: 18 Data size: 18 Basic stats: COMPLETE Column stats: NONE + Reduce Output Operator + sort order: + Statistics: Num rows: 18 Data size: 18 Basic stats: COMPLETE Column stats: NONE + value expressions: _col0 (type: int) + Map 4 + Map Operator Tree: + TableScan + alias: partitioned_table2 + Statistics: Num rows: 18 Data size: 18 Basic stats: COMPLETE Column stats: NONE + Select Operator + expressions: part_col (type: int) + outputColumnNames: _col0 + Statistics: Num rows: 18 Data size: 18 Basic stats: COMPLETE Column stats: NONE + Reduce Output Operator + sort order: + Statistics: Num rows: 18 Data size: 18 Basic stats: COMPLETE Column stats: NONE + value expressions: _col0 (type: int) + Reducer 2 + Local Work: + Map Reduce Local Work + Reduce Operator Tree: + Join Operator + condition map: + Inner Join 0 to 1 + keys: + 0 + 1 + outputColumnNames: _col0, _col1 + Statistics: Num rows: 324 Data size: 972 Basic stats: COMPLETE Column stats: NONE + Map Join Operator + condition map: + Inner Join 0 to 1 + keys: + 0 _col0 (type: int) + 1 _col0 (type: int) + outputColumnNames: _col1 + input vertices: + 1 Map 5 + Statistics: Num rows: 356 Data size: 1069 Basic stats: COMPLETE Column stats: NONE + Map Join Operator + condition map: + Inner Join 0 to 1 + keys: + 0 _col1 (type: int) + 1 _col0 (type: int) + input vertices: + 1 Map 6 + Statistics: Num rows: 391 Data size: 1175 Basic stats: COMPLETE Column stats: NONE + Group By Operator + aggregations: count() + mode: hash + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE + Reduce Output Operator + sort order: + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE + value expressions: _col0 (type: bigint) + Reducer 3 + Reduce Operator Tree: + Group By Operator + aggregations: count(VALUE._col0) + mode: mergepartial + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE + File Output Operator + compressed: false + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE + table: + input format: org.apache.hadoop.mapred.SequenceFileInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat + serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe + + Stage: Stage-0 + Fetch Operator + limit: -1 + Processor Tree: + ListSink + +Warning: Shuffle Join JOIN[14][tables = [$hdt$_0, $hdt$_1]] in Work 'Reducer 2' is a cross product +PREHOOK: query: SELECT count(*) +FROM partitioned_table1, + partitioned_table2, + regular_table1 rt1, + regular_table1 rt2 +WHERE rt1.col1 = partitioned_table1.part_col + AND rt2.col1 = partitioned_table2.part_col + AND rt1.col2 > 0 + AND rt2.col2 > 1 +PREHOOK: type: QUERY +PREHOOK: Input: default@partitioned_table1 +PREHOOK: Input: default@partitioned_table1@part_col=1 +PREHOOK: Input: default@partitioned_table1@part_col=2 +PREHOOK: Input: default@partitioned_table1@part_col=3 +PREHOOK: Input: default@partitioned_table2 +PREHOOK: Input: default@partitioned_table2@part_col=1 +PREHOOK: Input: default@partitioned_table2@part_col=2 +PREHOOK: Input: default@partitioned_table2@part_col=3 +PREHOOK: Input: default@regular_table1 +#### A masked pattern was here #### +POSTHOOK: query: SELECT count(*) +FROM partitioned_table1, + partitioned_table2, + regular_table1 rt1, + regular_table1 rt2 +WHERE rt1.col1 = partitioned_table1.part_col + AND rt2.col1 = partitioned_table2.part_col + AND rt1.col2 > 0 + AND rt2.col2 > 1 +POSTHOOK: type: QUERY +POSTHOOK: Input: default@partitioned_table1 +POSTHOOK: Input: default@partitioned_table1@part_col=1 +POSTHOOK: Input: default@partitioned_table1@part_col=2 +POSTHOOK: Input: default@partitioned_table1@part_col=3 +POSTHOOK: Input: default@partitioned_table2 +POSTHOOK: Input: default@partitioned_table2@part_col=1 +POSTHOOK: Input: default@partitioned_table2@part_col=2 +POSTHOOK: Input: default@partitioned_table2@part_col=3 +POSTHOOK: Input: default@regular_table1 +#### A masked pattern was here #### +72 +PREHOOK: query: EXPLAIN SELECT count(*) FROM regular_table1, partitioned_table1 WHERE regular_table1.col1 += partitioned_table1.part_col AND regular_table1.col2 BETWEEN 1 AND 3 +PREHOOK: type: QUERY +POSTHOOK: query: EXPLAIN SELECT count(*) FROM regular_table1, partitioned_table1 WHERE regular_table1.col1 += partitioned_table1.part_col AND regular_table1.col2 BETWEEN 1 AND 3 +POSTHOOK: type: QUERY +STAGE DEPENDENCIES: + Stage-2 is a root stage + Stage-1 depends on stages: Stage-2 + Stage-0 depends on stages: Stage-1 + +STAGE PLANS: + Stage: Stage-2 + Spark +#### A masked pattern was here #### + Vertices: + Map 1 + Map Operator Tree: + TableScan + alias: regular_table1 + Statistics: Num rows: 3 Data size: 9 Basic stats: COMPLETE Column stats: NONE + Filter Operator + predicate: (col2 BETWEEN 1 AND 3 and col1 is not null) (type: boolean) + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + Select Operator + expressions: col1 (type: int) + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + Spark HashTable Sink Operator + keys: + 0 _col0 (type: int) + 1 _col0 (type: int) + Select Operator + expressions: _col0 (type: int) + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + Group By Operator + keys: _col0 (type: int) + mode: hash + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + Spark Partition Pruning Sink Operator + Target column: part_col (int) + partition key expr: part_col + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + target work: Map 2 + Local Work: + Map Reduce Local Work + + Stage: Stage-1 + Spark + Edges: + Reducer 3 <- Map 2 (GROUP, 1) +#### A masked pattern was here #### + Vertices: + Map 2 + Map Operator Tree: + TableScan + alias: partitioned_table1 + Statistics: Num rows: 18 Data size: 18 Basic stats: COMPLETE Column stats: NONE + Select Operator + expressions: part_col (type: int) + outputColumnNames: _col0 + Statistics: Num rows: 18 Data size: 18 Basic stats: COMPLETE Column stats: NONE + Map Join Operator + condition map: + Inner Join 0 to 1 + keys: + 0 _col0 (type: int) + 1 _col0 (type: int) + input vertices: + 0 Map 1 + Statistics: Num rows: 19 Data size: 19 Basic stats: COMPLETE Column stats: NONE + Group By Operator + aggregations: count() + mode: hash + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE + Reduce Output Operator + sort order: + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE + value expressions: _col0 (type: bigint) + Local Work: + Map Reduce Local Work + Reducer 3 + Reduce Operator Tree: + Group By Operator + aggregations: count(VALUE._col0) + mode: mergepartial + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE + File Output Operator + compressed: false + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE + table: + input format: org.apache.hadoop.mapred.SequenceFileInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat + serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe + + Stage: Stage-0 + Fetch Operator + limit: -1 + Processor Tree: + ListSink + +PREHOOK: query: SELECT count(*) +FROM regular_table1, + partitioned_table1 +WHERE regular_table1.col1 = partitioned_table1.part_col + AND regular_table1.col2 BETWEEN 1 AND 3 +PREHOOK: type: QUERY +PREHOOK: Input: default@partitioned_table1 +PREHOOK: Input: default@partitioned_table1@part_col=1 +PREHOOK: Input: default@partitioned_table1@part_col=2 +PREHOOK: Input: default@partitioned_table1@part_col=3 +PREHOOK: Input: default@regular_table1 +#### A masked pattern was here #### +POSTHOOK: query: SELECT count(*) +FROM regular_table1, + partitioned_table1 +WHERE regular_table1.col1 = partitioned_table1.part_col + AND regular_table1.col2 BETWEEN 1 AND 3 +POSTHOOK: type: QUERY +POSTHOOK: Input: default@partitioned_table1 +POSTHOOK: Input: default@partitioned_table1@part_col=1 +POSTHOOK: Input: default@partitioned_table1@part_col=2 +POSTHOOK: Input: default@partitioned_table1@part_col=3 +POSTHOOK: Input: default@regular_table1 +#### A masked pattern was here #### +12 +PREHOOK: query: EXPLAIN WITH q1 AS (SELECT regular_table1.col1 AS col FROM regular_table1 WHERE +regular_table1.col2 > 1), q2 AS (SELECT partitioned_table1.part_col AS col FROM +partitioned_table1 WHERE partitioned_table1.col > 1) SELECT count(*) FROM q1 JOIN q2 ON +q1.col = q2.col +PREHOOK: type: QUERY +POSTHOOK: query: EXPLAIN WITH q1 AS (SELECT regular_table1.col1 AS col FROM regular_table1 WHERE +regular_table1.col2 > 1), q2 AS (SELECT partitioned_table1.part_col AS col FROM +partitioned_table1 WHERE partitioned_table1.col > 1) SELECT count(*) FROM q1 JOIN q2 ON +q1.col = q2.col +POSTHOOK: type: QUERY +STAGE DEPENDENCIES: + Stage-2 is a root stage + Stage-1 depends on stages: Stage-2 + Stage-0 depends on stages: Stage-1 + +STAGE PLANS: + Stage: Stage-2 + Spark +#### A masked pattern was here #### + Vertices: + Map 1 + Map Operator Tree: + TableScan + alias: regular_table1 + Statistics: Num rows: 3 Data size: 9 Basic stats: COMPLETE Column stats: NONE + Filter Operator + predicate: ((col2 > 1) and col1 is not null) (type: boolean) + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + Select Operator + expressions: col1 (type: int) + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + Spark HashTable Sink Operator + keys: + 0 _col0 (type: int) + 1 _col0 (type: int) + Select Operator + expressions: _col0 (type: int) + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + Group By Operator + keys: _col0 (type: int) + mode: hash + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + Spark Partition Pruning Sink Operator + Target column: part_col (int) + partition key expr: part_col + Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE + target work: Map 2 + Local Work: + Map Reduce Local Work + + Stage: Stage-1 + Spark + Edges: + Reducer 3 <- Map 2 (GROUP, 1) +#### A masked pattern was here #### + Vertices: + Map 2 + Map Operator Tree: + TableScan + alias: partitioned_table1 + Statistics: Num rows: 18 Data size: 18 Basic stats: COMPLETE Column stats: NONE + Filter Operator + predicate: (col > 1) (type: boolean) + Statistics: Num rows: 6 Data size: 6 Basic stats: COMPLETE Column stats: NONE + Select Operator + expressions: part_col (type: int) + outputColumnNames: _col0 + Statistics: Num rows: 6 Data size: 6 Basic stats: COMPLETE Column stats: NONE + Map Join Operator + condition map: + Inner Join 0 to 1 + keys: + 0 _col0 (type: int) + 1 _col0 (type: int) + input vertices: + 0 Map 1 + Statistics: Num rows: 6 Data size: 6 Basic stats: COMPLETE Column stats: NONE + Group By Operator + aggregations: count() + mode: hash + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE + Reduce Output Operator + sort order: + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE + value expressions: _col0 (type: bigint) + Local Work: + Map Reduce Local Work + Reducer 3 + Reduce Operator Tree: + Group By Operator + aggregations: count(VALUE._col0) + mode: mergepartial + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE + File Output Operator + compressed: false + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE + table: + input format: org.apache.hadoop.mapred.SequenceFileInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat + serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe + + Stage: Stage-0 + Fetch Operator + limit: -1 + Processor Tree: + ListSink + +PREHOOK: query: WITH q1 + AS (SELECT regular_table1.col1 AS col + FROM regular_table1 + WHERE regular_table1.col2 > 1), + q2 + AS (SELECT partitioned_table1.part_col AS col + FROM partitioned_table1 + WHERE partitioned_table1.col > 1) +SELECT count(*) +FROM q1 + JOIN q2 + ON q1.col = q2.col +PREHOOK: type: QUERY +PREHOOK: Input: default@partitioned_table1 +PREHOOK: Input: default@partitioned_table1@part_col=1 +PREHOOK: Input: default@partitioned_table1@part_col=2 +PREHOOK: Input: default@partitioned_table1@part_col=3 +PREHOOK: Input: default@regular_table1 +#### A masked pattern was here #### +POSTHOOK: query: WITH q1 + AS (SELECT regular_table1.col1 AS col + FROM regular_table1 + WHERE regular_table1.col2 > 1), + q2 + AS (SELECT partitioned_table1.part_col AS col + FROM partitioned_table1 + WHERE partitioned_table1.col > 1) +SELECT count(*) +FROM q1 + JOIN q2 + ON q1.col = q2.col +POSTHOOK: type: QUERY +POSTHOOK: Input: default@partitioned_table1 +POSTHOOK: Input: default@partitioned_table1@part_col=1 +POSTHOOK: Input: default@partitioned_table1@part_col=2 +POSTHOOK: Input: default@partitioned_table1@part_col=3 +POSTHOOK: Input: default@regular_table1 +#### A masked pattern was here #### +5 +PREHOOK: query: EXPLAIN SELECT count(*) FROM partitioned_table1, partitioned_table2 WHERE +partitioned_table1.part_col = partitioned_table2.part_col AND partitioned_table2.col > 1 +PREHOOK: type: QUERY +POSTHOOK: query: EXPLAIN SELECT count(*) FROM partitioned_table1, partitioned_table2 WHERE +partitioned_table1.part_col = partitioned_table2.part_col AND partitioned_table2.col > 1 +POSTHOOK: type: QUERY +STAGE DEPENDENCIES: + Stage-2 is a root stage + Stage-1 depends on stages: Stage-2 + Stage-0 depends on stages: Stage-1 + +STAGE PLANS: + Stage: Stage-2 + Spark +#### A masked pattern was here #### + Vertices: + Map 3 + Map Operator Tree: + TableScan + alias: partitioned_table2 + Statistics: Num rows: 18 Data size: 18 Basic stats: COMPLETE Column stats: NONE + Filter Operator + predicate: (col > 1) (type: boolean) + Statistics: Num rows: 6 Data size: 6 Basic stats: COMPLETE Column stats: NONE + Select Operator + expressions: part_col (type: int) + outputColumnNames: _col1 + Statistics: Num rows: 6 Data size: 6 Basic stats: COMPLETE Column stats: NONE + Spark HashTable Sink Operator + keys: + 0 _col0 (type: int) + 1 _col1 (type: int) + Select Operator + expressions: _col1 (type: int) + outputColumnNames: _col0 + Statistics: Num rows: 6 Data size: 6 Basic stats: COMPLETE Column stats: NONE + Group By Operator + keys: _col0 (type: int) + mode: hash + outputColumnNames: _col0 + Statistics: Num rows: 6 Data size: 6 Basic stats: COMPLETE Column stats: NONE + Spark Partition Pruning Sink Operator + Target column: part_col (int) + partition key expr: part_col + Statistics: Num rows: 6 Data size: 6 Basic stats: COMPLETE Column stats: NONE + target work: Map 1 + Local Work: + Map Reduce Local Work + + Stage: Stage-1 + Spark + Edges: + Reducer 2 <- Map 1 (GROUP, 1) +#### A masked pattern was here #### + Vertices: + Map 1 + Map Operator Tree: + TableScan + alias: partitioned_table1 + Statistics: Num rows: 18 Data size: 18 Basic stats: COMPLETE Column stats: NONE + Select Operator + expressions: part_col (type: int) + outputColumnNames: _col0 + Statistics: Num rows: 18 Data size: 18 Basic stats: COMPLETE Column stats: NONE + Map Join Operator + condition map: + Inner Join 0 to 1 + keys: + 0 _col0 (type: int) + 1 _col1 (type: int) + input vertices: + 1 Map 3 + Statistics: Num rows: 19 Data size: 19 Basic stats: COMPLETE Column stats: NONE + Group By Operator + aggregations: count() + mode: hash + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE + Reduce Output Operator + sort order: + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE + value expressions: _col0 (type: bigint) + Local Work: + Map Reduce Local Work + Reducer 2 + Reduce Operator Tree: + Group By Operator + aggregations: count(VALUE._col0) + mode: mergepartial + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE + File Output Operator + compressed: false + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE + table: + input format: org.apache.hadoop.mapred.SequenceFileInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat + serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe + + Stage: Stage-0 + Fetch Operator + limit: -1 + Processor Tree: + ListSink + +PREHOOK: query: SELECT count(*) +FROM partitioned_table1, + partitioned_table2 +WHERE partitioned_table1.part_col = partitioned_table2.part_col + AND partitioned_table2.col > 1 PREHOOK: type: QUERY -PREHOOK: Output: default@partitioned_table2@part_col=1 -POSTHOOK: query: insert into table partitioned_table2 partition (part_col = 1) values (1), (2), (3), (4), (5), (6), (7), (8), (9), (10) +PREHOOK: Input: default@partitioned_table1 +PREHOOK: Input: default@partitioned_table1@part_col=1 +PREHOOK: Input: default@partitioned_table1@part_col=2 +PREHOOK: Input: default@partitioned_table1@part_col=3 +PREHOOK: Input: default@partitioned_table2 +PREHOOK: Input: default@partitioned_table2@part_col=1 +PREHOOK: Input: default@partitioned_table2@part_col=2 +PREHOOK: Input: default@partitioned_table2@part_col=3 +#### A masked pattern was here #### +POSTHOOK: query: SELECT count(*) +FROM partitioned_table1, + partitioned_table2 +WHERE partitioned_table1.part_col = partitioned_table2.part_col + AND partitioned_table2.col > 1 POSTHOOK: type: QUERY -POSTHOOK: Output: default@partitioned_table2@part_col=1 -POSTHOOK: Lineage: partitioned_table2 PARTITION(part_col=1).col EXPRESSION [(values__tmp__table__3)values__tmp__table__3.FieldSchema(name:tmp_values_col1, type:string, comment:), ] -PREHOOK: query: explain select * from partitioned_table1, partitioned_table2 where partitioned_table1.part_col = partitioned_table2.part_col +POSTHOOK: Input: default@partitioned_table1 +POSTHOOK: Input: default@partitioned_table1@part_col=1 +POSTHOOK: Input: default@partitioned_table1@part_col=2 +POSTHOOK: Input: default@partitioned_table1@part_col=3 +POSTHOOK: Input: default@partitioned_table2 +POSTHOOK: Input: default@partitioned_table2@part_col=1 +POSTHOOK: Input: default@partitioned_table2@part_col=2 +POSTHOOK: Input: default@partitioned_table2@part_col=3 +#### A masked pattern was here #### +90 +PREHOOK: query: -- set auto convert size to a higher value so map-joins are triggered for the partitioned tables + +EXPLAIN SELECT count(*) FROM partitioned_table4, partitioned_table5 WHERE +partitioned_table4.part_col1 = partitioned_table5.part_col1 AND +partitioned_table4.part_col2 = partitioned_table5.part_col2 PREHOOK: type: QUERY -POSTHOOK: query: explain select * from partitioned_table1, partitioned_table2 where partitioned_table1.part_col = partitioned_table2.part_col +POSTHOOK: query: -- set auto convert size to a higher value so map-joins are triggered for the partitioned tables + +EXPLAIN SELECT count(*) FROM partitioned_table4, partitioned_table5 WHERE +partitioned_table4.part_col1 = partitioned_table5.part_col1 AND +partitioned_table4.part_col2 = partitioned_table5.part_col2 POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-2 is a root stage @@ -71,68 +1313,99 @@ STAGE PLANS: Spark #### A masked pattern was here #### Vertices: - Map 2 + Map 3 Map Operator Tree: TableScan - alias: partitioned_table2 - Statistics: Num rows: 10 Data size: 11 Basic stats: COMPLETE Column stats: NONE + alias: partitioned_table5 + Statistics: Num rows: 18 Data size: 18 Basic stats: COMPLETE Column stats: NONE Select Operator - expressions: col (type: int), part_col (type: int) + expressions: part_col1 (type: int), part_col2 (type: int) outputColumnNames: _col0, _col1 - Statistics: Num rows: 10 Data size: 11 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 18 Data size: 18 Basic stats: COMPLETE Column stats: NONE Spark HashTable Sink Operator keys: - 0 _col1 (type: int) - 1 _col1 (type: int) + 0 _col0 (type: int), _col1 (type: int) + 1 _col0 (type: int), _col1 (type: int) + Select Operator + expressions: _col0 (type: int) + outputColumnNames: _col0 + Statistics: Num rows: 18 Data size: 18 Basic stats: COMPLETE Column stats: NONE + Group By Operator + keys: _col0 (type: int) + mode: hash + outputColumnNames: _col0 + Statistics: Num rows: 18 Data size: 18 Basic stats: COMPLETE Column stats: NONE + Spark Partition Pruning Sink Operator + Target column: part_col1 (int) + partition key expr: part_col1 + Statistics: Num rows: 18 Data size: 18 Basic stats: COMPLETE Column stats: NONE + target work: Map 1 Select Operator expressions: _col1 (type: int) outputColumnNames: _col0 - Statistics: Num rows: 10 Data size: 11 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 18 Data size: 18 Basic stats: COMPLETE Column stats: NONE Group By Operator keys: _col0 (type: int) mode: hash outputColumnNames: _col0 - Statistics: Num rows: 10 Data size: 11 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 18 Data size: 18 Basic stats: COMPLETE Column stats: NONE Spark Partition Pruning Sink Operator - Target column: part_col (int) - partition key expr: part_col - Statistics: Num rows: 10 Data size: 11 Basic stats: COMPLETE Column stats: NONE + Target column: part_col2 (int) + partition key expr: part_col2 + Statistics: Num rows: 18 Data size: 18 Basic stats: COMPLETE Column stats: NONE target work: Map 1 Local Work: Map Reduce Local Work Stage: Stage-1 Spark + Edges: + Reducer 2 <- Map 1 (GROUP, 1) #### A masked pattern was here #### Vertices: Map 1 Map Operator Tree: TableScan - alias: partitioned_table1 - Statistics: Num rows: 10 Data size: 11 Basic stats: COMPLETE Column stats: NONE + alias: partitioned_table4 + Statistics: Num rows: 18 Data size: 18 Basic stats: COMPLETE Column stats: NONE Select Operator - expressions: col (type: int), part_col (type: int) + expressions: part_col1 (type: int), part_col2 (type: int) outputColumnNames: _col0, _col1 - Statistics: Num rows: 10 Data size: 11 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 18 Data size: 18 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 keys: - 0 _col1 (type: int) - 1 _col1 (type: int) - outputColumnNames: _col0, _col1, _col2, _col3 + 0 _col0 (type: int), _col1 (type: int) + 1 _col0 (type: int), _col1 (type: int) input vertices: - 1 Map 2 - Statistics: Num rows: 11 Data size: 12 Basic stats: COMPLETE Column stats: NONE - File Output Operator - compressed: false - Statistics: Num rows: 11 Data size: 12 Basic stats: COMPLETE Column stats: NONE - table: - input format: org.apache.hadoop.mapred.SequenceFileInputFormat - output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat - serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe + 1 Map 3 + Statistics: Num rows: 19 Data size: 19 Basic stats: COMPLETE Column stats: NONE + Group By Operator + aggregations: count() + mode: hash + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE + Reduce Output Operator + sort order: + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE + value expressions: _col0 (type: bigint) Local Work: Map Reduce Local Work + Reducer 2 + Reduce Operator Tree: + Group By Operator + aggregations: count(VALUE._col0) + mode: mergepartial + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE + File Output Operator + compressed: false + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE + table: + input format: org.apache.hadoop.mapred.SequenceFileInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat + serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Stage: Stage-0 Fetch Operator @@ -140,135 +1413,174 @@ STAGE PLANS: Processor Tree: ListSink -PREHOOK: query: explain select * from partitioned_table1 where partitioned_table1.part_col in (select regular_table.col from regular_table join partitioned_table2 on regular_table.col = partitioned_table2.part_col) +PREHOOK: query: SELECT count(*) +FROM partitioned_table4, + partitioned_table5 +WHERE partitioned_table4.part_col1 = partitioned_table5.part_col1 + AND partitioned_table4.part_col2 = partitioned_table5.part_col2 +PREHOOK: type: QUERY +PREHOOK: Input: default@partitioned_table4 +PREHOOK: Input: default@partitioned_table4@part_col1=1/part_col2=1 +PREHOOK: Input: default@partitioned_table4@part_col1=2/part_col2=2 +PREHOOK: Input: default@partitioned_table4@part_col1=3/part_col2=3 +PREHOOK: Input: default@partitioned_table5 +PREHOOK: Input: default@partitioned_table5@part_col1=1/part_col2=1 +PREHOOK: Input: default@partitioned_table5@part_col1=2/part_col2=2 +PREHOOK: Input: default@partitioned_table5@part_col1=3/part_col2=3 +#### A masked pattern was here #### +POSTHOOK: query: SELECT count(*) +FROM partitioned_table4, + partitioned_table5 +WHERE partitioned_table4.part_col1 = partitioned_table5.part_col1 + AND partitioned_table4.part_col2 = partitioned_table5.part_col2 +POSTHOOK: type: QUERY +POSTHOOK: Input: default@partitioned_table4 +POSTHOOK: Input: default@partitioned_table4@part_col1=1/part_col2=1 +POSTHOOK: Input: default@partitioned_table4@part_col1=2/part_col2=2 +POSTHOOK: Input: default@partitioned_table4@part_col1=3/part_col2=3 +POSTHOOK: Input: default@partitioned_table5 +POSTHOOK: Input: default@partitioned_table5@part_col1=1/part_col2=1 +POSTHOOK: Input: default@partitioned_table5@part_col1=2/part_col2=2 +POSTHOOK: Input: default@partitioned_table5@part_col1=3/part_col2=3 +#### A masked pattern was here #### +108 +PREHOOK: query: EXPLAIN SELECT count(*) FROM partitioned_table1 JOIN regular_table1 ON +partitioned_table1.part_col = regular_table1.col1 JOIN regular_table2 ON +regular_table1.col1 = regular_table2.col1 PREHOOK: type: QUERY -POSTHOOK: query: explain select * from partitioned_table1 where partitioned_table1.part_col in (select regular_table.col from regular_table join partitioned_table2 on regular_table.col = partitioned_table2.part_col) +POSTHOOK: query: EXPLAIN SELECT count(*) FROM partitioned_table1 JOIN regular_table1 ON +partitioned_table1.part_col = regular_table1.col1 JOIN regular_table2 ON +regular_table1.col1 = regular_table2.col1 POSTHOOK: type: QUERY STAGE DEPENDENCIES: - Stage-3 is a root stage - Stage-2 depends on stages: Stage-3 + Stage-2 is a root stage Stage-1 depends on stages: Stage-2 Stage-0 depends on stages: Stage-1 STAGE PLANS: - Stage: Stage-3 + Stage: Stage-2 Spark #### A masked pattern was here #### Vertices: - Map 2 + Map 3 Map Operator Tree: TableScan - alias: regular_table - Statistics: Num rows: 1 Data size: 1 Basic stats: COMPLETE Column stats: NONE + alias: regular_table1 + Statistics: Num rows: 3 Data size: 9 Basic stats: COMPLETE Column stats: NONE Filter Operator - predicate: col is not null (type: boolean) - Statistics: Num rows: 1 Data size: 1 Basic stats: COMPLETE Column stats: NONE + predicate: col1 is not null (type: boolean) + Statistics: Num rows: 3 Data size: 9 Basic stats: COMPLETE Column stats: NONE Select Operator - expressions: col (type: int) + expressions: col1 (type: int) outputColumnNames: _col0 - Statistics: Num rows: 1 Data size: 1 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3 Data size: 9 Basic stats: COMPLETE Column stats: NONE Spark HashTable Sink Operator keys: 0 _col0 (type: int) 1 _col0 (type: int) + 2 _col0 (type: int) Select Operator expressions: _col0 (type: int) outputColumnNames: _col0 - Statistics: Num rows: 1 Data size: 1 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3 Data size: 9 Basic stats: COMPLETE Column stats: NONE Group By Operator keys: _col0 (type: int) mode: hash outputColumnNames: _col0 - Statistics: Num rows: 1 Data size: 1 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3 Data size: 9 Basic stats: COMPLETE Column stats: NONE Spark Partition Pruning Sink Operator Target column: part_col (int) partition key expr: part_col - Statistics: Num rows: 1 Data size: 1 Basic stats: COMPLETE Column stats: NONE - target work: Map 3 + Statistics: Num rows: 3 Data size: 9 Basic stats: COMPLETE Column stats: NONE + target work: Map 1 Local Work: Map Reduce Local Work - - Stage: Stage-2 - Spark -#### A masked pattern was here #### - Vertices: - Map 3 + Map 4 Map Operator Tree: TableScan - alias: partitioned_table2 - Statistics: Num rows: 10 Data size: 11 Basic stats: COMPLETE Column stats: NONE - Select Operator - expressions: part_col (type: int) - outputColumnNames: _col0 - Statistics: Num rows: 10 Data size: 11 Basic stats: COMPLETE Column stats: NONE - Map Join Operator - condition map: - Inner Join 0 to 1 - keys: - 0 _col0 (type: int) - 1 _col0 (type: int) + alias: regular_table2 + Statistics: Num rows: 3 Data size: 9 Basic stats: COMPLETE Column stats: NONE + Filter Operator + predicate: col1 is not null (type: boolean) + Statistics: Num rows: 3 Data size: 9 Basic stats: COMPLETE Column stats: NONE + Select Operator + expressions: col1 (type: int) outputColumnNames: _col0 - input vertices: - 0 Map 2 - Statistics: Num rows: 11 Data size: 12 Basic stats: COMPLETE Column stats: NONE - Group By Operator - keys: _col0 (type: int) - mode: hash + Statistics: Num rows: 3 Data size: 9 Basic stats: COMPLETE Column stats: NONE + Spark HashTable Sink Operator + keys: + 0 _col0 (type: int) + 1 _col0 (type: int) + 2 _col0 (type: int) + Select Operator + expressions: _col0 (type: int) outputColumnNames: _col0 - Statistics: Num rows: 11 Data size: 12 Basic stats: COMPLETE Column stats: NONE - Spark HashTable Sink Operator - keys: - 0 _col1 (type: int) - 1 _col0 (type: int) - Select Operator - expressions: _col0 (type: int) + Statistics: Num rows: 3 Data size: 9 Basic stats: COMPLETE Column stats: NONE + Group By Operator + keys: _col0 (type: int) + mode: hash outputColumnNames: _col0 - Statistics: Num rows: 11 Data size: 12 Basic stats: COMPLETE Column stats: NONE - Group By Operator - keys: _col0 (type: int) - mode: hash - outputColumnNames: _col0 - Statistics: Num rows: 11 Data size: 12 Basic stats: COMPLETE Column stats: NONE - Spark Partition Pruning Sink Operator - Target column: part_col (int) - partition key expr: part_col - Statistics: Num rows: 11 Data size: 12 Basic stats: COMPLETE Column stats: NONE - target work: Map 1 + Statistics: Num rows: 3 Data size: 9 Basic stats: COMPLETE Column stats: NONE + Spark Partition Pruning Sink Operator + Target column: part_col (int) + partition key expr: part_col + Statistics: Num rows: 3 Data size: 9 Basic stats: COMPLETE Column stats: NONE + target work: Map 1 Local Work: Map Reduce Local Work Stage: Stage-1 Spark + Edges: + Reducer 2 <- Map 1 (GROUP, 1) #### A masked pattern was here #### Vertices: Map 1 Map Operator Tree: TableScan alias: partitioned_table1 - Statistics: Num rows: 10 Data size: 11 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 18 Data size: 18 Basic stats: COMPLETE Column stats: NONE Select Operator - expressions: col (type: int), part_col (type: int) - outputColumnNames: _col0, _col1 - Statistics: Num rows: 10 Data size: 11 Basic stats: COMPLETE Column stats: NONE + expressions: part_col (type: int) + outputColumnNames: _col0 + Statistics: Num rows: 18 Data size: 18 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: - Left Semi Join 0 to 1 + Inner Join 0 to 1 + Inner Join 1 to 2 keys: - 0 _col1 (type: int) + 0 _col0 (type: int) 1 _col0 (type: int) - outputColumnNames: _col0, _col1 + 2 _col0 (type: int) input vertices: 1 Map 3 - Statistics: Num rows: 12 Data size: 13 Basic stats: COMPLETE Column stats: NONE - File Output Operator - compressed: false - Statistics: Num rows: 12 Data size: 13 Basic stats: COMPLETE Column stats: NONE - table: - input format: org.apache.hadoop.mapred.SequenceFileInputFormat - output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat - serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe + 2 Map 4 + Statistics: Num rows: 39 Data size: 39 Basic stats: COMPLETE Column stats: NONE + Group By Operator + aggregations: count() + mode: hash + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE + Reduce Output Operator + sort order: + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE + value expressions: _col0 (type: bigint) Local Work: Map Reduce Local Work + Reducer 2 + Reduce Operator Tree: + Group By Operator + aggregations: count(VALUE._col0) + mode: mergepartial + outputColumnNames: _ <TRUNCATED>
