[jira] [Updated] (HIVE-19850) Dynamic partition pruning in Tez is leading to 'No work found for tablescan' error
[
https://issues.apache.org/jira/browse/HIVE-19850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-19850:
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks, Ganesha!
> Dynamic partition pruning in Tez is leading to 'No work found for tablescan'
> error
> --
>
> Key: HIVE-19850
> URL: https://issues.apache.org/jira/browse/HIVE-19850
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 3.0.0
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-19850.patch
>
>
>
> When multiple views are used along with union all, it is resulting in the
> following error when dynamic partition pruning is enabled in tez.
>
> {code:java}
> Exception in thread "main" java.lang.AssertionError: No work found for
> tablescan TS[8]
> at
> org.apache.hadoop.hive.ql.parse.GenTezUtils.processAppMasterEvent(GenTezUtils.java:408)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:383)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:205)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10371)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:208)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:239)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:479)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:347)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1203)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1257)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1140)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1130)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:204)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:433)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:894)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:825)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:726)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:223)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136){code}
>
> *Steps to reproduce:*
> set hive.execution.engine=tez;
> set hive.tez.dynamic.partition.pruning=true;
> CREATE TABLE t1(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
> CREATE TABLE t2(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
> CREATE TABLE t3(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
>
> insert into table t1 partition(dt='2018') values ('k1','v1',1,1.0,true);
> insert into table t2 partition(dt='2018') values ('k2','v2',2,2.0,true);
> insert into table t3 partition(dt='2018') values ('k3','v3',3,3.0,true);
>
> CREATE VIEW `view1` AS select
> `t1`.`key`,`t1`.`value`,`t1`.`c_int`,`t1`.`c_float`,`t1`.`c_boolean`,`t1`.`dt`
> from `t1` union all select
> `t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_boolean`,`t2`.`dt`
> from `t2`;
> CREATE VIEW `view2` AS select
> `t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_boolean`,`t2`.`dt`
> from `t2` union all select
> `t3`.`key`,`t3`.`value`,`t3`.`c_int`,`t3`.`c_float`,`t3`.`c_boolean`,`t3`.`dt`
> from `t3`;
> create table t4 as select key,value,c_int,c_float,c_boolean,dt from t1 union
> all select v1.key,v1.value,v1.c_int,v1.c_float,v1.c_boolean,v1.dt from view1
> v1 join view2 v2 on v1.dt=v2.dt;
> CREATE VIEW `view3` AS select
> `t4`.`key`,`t4`.`value`,`t4`.`c_int`,`t4`.`c_float`,`t4`.`c_boolean`,`t4`.`dt`
> from `t4` union all select
> `t1`.`key`,`t1`.`value`,`t1`.`c_int`,`t1`.`c_float`,`t1`.`c_boolean`,`t1`.`dt`
> from `t1`;
>
> select count(0) from view2 v2 join view3 v3 on v2.dt=v3.dt; // Throws No work
> found for tablescan error
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19850) Dynamic partition pruning in Tez is leading to 'No work found for tablescan' error
[
https://issues.apache.org/jira/browse/HIVE-19850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ganesha Shreedhara updated HIVE-19850:
--
Attachment: HIVE-19850.patch
> Dynamic partition pruning in Tez is leading to 'No work found for tablescan'
> error
> --
>
> Key: HIVE-19850
> URL: https://issues.apache.org/jira/browse/HIVE-19850
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 3.0.0
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-19850.patch
>
>
>
> When multiple views are used along with union all, it is resulting in the
> following error when dynamic partition pruning is enabled in tez.
>
> {code:java}
> Exception in thread "main" java.lang.AssertionError: No work found for
> tablescan TS[8]
> at
> org.apache.hadoop.hive.ql.parse.GenTezUtils.processAppMasterEvent(GenTezUtils.java:408)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:383)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:205)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10371)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:208)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:239)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:479)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:347)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1203)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1257)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1140)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1130)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:204)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:433)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:894)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:825)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:726)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:223)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136){code}
>
> *Steps to reproduce:*
> set hive.execution.engine=tez;
> set hive.tez.dynamic.partition.pruning=true;
> CREATE TABLE t1(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
> CREATE TABLE t2(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
> CREATE TABLE t3(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
>
> insert into table t1 partition(dt='2018') values ('k1','v1',1,1.0,true);
> insert into table t2 partition(dt='2018') values ('k2','v2',2,2.0,true);
> insert into table t3 partition(dt='2018') values ('k3','v3',3,3.0,true);
>
> CREATE VIEW `view1` AS select
> `t1`.`key`,`t1`.`value`,`t1`.`c_int`,`t1`.`c_float`,`t1`.`c_boolean`,`t1`.`dt`
> from `t1` union all select
> `t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_boolean`,`t2`.`dt`
> from `t2`;
> CREATE VIEW `view2` AS select
> `t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_boolean`,`t2`.`dt`
> from `t2` union all select
> `t3`.`key`,`t3`.`value`,`t3`.`c_int`,`t3`.`c_float`,`t3`.`c_boolean`,`t3`.`dt`
> from `t3`;
> create table t4 as select key,value,c_int,c_float,c_boolean,dt from t1 union
> all select v1.key,v1.value,v1.c_int,v1.c_float,v1.c_boolean,v1.dt from view1
> v1 join view2 v2 on v1.dt=v2.dt;
> CREATE VIEW `view3` AS select
> `t4`.`key`,`t4`.`value`,`t4`.`c_int`,`t4`.`c_float`,`t4`.`c_boolean`,`t4`.`dt`
> from `t4` union all select
> `t1`.`key`,`t1`.`value`,`t1`.`c_int`,`t1`.`c_float`,`t1`.`c_boolean`,`t1`.`dt`
> from `t1`;
>
> select count(0) from view2 v2 join view3 v3 on v2.dt=v3.dt; // Throws No work
> found for tablescan error
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19850) Dynamic partition pruning in Tez is leading to 'No work found for tablescan' error
[
https://issues.apache.org/jira/browse/HIVE-19850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ganesha Shreedhara updated HIVE-19850:
--
Attachment: (was: HIVE-19850.patch)
> Dynamic partition pruning in Tez is leading to 'No work found for tablescan'
> error
> --
>
> Key: HIVE-19850
> URL: https://issues.apache.org/jira/browse/HIVE-19850
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 3.0.0
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-19850.patch
>
>
>
> When multiple views are used along with union all, it is resulting in the
> following error when dynamic partition pruning is enabled in tez.
>
> {code:java}
> Exception in thread "main" java.lang.AssertionError: No work found for
> tablescan TS[8]
> at
> org.apache.hadoop.hive.ql.parse.GenTezUtils.processAppMasterEvent(GenTezUtils.java:408)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:383)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:205)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10371)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:208)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:239)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:479)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:347)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1203)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1257)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1140)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1130)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:204)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:433)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:894)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:825)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:726)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:223)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136){code}
>
> *Steps to reproduce:*
> set hive.execution.engine=tez;
> set hive.tez.dynamic.partition.pruning=true;
> CREATE TABLE t1(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
> CREATE TABLE t2(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
> CREATE TABLE t3(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
>
> insert into table t1 partition(dt='2018') values ('k1','v1',1,1.0,true);
> insert into table t2 partition(dt='2018') values ('k2','v2',2,2.0,true);
> insert into table t3 partition(dt='2018') values ('k3','v3',3,3.0,true);
>
> CREATE VIEW `view1` AS select
> `t1`.`key`,`t1`.`value`,`t1`.`c_int`,`t1`.`c_float`,`t1`.`c_boolean`,`t1`.`dt`
> from `t1` union all select
> `t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_boolean`,`t2`.`dt`
> from `t2`;
> CREATE VIEW `view2` AS select
> `t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_boolean`,`t2`.`dt`
> from `t2` union all select
> `t3`.`key`,`t3`.`value`,`t3`.`c_int`,`t3`.`c_float`,`t3`.`c_boolean`,`t3`.`dt`
> from `t3`;
> create table t4 as select key,value,c_int,c_float,c_boolean,dt from t1 union
> all select v1.key,v1.value,v1.c_int,v1.c_float,v1.c_boolean,v1.dt from view1
> v1 join view2 v2 on v1.dt=v2.dt;
> CREATE VIEW `view3` AS select
> `t4`.`key`,`t4`.`value`,`t4`.`c_int`,`t4`.`c_float`,`t4`.`c_boolean`,`t4`.`dt`
> from `t4` union all select
> `t1`.`key`,`t1`.`value`,`t1`.`c_int`,`t1`.`c_float`,`t1`.`c_boolean`,`t1`.`dt`
> from `t1`;
>
> select count(0) from view2 v2 join view3 v3 on v2.dt=v3.dt; // Throws No work
> found for tablescan error
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19850) Dynamic partition pruning in Tez is leading to 'No work found for tablescan' error
[
https://issues.apache.org/jira/browse/HIVE-19850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ganesha Shreedhara updated HIVE-19850:
--
Attachment: HIVE-19850.patch
> Dynamic partition pruning in Tez is leading to 'No work found for tablescan'
> error
> --
>
> Key: HIVE-19850
> URL: https://issues.apache.org/jira/browse/HIVE-19850
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 3.0.0
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-19850.patch
>
>
>
> When multiple views are used along with union all, it is resulting in the
> following error when dynamic partition pruning is enabled in tez.
>
> {code:java}
> Exception in thread "main" java.lang.AssertionError: No work found for
> tablescan TS[8]
> at
> org.apache.hadoop.hive.ql.parse.GenTezUtils.processAppMasterEvent(GenTezUtils.java:408)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:383)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:205)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10371)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:208)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:239)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:479)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:347)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1203)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1257)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1140)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1130)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:204)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:433)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:894)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:825)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:726)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:223)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136){code}
>
> *Steps to reproduce:*
> set hive.execution.engine=tez;
> set hive.tez.dynamic.partition.pruning=true;
> CREATE TABLE t1(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
> CREATE TABLE t2(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
> CREATE TABLE t3(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
>
> insert into table t1 partition(dt='2018') values ('k1','v1',1,1.0,true);
> insert into table t2 partition(dt='2018') values ('k2','v2',2,2.0,true);
> insert into table t3 partition(dt='2018') values ('k3','v3',3,3.0,true);
>
> CREATE VIEW `view1` AS select
> `t1`.`key`,`t1`.`value`,`t1`.`c_int`,`t1`.`c_float`,`t1`.`c_boolean`,`t1`.`dt`
> from `t1` union all select
> `t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_boolean`,`t2`.`dt`
> from `t2`;
> CREATE VIEW `view2` AS select
> `t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_boolean`,`t2`.`dt`
> from `t2` union all select
> `t3`.`key`,`t3`.`value`,`t3`.`c_int`,`t3`.`c_float`,`t3`.`c_boolean`,`t3`.`dt`
> from `t3`;
> create table t4 as select key,value,c_int,c_float,c_boolean,dt from t1 union
> all select v1.key,v1.value,v1.c_int,v1.c_float,v1.c_boolean,v1.dt from view1
> v1 join view2 v2 on v1.dt=v2.dt;
> CREATE VIEW `view3` AS select
> `t4`.`key`,`t4`.`value`,`t4`.`c_int`,`t4`.`c_float`,`t4`.`c_boolean`,`t4`.`dt`
> from `t4` union all select
> `t1`.`key`,`t1`.`value`,`t1`.`c_int`,`t1`.`c_float`,`t1`.`c_boolean`,`t1`.`dt`
> from `t1`;
>
> select count(0) from view2 v2 join view3 v3 on v2.dt=v3.dt; // Throws No work
> found for tablescan error
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19850) Dynamic partition pruning in Tez is leading to 'No work found for tablescan' error
[
https://issues.apache.org/jira/browse/HIVE-19850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ganesha Shreedhara updated HIVE-19850:
--
Attachment: (was: HIVE-19850.patch)
> Dynamic partition pruning in Tez is leading to 'No work found for tablescan'
> error
> --
>
> Key: HIVE-19850
> URL: https://issues.apache.org/jira/browse/HIVE-19850
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 3.0.0
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-19850.patch
>
>
>
> When multiple views are used along with union all, it is resulting in the
> following error when dynamic partition pruning is enabled in tez.
>
> {code:java}
> Exception in thread "main" java.lang.AssertionError: No work found for
> tablescan TS[8]
> at
> org.apache.hadoop.hive.ql.parse.GenTezUtils.processAppMasterEvent(GenTezUtils.java:408)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:383)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:205)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10371)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:208)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:239)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:479)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:347)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1203)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1257)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1140)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1130)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:204)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:433)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:894)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:825)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:726)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:223)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136){code}
>
> *Steps to reproduce:*
> set hive.execution.engine=tez;
> set hive.tez.dynamic.partition.pruning=true;
> CREATE TABLE t1(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
> CREATE TABLE t2(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
> CREATE TABLE t3(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
>
> insert into table t1 partition(dt='2018') values ('k1','v1',1,1.0,true);
> insert into table t2 partition(dt='2018') values ('k2','v2',2,2.0,true);
> insert into table t3 partition(dt='2018') values ('k3','v3',3,3.0,true);
>
> CREATE VIEW `view1` AS select
> `t1`.`key`,`t1`.`value`,`t1`.`c_int`,`t1`.`c_float`,`t1`.`c_boolean`,`t1`.`dt`
> from `t1` union all select
> `t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_boolean`,`t2`.`dt`
> from `t2`;
> CREATE VIEW `view2` AS select
> `t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_boolean`,`t2`.`dt`
> from `t2` union all select
> `t3`.`key`,`t3`.`value`,`t3`.`c_int`,`t3`.`c_float`,`t3`.`c_boolean`,`t3`.`dt`
> from `t3`;
> create table t4 as select key,value,c_int,c_float,c_boolean,dt from t1 union
> all select v1.key,v1.value,v1.c_int,v1.c_float,v1.c_boolean,v1.dt from view1
> v1 join view2 v2 on v1.dt=v2.dt;
> CREATE VIEW `view3` AS select
> `t4`.`key`,`t4`.`value`,`t4`.`c_int`,`t4`.`c_float`,`t4`.`c_boolean`,`t4`.`dt`
> from `t4` union all select
> `t1`.`key`,`t1`.`value`,`t1`.`c_int`,`t1`.`c_float`,`t1`.`c_boolean`,`t1`.`dt`
> from `t1`;
>
> select count(0) from view2 v2 join view3 v3 on v2.dt=v3.dt; // Throws No work
> found for tablescan error
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19850) Dynamic partition pruning in Tez is leading to 'No work found for tablescan' error
[
https://issues.apache.org/jira/browse/HIVE-19850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ganesha Shreedhara updated HIVE-19850:
--
Affects Version/s: (was: 2.3.2)
3.0.0
Status: Patch Available (was: Open)
Found that the value for TableScanOperator is not getting retrieved from
rootToWorkMap even though operatorId/name/identifier is matching. It is not a
good idea to override equals/hashcode in Operator class as it can break other
things, so added one more level of comparing the table scan operators based on
operatorId if the objects comparison doesn't work. Please review and let me
know if there is a better way of handling this.
> Dynamic partition pruning in Tez is leading to 'No work found for tablescan'
> error
> --
>
> Key: HIVE-19850
> URL: https://issues.apache.org/jira/browse/HIVE-19850
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 3.0.0
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-19850.patch
>
>
>
> When multiple views are used along with union all, it is resulting in the
> following error when dynamic partition pruning is enabled in tez.
>
> {code:java}
> Exception in thread "main" java.lang.AssertionError: No work found for
> tablescan TS[8]
> at
> org.apache.hadoop.hive.ql.parse.GenTezUtils.processAppMasterEvent(GenTezUtils.java:408)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:383)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:205)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10371)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:208)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:239)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:479)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:347)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1203)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1257)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1140)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1130)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:204)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:433)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:894)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:825)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:726)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:223)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136){code}
>
> *Steps to reproduce:*
> set hive.execution.engine=tez;
> set hive.tez.dynamic.partition.pruning=true;
> CREATE TABLE t1(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
> CREATE TABLE t2(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
> CREATE TABLE t3(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
>
> insert into table t1 partition(dt='2018') values ('k1','v1',1,1.0,true);
> insert into table t2 partition(dt='2018') values ('k2','v2',2,2.0,true);
> insert into table t3 partition(dt='2018') values ('k3','v3',3,3.0,true);
>
> CREATE VIEW `view1` AS select
> `t1`.`key`,`t1`.`value`,`t1`.`c_int`,`t1`.`c_float`,`t1`.`c_boolean`,`t1`.`dt`
> from `t1` union all select
> `t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_boolean`,`t2`.`dt`
> from `t2`;
> CREATE VIEW `view2` AS select
> `t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_boolean`,`t2`.`dt`
> from `t2` union all select
> `t3`.`key`,`t3`.`value`,`t3`.`c_int`,`t3`.`c_float`,`t3`.`c_boolean`,`t3`.`dt`
> from `t3`;
> create table t4 as select key,value,c_int,c_float,c_boolean,dt from t1 union
> all select v1.key,v1.value,v1.c_int,v1.c_float,v1.c_boolean,v1.dt from view1
> v1 join view2 v2 on v1.dt=v2.dt;
> CREATE VIEW `view3` AS select
> `t4`.`key`,`t4`.`value`,`t4`.`c_int`,`t4`.`c_float`,`t4`.`c_boolean`,`t4`.`dt`
> from `t4` union all select
>
[jira] [Updated] (HIVE-19850) Dynamic partition pruning in Tez is leading to 'No work found for tablescan' error
[
https://issues.apache.org/jira/browse/HIVE-19850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ganesha Shreedhara updated HIVE-19850:
--
Attachment: HIVE-19850.patch
> Dynamic partition pruning in Tez is leading to 'No work found for tablescan'
> error
> --
>
> Key: HIVE-19850
> URL: https://issues.apache.org/jira/browse/HIVE-19850
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 2.3.2
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-19850.patch
>
>
>
> When multiple views are used along with union all, it is resulting in the
> following error when dynamic partition pruning is enabled in tez.
>
> {code:java}
> Exception in thread "main" java.lang.AssertionError: No work found for
> tablescan TS[8]
> at
> org.apache.hadoop.hive.ql.parse.GenTezUtils.processAppMasterEvent(GenTezUtils.java:408)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:383)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:205)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10371)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:208)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:239)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:479)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:347)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1203)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1257)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1140)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1130)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:204)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:433)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:894)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:825)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:726)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:223)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136){code}
>
> *Steps to reproduce:*
> set hive.execution.engine=tez;
> set hive.tez.dynamic.partition.pruning=true;
> CREATE TABLE t1(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
> CREATE TABLE t2(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
> CREATE TABLE t3(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
>
> insert into table t1 partition(dt='2018') values ('k1','v1',1,1.0,true);
> insert into table t2 partition(dt='2018') values ('k2','v2',2,2.0,true);
> insert into table t3 partition(dt='2018') values ('k3','v3',3,3.0,true);
>
> CREATE VIEW `view1` AS select
> `t1`.`key`,`t1`.`value`,`t1`.`c_int`,`t1`.`c_float`,`t1`.`c_boolean`,`t1`.`dt`
> from `t1` union all select
> `t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_boolean`,`t2`.`dt`
> from `t2`;
> CREATE VIEW `view2` AS select
> `t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_boolean`,`t2`.`dt`
> from `t2` union all select
> `t3`.`key`,`t3`.`value`,`t3`.`c_int`,`t3`.`c_float`,`t3`.`c_boolean`,`t3`.`dt`
> from `t3`;
> create table t4 as select key,value,c_int,c_float,c_boolean,dt from t1 union
> all select v1.key,v1.value,v1.c_int,v1.c_float,v1.c_boolean,v1.dt from view1
> v1 join view2 v2 on v1.dt=v2.dt;
> CREATE VIEW `view3` AS select
> `t4`.`key`,`t4`.`value`,`t4`.`c_int`,`t4`.`c_float`,`t4`.`c_boolean`,`t4`.`dt`
> from `t4` union all select
> `t1`.`key`,`t1`.`value`,`t1`.`c_int`,`t1`.`c_float`,`t1`.`c_boolean`,`t1`.`dt`
> from `t1`;
>
> select count(0) from view2 v2 join view3 v3 on v2.dt=v3.dt; // Throws No work
> found for tablescan error
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19850) Dynamic partition pruning in Tez is leading to 'No work found for tablescan' error
[
https://issues.apache.org/jira/browse/HIVE-19850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ganesha Shreedhara updated HIVE-19850:
--
Description:
When multiple views are used along with union all, it is resulting in the
following error when dynamic partition pruning is enabled in tez.
{code:java}
Exception in thread "main" java.lang.AssertionError: No work found for
tablescan TS[8]
at
org.apache.hadoop.hive.ql.parse.GenTezUtils.processAppMasterEvent(GenTezUtils.java:408)
at
org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:383)
at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:205)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10371)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:208)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:239)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:479)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:347)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1203)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1257)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1140)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1130)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:204)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:433)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:894)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:825)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:726)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:223)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136){code}
*Steps to reproduce:*
set hive.execution.engine=tez;
set hive.tez.dynamic.partition.pruning=true;
CREATE TABLE t1(key string, value string, c_int int, c_float float, c_boolean
boolean) partitioned by (dt string);
CREATE TABLE t2(key string, value string, c_int int, c_float float, c_boolean
boolean) partitioned by (dt string);
CREATE TABLE t3(key string, value string, c_int int, c_float float, c_boolean
boolean) partitioned by (dt string);
insert into table t1 partition(dt='2018') values ('k1','v1',1,1.0,true);
insert into table t2 partition(dt='2018') values ('k2','v2',2,2.0,true);
insert into table t3 partition(dt='2018') values ('k3','v3',3,3.0,true);
CREATE VIEW `view1` AS select
`t1`.`key`,`t1`.`value`,`t1`.`c_int`,`t1`.`c_float`,`t1`.`c_boolean`,`t1`.`dt`
from `t1` union all select
`t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_boolean`,`t2`.`dt`
from `t2`;
CREATE VIEW `view2` AS select
`t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_boolean`,`t2`.`dt`
from `t2` union all select
`t3`.`key`,`t3`.`value`,`t3`.`c_int`,`t3`.`c_float`,`t3`.`c_boolean`,`t3`.`dt`
from `t3`;
create table t4 as select key,value,c_int,c_float,c_boolean,dt from t1 union
all select v1.key,v1.value,v1.c_int,v1.c_float,v1.c_boolean,v1.dt from view1 v1
join view2 v2 on v1.dt=v2.dt;
CREATE VIEW `view3` AS select
`t4`.`key`,`t4`.`value`,`t4`.`c_int`,`t4`.`c_float`,`t4`.`c_boolean`,`t4`.`dt`
from `t4` union all select
`t1`.`key`,`t1`.`value`,`t1`.`c_int`,`t1`.`c_float`,`t1`.`c_boolean`,`t1`.`dt`
from `t1`;
select count(0) from view2 v2 join view3 v3 on v2.dt=v3.dt; // Throws No work
found for tablescan error
was:
When multiple views are used along with union all, it is resulting in the
following error when dynamic partition pruning is enabled in tez.
{code:java}
Exception in thread "main" java.lang.AssertionError: No work found for
tablescan TS[8]
at
org.apache.hadoop.hive.ql.parse.GenTezUtils.processAppMasterEvent(GenTezUtils.java:408)
at
org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:383)
at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:205)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10371)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:208)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:239)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:479)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:347)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1203)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1257)
at
[jira] [Updated] (HIVE-19850) Dynamic partition pruning in Tez is leading to 'No work found for tablescan' error
[
https://issues.apache.org/jira/browse/HIVE-19850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ganesha Shreedhara updated HIVE-19850:
--
Component/s: Tez
> Dynamic partition pruning in Tez is leading to 'No work found for tablescan'
> error
> --
>
> Key: HIVE-19850
> URL: https://issues.apache.org/jira/browse/HIVE-19850
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 2.3.2
>Reporter: Ganesha Shreedhara
>Priority: Major
>
>
> When multiple views are used along with union all, it is resulting in the
> following error when dynamic partition pruning is enabled in tez.
>
> {code:java}
> Exception in thread "main" java.lang.AssertionError: No work found for
> tablescan TS[8]
> at
> org.apache.hadoop.hive.ql.parse.GenTezUtils.processAppMasterEvent(GenTezUtils.java:408)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:383)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:205)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10371)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:208)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:239)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:479)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:347)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1203)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1257)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1140)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1130)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:204)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:433)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:894)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:825)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:726)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:223)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136){code}
>
> *Steps to reproduce:*
> set hive.execution.engine=tez;
> hive.tez.dynamic.partition.pruning=true;
> CREATE TABLE t1(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
> CREATE TABLE t2(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
> CREATE TABLE t3(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
>
> insert into table t1 partition(dt='2018') values ('k1','v1',1,1.0,true);
> insert into table t2 partition(dt='2018') values ('k2','v2',2,2.0,true);
> insert into table t3 partition(dt='2018') values ('k3','v3',3,3.0,true);
>
> CREATE VIEW `view1` AS select
> `t1`.`key`,`t1`.`value`,`t1`.`c_int`,`t1`.`c_float`,`t1`.`c_boolean`,`t1`.`dt`
> from `t1` union all select
> `t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_boolean`,`t2`.`dt`
> from `t2`;
> CREATE VIEW `view2` AS select
> `t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_boolean`,`t2`.`dt`
> from `t2` union all select
> `t3`.`key`,`t3`.`value`,`t3`.`c_int`,`t3`.`c_float`,`t3`.`c_boolean`,`t3`.`dt`
> from `t3`;
> create table t4 as select key,value,c_int,c_float,c_boolean,dt from t1 union
> all select v1.key,v1.value,v1.c_int,v1.c_float,v1.c_boolean,v1.dt from view1
> v1 join view2 v2 on v1.dt=v2.dt;
> CREATE VIEW `view3` AS select
> `t4`.`key`,`t4`.`value`,`t4`.`c_int`,`t4`.`c_float`,`t4`.`c_boolean`,`t4`.`dt`
> from `t4` union all select
> `t1`.`key`,`t1`.`value`,`t1`.`c_int`,`t1`.`c_float`,`t1`.`c_boolean`,`t1`.`dt`
> from `t1`;
>
> select count(0) from view2 v2 join view3 v3 on v2.dt=v3.dt; // Throws No work
> found for tablescan error
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19850) Dynamic partition pruning in Tez is leading to 'No work found for tablescan' error
[
https://issues.apache.org/jira/browse/HIVE-19850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ganesha Shreedhara updated HIVE-19850:
--
Affects Version/s: 2.3.2
> Dynamic partition pruning in Tez is leading to 'No work found for tablescan'
> error
> --
>
> Key: HIVE-19850
> URL: https://issues.apache.org/jira/browse/HIVE-19850
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.2
>Reporter: Ganesha Shreedhara
>Priority: Major
>
>
> When multiple views are used along with union all, it is resulting in the
> following error when dynamic partition pruning is enabled in tez.
>
> {code:java}
> Exception in thread "main" java.lang.AssertionError: No work found for
> tablescan TS[8]
> at
> org.apache.hadoop.hive.ql.parse.GenTezUtils.processAppMasterEvent(GenTezUtils.java:408)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:383)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:205)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10371)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:208)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:239)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:479)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:347)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1203)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1257)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1140)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1130)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:204)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:433)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:894)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:825)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:726)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:223)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136){code}
>
> *Steps to reproduce:*
> set hive.execution.engine=tez;
> hive.tez.dynamic.partition.pruning=true;
> CREATE TABLE t1(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
> CREATE TABLE t2(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
> CREATE TABLE t3(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
>
> insert into table t1 partition(dt='2018') values ('k1','v1',1,1.0,true);
> insert into table t2 partition(dt='2018') values ('k2','v2',2,2.0,true);
> insert into table t3 partition(dt='2018') values ('k3','v3',3,3.0,true);
>
> CREATE VIEW `view1` AS select
> `t1`.`key`,`t1`.`value`,`t1`.`c_int`,`t1`.`c_float`,`t1`.`c_boolean`,`t1`.`dt`
> from `t1` union all select
> `t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_boolean`,`t2`.`dt`
> from `t2`;
> CREATE VIEW `view2` AS select
> `t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_boolean`,`t2`.`dt`
> from `t2` union all select
> `t3`.`key`,`t3`.`value`,`t3`.`c_int`,`t3`.`c_float`,`t3`.`c_boolean`,`t3`.`dt`
> from `t3`;
> create table t4 as select key,value,c_int,c_float,c_boolean,dt from t1 union
> all select v1.key,v1.value,v1.c_int,v1.c_float,v1.c_boolean,v1.dt from view1
> v1 join view2 v2 on v1.dt=v2.dt;
> CREATE VIEW `view3` AS select
> `t4`.`key`,`t4`.`value`,`t4`.`c_int`,`t4`.`c_float`,`t4`.`c_boolean`,`t4`.`dt`
> from `t4` union all select
> `t1`.`key`,`t1`.`value`,`t1`.`c_int`,`t1`.`c_float`,`t1`.`c_boolean`,`t1`.`dt`
> from `t1`;
>
> select count(0) from view2 v2 join view3 v3 on v2.dt=v3.dt; // Throws No work
> found for tablescan error
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
