[ https://issues.apache.org/jira/browse/HIVE-2820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2820: ------------------------------ Attachment: HIVE-2820.D1935.1.patch navis requested code review of "HIVE-2820 [jira] Invalid tag is used for MapJoinProcessor". Reviewers: JIRA DPAL-889 Invalid tag is used for MapJoinProcessor Testing HIVE-2810, I've found tag and alias are used in very confusing manner. For example, query below fails.. hive> set hive.auto.convert.join=true; hive> select /*+ STREAMTABLE(a) */ * from myinput1 a join myinput1 b on a.key=b.key join myinput1 c on a.key=c.key; Total MapReduce jobs = 4 Ended Job = 1667415037, job is filtered out (removed at runtime). Ended Job = 1739566906, job is filtered out (removed at runtime). Ended Job = 1113337780, job is filtered out (removed at runtime). 12/02/24 10:27:14 WARN conf.HiveConf: DEPRECATED: Ignoring hive-default.xml found on the CLASSPATH at /home/navis/hive/conf/hive-default.xml Execution log at: /tmp/navis/navis_20120224102727_cafe0d8d-9b21-441d-bd4e-b83303b31cdc.log 2012-02-24 10:27:14 Starting to launch local task to process map join; maximum memory = 932118528 java.lang.NullPointerException at org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.processOp(HashTableSinkOperator.java:312) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762) at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.MapredLocalTask.startForward(MapredLocalTask.java:325) at org.apache.hadoop.hive.ql.exec.MapredLocalTask.executeFromChildJVM(MapredLocalTask.java:272) at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:685) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:186) Execution failed with exit status: 2 Obtaining error information Failed task has a plan which doesn't make sense. Stage: Stage-8 Map Reduce Local Work Alias -> Map Local Tables: b Fetch Operator limit: -1 c Fetch Operator limit: -1 Alias -> Map Local Operator Tree: b TableScan alias: b HashTable Sink Operator condition expressions: 0 {key} {value} 1 {key} {value} 2 {key} {value} handleSkewJoin: false keys: 0 [Column[key]] 1 [Column[key]] 2 [Column[key]] Position of Big Table: 0 c TableScan alias: c Map Join Operator condition map: Inner Join 0 to 1 Inner Join 0 to 2 condition expressions: 0 {key} {value} 1 {key} {value} 2 {key} {value} handleSkewJoin: false keys: 0 [Column[key]] 1 [Column[key]] 2 [Column[key]] outputColumnNames: _col0, _col1, _col4, _col5, _col8, _col9 Position of Big Table: 0 Select Operator expressions: expr: _col0 type: int expr: _col1 type: int expr: _col4 type: int expr: _col5 type: int expr: _col8 type: int expr: _col9 type: int outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5 File Output Operator compressed: false GlobalTableId: 0 table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Stage: Stage-4 Map Reduce Alias -> Map Operator Tree: a TableScan alias: a HashTable Sink Operator condition expressions: 0 {key} {value} 1 {key} {value} 2 {key} {value} handleSkewJoin: false keys: 0 [Column[key]] 1 [Column[key]] 2 [Column[key]] Position of Big Table: 0 Local Work: Map Reduce Local Work TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1935 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractMapJoinOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinResolver.java ql/src/java/org/apache/hadoop/hive/ql/plan/HashTableSinkDesc.java ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java ql/src/test/queries/clientpositive/join_reorder2.q ql/src/test/results/clientpositive/join_reorder2.q.out MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/4101/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. > Invalid tag is used for MapJoinProcessor > ---------------------------------------- > > Key: HIVE-2820 > URL: https://issues.apache.org/jira/browse/HIVE-2820 > Project: Hive > Issue Type: Bug > Components: Query Processor > Affects Versions: 0.9.0 > Environment: ubuntu > Reporter: Navis > Assignee: Navis > Fix For: 0.9.0 > > Attachments: HIVE-2820.D1935.1.patch > > > Testing HIVE-2810, I've found tag and alias are used in very confusing > manner. For example, query below fails.. > {code} > hive> set hive.auto.convert.join=true; > > hive> select /*+ STREAMTABLE(a) */ * from myinput1 a join myinput1 b on > a.key=b.key join myinput1 c on a.key=c.key; > Total MapReduce jobs = 4 > Ended Job = 1667415037, job is filtered out (removed at runtime). > Ended Job = 1739566906, job is filtered out (removed at runtime). > Ended Job = 1113337780, job is filtered out (removed at runtime). > 12/02/24 10:27:14 WARN conf.HiveConf: DEPRECATED: Ignoring hive-default.xml > found on the CLASSPATH at /home/navis/hive/conf/hive-default.xml > Execution log at: > /tmp/navis/navis_20120224102727_cafe0d8d-9b21-441d-bd4e-b83303b31cdc.log > 2012-02-24 10:27:14 Starting to launch local task to process map join; > maximum memory = 932118528 > java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.processOp(HashTableSinkOperator.java:312) > at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) > at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762) > at > org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83) > at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) > at > org.apache.hadoop.hive.ql.exec.MapredLocalTask.startForward(MapredLocalTask.java:325) > at > org.apache.hadoop.hive.ql.exec.MapredLocalTask.executeFromChildJVM(MapredLocalTask.java:272) > at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:685) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:186) > Execution failed with exit status: 2 > Obtaining error information > {code} > Failed task has a plan which doesn't make sense. > {noformat} > Stage: Stage-8 > Map Reduce Local Work > Alias -> Map Local Tables: > b > Fetch Operator > limit: -1 > c > Fetch Operator > limit: -1 > Alias -> Map Local Operator Tree: > b > TableScan > alias: b > HashTable Sink Operator > condition expressions: > 0 {key} {value} > 1 {key} {value} > 2 {key} {value} > handleSkewJoin: false > keys: > 0 [Column[key]] > 1 [Column[key]] > 2 [Column[key]] > Position of Big Table: 0 > c > TableScan > alias: c > Map Join Operator > condition map: > Inner Join 0 to 1 > Inner Join 0 to 2 > condition expressions: > 0 {key} {value} > 1 {key} {value} > 2 {key} {value} > handleSkewJoin: false > keys: > 0 [Column[key]] > 1 [Column[key]] > 2 [Column[key]] > outputColumnNames: _col0, _col1, _col4, _col5, _col8, _col9 > Position of Big Table: 0 > Select Operator > expressions: > expr: _col0 > type: int > expr: _col1 > type: int > expr: _col4 > type: int > expr: _col5 > type: int > expr: _col8 > type: int > expr: _col9 > type: int > outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5 > File Output Operator > compressed: false > GlobalTableId: 0 > table: > input format: org.apache.hadoop.mapred.TextInputFormat > output format: > org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat > Stage: Stage-4 > Map Reduce > Alias -> Map Operator Tree: > a > TableScan > alias: a > HashTable Sink Operator > condition expressions: > 0 {key} {value} > 1 {key} {value} > 2 {key} {value} > handleSkewJoin: false > keys: > 0 [Column[key]] > 1 [Column[key]] > 2 [Column[key]] > Position of Big Table: 0 > Local Work: > Map Reduce Local Work > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira