[jira] [Commented] (HIVE-11693) CommonMergeJoinOperator throws exception with tez
[ https://issues.apache.org/jira/browse/HIVE-11693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15662437#comment-15662437 ] Hive QA commented on HIVE-11693: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12766760/HIVE-11693.1.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10692 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_acid_non_acid] (batchId=150) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=145) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=91) org.apache.hive.spark.client.TestSparkClient.testJobSubmission (batchId=272) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2103/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2103/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2103/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12766760 - PreCommit-HIVE-Build > CommonMergeJoinOperator throws exception with tez > - > > Key: HIVE-11693 > URL: https://issues.apache.org/jira/browse/HIVE-11693 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Selina Zhang > Attachments: HIVE-11693.1.patch > > > Got this when executing a simple query with latest hive build + tez latest > version. > {noformat} > Error: Failure while running task: > attempt_1439860407967_0291_2_03_45_0:java.lang.RuntimeException: > java.lang.RuntimeException: Hive Runtime Error while closing operators: > java.lang.RuntimeException: java.io.IOException: Please check if you are > invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171) > at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:349) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:35) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.RuntimeException: Hive Runtime Error while closing > operators: java.lang.RuntimeException: java.io.IOException: Please check if > you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:316) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:162) > ... 14 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.RuntimeException: java.io.IOException: Please check if you are > invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:412) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchNextGroup(CommonMergeJoinOperator.java:375) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.doFirstFetchIfNeeded(CommonMergeJoinOperator.java:482) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinFinalLeftData(CommonMergeJoinOperator.java:434) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.closeOp(CommonMergeJoinOperator.java:384) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:616) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:292) > ... 15 more > Ca
[jira] [Commented] (HIVE-11693) CommonMergeJoinOperator throws exception with tez
[ https://issues.apache.org/jira/browse/HIVE-11693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15662346#comment-15662346 ] Premal Shah commented on HIVE-11693: [~selinazh] Is there a patch available? After applying the patch (https://issues.apache.org/jira/secure/attachment/12766760/HIVE-11693.1.patch), I noticed incorrect data after a left outer join. This query returns the right number of rows {noformat} SELECT COUNT(*) FROM account LEFT JOIN contact ON contact.account_id = account.id ; {noformat} But, if I have a complex sub-query on the right, then I get 0 rows back {noformat} SELECT COUNT(*) FROM account LEFT JOIN ( SELECT DISTINCT contact.email_domain AS email_domain, contact.account_id FROM contact JOIN ( SELECT account_id, COUNT(DISTINCT email_domain) AS email_domain_count FROM contact GROUP BY account_id ) email_domain_counts ON contact.account_id = email_domain_counts.account_id AND email_domain_counts.email_domain_count = 1 WHERE contact.account_id != '' ) contact ON contact.account_id = account.id ; {noformat} If, I change hive.execution.engine to mr, then it returns the number of rows in map_account > CommonMergeJoinOperator throws exception with tez > - > > Key: HIVE-11693 > URL: https://issues.apache.org/jira/browse/HIVE-11693 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Selina Zhang > Attachments: HIVE-11693.1.patch > > > Got this when executing a simple query with latest hive build + tez latest > version. > {noformat} > Error: Failure while running task: > attempt_1439860407967_0291_2_03_45_0:java.lang.RuntimeException: > java.lang.RuntimeException: Hive Runtime Error while closing operators: > java.lang.RuntimeException: java.io.IOException: Please check if you are > invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171) > at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:349) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:35) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.RuntimeException: Hive Runtime Error while closing > operators: java.lang.RuntimeException: java.io.IOException: Please check if > you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:316) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:162) > ... 14 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.RuntimeException: java.io.IOException: Please check if you are > invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:412) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchNextGroup(CommonMergeJoinOperator.java:375) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.doFirstFetchIfNeeded(CommonMergeJoinOperator.java:482) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinFinalLeftData(CommonMergeJoinOperator.java:434) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.closeOp(CommonMergeJoinOperator.java:384) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:616) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:292) > ... 15 more > Caused by: java.lang.RuntimeException: java.io.IOException: Ple
[jira] [Commented] (HIVE-11693) CommonMergeJoinOperator throws exception with tez
[ https://issues.apache.org/jira/browse/HIVE-11693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418137#comment-15418137 ] Hive QA commented on HIVE-11693: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12766760/HIVE-11693.1.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10418 tests executed *Failed tests:* {noformat} TestMsgBusConnection - did not produce a TEST-*.xml file TestQueryLifeTimeHook - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_llap_counters org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_1 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_2 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_join_hash {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/857/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/857/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-857/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 6 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12766760 - PreCommit-HIVE-MASTER-Build > CommonMergeJoinOperator throws exception with tez > - > > Key: HIVE-11693 > URL: https://issues.apache.org/jira/browse/HIVE-11693 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Selina Zhang > Attachments: HIVE-11693.1.patch > > > Got this when executing a simple query with latest hive build + tez latest > version. > {noformat} > Error: Failure while running task: > attempt_1439860407967_0291_2_03_45_0:java.lang.RuntimeException: > java.lang.RuntimeException: Hive Runtime Error while closing operators: > java.lang.RuntimeException: java.io.IOException: Please check if you are > invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171) > at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:349) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:35) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.RuntimeException: Hive Runtime Error while closing > operators: java.lang.RuntimeException: java.io.IOException: Please check if > you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:316) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:162) > ... 14 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.RuntimeException: java.io.IOException: Please check if you are > invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:412) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchNextGroup(CommonMergeJoinOperator.java:375) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.doFirstFetchIfNeeded(CommonMergeJoinOperator.java:482) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinFinalLeftData(CommonMergeJoinOperator.java:434) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.closeOp(CommonMergeJoinOperator.java:384) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:616) > at >
[jira] [Commented] (HIVE-11693) CommonMergeJoinOperator throws exception with tez
[ https://issues.apache.org/jira/browse/HIVE-11693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417699#comment-15417699 ] Mithun Radhakrishnan commented on HIVE-11693: - [~selinazh], could we please post our solution to this JIRA? > CommonMergeJoinOperator throws exception with tez > - > > Key: HIVE-11693 > URL: https://issues.apache.org/jira/browse/HIVE-11693 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Selina Zhang > Attachments: HIVE-11693.1.patch > > > Got this when executing a simple query with latest hive build + tez latest > version. > {noformat} > Error: Failure while running task: > attempt_1439860407967_0291_2_03_45_0:java.lang.RuntimeException: > java.lang.RuntimeException: Hive Runtime Error while closing operators: > java.lang.RuntimeException: java.io.IOException: Please check if you are > invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171) > at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:349) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:35) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.RuntimeException: Hive Runtime Error while closing > operators: java.lang.RuntimeException: java.io.IOException: Please check if > you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:316) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:162) > ... 14 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.RuntimeException: java.io.IOException: Please check if you are > invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:412) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchNextGroup(CommonMergeJoinOperator.java:375) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.doFirstFetchIfNeeded(CommonMergeJoinOperator.java:482) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinFinalLeftData(CommonMergeJoinOperator.java:434) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.closeOp(CommonMergeJoinOperator.java:384) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:616) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:292) > ... 15 more > Caused by: java.lang.RuntimeException: java.io.IOException: Please check if > you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:291) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:400) > ... 21 more > Caused by: java.io.IOException: Please check if you are invoking moveToNext() > even after it returned false. > at > org.apache.tez.runtime.library.common.ValuesIterator.hasCompletedProcessing(ValuesIterator.java:223) > at > org.apache.tez.runtime.library.common.ValuesIterator.moveToNext(ValuesIterator.java:105) > at > org.apache.tez.runtime.library.input.OrderedGroupedKVInput$OrderedGroupedKeyValuesReader.next(OrderedGroupedKVInput.java:308) > at > org.apache.hadoop.hive.ql.exec.tez.KeyValuesFromKeyValues.next(KeyValuesFromKeyValues.java:46) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:249) > ... 22 more > {noformat} > Not sure if this is related to HIVE-11016. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11693) CommonMergeJoinOperator throws exception with tez
[ https://issues.apache.org/jira/browse/HIVE-11693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15364738#comment-15364738 ] Mithun Radhakrishnan commented on HIVE-11693: - Kewl. Assigned to [~selinazh]. Should have our patch out soon. > CommonMergeJoinOperator throws exception with tez > - > > Key: HIVE-11693 > URL: https://issues.apache.org/jira/browse/HIVE-11693 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Selina Zhang > Attachments: HIVE-11693.1.patch > > > Got this when executing a simple query with latest hive build + tez latest > version. > {noformat} > Error: Failure while running task: > attempt_1439860407967_0291_2_03_45_0:java.lang.RuntimeException: > java.lang.RuntimeException: Hive Runtime Error while closing operators: > java.lang.RuntimeException: java.io.IOException: Please check if you are > invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171) > at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:349) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:35) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.RuntimeException: Hive Runtime Error while closing > operators: java.lang.RuntimeException: java.io.IOException: Please check if > you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:316) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:162) > ... 14 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.RuntimeException: java.io.IOException: Please check if you are > invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:412) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchNextGroup(CommonMergeJoinOperator.java:375) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.doFirstFetchIfNeeded(CommonMergeJoinOperator.java:482) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinFinalLeftData(CommonMergeJoinOperator.java:434) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.closeOp(CommonMergeJoinOperator.java:384) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:616) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:292) > ... 15 more > Caused by: java.lang.RuntimeException: java.io.IOException: Please check if > you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:291) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:400) > ... 21 more > Caused by: java.io.IOException: Please check if you are invoking moveToNext() > even after it returned false. > at > org.apache.tez.runtime.library.common.ValuesIterator.hasCompletedProcessing(ValuesIterator.java:223) > at > org.apache.tez.runtime.library.common.ValuesIterator.moveToNext(ValuesIterator.java:105) > at > org.apache.tez.runtime.library.input.OrderedGroupedKVInput$OrderedGroupedKeyValuesReader.next(OrderedGroupedKVInput.java:308) > at > org.apache.hadoop.hive.ql.exec.tez.KeyValuesFromKeyValues.next(KeyValuesFromKeyValues.java:46) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:249) > ... 22 more > {noformat} > Not sure if this is related to HIVE-11016. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11693) CommonMergeJoinOperator throws exception with tez
[ https://issues.apache.org/jira/browse/HIVE-11693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15364712#comment-15364712 ] Gopal V commented on HIVE-11693: That sounds reasonable - since this is unassigned and AFAIK [~rajesh.balamohan] has been dragged into hadoop-common/hdfs, can [~selinazh] take over and provide a patch? For now, this is +1 - tests pending. > CommonMergeJoinOperator throws exception with tez > - > > Key: HIVE-11693 > URL: https://issues.apache.org/jira/browse/HIVE-11693 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan > Attachments: HIVE-11693.1.patch > > > Got this when executing a simple query with latest hive build + tez latest > version. > {noformat} > Error: Failure while running task: > attempt_1439860407967_0291_2_03_45_0:java.lang.RuntimeException: > java.lang.RuntimeException: Hive Runtime Error while closing operators: > java.lang.RuntimeException: java.io.IOException: Please check if you are > invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171) > at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:349) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:35) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.RuntimeException: Hive Runtime Error while closing > operators: java.lang.RuntimeException: java.io.IOException: Please check if > you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:316) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:162) > ... 14 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.RuntimeException: java.io.IOException: Please check if you are > invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:412) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchNextGroup(CommonMergeJoinOperator.java:375) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.doFirstFetchIfNeeded(CommonMergeJoinOperator.java:482) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinFinalLeftData(CommonMergeJoinOperator.java:434) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.closeOp(CommonMergeJoinOperator.java:384) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:616) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:292) > ... 15 more > Caused by: java.lang.RuntimeException: java.io.IOException: Please check if > you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:291) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:400) > ... 21 more > Caused by: java.io.IOException: Please check if you are invoking moveToNext() > even after it returned false. > at > org.apache.tez.runtime.library.common.ValuesIterator.hasCompletedProcessing(ValuesIterator.java:223) > at > org.apache.tez.runtime.library.common.ValuesIterator.moveToNext(ValuesIterator.java:105) > at > org.apache.tez.runtime.library.input.OrderedGroupedKVInput$OrderedGroupedKeyValuesReader.next(OrderedGroupedKVInput.java:308) > at > org.apache.hadoop.hive.ql.exec.tez.KeyValuesFromKeyValues.next(KeyValuesFromKeyValues.java:46) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:249) > ... 22 more > {noformat} > Not sure if this is related to HIVE-11016. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11693) CommonMergeJoinOperator throws exception with tez
[ https://issues.apache.org/jira/browse/HIVE-11693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15364709#comment-15364709 ] Mithun Radhakrishnan commented on HIVE-11693: - [~rajesh.balamohan], et al., [~selinazh]'s analysis here seems accurate. Wouldn't her suggestion (i.e. to move {{posBigTable = (byte) conf.getBigTablePosition();}} to {{initializeOp()}}) fix the problem? Would anyone else like to comment? > CommonMergeJoinOperator throws exception with tez > - > > Key: HIVE-11693 > URL: https://issues.apache.org/jira/browse/HIVE-11693 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan > Attachments: HIVE-11693.1.patch > > > Got this when executing a simple query with latest hive build + tez latest > version. > {noformat} > Error: Failure while running task: > attempt_1439860407967_0291_2_03_45_0:java.lang.RuntimeException: > java.lang.RuntimeException: Hive Runtime Error while closing operators: > java.lang.RuntimeException: java.io.IOException: Please check if you are > invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171) > at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:349) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:35) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.RuntimeException: Hive Runtime Error while closing > operators: java.lang.RuntimeException: java.io.IOException: Please check if > you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:316) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:162) > ... 14 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.RuntimeException: java.io.IOException: Please check if you are > invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:412) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchNextGroup(CommonMergeJoinOperator.java:375) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.doFirstFetchIfNeeded(CommonMergeJoinOperator.java:482) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinFinalLeftData(CommonMergeJoinOperator.java:434) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.closeOp(CommonMergeJoinOperator.java:384) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:616) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:292) > ... 15 more > Caused by: java.lang.RuntimeException: java.io.IOException: Please check if > you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:291) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:400) > ... 21 more > Caused by: java.io.IOException: Please check if you are invoking moveToNext() > even after it returned false. > at > org.apache.tez.runtime.library.common.ValuesIterator.hasCompletedProcessing(ValuesIterator.java:223) > at > org.apache.tez.runtime.library.common.ValuesIterator.moveToNext(ValuesIterator.java:105) > at > org.apache.tez.runtime.library.input.OrderedGroupedKVInput$OrderedGroupedKeyValuesReader.next(OrderedGroupedKVInput.java:308) > at > org.apache.hadoop.hive.ql.exec.tez.KeyValuesFromKeyValues.next(KeyValuesFromKeyValues.java:46) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:249) > ... 22 more > {noformat} > Not sure if this is related to HIVE-11016. -- Th
[jira] [Commented] (HIVE-11693) CommonMergeJoinOperator throws exception with tez
[ https://issues.apache.org/jira/browse/HIVE-11693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15302701#comment-15302701 ] Selina Zhang commented on HIVE-11693: - [~rajesh.balamohan], we hit the same issue recently. But I think the patch you attached did not fix the root problem. The issue is actually CommonMergeJoinOperator only set big table position when it has inputs for big table. {code:title=CommonMergeJoinOperator.java} @Override public void process(Object row, int tag) throws HiveException { posBigTable = (byte) conf.getBigTablePosition(); ... {code} If the input is empty, the above method will not be called. In the query you listed, a subquery is involved. The generated table is tagged as 0, while the left table is tagged as 1. GenTezWork.java set the big table position as 1 for both reduce work and CommonJoinOperator. In reduce phase, when ReduceRecordProcessor got executed, it retrieves the record from big table: {code:title=ReduceRecordProcessor.java} @Override void run() throws Exception { // run the operator pipeline while (sources[bigTablePosition].pushRecord()) { } } {code} The big table position here is 1. If the input from the big table is empty, this is the only place pushRecord() be called to read big table. However, because the CommonMergeJoinOperator missed set big table position, in closeOp() part, it will think tag 1 is small table, so another pushRecord() is called to retrieve table content. Then we see the exception listed in this JIRA. Please let me know if my analysis has problem. If you think it is correct, can you update the patch? Thanks > CommonMergeJoinOperator throws exception with tez > - > > Key: HIVE-11693 > URL: https://issues.apache.org/jira/browse/HIVE-11693 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan > Attachments: HIVE-11693.1.patch > > > Got this when executing a simple query with latest hive build + tez latest > version. > {noformat} > Error: Failure while running task: > attempt_1439860407967_0291_2_03_45_0:java.lang.RuntimeException: > java.lang.RuntimeException: Hive Runtime Error while closing operators: > java.lang.RuntimeException: java.io.IOException: Please check if you are > invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171) > at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:349) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:35) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.RuntimeException: Hive Runtime Error while closing > operators: java.lang.RuntimeException: java.io.IOException: Please check if > you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:316) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:162) > ... 14 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.RuntimeException: java.io.IOException: Please check if you are > invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:412) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchNextGroup(CommonMergeJoinOperator.java:375) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.doFirstFetchIfNeeded(CommonMergeJoinOperator.java:482) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinFinalLeftData(CommonMergeJoinOperator.java:434) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.closeOp(CommonMergeJoinOperator.java:384) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:61
[jira] [Commented] (HIVE-11693) CommonMergeJoinOperator throws exception with tez
[ https://issues.apache.org/jira/browse/HIVE-11693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14960782#comment-14960782 ] Hive QA commented on HIVE-11693: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12766760/HIVE-11693.1.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 9694 tests executed *Failed tests:* {noformat} org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation org.apache.hive.jdbc.TestSSL.testSSLVersion {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5676/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5676/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5676/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12766760 - PreCommit-HIVE-TRUNK-Build > CommonMergeJoinOperator throws exception with tez > - > > Key: HIVE-11693 > URL: https://issues.apache.org/jira/browse/HIVE-11693 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan > Attachments: HIVE-11693.1.patch > > > Got this when executing a simple query with latest hive build + tez latest > version. > {noformat} > Error: Failure while running task: > attempt_1439860407967_0291_2_03_45_0:java.lang.RuntimeException: > java.lang.RuntimeException: Hive Runtime Error while closing operators: > java.lang.RuntimeException: java.io.IOException: Please check if you are > invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171) > at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:349) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:35) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.RuntimeException: Hive Runtime Error while closing > operators: java.lang.RuntimeException: java.io.IOException: Please check if > you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:316) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:162) > ... 14 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.RuntimeException: java.io.IOException: Please check if you are > invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:412) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchNextGroup(CommonMergeJoinOperator.java:375) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.doFirstFetchIfNeeded(CommonMergeJoinOperator.java:482) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinFinalLeftData(CommonMergeJoinOperator.java:434) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.closeOp(CommonMergeJoinOperator.java:384) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:616) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:292) > ... 15 more > Caused by: java.lang.RuntimeException: java.io.IOException: Please check if > you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.
[jira] [Commented] (HIVE-11693) CommonMergeJoinOperator throws exception with tez
[ https://issues.apache.org/jira/browse/HIVE-11693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14724365#comment-14724365 ] Sergey Shelukhin commented on HIVE-11693: - Well, then it's some other issue > CommonMergeJoinOperator throws exception with tez > - > > Key: HIVE-11693 > URL: https://issues.apache.org/jira/browse/HIVE-11693 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan > > Got this when executing a simple query with latest hive build + tez latest > version. > {noformat} > Error: Failure while running task: > attempt_1439860407967_0291_2_03_45_0:java.lang.RuntimeException: > java.lang.RuntimeException: Hive Runtime Error while closing operators: > java.lang.RuntimeException: java.io.IOException: Please check if you are > invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171) > at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:349) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:35) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.RuntimeException: Hive Runtime Error while closing > operators: java.lang.RuntimeException: java.io.IOException: Please check if > you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:316) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:162) > ... 14 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.RuntimeException: java.io.IOException: Please check if you are > invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:412) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchNextGroup(CommonMergeJoinOperator.java:375) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.doFirstFetchIfNeeded(CommonMergeJoinOperator.java:482) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinFinalLeftData(CommonMergeJoinOperator.java:434) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.closeOp(CommonMergeJoinOperator.java:384) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:616) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:292) > ... 15 more > Caused by: java.lang.RuntimeException: java.io.IOException: Please check if > you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:291) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:400) > ... 21 more > Caused by: java.io.IOException: Please check if you are invoking moveToNext() > even after it returned false. > at > org.apache.tez.runtime.library.common.ValuesIterator.hasCompletedProcessing(ValuesIterator.java:223) > at > org.apache.tez.runtime.library.common.ValuesIterator.moveToNext(ValuesIterator.java:105) > at > org.apache.tez.runtime.library.input.OrderedGroupedKVInput$OrderedGroupedKeyValuesReader.next(OrderedGroupedKVInput.java:308) > at > org.apache.hadoop.hive.ql.exec.tez.KeyValuesFromKeyValues.next(KeyValuesFromKeyValues.java:46) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:249) > ... 22 more > {noformat} > Not sure if this is related to HIVE-11016. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11693) CommonMergeJoinOperator throws exception with tez
[ https://issues.apache.org/jira/browse/HIVE-11693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14724349#comment-14724349 ] Rajesh Balamohan commented on HIVE-11693: - [~sershe] - Are you referring to "LOG.warn("fetchDone[" + tag + "] was set to true (by a recursive call) and will be reset");" ?. No, I do not find that in the logs. > CommonMergeJoinOperator throws exception with tez > - > > Key: HIVE-11693 > URL: https://issues.apache.org/jira/browse/HIVE-11693 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan > > Got this when executing a simple query with latest hive build + tez latest > version. > {noformat} > Error: Failure while running task: > attempt_1439860407967_0291_2_03_45_0:java.lang.RuntimeException: > java.lang.RuntimeException: Hive Runtime Error while closing operators: > java.lang.RuntimeException: java.io.IOException: Please check if you are > invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171) > at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:349) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:35) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.RuntimeException: Hive Runtime Error while closing > operators: java.lang.RuntimeException: java.io.IOException: Please check if > you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:316) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:162) > ... 14 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.RuntimeException: java.io.IOException: Please check if you are > invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:412) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchNextGroup(CommonMergeJoinOperator.java:375) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.doFirstFetchIfNeeded(CommonMergeJoinOperator.java:482) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinFinalLeftData(CommonMergeJoinOperator.java:434) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.closeOp(CommonMergeJoinOperator.java:384) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:616) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:292) > ... 15 more > Caused by: java.lang.RuntimeException: java.io.IOException: Please check if > you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:291) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:400) > ... 21 more > Caused by: java.io.IOException: Please check if you are invoking moveToNext() > even after it returned false. > at > org.apache.tez.runtime.library.common.ValuesIterator.hasCompletedProcessing(ValuesIterator.java:223) > at > org.apache.tez.runtime.library.common.ValuesIterator.moveToNext(ValuesIterator.java:105) > at > org.apache.tez.runtime.library.input.OrderedGroupedKVInput$OrderedGroupedKeyValuesReader.next(OrderedGroupedKVInput.java:308) > at > org.apache.hadoop.hive.ql.exec.tez.KeyValuesFromKeyValues.next(KeyValuesFromKeyValues.java:46) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:249) > ... 22 more > {noformat} > Not sure if this is related to HIVE-11016. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11693) CommonMergeJoinOperator throws exception with tez
[ https://issues.apache.org/jira/browse/HIVE-11693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14723754#comment-14723754 ] Sergey Shelukhin commented on HIVE-11693: - Is there a warning log added in HIVE-11016 present in the logs? > CommonMergeJoinOperator throws exception with tez > - > > Key: HIVE-11693 > URL: https://issues.apache.org/jira/browse/HIVE-11693 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan > > Got this when executing a simple query with latest hive build + tez latest > version. > {noformat} > Error: Failure while running task: > attempt_1439860407967_0291_2_03_45_0:java.lang.RuntimeException: > java.lang.RuntimeException: Hive Runtime Error while closing operators: > java.lang.RuntimeException: java.io.IOException: Please check if you are > invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171) > at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:349) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:35) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.RuntimeException: Hive Runtime Error while closing > operators: java.lang.RuntimeException: java.io.IOException: Please check if > you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:316) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:162) > ... 14 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.RuntimeException: java.io.IOException: Please check if you are > invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:412) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchNextGroup(CommonMergeJoinOperator.java:375) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.doFirstFetchIfNeeded(CommonMergeJoinOperator.java:482) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinFinalLeftData(CommonMergeJoinOperator.java:434) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.closeOp(CommonMergeJoinOperator.java:384) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:616) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:292) > ... 15 more > Caused by: java.lang.RuntimeException: java.io.IOException: Please check if > you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:291) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:400) > ... 21 more > Caused by: java.io.IOException: Please check if you are invoking moveToNext() > even after it returned false. > at > org.apache.tez.runtime.library.common.ValuesIterator.hasCompletedProcessing(ValuesIterator.java:223) > at > org.apache.tez.runtime.library.common.ValuesIterator.moveToNext(ValuesIterator.java:105) > at > org.apache.tez.runtime.library.input.OrderedGroupedKVInput$OrderedGroupedKeyValuesReader.next(OrderedGroupedKVInput.java:308) > at > org.apache.hadoop.hive.ql.exec.tez.KeyValuesFromKeyValues.next(KeyValuesFromKeyValues.java:46) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:249) > ... 22 more > {noformat} > Not sure if this is related to HIVE-11016. -- This message was sent by Atlassian JIRA (v6.3.4#6332)