According to the Jira ticket you linked, it was fixed in 1.2.1. Any way you can upgrade to that version? Otherwise, looking at the comment posted by Sergey, the issue sounds very low-level in the join logic. Not sure if there are any workarounds...
-Doug On Sat, Apr 16, 2016 at 11:32 PM, Mahender Sarangam < [email protected]> wrote: > Ping. Urgent help on this... > > ------------------------------ > From: [email protected] > To: [email protected]; [email protected] > Subject: Having Issue with Join Statement > Date: Sun, 17 Apr 2016 00:36:24 +0000 > > Hi, > > I'm doing join of 2 large tables along join with couple of CTEs. I'm > getting following error *" Failure while running > task:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime > Error while closing operators: java.lang.RuntimeException: > java.io.IOException: Please check if you are invoking moveToNext() even > after it returned false."* > > Has any one faced or came across with same issue. I see JIRA that it has > been already fixed. We are using Hive 1.2 version. > https://issues.apache.org/jira/browse/HIVE-11016 > > > Is there any work around for this issue. or what reason for this error. Is > there any Missing key. > > NFO : Map 1: 1/1 Map 11: 1/1 Map 12: 1/1 Map 13: 1/1 Map 16: 1/1 Map 17: > 1/1 Map 18: 26/26 Map 19: 1/1 Map 20: 1/1 Map 22: 145/145 Map 25: 4/4 Map > 26: 4/4 Map 27: 39/39 Map 28: 1/1 Map 29: 1/1 Map 30: 1/1 Map 31: 1/1 Map > 32: 1/1 Reducer 10: 0/1009 Reducer 14: 1/1 Reducer 15: 1/1 Reducer 2: > 1/1 Reducer 21: 270(+79,-3)/469 Reducer 23: 134/134 Reducer 24: > 427/427 Reducer 3: 1/1 Reducer 4: 0/1009 Reducer 5: 0/1009 Reducer 6: > 0/1009 Reducer 7: 0/1009 Reducer 8: 0/1009 Reducer 9: 0/1009 > INFO : Map 1: 1/1 Map 11: 1/1 Map 12: 1/1 Map 13: 1/1 Map 16: 1/1 Map 17: > 1/1 Map 18: 26/26 Map 19: 1/1 Map 20: 1/1 Map 22: 145/145 Map 25: 4/4 Map > 26: 4/4 Map 27: 39/39 Map 28: 1/1 Map 29: 1/1 Map 30: 1/1 Map 31: 1/1 Map > 32: 1/1 Reducer 10: 0/1009 Reducer 14: 1/1 Reducer 15: 1/1 Reducer 2: > 1/1 Reducer 21: 289(+82,-3)/469 Reducer 23: 134/134 Reducer 24: > 427/427 Reducer 3: 1/1 Reducer 4: 0/1009 Reducer 5: 0/1009 Reducer 6: > 0/1009 Reducer 7: 0/1009 Reducer 8: 0/1009 Reducer 9: 0/1009 > INFO : Map 1: 1/1 Map 11: 1/1 Map 12: 1/1 Map 13: 1/1 Map 16: 1/1 Map 17: > 1/1 Map 18: 26/26 Map 19: 1/1 Map 20: 1/1 Map 22: 145/145 Map 25: 4/4 Map > 26: 4/4 Map 27: 39/39 Map 28: 1/1 Map 29: 1/1 Map 30: 1/1 Map 31: 1/1 Map > 32: 1/1 Reducer 10: 0/1009 Reducer 14: 1/1 Reducer 15: 1/1 Reducer 2: > 1/1 Reducer 21: 309(+84,-3)/469 Reducer 23: 134/134 Reducer 24: > 427/427 Reducer 3: 1/1 Reducer 4: 0/1009 Reducer 5: 0/1009 Reducer 6: > 0/1009 Reducer 7: 0/1009 Reducer 8: 0/1009 Reducer 9: 0/1009 > INFO : Map 1: 1/1 Map 11: 1/1 Map 12: 1/1 Map 13: 1/1 Map 16: 1/1 Map 17: > 1/1 Map 18: 26/26 Map 19: 1/1 Map 20: 1/1 Map 22: 145/145 Map 25: 4/4 Map > 26: 4/4 Map 27: 39/39 Map 28: 1/1 Map 29: 1/1 Map 30: 1/1 Map 31: 1/1 Map > 32: 1/1 Reducer 10: 0/1009 Reducer 14: 1/1 Reducer 15: 1/1 Reducer 2: > 1/1 Reducer 21: 312(+0,-4)/469 Reducer 23: 134/134 Reducer 24: > 427/427 Reducer 3: 1/1 Reducer 4: 0/1009 Reducer 5: 0/1009 Reducer 6: > 0/1009 Reducer 7: 0/1009 Reducer 8: 0/1009 Reducer 9: 0/1009 > ERROR : Status: Failed > ERROR : Vertex failed, vertexName=Reducer 21, > vertexId=vertex_1460713464534_0363_1_09, diagnostics=[Task failed, > taskId=task_1460713464534_0363_1_09_000232, diagnostics=[TaskAttempt 0 > failed, info=[Error: Failure while running task:java.lang.RuntimeException: > java.lang.RuntimeException: Hive Runtime Error while closing operators: > java.lang.RuntimeException: java.io.IOException: Please check if you are > invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.RuntimeException: Hive Runtime Error while closing > operators: java.lang.RuntimeException: java.io.IOException: Please check if > you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:310) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:162) > ... 14 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.RuntimeException: java.io.IOException: Please check if you are > invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:419) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchNextGroup(CommonMergeJoinOperator.java:387) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinFinalLeftData(CommonMergeJoinOperator.java:456) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.closeOp(CommonMergeJoinOperator.java:396) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:617) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:287) > ... 15 more > Caused by: java.lang.RuntimeException: java.io.IOException: Please check > if you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:302) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:411) > ... 20 more > Caused by: java.io.IOException: Please check if you are invoking > moveToNext() even after it returned false. > at > org.apache.tez.runtime.library.common.ValuesIterator.hasCompletedProcessing(ValuesIterator.java:221) > at > org.apache.tez.runtime.library.common.ValuesIterator.moveToNext(ValuesIterator.java:103) > at > org.apache.tez.runtime.library.input.OrderedGroupedKVInput$OrderedGroupedKeyValuesReader.next(OrderedGroupedKVInput.java:308) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:260) > ... 21 more > ], TaskAttempt 1 failed, info=[Error: Failure while running > task:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime > Error while closing operators: java.lang.RuntimeException: > java.io.IOException: Please check if you are invoking moveToNext() even > after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.RuntimeException: Hive Runtime Error while closing > operators: java.lang.RuntimeException: java.io.IOException: Please check if > you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:310) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:162) > ... 14 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.RuntimeException: java.io.IOException: Please check if you are > invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:419) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchNextGroup(CommonMergeJoinOperator.java:387) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinFinalLeftData(CommonMergeJoinOperator.java:456) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.closeOp(CommonMergeJoinOperator.java:396) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:617) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:287) > ... 15 more > Caused by: java.lang.RuntimeException: java.io.IOException: Please check > if you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:302) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:411) > ... 20 more > Caused by: java.io.IOException: Please check if you are invoking > moveToNext() even after it returned false. > at > org.apache.tez.runtime.library.common.ValuesIterator.hasCompletedProcessing(ValuesIterator.java:221) > at > org.apache.tez.runtime.library.common.ValuesIterator.moveToNext(ValuesIterator.java:103) > at > org.apache.tez.runtime.library.input.OrderedGroupedKVInput$OrderedGroupedKeyValuesReader.next(OrderedGroupedKVInput.java:308) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:260) > ... 21 more > ], TaskAttempt 2 failed, info=[Error: Failure while running > task:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime > Error while closing operators: java.lang.RuntimeException: > java.io.IOException: Please check if you are invoking moveToNext() even > after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.RuntimeException: Hive Runtime Error while closing > operators: java.lang.RuntimeException: java.io.IOException: Please check if > you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:310) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:162) > ... 14 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.RuntimeException: java.io.IOException: Please check if you are > invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:419) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchNextGroup(CommonMergeJoinOperator.java:387) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinFinalLeftData(CommonMergeJoinOperator.java:456) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.closeOp(CommonMergeJoinOperator.java:396) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:617) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:287) > ... 15 more > Caused by: java.lang.RuntimeException: java.io.IOException: Please check > if you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:302) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:411) > ... 20 more > Caused by: java.io.IOException: Please check if you are invoking > moveToNext() even after it returned false. > at > org.apache.tez.runtime.library.common.ValuesIterator.hasCompletedProcessing(ValuesIterator.java:221) > at > org.apache.tez.runtime.library.common.ValuesIterator.moveToNext(ValuesIterator.java:103) > at > org.apache.tez.runtime.library.input.OrderedGroupedKVInput$OrderedGroupedKeyValuesReader.next(OrderedGroupedKVInput.java:308) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:260) > ... 21 more > ], TaskAttempt 3 failed, info=[Error: Failure while running > task:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime > Error while closing operators: java.lang.RuntimeException: > java.io.IOException: Please check if you are invoking moveToNext() even > after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.RuntimeException: Hive Runtime Error while closing > operators: java.lang.RuntimeException: java.io.IOException: Please check if > you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:310) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:162) > ... 14 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.RuntimeException: java.io.IOException: Please check if you are > invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:419) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchNextGroup(CommonMergeJoinOperator.java:387) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinFinalLeftData(CommonMergeJoinOperator.java:456) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.closeOp(CommonMergeJoinOperator.java:396) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:617) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:287) > ... 15 more > Caused by: java.lang.RuntimeException: java.io.IOException: Please check > if you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:302) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:411) > ... 20 more > Caused by: java.io.IOException: Please check if you are invoking > moveToNext() even after it returned false. > at > org.apache.tez.runtime.library.common.ValuesIterator.hasCompletedProcessing(ValuesIterator.java:221) > at > org.apache.tez.runtime.library.common.ValuesIterator.moveToNext(ValuesIterator.java:103) > at > org.apache.tez.runtime.library.input.OrderedGroupedKVInput$OrderedGroupedKeyValuesReader.next(OrderedGroupedKVInput.java:308) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:260) > ... 21 more > ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 > killedTasks:156, Vertex vertex_1460713464534_0363_1_09 [Reducer 21] > killed/failed due to:OWN_TASK_FAILURE] > ERROR : Vertex killed, vertexName=Reducer 8, > vertexId=vertex_1460713464534_0363_1_29, diagnostics=[Vertex received Kill > while in RUNNING state., Vertex did not succeed due to > OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1009, Vertex > vertex_1460713464534_0363_1_29 [Reducer 8] killed/failed due > to:OTHER_VERTEX_FAILURE] > ERROR : Vertex killed, vertexName=Reducer 9, > vertexId=vertex_1460713464534_0363_1_30, diagnostics=[Vertex received Kill > while in RUNNING state., Vertex did not succeed due to > OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1009, Vertex > vertex_1460713464534_0363_1_30 [Reducer 9] killed/failed due > to:OTHER_VERTEX_FAILURE] > ERROR : Vertex killed, vertexName=Reducer 10, > vertexId=vertex_1460713464534_0363_1_31, diagnostics=[Vertex received Kill > while in RUNNING state., Vertex did not succeed due to > OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1009, Vertex > vertex_1460713464534_0363_1_31 [Reducer 10] killed/failed due > to:OTHER_VERTEX_FAILURE] > ERROR : Vertex killed, vertexName=Reducer 5, > vertexId=vertex_1460713464534_0363_1_26, diagnostics=[Vertex received Kill > while in RUNNING state., Vertex did not succeed due to > OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1009, Vertex > vertex_1460713464534_0363_1_26 [Reducer 5] killed/failed due > to:OTHER_VERTEX_FAILURE] > ERROR : Vertex killed, vertexName=Reducer 6, > vertexId=vertex_1460713464534_0363_1_27, diagnostics=[Vertex received Kill > while in RUNNING state., Vertex did not succeed due to > OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1009, Vertex > vertex_1460713464534_0363_1_27 [Reducer 6] killed/failed due > to:OTHER_VERTEX_FAILURE] > ERROR : Vertex killed, vertexName=Reducer 7, > vertexId=vertex_1460713464534_0363_1_28, diagnostics=[Vertex received Kill > while in RUNNING state., Vertex did not succeed due to > OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1009, Vertex > vertex_1460713464534_0363_1_28 [Reducer 7] killed/failed due > to:OTHER_VERTEX_FAILURE] > ERROR : Vertex killed, vertexName=Reducer 4, > vertexId=vertex_1460713464534_0363_1_25, diagnostics=[Vertex received Kill > while in RUNNING state., Vertex did not succeed due to > OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1009, Vertex > vertex_1460713464534_0363_1_25 [Reducer 4] killed/failed due > to:OTHER_VERTEX_FAILURE] > ERROR : DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 > killedVertices:7 > Microsoft.HDInsight.Hive.Data.HiveException (0x80004005): Error while > processing statement: FAILED: Execution Error, return code 2 from > org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, > vertexName=Reducer 21, vertexId=vertex_1460713464534_0363_1_09, > diagnostics=[Task failed, taskId=task_1460713464534_0363_1_09_000232, > diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running > task:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime > Error while closing operators: java.lang.RuntimeException: > java.io.IOException: Please check if you are invoking moveToNext() even > after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.RuntimeException: Hive Runtime Error while closing > operators: java.lang.RuntimeException: java.io.IOException: Please check if > you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:310) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:162) > ... 14 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.RuntimeException: java.io.IOException: Please check if you are > invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:419) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchNextGroup(CommonMergeJoinOperator.java:387) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinFinalLeftData(CommonMergeJoinOperator.java:456) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.closeOp(CommonMergeJoinOperator.java:396) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:617) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:287) > ... 15 more > Caused by: java.lang.RuntimeException: java.io.IOException: Please check > if you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:302) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:411) > ... 20 more > Caused by: java.io.IOException: Please check if you are invoking > moveToNext() even after it returned false. > at > org.apache.tez.runtime.library.common.ValuesIterator.hasCompletedProcessing(ValuesIterator.java:221) > at > org.apache.tez.runtime.library.common.ValuesIterator.moveToNext(ValuesIterator.java:103) > at > org.apache.tez.runtime.library.input.OrderedGroupedKVInput$OrderedGroupedKeyValuesReader.next(OrderedGroupedKVInput.java:308) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:260) > ... 21 more > ], TaskAttempt 1 failed, info=[Error: Failure while running > task:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime > Error while closing operators: java.lang.RuntimeException: > java.io.IOException: Please check if you are invoking moveToNext() even > after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.RuntimeException: Hive Runtime Error while closing > operators: java.lang.RuntimeException: java.io.IOException: Please check if > you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:310) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:162) > ... 14 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.RuntimeException: java.io.IOException: Please check if you are > invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:419) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchNextGroup(CommonMergeJoinOperator.java:387) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinFinalLeftData(CommonMergeJoinOperator.java:456) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.closeOp(CommonMergeJoinOperator.java:396) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:617) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:287) > ... 15 more > Caused by: java.lang.RuntimeException: java.io.IOException: Please check > if you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:302) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:411) > ... 20 more > Caused by: java.io.IOException: Please check if you are invoking > moveToNext() even after it returned false. > at > org.apache.tez.runtime.library.common.ValuesIterator.hasCompletedProcessing(ValuesIterator.java:221) > at > org.apache.tez.runtime.library.common.ValuesIterator.moveToNext(ValuesIterator.java:103) > at > org.apache.tez.runtime.library.input.OrderedGroupedKVInput$OrderedGroupedKeyValuesReader.next(OrderedGroupedKVInput.java:308) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:260) > ... 21 more > ], TaskAttempt 2 failed, info=[Error: Failure while running > task:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime > Error while closing operators: java.lang.RuntimeException: > java.io.IOException: Please check if you are invoking moveToNext() even > after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.RuntimeException: Hive Runtime Error while closing > operators: java.lang.RuntimeException: java.io.IOException: Please check if > you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:310) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:162) > ... 14 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.RuntimeException: java.io.IOException: Please check if you are > invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:419) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchNextGroup(CommonMergeJoinOperator.java:387) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinFinalLeftData(CommonMergeJoinOperator.java:456) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.closeOp(CommonMergeJoinOperator.java:396) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:617) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:287) > ... 15 more > Caused by: java.lang.RuntimeException: java.io.IOException: Please check > if you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:302) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:411) > ... 20 more > Caused by: java.io.IOException: Please check if you are invoking > moveToNext() even after it returned false. > at > org.apache.tez.runtime.library.common.ValuesIterator.hasCompletedProcessing(ValuesIterator.java:221) > at > org.apache.tez.runtime.library.common.ValuesIterator.moveToNext(ValuesIterator.java:103) > at > org.apache.tez.runtime.library.input.OrderedGroupedKVInput$OrderedGroupedKeyValuesReader.next(OrderedGroupedKVInput.java:308) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:260) > ... 21 more > ], TaskAttempt 3 failed, info=[Error: Failure while running > task:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime > Error while closing operators: java.lang.RuntimeException: > java.io.IOException: Please check if you are invoking moveToNext() even > after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.RuntimeException: Hive Runtime Error while closing > operators: java.lang.RuntimeException: java.io.IOException: Please check if > you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:310) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:162) > ... 14 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.RuntimeException: java.io.IOException: Please check if you are > invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:419) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchNextGroup(CommonMergeJoinOperator.java:387) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinFinalLeftData(CommonMergeJoinOperator.java:456) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.closeOp(CommonMergeJoinOperator.java:396) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:617) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:287) > ... 15 more > Caused by: java.lang.RuntimeException: java.io.IOException: Please check > if you are invoking moveToNext() even after it returned false. > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:302) > at > org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:411) > ... 20 more > Caused by: java.io.IOException: Please check if you are invoking > moveToNext() even after it returned false. > at > org.apache.tez.runtime.library.common.ValuesIterator.hasCompletedProcessing(ValuesIterator.java:221) > at > org.apache.tez.runtime.library.common.ValuesIterator.moveToNext(ValuesIterator.java:103) > at > org.apache.tez.runtime.library.input.OrderedGroupedKVInput$OrderedGroupedKeyValuesReader.next(OrderedGroupedKVInput.java:308) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:260) > ... 21 more > ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 > killedTasks:156, Vertex vertex_1460713464534_0363_1_09 [Reducer 21] > killed/failed due to:OWN_TASK_FAILURE]Vertex killed, vertexName=Reducer 8, > vertexId=vertex_1460713464534_0363_1_29, diagnostics=[Vertex received Kill > while in RUNNING state., Vertex did not succeed due to > OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1009, Vertex > vertex_1460713464534_0363_1_29 [Reducer 8] killed/failed due > to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Reducer 9, > vertexId=vertex_1460713464534_0363_1_30, diagnostics=[Vertex received Kill > while in RUNNING state., Vertex did not succeed due to > OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1009, Vertex > vertex_1460713464534_0363_1_30 [Reducer 9] killed/failed due > to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Reducer 10, > vertexId=vertex_1460713464534_0363_1_31, diagnostics=[Vertex received Kill > while in RUNNING state., Vertex did not succeed due to > OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1009, Vertex > vertex_1460713464534_0363_1_31 [Reducer 10] killed/failed due > to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Reducer 5, > vertexId=vertex_1460713464534_0363_1_26, diagnostics=[Vertex received Kill > while in RUNNING state., Vertex did not succeed due to > OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1009, Vertex > vertex_1460713464534_0363_1_26 [Reducer 5] killed/failed due > to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Reducer 6, > vertexId=vertex_1460713464534_0363_1_27, diagnostics=[Vertex received Kill > while in RUNNING state., Vertex did not succeed due to > OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1009, Vertex > vertex_1460713464534_0363_1_27 [Reducer 6] killed/failed due > to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Reducer 7, > vertexId=vertex_1460713464534_0363_1_28, diagnostics=[Vertex received Kill > while in RUNNING state., Vertex did not succeed due to > OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1009, Vertex > vertex_1460713464534_0363_1_28 [Reducer 7] killed/failed due > to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Reducer 4, > vertexId=vertex_1460713464534_0363_1_25, diagnostics=[Vertex received Kill > while in RUNNING state., Vertex did not succeed due to > OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1009, Vertex > vertex_1460713464534_0363_1_25 [Reducer 4] killed/failed due > to:OTHER_VERTEX_FAILURE]DAG did not succeed due to VERTEX_FAILURE. > failedVertices:1 killedVertices:7 > at > Microsoft.HDInsight.Hive.Data.HiveCommand.PrivateEndExecute(IAsyncResult > hiveCommandAsyncResult, Boolean closeCommand) > at > Microsoft.HDInsight.Hive.Data.HiveCommand.EndExecuteNonQuery(IAsyncResult > hiveCommandAsyncResult) > at > Microsoft.WindowsAzure.HDIStudio.UI.ViewModel.ADOSubmissionViewModel.RunCommand(HiveCommand > command) > at > Microsoft.WindowsAzure.HDIStudio.UI.ViewModel.ADOSubmissionViewModel.RunScript() > Errors occurred during execution. Some of the Hive queries in the script > might not be executed. > > > > >
