[
https://issues.apache.org/jira/browse/TEZ-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated TEZ-4356:
------------------------------
Description:
with HIVE-24207, we can be in a situation where there are input related events
to process when the task is already CLOSED
in this case, we simply don't care about certain kinds of exceptions and should
ignore them in order to avoid task failure
thinking about whether it is unsafe to ignore these exceptions...I'm assuming
that processing events for a closed task is just a best-effort behavior and we
never rely on that
{code}
java.lang.RuntimeException: java.io.IOException: java.io.IOException:
java.io.IOException: Failed on local exception:
java.nio.channels.ClosedByInterruptException; Host Details : local host is:
"lbodor-MBP16.local/192.168.0.53"; destination host is: "localhost":54688;
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:200)
~[tez-mapreduce-0.10.1.jar:0.10.1]
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.<init>(TezGroupedSplitsInputFormat.java:139)
~[tez-mapreduce-0.10.1.jar:0.10.1]
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:105)
~[tez-mapreduce-0.10.1.jar:0.10.1]
at
org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:164)
~[tez-mapreduce-0.10.1.jar:0.10.1]
at
org.apache.tez.mapreduce.lib.MRReaderMapred.<init>(MRReaderMapred.java:76)
~[tez-mapreduce-0.10.1.jar:0.10.1]
at
org.apache.tez.mapreduce.input.MultiMRInput.initFromEvent(MultiMRInput.java:196)
~[tez-mapreduce-0.10.1.jar:0.10.1]
at
org.apache.tez.mapreduce.input.MultiMRInput.handleEvents(MultiMRInput.java:154)
~[tez-mapreduce-0.10.1.jar:0.10.1]
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.handleEvent(LogicalIOProcessorRuntimeTask.java:729)
[tez-runtime-internals-0.10.1.jar:0.10.1]
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.access$600(LogicalIOProcessorRuntimeTask.java:110)
[tez-runtime-internals-0.10.1.jar:0.10.1]
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$1.runInternal(LogicalIOProcessorRuntimeTask.java:817)
[tez-runtime-internals-0.10.1.jar:0.10.1]
at org.apache.tez.common.RunnableWithNdc.run(RunnableWithNdc.java:35)
[tez-common-0.10.1.jar:0.10.1]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
Caused by: java.io.IOException: java.io.IOException: java.io.IOException:
Failed on local exception: java.nio.channels.ClosedByInterruptException; Host
Details : local host is: "lbodor-MBP16.local/192.168.0.53"; destination host
is: "localhost":54688;
at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
~[hive-shims-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
~[hive-shims-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:449)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:197)
~[tez-mapreduce-0.10.1.jar:0.10.1]
... 11 more
Caused by: java.io.IOException: java.io.IOException: Failed on local exception:
java.nio.channels.ClosedByInterruptException; Host Details : local host is:
"lbodor-MBP16.local/192.168.0.53"; destination host is: "localhost":54688;
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat.getRecordReader(LlapInputFormat.java:141)
~[hive-llap-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.io.RecordReaderWrapper.create(RecordReaderWrapper.java:72)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:446)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:197)
~[tez-mapreduce-0.10.1.jar:0.10.1]
... 11 more
{code}
was:
with HIVE-24207, we can be in a situation where there are input related events
to process when the task is already CLOSED
in this case, we simply don't care about certain kinds of exceptions, and
should ignore them in order to avoid task failure
{code}
java.lang.RuntimeException: java.io.IOException: java.io.IOException:
java.io.IOException: Failed on local exception:
java.nio.channels.ClosedByInterruptException; Host Details : local host is:
"lbodor-MBP16.local/192.168.0.53"; destination host is: "localhost":54688;
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:200)
~[tez-mapreduce-0.10.1.jar:0.10.1]
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.<init>(TezGroupedSplitsInputFormat.java:139)
~[tez-mapreduce-0.10.1.jar:0.10.1]
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:105)
~[tez-mapreduce-0.10.1.jar:0.10.1]
at
org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:164)
~[tez-mapreduce-0.10.1.jar:0.10.1]
at
org.apache.tez.mapreduce.lib.MRReaderMapred.<init>(MRReaderMapred.java:76)
~[tez-mapreduce-0.10.1.jar:0.10.1]
at
org.apache.tez.mapreduce.input.MultiMRInput.initFromEvent(MultiMRInput.java:196)
~[tez-mapreduce-0.10.1.jar:0.10.1]
at
org.apache.tez.mapreduce.input.MultiMRInput.handleEvents(MultiMRInput.java:154)
~[tez-mapreduce-0.10.1.jar:0.10.1]
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.handleEvent(LogicalIOProcessorRuntimeTask.java:729)
[tez-runtime-internals-0.10.1.jar:0.10.1]
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.access$600(LogicalIOProcessorRuntimeTask.java:110)
[tez-runtime-internals-0.10.1.jar:0.10.1]
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$1.runInternal(LogicalIOProcessorRuntimeTask.java:817)
[tez-runtime-internals-0.10.1.jar:0.10.1]
at org.apache.tez.common.RunnableWithNdc.run(RunnableWithNdc.java:35)
[tez-common-0.10.1.jar:0.10.1]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
Caused by: java.io.IOException: java.io.IOException: java.io.IOException:
Failed on local exception: java.nio.channels.ClosedByInterruptException; Host
Details : local host is: "lbodor-MBP16.local/192.168.0.53"; destination host
is: "localhost":54688;
at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
~[hive-shims-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
~[hive-shims-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:449)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:197)
~[tez-mapreduce-0.10.1.jar:0.10.1]
... 11 more
Caused by: java.io.IOException: java.io.IOException: Failed on local exception:
java.nio.channels.ClosedByInterruptException; Host Details : local host is:
"lbodor-MBP16.local/192.168.0.53"; destination host is: "localhost":54688;
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat.getRecordReader(LlapInputFormat.java:141)
~[hive-llap-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.io.RecordReaderWrapper.create(RecordReaderWrapper.java:72)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:446)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:197)
~[tez-mapreduce-0.10.1.jar:0.10.1]
... 11 more
{code}
> Ignore some exceptions when the task is already in CLOSED state
> ---------------------------------------------------------------
>
> Key: TEZ-4356
> URL: https://issues.apache.org/jira/browse/TEZ-4356
> Project: Apache Tez
> Issue Type: Bug
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
>
> with HIVE-24207, we can be in a situation where there are input related
> events to process when the task is already CLOSED
> in this case, we simply don't care about certain kinds of exceptions and
> should ignore them in order to avoid task failure
> thinking about whether it is unsafe to ignore these exceptions...I'm assuming
> that processing events for a closed task is just a best-effort behavior and
> we never rely on that
> {code}
> java.lang.RuntimeException: java.io.IOException: java.io.IOException:
> java.io.IOException: Failed on local exception:
> java.nio.channels.ClosedByInterruptException; Host Details : local host is:
> "lbodor-MBP16.local/192.168.0.53"; destination host is: "localhost":54688;
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:200)
> ~[tez-mapreduce-0.10.1.jar:0.10.1]
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.<init>(TezGroupedSplitsInputFormat.java:139)
> ~[tez-mapreduce-0.10.1.jar:0.10.1]
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:105)
> ~[tez-mapreduce-0.10.1.jar:0.10.1]
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:164)
> ~[tez-mapreduce-0.10.1.jar:0.10.1]
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.<init>(MRReaderMapred.java:76)
> ~[tez-mapreduce-0.10.1.jar:0.10.1]
> at
> org.apache.tez.mapreduce.input.MultiMRInput.initFromEvent(MultiMRInput.java:196)
> ~[tez-mapreduce-0.10.1.jar:0.10.1]
> at
> org.apache.tez.mapreduce.input.MultiMRInput.handleEvents(MultiMRInput.java:154)
> ~[tez-mapreduce-0.10.1.jar:0.10.1]
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.handleEvent(LogicalIOProcessorRuntimeTask.java:729)
> [tez-runtime-internals-0.10.1.jar:0.10.1]
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.access$600(LogicalIOProcessorRuntimeTask.java:110)
> [tez-runtime-internals-0.10.1.jar:0.10.1]
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$1.runInternal(LogicalIOProcessorRuntimeTask.java:817)
> [tez-runtime-internals-0.10.1.jar:0.10.1]
> at org.apache.tez.common.RunnableWithNdc.run(RunnableWithNdc.java:35)
> [tez-common-0.10.1.jar:0.10.1]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
> Caused by: java.io.IOException: java.io.IOException: java.io.IOException:
> Failed on local exception: java.nio.channels.ClosedByInterruptException; Host
> Details : local host is: "lbodor-MBP16.local/192.168.0.53"; destination host
> is: "localhost":54688;
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
> ~[hive-shims-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
> ~[hive-shims-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:449)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:197)
> ~[tez-mapreduce-0.10.1.jar:0.10.1]
> ... 11 more
> Caused by: java.io.IOException: java.io.IOException: Failed on local
> exception: java.nio.channels.ClosedByInterruptException; Host Details : local
> host is: "lbodor-MBP16.local/192.168.0.53"; destination host is:
> "localhost":54688;
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat.getRecordReader(LlapInputFormat.java:141)
> ~[hive-llap-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.RecordReaderWrapper.create(RecordReaderWrapper.java:72)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:446)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:197)
> ~[tez-mapreduce-0.10.1.jar:0.10.1]
> ... 11 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
