[
https://issues.apache.org/jira/browse/TEZ-3416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhiyuan Yang updated TEZ-3416:
------------------------------
Description:
This happened in Hive on Tez. The query is select count(\*) from f as a, f as
b. When I manually killed AM, all vertices except the one join the data
finished reconfiguration. After recovery, DAG failed with
ArrayIndexOutOfBoundsException in ScatterGatherEdgeManager of the counting
vertex. It doesn't happen every time.
{code:java}
org.apache.tez.dag.app.dag.impl.AMUserCodeException: Fail to
maybeAddTezEventForDestinationTask,
event:org.apache.tez.runtime.api.events.CompositeDataMovementEvent@5eca05b9,
sourceInfo:{ producerConsumerType=OUTPUT, taskVertexName=Reducer 2,
edgeVertexName=Reducer 3,
taskAttemptId=attempt_1471650900010_0018_1_02_000003_0 }, destinationInfo:null,
EdgeInfo: sourceVertexName=Reducer 2, destinationVertexName=Reducer 3
at
org.apache.tez.dag.app.dag.impl.Edge.maybeAddTezEventForDestinationTask(Edge.java:659)
at
org.apache.tez.dag.app.dag.impl.VertexImpl.getTaskAttemptTezEvents(VertexImpl.java:3720)
at
org.apache.tez.dag.app.TaskCommunicatorManager.heartbeat(TaskCommunicatorManager.java:363)
at
org.apache.tez.dag.app.TaskCommunicatorContextImpl.heartbeat(TaskCommunicatorContextImpl.java:98)
at
org.apache.tez.dag.app.TezTaskCommunicatorImpl$TezTaskUmbilicalProtocolImpl.heartbeat(TezTaskCommunicatorImpl.java:384)
at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:514)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 3
at
org.apache.tez.dag.app.dag.impl.ScatterGatherEdgeManager.routeCompositeDataMovementEventToDestination(ScatterGatherEdgeManager.java:120)
at
org.apache.tez.dag.app.dag.impl.Edge.maybeAddTezEventForDestinationTask(Edge.java:560)
... 15 more
{code}
was:
This happened in Hive on Tez. The query is select count(\*) from f as a, f as
b. When I manually killed AM, all vertices except the one join the data
finished reconfiguration. After recovery, DAG failed with
ArrayIndexOutOfBoundsException in ScatterGatherEdgeManager of the counting
vertex. It doesn't happen every time.
2016-08-22 22:30:54,796 [ERROR] [IPC Server handler 12 on 58819]
|impl.VertexImpl|: Exception in EdgeManager,
vertex=vertex_1471650900010_0018_1_03 [Reducer 3]
org.apache.tez.dag.app.dag.impl.AMUserCodeException: Fail to
maybeAddTezEventForDestinationTask,
event:org.apache.tez.runtime.api.events.CompositeDataMovementEvent@5eca05b9,
sourceInfo:{ producerConsumerType=OUTPUT, taskVertexName=Reducer 2,
edgeVertexName=Reducer 3,
taskAttemptId=attempt_1471650900010_0018_1_02_000003_0 }, destinationInfo:null,
EdgeInfo: sourceVertexName=Reducer 2, destinationVertexName=Reducer 3
at
org.apache.tez.dag.app.dag.impl.Edge.maybeAddTezEventForDestinationTask(Edge.java:659)
at
org.apache.tez.dag.app.dag.impl.VertexImpl.getTaskAttemptTezEvents(VertexImpl.java:3720)
at
org.apache.tez.dag.app.TaskCommunicatorManager.heartbeat(TaskCommunicatorManager.java:363)
at
org.apache.tez.dag.app.TaskCommunicatorContextImpl.heartbeat(TaskCommunicatorContextImpl.java:98)
at
org.apache.tez.dag.app.TezTaskCommunicatorImpl$TezTaskUmbilicalProtocolImpl.heartbeat(TezTaskCommunicatorImpl.java:384)
at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:514)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 3
at
org.apache.tez.dag.app.dag.impl.ScatterGatherEdgeManager.routeCompositeDataMovementEventToDestination(ScatterGatherEdgeManager.java:120)
at
org.apache.tez.dag.app.dag.impl.Edge.maybeAddTezEventForDestinationTask(Edge.java:560)
... 15 more
> ArrayIndexOutOfBoundsException happens in ScatterGatherEdgeManager after DAG
> recovery
> -------------------------------------------------------------------------------------
>
> Key: TEZ-3416
> URL: https://issues.apache.org/jira/browse/TEZ-3416
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Zhiyuan Yang
> Assignee: Zhiyuan Yang
> Attachments: applog
>
>
> This happened in Hive on Tez. The query is select count(\*) from f as a, f as
> b. When I manually killed AM, all vertices except the one join the data
> finished reconfiguration. After recovery, DAG failed with
> ArrayIndexOutOfBoundsException in ScatterGatherEdgeManager of the counting
> vertex. It doesn't happen every time.
> {code:java}
> org.apache.tez.dag.app.dag.impl.AMUserCodeException: Fail to
> maybeAddTezEventForDestinationTask,
> event:org.apache.tez.runtime.api.events.CompositeDataMovementEvent@5eca05b9,
> sourceInfo:{ producerConsumerType=OUTPUT, taskVertexName=Reducer 2,
> edgeVertexName=Reducer 3,
> taskAttemptId=attempt_1471650900010_0018_1_02_000003_0 },
> destinationInfo:null, EdgeInfo: sourceVertexName=Reducer 2,
> destinationVertexName=Reducer 3
> at
> org.apache.tez.dag.app.dag.impl.Edge.maybeAddTezEventForDestinationTask(Edge.java:659)
> at
> org.apache.tez.dag.app.dag.impl.VertexImpl.getTaskAttemptTezEvents(VertexImpl.java:3720)
> at
> org.apache.tez.dag.app.TaskCommunicatorManager.heartbeat(TaskCommunicatorManager.java:363)
> at
> org.apache.tez.dag.app.TaskCommunicatorContextImpl.heartbeat(TaskCommunicatorContextImpl.java:98)
> at
> org.apache.tez.dag.app.TezTaskCommunicatorImpl$TezTaskUmbilicalProtocolImpl.heartbeat(TezTaskCommunicatorImpl.java:384)
> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:514)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 3
> at
> org.apache.tez.dag.app.dag.impl.ScatterGatherEdgeManager.routeCompositeDataMovementEventToDestination(ScatterGatherEdgeManager.java:120)
> at
> org.apache.tez.dag.app.dag.impl.Edge.maybeAddTezEventForDestinationTask(Edge.java:560)
> ... 15 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
