[
https://issues.apache.org/jira/browse/TEZ-2020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated TEZ-2020:
----------------------------------
Attachment: synthetic_job.png
[~bikassaha] - I was testing a synthetic job with patch on local-vm. All
vertices except root had parallelism set to 10.
{code}
2015-02-04 00:22:28,587 INFO [Dispatcher thread: Central] impl.VertexImpl:
Setting vertexManager to InputReadyVertexManager for
vertex_1422270854961_0140_1_04 [Summation1]
...
2015-02-04 00:22:36,447 INFO [Dispatcher thread: Central]
vertexmanager.ShuffleVertexManager: Reduce auto parallelism for vertex:
ordered_vertex to 1 from 10 . Expected output: 0 based on actual output: 0 from
1 vertex manager events. desiredTaskInputSize: 1 max slow start tasks:1.0 num
sources completed:1
...
...
2015-02-04 00:22:36,491 INFO [Dispatcher thread: Central]
vertexmanager.InputReadyVertexManager: Received configured signal from:
ordered_vertex numConfiguredSources: 1 needed: 2
2015-02-04 00:22:36,492 INFO [Dispatcher thread: Central]
vertexmanager.InputReadyVertexManager: Received configured signal from:
unordered_vertex numConfiguredSources: 2 needed: 2
2015-02-04 00:22:36,492 INFO [Dispatcher thread: Central]
vertexmanager.InputReadyVertexManager: Managing 10 tasks for vertex:
Summation1..
..
..
2015-02-04 00:22:36,497 ERROR [Dispatcher thread: Central] impl.VertexImpl:
Exception in VertexManager, vertex:vertex_1422270854961_0140_1_04 [Summation1]
org.apache.tez.dag.app.dag.impl.AMUserCodeException:
org.apache.tez.dag.api.TezUncheckedException: All 1-1 source vertices must have
identical concurrency
at
org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerPluginContextImpl.onStateUpdated(VertexManager.java:304)
at
org.apache.tez.dag.app.dag.StateChangeNotifier$ListenerContainer.sendStateUpdate(StateChangeNotifier.java:138)
at
org.apache.tez.dag.app.dag.StateChangeNotifier$ListenerContainer.access$100(StateChangeNotifier.java:122)
at
org.apache.tez.dag.app.dag.StateChangeNotifier.sendStateUpdate(StateChangeNotifier.java:116)
at
org.apache.tez.dag.app.dag.StateChangeNotifier.stateChanged(StateChangeNotifier.java:106)
at
org.apache.tez.dag.app.dag.impl.VertexImpl.doneReconfiguringVertex(VertexImpl.java:1525)
at
org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerPluginContextImpl.doneReconfiguringVertex(VertexManager.java:287)
at
org.apache.tez.dag.library.vertexmanager.ShuffleVertexManager.schedulePendingTasks(ShuffleVertexManager.java:531)
at
org.apache.tez.dag.library.vertexmanager.ShuffleVertexManager.schedulePendingTasks(ShuffleVertexManager.java:633)
at
org.apache.tez.dag.library.vertexmanager.ShuffleVertexManager.onSourceTaskCompleted(ShuffleVertexManager.java:367)
at
org.apache.tez.dag.app.dag.impl.VertexManager.onSourceTaskCompleted(VertexManager.java:378)
at
org.apache.tez.dag.app.dag.impl.VertexImpl$SourceTaskAttemptCompletedEventTransition.transition(VertexImpl.java:3458)
at
org.apache.tez.dag.app.dag.impl.VertexImpl$SourceTaskAttemptCompletedEventTransition.transition(VertexImpl.java:3439)
at
org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
at
org.apache.tez.state.StateMachineTez.doTransition(StateMachineTez.java:57)
at
org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:1547)
at
org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:181)
at
org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:1768)
at
org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:1754)
at
org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:184)
at org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:116)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.tez.dag.api.TezUncheckedException: All 1-1 source
vertices must have identical concurrency
at
org.apache.tez.dag.library.vertexmanager.InputReadyVertexManager.configure(InputReadyVertexManager.java:104)
at
org.apache.tez.dag.library.vertexmanager.InputReadyVertexManager.onVertexStateUpdated(InputReadyVertexManager.java:175)
at
org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerPluginContextImpl.onStateUpdated(VertexManager.java:301)
... 24 more
{code}
Is this a valid usecase?
> For 1-1 edge vertex configured event may be sent incorrectly
> ------------------------------------------------------------
>
> Key: TEZ-2020
> URL: https://issues.apache.org/jira/browse/TEZ-2020
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: TEZ-2020.1.patch, synthetic_job.png
>
>
> For a fully configured 1-1 vertex configured event will be sent after init
> because the vertex manager does not signal intent to reconfigure. However a
> change in parallelism upstream on the 1-1 edge can cause this vertex's
> parallelism to be updated accordingly. This breaks the guarantee that
> configured should be sent after vertex is fully configured. This is because
> update of parallelism on 1-1 edge has until now been hard-coded in the Vertex
> code. This can be removed now with the InputReadyVertexManager handling the
> 1-1 edge and the addition of vertex state updates. Not only will this fix the
> issue but also remove the special handling for 1-1 edges in the AM which
> should not be there for cleanliness.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
