[
https://issues.apache.org/jira/browse/TEZ-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022893#comment-16022893
]
Kuhu Shukla commented on TEZ-3605:
----------------------------------
The test failure is irreproducible locally and is due to an unexpected state
transition. I ran the same test in a loop and did not see it failing even once.
{code}
2017-05-22 20:58:09,104 ERROR [Dispatcher thread {Central}]
impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(861)) - Can't handle this
event at current state for attempt_1495486688894_0001_1_00_000003_1
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
TA_SUBMITTED at KILL_IN_PROGRESS
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
at
org.apache.tez.dag.app.dag.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:859)
at
org.apache.tez.dag.app.dag.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:124)
at
org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher.handle(DAGAppMaster.java:2299)
at
org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher.handle(DAGAppMaster.java:2284)
at
org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:180)
at org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:115)
at java.lang.Thread.run(Thread.java:745)
{code}
Would appreciate any comments/review on the latest patch. [~sseth]/
[~jeagles]/[~rajesh.balamohan]. Thanks a lot!
> Detect and prune empty partitions for the Ordered case
> ------------------------------------------------------
>
> Key: TEZ-3605
> URL: https://issues.apache.org/jira/browse/TEZ-3605
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Kuhu Shukla
> Assignee: Kuhu Shukla
> Attachments: TEZ-3605.001.patch, TEZ-3605.002.patch,
> TEZ-3605.003.patch, TEZ-3605.004.patch, TEZ-3605.005.patch,
> TEZ-3605.006.patch, TEZ-3605.007.patch, TEZ-3605.008.patch, TEZ-3605.009.patch
>
>
> Analogous to the Unordered case we should not have empty partition
> entries/segments in the Ordered/DefaultSorter case. This will save writing
> unnecessary data.
> Additionally, with tez_shuffle feature (TEZ-3334), in a heavily auto reduced
> job, this change would allow not fetching empty partitions and then throwing
> them away.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
