[
https://issues.apache.org/jira/browse/TEZ-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14530259#comment-14530259
]
Jeff Zhang commented on TEZ-2404:
---------------------------------
bq. This does still give us most of the benefits of TEZ-2325, since
TaskComplete events are received once per task - but TASK_STATUS_UPDATES are
received every 100ms / heartbeat-interval - which can amount to a large number
of events for even short running tasks.
+1 on this. Recovery only depend on TaskAttemptFinishedEvent &
DataMovementEvent and require DataMovementEvent logged before
TaskAttemptFinishedEvent. The patch should be able to gurantee
TaskAttemptFinishedEvent is routed to TaskAttempt after the TaskStatusUpdate.
Any other ordering issues in your mind ? [~bikassaha]
> Handle DataMovementEvent before its TaskAttemptCompletedEvent
> -------------------------------------------------------------
>
> Key: TEZ-2404
> URL: https://issues.apache.org/jira/browse/TEZ-2404
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Priority: Critical
> Attachments: TEZ-2404-1.patch, TEZ-2404-2.patch
>
>
> TEZ-2325 route TASK_ATTEMPT_COMPLETED_EVENT directly to the attempt, but it
> would cause recovery issue. Recovery need that DataMovement event is handled
> before TaskAttemptCompletedEvent, otherwise DataMovement event may be lost in
> recovering and cause the its dependent tasks hang.
> 2 Ways to fix this issue.
> 1. Still route TaskAtttemptCompletedEvent in Vertex
> 2. route DataMovementEvent before TaskAttemptCompeltedEvent in
> TezTaskAttemptListener
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
