Bikas Saha created TEZ-2599:
-------------------------------
Summary: Dont send obsoleted data movement events to tasks
Key: TEZ-2599
URL: https://issues.apache.org/jira/browse/TEZ-2599
Project: Apache Tez
Issue Type: Improvement
Reporter: Bikas Saha
Assignee: Bikas Saha
Since events are sent in bulk and in sequential order to tasks they can end up
getting a bunch of data movement events in round 1 and then an input failed
event in round 2. The task may end up scheduling fetches for these obsoleted
events before round 2, leading to wastage.
Given that an input failed event and data movement event can be matched via
their source task attempt ids, the AM can use this match to obsolete the data
movements events ahead of time and not send them to the tasks in the first
place.
The input failed events still need to be sent to the tasks, so that they can
obsolete any data movements events that they may have received much earlier
from the failed task attempt.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
