[
https://issues.apache.org/jira/browse/TEZ-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bikas Saha updated TEZ-2599:
----------------------------
Attachment: TEZ-2599.1.patch
[~hitesh] [~rajesh.balamohan] [~pramachandran] Please review. Tests added.
> Dont send obsoleted data movement events to tasks
> -------------------------------------------------
>
> Key: TEZ-2599
> URL: https://issues.apache.org/jira/browse/TEZ-2599
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: TEZ-2599.1.patch
>
>
> Since events are sent in bulk and in sequential order to tasks they can end
> up getting a bunch of data movement events in round 1 and then an input
> failed event in round 2. The task may end up scheduling fetches for these
> obsoleted events before round 2, leading to wastage.
> Given that an input failed event and data movement event can be matched via
> their source task attempt ids, the AM can use this match to obsolete the data
> movements events ahead of time and not send them to the tasks in the first
> place.
> The input failed events still need to be sent to the tasks, so that they can
> obsolete any data movements events that they may have received much earlier
> from the failed task attempt.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
