[
https://issues.apache.org/jira/browse/TEZ-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14613600#comment-14613600
]
Bikas Saha commented on TEZ-2599:
---------------------------------
bq. edgeManagerOnDemand.routeInputSourceTaskFailedEventToDestination' itself as
the pendingRoute will anyways be null for INPUT_FAILED_EVENT
It may not be null because they can be expanded too (e.g. custom shuffle edge
manager)
Thanks for the review. Committing latest patch that has a test fix. Ran tests
locally since pre-commit seems to be down.
This only adds an if check on the event to check for obsoletion before sending
it.
> Dont send obsoleted data movement events to tasks
> -------------------------------------------------
>
> Key: TEZ-2599
> URL: https://issues.apache.org/jira/browse/TEZ-2599
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: TEZ-2599.1.patch, TEZ-2599.2.patch
>
>
> Since events are sent in bulk and in sequential order to tasks they can end
> up getting a bunch of data movement events in round 1 and then an input
> failed event in round 2. The task may end up scheduling fetches for these
> obsoleted events before round 2, leading to wastage.
> Given that an input failed event and data movement event can be matched via
> their source task attempt ids, the AM can use this match to obsolete the data
> movements events ahead of time and not send them to the tasks in the first
> place.
> The input failed events still need to be sent to the tasks, so that they can
> obsolete any data movements events that they may have received much earlier
> from the failed task attempt.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
