[
https://issues.apache.org/jira/browse/TEZ-3910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518664#comment-16518664
]
Jonathan Eagles commented on TEZ-3910:
--------------------------------------
Talked offline with [~kshukla]. Trying to capture our conversation here.
Eventually, we will need to refactor the fetch failure retry logic, but that
can wait for another jira.
Couple of minor things.
-TEZ_AM_MAX_DOWNSTREAM_TASK_FAILURES //configuration key name needs to change.
-attempt.getVertex().getTask(failedDestTaId.getTaskID()); //completely
unnecessary line
Design. The sending of isShuffleHealthy is attempting to keep the downstream
task from killing itself and giving that power over to the AM to decide.
Having the AM decide can be good. I'm wondering if we should give that power
over to the upstream task. When the task reaches over to get the states of the
downstream failed attempts, it points to the logic possibly in the incorrect
place.
I worry about hangs of the downstream attempt where where the threshold isn't
met in either upstream or downstream attempt.
I wonder about alternative solutions but have to keep them from assuming
scatter gather solutions. Do we even need a configuration for this or should it
always be one less than the number of max attempts. Could we extend the
isShuffleHealthy to give more knowledge to the AM, or perhaps get ride of
isShuffleHealthy altogether. Also, how do we think about this considering other
shuffle problems (single task vertex hangs where max threshold is never met).
> Single node can cause Tez job to fail during shuffle
> ----------------------------------------------------
>
> Key: TEZ-3910
> URL: https://issues.apache.org/jira/browse/TEZ-3910
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.9.1
> Reporter: Kuhu Shukla
> Assignee: Kuhu Shukla
> Priority: Major
> Attachments: TEZ-3910.001.patch, TEZ-3910.002.patch,
> TEZ-3910.003.patch
>
>
> There is a race where a downstream task that is running into fetch failures
> due to bad output from the upstream task can continue to blame itself for the
> failure before the AM can do a re-run of the upstream offending task and fix
> the fetch failure. This causes the DAG to fail even if a single node fails.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
