[
https://issues.apache.org/jira/browse/TEZ-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16614002#comment-16614002
]
Jonathan Eagles commented on TEZ-3990:
--------------------------------------
The penalties queue keeps shuffle retries going, but it is also generates the
AMs signal from the attempts on shuffle health of the system. If we stop
sending the signal from the attempt, we limit the AMs ability to correctly
determine the shuffle health of the system and its ability to decide which
tasks to retry. What can we do instead that keeps the signal going so that the
AM can continue its system health monitoring?
> The number of shuffle penalties for a host/inputAttemptIdentifier should be
> capped
> ----------------------------------------------------------------------------------
>
> Key: TEZ-3990
> URL: https://issues.apache.org/jira/browse/TEZ-3990
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.9.1, 0.10.0
> Reporter: Kuhu Shukla
> Assignee: Kuhu Shukla
> Priority: Major
> Attachments: TEZ-3990.001.patch
>
>
> In a scenario where the same mapId fetches fail, the penalty code allows
> adding the same Host/InputAttemptIdentifier over and over with revised
> penalty time that grows exponentially. It should at some point drop the
> retrying and report failure to the AM asap to allow the job to rectify the
> upstream output.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
