[
https://issues.apache.org/jira/browse/MAPREDUCE-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17534501#comment-17534501
]
Chris Nauroth commented on MAPREDUCE-7369:
------------------------------------------
Hello [~ste...@apache.org] and [~groot]. I wanted to call out a potential
unintended side effect in behavior resulting from this change.
Right now, "liveness" inherently demands some sort of forward progress by the
task, like continuous processing of input records or manual interaction with
the {{Progressable}} interface if the job has been custom coded to do so. It is
not sufficient that the OS process is merely alive and sending pings. That
means that in some gray failure scenarios, where the process is running and
pinging but not making meaningful progress (e.g. horrendously degraded but not
completely failed disk), the framework gets to detect that, kill the task, and
schedule a reattempt elsewhere. Speculative execution helps with this too, but
if users have disabled speculative execution for some reason, then this is an
important protection mechanism.
I understand what motivates this change: slow large rename operations against
cloud storage that don't appear to signal meaningful progress. However, if we
equate ping with liveness, then I think we also lose the resilience described
above. For that reason, I would suggest putting the new behavior behind a
configuration flag, something like the 001 patch file attached to the issue
here rather than the current GitHub PR.
Ideally, we'd get these long renames interacting with the {{Progressable}}
somehow, but that's a much bigger topic.
Thanks!
> MapReduce tasks timing out when spends more time on MultipleOutputs#close
> -------------------------------------------------------------------------
>
> Key: MAPREDUCE-7369
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7369
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 3.3.1
> Reporter: Prabhu Joseph
> Assignee: Ashutosh Gupta
> Priority: Major
> Labels: pull-request-available
> Attachments: MAPREDUCE-7369.001.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> MapReduce tasks timing out when spends more time on MultipleOutputs#close.
> MultipleOutputs#closes takes more time when there are multiple files to be
> closed & there is a high latency in closing a stream.
> {code}
> 2021-11-01 02:45:08,312 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics
> report from attempt_1634949471086_61268_m_001115_0:
> AttemptID:attempt_1634949471086_61268_m_001115_0 Timed out after 300 secs
> {code}
> MapReduce task timeout can be increased but it is tough to set the right
> timeout value. The timeout can be disabled with 0 but that might lead to
> hanging tasks not getting killed.
> The tasks are sending the ping every 3 seconds which are not honored by
> ApplicationMaster. It expects the status information which won't be send
> during MultipleOutputs#close. This jira is to add a config which considers
> the ping from task as part of Task Liveliness Check in the ApplicationMaster.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
