[
https://issues.apache.org/jira/browse/TEZ-3462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Badger updated TEZ-3462:
-----------------------------
Attachment: TEZ-3462.001.patch
Attaching a patch that only communicates the task failure to the AM if the
container is not currently shutting down. This is the same as what
MAPREDUCE-6002 does.
The unit test creates a task that will throw an IOException. It overrides the
{{isShuttingDown()}} method that was added and forces it to always return true.
Then it checks to make sure that no {{TaskAttemptFailedEvent}} or
{{TaskAttemptCompletedEvent}} events were sent over the umbilical. The test
passes consistently with {{isShuttingDown()}} set to true and fails
consistently with {{isShuttingDown()}} set to false.
[~jlowe], could you review this? Thanks!
> Task attempt failure during container shutdown loses useful container
> diagnostics
> ---------------------------------------------------------------------------------
>
> Key: TEZ-3462
> URL: https://issues.apache.org/jira/browse/TEZ-3462
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: Jason Lowe
> Assignee: Eric Badger
> Attachments: TEZ-3462.001.patch
>
>
> When a nodemanager kills a task attempt due to excessive memory usage it will
> send a SIGTERM followed by a SIGKILL. It also sends a useful diagnostic
> message with the container completion event to the RM which will eventually
> make it to the AM on a subsequent heartbeat.
> However if the JVM shutdown processing causes an error in the task (e.g.:
> filesystem being closed by shutdown hook) then the task attempt can report a
> failure before the useful NM diagnostic makes it to the AM. The AM then
> records some other error as the task failure reason, and by the time the
> container completion status makes it to the AM it does not associate that
> error with the task attempt and the useful information is lost.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
