[
https://issues.apache.org/jira/browse/MAPREDUCE-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250387#comment-13250387
]
Siddharth Seth commented on MAPREDUCE-4099:
-------------------------------------------
bq. Initially I thought a quick fix would be to change MRAppMaster so the call
to cleanupStagingDir() occurs before stopping all the services (and therefore
before the RM tries to kill the AM). However this introduces another problem:
if something goes wrong with the AM (killed, crashed, hung) between the time it
has removed the staging directory and before it has notified the RM then the RM
will think the AM did not complete successfully and it will either report the
job as failed (after the AM already told the client it was successful) or
worse, the RM will launch another AM attempt and fail because the staging
directory has been removed.
A similar situation would still exist. During service shutdown, all exceptions
are ignored (logged) - and an attempt is made to shutdown all subsequent
services. After service stop - the AM would proceed and delete the staging
directory.
A really bad case is if there was an error talking to the RM during the
unregister. The client has already been told that the job is successful - the
RM would have no idea, and the AM would eventually delete the staging directory
and exit.
Deleting the staging directory just before the RMCommunicator is stopped would
be a much simpler change. It would however have the same problem in case of a
failed unregister. Subsequent services do not matter.
The HistoryEventHandler had a similar race - where the AM was being shutdown
before the history handler was stopped. That was fixed by ensuring the history
service was registered after the container allocator - so that it shuts down
first.
Handling the situation where the client thinks the job is successful, and the
RM has no idea about the job seems like a separate Jira. Some way to have the
RM pick up the state of the job if and when it restarts the next AM.
> ApplicationMaster may fail to remove staging directory
> ------------------------------------------------------
>
> Key: MAPREDUCE-4099
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4099
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.2
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Critical
> Attachments: MAPREDUCE-4099.patch, MAPREDUCE-4099.patch
>
>
> When the ApplicationMaster shuts down it's supposed to remove the staging
> directory, assuming properties weren't set to override this behavior. During
> shutdown the AM tells the ResourceManager that it has finished before it
> cleans up the staging directory. However upon hearing the AM has finished,
> the RM turns right around and kills the AM container. If the AM is too slow,
> the AM will be killed before the staging directory is removed.
> We're seeing the AM lose this race fairly consistently on our clusters, and
> the lack of staging directory cleanup quickly leads to filesystem quota
> issues for some users.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
