[
https://issues.apache.org/jira/browse/MAPREDUCE-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13245751#comment-13245751
]
Jason Lowe commented on MAPREDUCE-4099:
---------------------------------------
Initially I thought a quick fix would be to change MRAppMaster so the call to
cleanupStagingDir() occurs before stopping all the services (and therefore
before the RM tries to kill the AM). However this introduces another problem:
if something goes wrong with the AM (killed, crashed, hung) between the time it
has removed the staging directory and before it has notified the RM then the RM
will think the AM did not complete successfully and it will either report the
job as failed (after the AM already told the client it was successful) or
worse, the RM will launch another AM attempt and fail because the staging
directory has been removed.
Seems like we need another RMApp state to track the fact that we've succeeded
but are still cleaning up. For example, when the AM unregisters, we go from
RUNNING into a new FINISHING state. The RM then gives the AM so many seconds
to exit on its own. If the AM container doesn't exit on its own within the
time limit then the RM kills the container, but in either case we move to the
FINISHED state (i.e.: once we're FINISHING, we're going to get to FINISHED one
way or another).
I'm not thrilled with the idea of adding yet another state to app/attempts, but
other alternatives seem to open doors to the AM failing at just the wrong time
and we end up failing the job after the AM has already told the client the job
was successful.
> ApplicationMaster may fail to remove staging directory
> ------------------------------------------------------
>
> Key: MAPREDUCE-4099
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4099
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.2
> Reporter: Jason Lowe
> Priority: Critical
>
> When the ApplicationMaster shuts down it's supposed to remove the staging
> directory, assuming properties weren't set to override this behavior. During
> shutdown the AM tells the ResourceManager that it has finished before it
> cleans up the staging directory. However upon hearing the AM has finished,
> the RM turns right around and kills the AM container. If the AM is too slow,
> the AM will be killed before the staging directory is removed.
> We're seeing the AM lose this race fairly consistently on our clusters, and
> the lack of staging directory cleanup quickly leads to filesystem quota
> issues for some users.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
