[
https://issues.apache.org/jira/browse/YARN-6319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15926799#comment-15926799
]
Haibo Chen commented on YARN-6319:
----------------------------------
[~zhiguohong] Sorry for my misunderstanding of the issue. So this is a problem
for both LCE and DefautContainerExecutor?
I suppose the callback only needs to be added to the last container cleanup
task, which immediately requires some special case checking code. In addition,
relying the callback can also lead to the possibility of leaving out
application dir. Say for some reason, the deletion call back is not executed,
then we are gonna have the application folder not cleaned up. Having an
independent and robust deletion task for application folder will serve as a
safety net, should any container dir cleanup fail.
> race condition between deleting app dir and deleting container dir
> ------------------------------------------------------------------
>
> Key: YARN-6319
> URL: https://issues.apache.org/jira/browse/YARN-6319
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Reporter: Hong Zhiguo
> Assignee: Hong Zhiguo
>
> Last container (on one node) of one app complete
> | --> triggers async deletion of container dir (container cleanup)
> | --> triggers async deletion of app dir (app cleanup)
> For LCE, deletion is done by container-executor. The "app cleanup" lists
> sub-dir (step 1), and then unlink items one by one(step 2). If a file is
> deleted by "container cleanup" between step 1 and step2, it'll report below
> error and breaks the deletion.
> {code}
> ContainerExecutor: Couldn't delete file
> $LOCAL/usercache/$USER/appcache/application_1481785469354_353539/container_1481785469354_353539_01_000028/$FILE
> - No such file or directory
> {code}
> This app dir then escape the cleanup. And that's why we always have many app
> dirs left there.
> solution 1: just ignore the error without breaking in
> container-executor.c::delete_path()
> solution 2: use a lock to serialize the cleanup of same app dir.
> solution 3: backoff and retry on error
> Comments are welcome.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
