[
https://issues.apache.org/jira/browse/YARN-9647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16883948#comment-16883948
]
Eric Badger commented on YARN-9647:
-----------------------------------
Hi [~magnum], thanks for the patch.
Could you explain the failure you're seeing a little bit more? The only way I
can see this specific scenario happening is if the disk goes bad between the
time when the Java code grabs the good local/log dirs and when the
container-executor actually goes to validate the mounts. That seems like a
pretty small race, and if the disk has gone bad in that time, the container
should fail. There could be distributed cache files on the disk that went bad.
So if we just skip mounting that disk, then we could fail anyway from not
having those distributed cache files. Additionally, the java code that will be
launched inside of the container already has a list of local and log dirs that
it can use. If we blindly ignore mounting those directories then the java code
could try to write to them and fail because the mounts don't exist. Both of
these failures would probably look weird and confusing and I think failing the
container upright is a much better decision than hoping that the container
somehow succeeds.
> Docker launch fails when local-dirs or log-dirs is unhealthy.
> -------------------------------------------------------------
>
> Key: YARN-9647
> URL: https://issues.apache.org/jira/browse/YARN-9647
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 3.1.2
> Reporter: KWON BYUNGCHANG
> Priority: Major
> Attachments: YARN-9647.001.patch, YARN-9647.002.patch
>
>
> my /etc/hadoop/conf/container-executor.cfg
> {code}
> [docker]
> docker.allowed.ro-mounts=/data1/hadoop/yarn/local,/data2/hadoop/yarn/local
> docker.allowed.rw-mounts=/data1/hadoop/yarn/local,/data2/hadoop/yarn/local
> {code}
> if /data2 is unhealthy, docker launch fails although container can use
> /data1 as local-dir, log-dir
> error message is below
> {code}
> [2019-06-25 14:55:26.168]Exception from container-launch. Container id:
> container_e50_1561100493387_5185_01_000597 Exit code: 29 Exception message:
> Launch container failed Shell error output: Could not determine real path of
> mount '/data2/hadoop/yarn/local' Could not determine real path of mount
> '/data2/hadoop/yarn/local' Unable to find permitted docker mounts on disk
> Error constructing docker command, docker error code=16, error message='Mount
> access error' Shell output: main : command provided 4 main : run as user is
> magnum main : requested yarn user is magnum Creating script paths... Creating
> local dirs... [2019-06-25 14:55:26.189]Container exited with a non-zero exit
> code 29. [2019-06-25 14:55:26.192]Container exited with a non-zero exit code
> 29.
> {code}
> root cause is that normalize_mounts() in docker-util.c return -1 because it
> cannot resolve real path of /data2/hadoop/yarn/local.(note that /data2 is
> disk fault at this point)
> however disk of nm local dirs and nm log dirs can fail at any time.
> docker launch should succeed if there are available local dirs and log dirs.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
