[
https://issues.apache.org/jira/browse/YARN-8485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated YARN-8485:
----------------------------
Attachment: YARN-8485.002.patch
> Priviledged container app launch is failing intermittently
> ----------------------------------------------------------
>
> Key: YARN-8485
> URL: https://issues.apache.org/jira/browse/YARN-8485
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-native-services
> Environment: Debian
> Reporter: Yesha Vora
> Assignee: Eric Yang
> Priority: Major
> Attachments: YARN-8485.001.patch, YARN-8485.002.patch
>
>
> Privileged application fails intermittently
> {code:java}
> yarn jar
> /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell-*.jar
> -shell_command "sleep 30" -num_containers 1 -shell_env
> YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env
> YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=xxx -shell_env
> YARN_CONTAINER_RUNTIME_DOCKER_RUN_PRIVILEGED_CONTAINER=true -jar
> /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell-*.jar{code}
> Here, container launch fails with 'Privileged containers are disabled' even
> though Docker privilege container is enabled in the cluster
> {code:java|title=nm log}
> 2018-06-28 21:21:15,647 INFO runtime.DockerLinuxContainerRuntime
> (DockerLinuxContainerRuntime.java:allowPrivilegedContainerExecution(664)) -
> All checks pass. Launching privileged container for :
> container_e01_1530220647587_0001_01_000002
> 2018-06-28 21:21:15,665 WARN nodemanager.LinuxContainerExecutor
> (LinuxContainerExecutor.java:handleExitCode(593)) - Exit code from container
> container_e01_1530220647587_0001_01_000002 is : 29
> 2018-06-28 21:21:15,666 WARN nodemanager.LinuxContainerExecutor
> (LinuxContainerExecutor.java:handleExitCode(599)) - Exception from
> container-launch with container ID:
> container_e01_1530220647587_0001_01_000002 and exit code: 29
> org.apache.hadoop.yarn.server.nodemanager.containermanager.runtime.ContainerExecutionException:
> Launch container failed
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(DockerLinuxContainerRuntime.java:958)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime.launchContainer(DelegatingLinuxContainerRuntime.java:141)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.handleLaunchForLaunchType(LinuxContainerExecutor.java:564)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:479)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:494)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:306)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:103)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor
> (ContainerExecutor.java:logOutput(541)) - Exception from container-launch.
> 2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor
> (ContainerExecutor.java:logOutput(541)) - Container id:
> container_e01_1530220647587_0001_01_000002
> 2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor
> (ContainerExecutor.java:logOutput(541)) - Exit code: 29
> 2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor
> (ContainerExecutor.java:logOutput(541)) - Exception message: Launch container
> failed
> 2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor
> (ContainerExecutor.java:logOutput(541)) - Shell error output: check
> privileges failed for user: hrt_qa, error code: 0
> 2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor
> (ContainerExecutor.java:logOutput(541)) - Privileged containers are disabled
> for user: hrt_qa
> 2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor
> (ContainerExecutor.java:logOutput(541)) - Error constructing docker command,
> docker error code=11, error message='Privileged containers are disabled'
> 2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor
> (ContainerExecutor.java:logOutput(541)) -
> 2018-06-28 21:21:15,669 INFO nodemanager.ContainerExecutor
> (ContainerExecutor.java:logOutput(541)) - Shell output: main : command
> provided 4
> 2018-06-28 21:21:15,669 INFO nodemanager.ContainerExecutor
> (ContainerExecutor.java:logOutput(541)) - main : run as user is hrt_qa
> 2018-06-28 21:21:15,669 INFO nodemanager.ContainerExecutor
> (ContainerExecutor.java:logOutput(541)) - main : requested yarn user is hrt_qa
> 2018-06-28 21:21:15,669 INFO nodemanager.ContainerExecutor
> (ContainerExecutor.java:logOutput(541)) - Creating script paths...
> 2018-06-28 21:21:15,669 INFO nodemanager.ContainerExecutor
> (ContainerExecutor.java:logOutput(541)) - Creating local dirs...
> 2018-06-28 21:21:15,669 INFO nodemanager.ContainerExecutor
> (ContainerExecutor.java:logOutput(541)) - Creating script paths...
> 2018-06-28 21:21:15,669 INFO nodemanager.ContainerExecutor
> (ContainerExecutor.java:logOutput(541)) - Creating local dirs...
> 2018-06-28 21:21:15,693 WARN launcher.ContainerLaunch
> (ContainerLaunch.java:handleContainerExitWithFailure(598)) - Container launch
> failed : Container exited with a non-zero exit code 29.
> 2018-06-28 21:21:15,693 ERROR launcher.ContainerLaunch
> (ContainerLaunch.java:handleContainerExitWithFailure(623)) - Failed to get
> tail of the container's prelaunch error log file
> java.io.FileNotFoundException: File
> /grid/0/hadoop/yarn/log/application_1530220647587_0001/container_e01_1530220647587_0001_01_000002/prelaunch.err
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:641)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:930)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.handleContainerExitWithFailure(ContainerLaunch.java:609)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.handleContainerExitCode(ContainerLaunch.java:575)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:340)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:103)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2018-06-28 21:21:15,704 INFO container.ContainerImpl
> (ContainerImpl.java:handle(2093)) - Container
> container_e01_1530220647587_0001_01_000002 transitioned from RUNNING to
> EXITED_WITH_FAILURE{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
