[
https://issues.apache.org/jira/browse/YARN-8064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16444931#comment-16444931
]
Eric Yang commented on YARN-8064:
---------------------------------
[~ebadger] I am getting errors when trying out the patch 008:
{code}
Invalid conf file provided, unable to open file :
/tmp/hadoop-yarn/nm-local-dir/nmPrivate/application_1524177007026_0001/container_1524177007026_0001_01_000003/docker.container_1524177007026_0001_01_0000031851872809113698536.cmd
Error constructing docker command, docker error code=1, error message='Invalid
command file passed'
Stdout: main : command provided 4
main : run as user is hbase
main : requested yarn user is hbase
Creating script paths...
Creating local dirs...
Full command array for failed execution:
[/usr/local/hadoop-3.2.0-SNAPSHOT/bin/container-executor, hbase, hbase, 4,
application_1524177007026_0001, container_1524177007026_0001_01_000003,
/tmp/hadoop-yarn/nm-local-dir/usercache/hbase/appcache/application_1524177007026_0001/container_1524177007026_0001_01_000003,
/tmp/hadoop-yarn/nm-local-dir/nmPrivate/application_1524177007026_0001/container_1524177007026_0001_01_000003/launch_container.sh,
/tmp/hadoop-yarn/nm-local-dir/nmPrivate/application_1524177007026_0001/container_1524177007026_0001_01_000003/container_1524177007026_0001_01_000003.tokens,
/tmp/hadoop-yarn/nm-local-dir/nmPrivate/application_1524177007026_0001/container_1524177007026_0001_01_000003/container_1524177007026_0001_01_000003.pid,
/tmp/hadoop-yarn/nm-local-dir, /usr/local/hadoop-3.2.0-SNAPSHOT/logs/userlogs,
/tmp/hadoop-yarn/nm-local-dir/nmPrivate/application_1524177007026_0001/container_1524177007026_0001_01_000003/docker.container_1524177007026_0001_01_0000031851872809113698536.cmd,
cgroups=none]
2018-04-19 22:31:31,093 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime:
Launch container failed. Exception:
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException:
ExitCodeException exitCode=29: Invalid conf file provided, unable to open file
:
/tmp/hadoop-yarn/nm-local-dir/nmPrivate/application_1524177007026_0001/container_1524177007026_0001_01_000003/docker.container_1524177007026_0001_01_0000031851872809113698536.cmd
Error constructing docker command, docker error code=1, error message='Invalid
command file passed'
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:180)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(DockerLinuxContainerRuntime.java:910)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime.launchContainer(DelegatingLinuxContainerRuntime.java:141)
at
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.handleLaunchForLaunchType(LinuxContainerExecutor.java:564)
at
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:479)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:492)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:304)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: ExitCodeException exitCode=29: Invalid conf file provided, unable to
open file :
/tmp/hadoop-yarn/nm-local-dir/nmPrivate/application_1524177007026_0001/container_1524177007026_0001_01_000003/docker.container_1524177007026_0001_01_0000031851872809113698536.cmd
Error constructing docker command, docker error code=1, error message='Invalid
command file passed'
at org.apache.hadoop.util.Shell.runCommand(Shell.java:1009)
at org.apache.hadoop.util.Shell.run(Shell.java:902)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1227)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:152)
... 11 more
{code}
After the first occurrence of logged exception happens, I don't see cmd files
created for other containers.
> Docker ".cmd" files should not be put in hadoop.tmp.dir
> -------------------------------------------------------
>
> Key: YARN-8064
> URL: https://issues.apache.org/jira/browse/YARN-8064
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Eric Badger
> Assignee: Eric Badger
> Priority: Critical
> Attachments: YARN-8064.001.patch, YARN-8064.002.patch,
> YARN-8064.003.patch, YARN-8064.004.patch, YARN-8064.005.patch,
> YARN-8064.006.patch, YARN-8064.007.patch, YARN-8064.008.patch
>
>
> Currently all of the docker command files are being put into
> {{hadoop.tmp.dir}}, which doesn't get cleaned up. So, eventually all of the
> inodes will fill up and no more tasks will be able to run
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
