Hi there, I’m trying to work with DockerContainerExecutor in Yarn, which is described in https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html
I followed the document and prepared everything, but failed to run even a simple wordcount job with DockerContainerExecutor. The console shows the failure: > Diagnostics: Exception from container-launch: > ExitCodeException exitCode=126: Error: No such image or container: > container_1461317429669_0001_02_000001 > Usage of loopback devices is strongly discouraged for production use. Either > use `--storage-opt dm.thinpooldev` or use `--storage-opt > dm.no_warn_on_loop_devices=true` to suppress this warning. > bash: > /tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1461317429669_0001/container_1461317429669_0001_02_000001/launch_container.sh: > Permission denied > > at org.apache.hadoop.util.Shell.runCommand(Shell.java:545) > at org.apache.hadoop.util.Shell.run(Shell.java:456) > ... > Container exited with a non-zero exit code 126 > Failing this attempt. Failing the application. The log of nodemanager shows: > 2016-04-22 17:31:40,976 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: > Container container_1461317429669_0001_01_000001 transitioned from LOCALIZED > to RUNNING > 2016-04-22 17:31:40,992 DEBUG > org.apache.hadoop.yarn.server.nodemanager.DockerContainerExecutor: > launchContainer: /usr/bin/docker run --rm --net=host --name > container_1461317429669_0001_01_000001 -v > /tmp/hadoop-root/nm-local-dir:/tmp/hadoop-root/nm-local-dir -v > /root/hadoop-2.7.2/logs/userlogs:/root/hadoop-2.7.2/logs/userlogs -v > /tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1461317429669_0001/container_1461317429669_0001_01_000001:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1461317429669_0001/container_1461317429669_0001_01_000001 > sequenceiq/hadoop-docker:2.7.1 bash > /tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1461317429669_0001/container_1461317429669_0001_01_000001/docker_container_executor.sh > 2016-04-22 17:31:43,575 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > Starting resource-monitoring for container_1461317429669_0001_01_000001 > 2016-04-22 17:31:45,832 WARN > org.apache.hadoop.yarn.server.nodemanager.DockerContainerExecutor: Exit code > from container container_1461317429669_0001_01_000001 is : 126 > 2016-04-22 17:31:45,834 WARN > org.apache.hadoop.yarn.server.nodemanager.DockerContainerExecutor: Exception > from container-launch with container ID: > container_1461317429669_0001_01_000001 and exit code: 126 > ExitCodeException exitCode=126: Error: No such image or container: > container_1461317429669_0001_01_000001 > Usage of loopback devices is strongly discouraged for production use. Either > use `--storage-opt dm.thinpooldev` or use `--storage-opt > dm.no_warn_on_loop_devices=true` to suppress this warning. > bash: > /tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1461317429669_0001/container_1461317429669_0001_01_000001/launch_container.sh: > Permission denied > > at org.apache.hadoop.util.Shell.runCommand(Shell.java:545) > at org.apache.hadoop.util.Shell.run(Shell.java:456) > ... > 2016-04-22 17:31:45,836 INFO > org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: > 2016-04-22 17:31:45,837 WARN > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: > Container exited with a non-zero exit code 126 > 2016-04-22 17:31:45,839 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: > Container container_1461317429669_0001_01_000001 transitioned from RUNNING > to EXITED_WITH_FAILURE I tried to run the launching command by manual, but got the same error: Permission denied. But it differs when I append the "--privileged" options. > /usr/bin/docker run --privileged --rm --net=host --name > container_1461317429669_0001_01_000001 -v > /tmp/hadoop-root/nm-local-dir:/tmp/hadoop-root/nm-local-dir -v > /root/hadoop-2.7.2/logs/userlogs:/root/hadoop-2.7.2/logs/userlogs -v > /tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1461317429669_0001/container_1461317429669_0001_01_000001:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1461317429669_0001/container_1461317429669_0001_01_000001 > sequenceiq/hadoop-docker:2.7.1 bash > /tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1461317429669_0001/container_1461317429669_0001_01_000001/docker_container_executor.sh > Usage of loopback devices is strongly discouraged for production use. Either > use `--storage-opt dm.thinpooldev` or use `--storage-opt > dm.no_warn_on_loop_devices=true` to suppress this warning. > /tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1461317429669_0001/container_1461317429669_0001_01_000001/docker_container_executor_session.sh: > line 3: /usr/bin/docker: No such file or directory > /tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1461317429669_0001/container_1461317429669_0001_01_000001/docker_container_executor_session.sh: > line 5: /usr/bin/docker: No such file or directory Why the launching script need to run docker command within a docker container? Is there anybody can show me a way to run jobs with DockerContainerExecutor? Environment: * CentOS 7 * Hadoop 2.7.2 * Docker 1.9.1 * Docker Image: sequenceiq/hadoop-docker:2.7.1 Reproduce steps: * Download & install CentOS7 minimal ISO with single root user: http://isoredirect.centos.org/centos/7/isos/x86_64/CentOS-7-x86_64-Minimal-1511.iso * yum install docker java-1.7.0-openjdk * Download Hadoop 2.7.2: http://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.7.2/hadoop-2.7.2.tar.gz * Prepare configuration files from here: https://onedrive.live.com/redir?resid=F41C57B4F6E1B6C9!1618&authkey=!AEvln9poXczhSX4&ithint=file%2cgz * Setup JAVA_HOME & HADOOP_PREFIX * docker pull sequenceiq/hadoop-docker:2.7.1 * $HADOOP_PREFIX/bin/hdfs namenode -format * $HADOOP_PREFIX/sbin/start-all.sh * $HADOOP_PREFIX/bin/hdfs dfs -mkdir -p /user/root * $HADOOP_PREFIX/bin/hdfs dfs -put $HADOOP_PREFIX/LICENSE.txt * $HADOOP_PREFIX/bin/hadoop jar $HADOOP_PREFIX/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar wordcount -Dmapreduce.map.env="yarn.nodemanager.docker-container-executor.image-name=sequenceiq/hadoop-docker:2.7.1" -Dyarn.app.mapreduce.am.env="yarn.nodemanager.docker-container-executor.image-name=sequenceiq/hadoop-docker:2.7.1" LICENSE.txt wc_out Additional info: Related logs & configuration files & generated shell scripts are placed here: https://onedrive.live.com/redir?resid=F41C57B4F6E1B6C9!1621&authkey=!AGZttYYahRTc0KQ&ithint=folder%2cgz Regards, Shuai Zhang --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
