[
https://issues.apache.org/jira/browse/YARN-8181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16444862#comment-16444862
]
Seyyed Ahmad Javadi commented on YARN-8181:
-------------------------------------------
{color:#333333}Thank you so much Shane,{color}
{color:#333333}I did what you have suggested and now I am getting another
error, I could not solve it, thus really appropriated if you give your
suggestion on that as well?{color}
{color:#333333}Is this fine that my $HADOOP_HOME is /home/ubuntu/hadoop-3.1.0 ?
{color}
{color:#333333}I am using use ubuntu and did create new user/group for hadoop
installation purpose. {color}
{color:#333333}container-executor.cfg{color}
{code:java}
yarn.nodemanager.linux-container-executor.group=ubuntu
min.user.id=0
#feature.tc.enabled=1
#feature.docker.enabled=1
allowed.system.users=ubuntu
# The configs below deal with settings for Docker
[docker]
module.enabled=true
docker.privileged-containers.enabled=true
docker.binary=/usr/bin/docker
docker.allowed.capabilities=SYS_CHROOT,MKNOD,SETFCAP,SETPCAP,FSETID,CHOWN,AUDIT_WRITE,SETGID,NET_RAW,FOWNER,SETUID,DAC_OVERRIDE,KILL,NET_BIND_SERVICE
# docker.allowed.devices=## comma seperated list of devices that can be
mounted into a container
docker.allowed.networks=bridge,host,none
docker.allowed.ro-mounts=/sys/fs/cgroup
docker.privileged-containers.registries=local
#docker.host-pid-namespace.enabled=false
docker.allowed.rw-mounts=/home/ubuntu/hadoop-3.1.0,/home/ubuntu/hadoop-3.1.0/logs
#docker.privileged-containers.enabled=true
#docker.allowed.volume-drivers=## comma seperated list of allowed volume-drivers
# The configs below deal with settings for FPGA resource
#[fpga]
# module.enabled=## Enable/Disable the FPGA resource handler module. set to
"true" to enable, disabled by default
# fpga.major-device-number=## Major device number of FPGA, by default is 246.
Strongly recommend setting this
# fpga.allowed-device-minor-numbers=## Comma separated allowed minor device
numbers, empty means all FPGA devices managed by YARN.
#[docker]
# module.enabled=true
# docker.privileged-containers.enabled=true
# docker.privileged-containers.registries=centos
#
docker.allowed.capabilities=SYS_CHROOT,MKNOD,SETFCAP,SETPCAP,FSETID,CHOWN,AUDIT_WRITE,SETGID,NET_RAW,FOWNER,SETUID,DAC_OVERRIDE,KILL,NET_BIND_SERVICE
# docker.allowed.networks=bridge,host,none
# docker.allowed.ro-mounts=/sys/fs/cgroup
# docker.allowed.rw-mounts=/var/hadoop/yarn/local-dir,/var/hadoop/yarn/log-dir
~
{code}
{code:java}
.. ~/hadoop-3.1.0/etc/hadoop$ docker images
REPOSITORY TAG IMAGE ID CREATED
SIZE
local/hadoop-ubuntu latest d8335693084b 7 hours ago
2.06GB
hadoop-ubuntu latest d8335693084b 7 hours ago
2.06GB
ubuntu 16.04 c9d990395902 7 days ago
113MB
{code}
{code:java}
2018-04-19 17:29:09,413 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth
successful for appattempt_1524172890453_0002_000002 (auth:SIMPLE)
2018-04-19 17:29:09,425 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
Start request for container_1524172890453_0002_02_000001 by user ubuntu
2018-04-19 17:29:09,448 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
Creating a new application reference for app application_1524172890453_0002
2018-04-19 17:29:09,449 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
Application application_1524172890453_0002 transitioned from NEW to INITING
2018-04-19 17:29:09,450 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
Adding container_1524172890453_0002_02_000001 to application
application_1524172890453_0002
2018-04-19 17:29:09,450 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
Application application_1524172890453_0002 transitioned from INITING to RUNNING
2018-04-19 17:29:09,451 INFO
org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=ubuntu
IP=130.245.127.176 OPERATION=Start Container Request
TARGET=ContainerManageImpl RESULT=SUCCESS
APPID=application_1524172890453_0002
CONTAINERID=container_1524172890453_0002_02_000001
2018-04-19 17:29:09,454 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
Container container_1524172890453_0002_02_000001 transitioned from NEW to
LOCALIZING
2018-04-19 17:29:09,454 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
event CONTAINER_INIT for appId application_1524172890453_0002
2018-04-19 17:29:09,455 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
Created localizer for container_1524172890453_0002_02_000001
2018-04-19 17:29:09,472 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
Writing credentials to the nmPrivate file
/tmp/hadoop-ubuntu/nm-local-dir/nmPrivate/container_1524172890453_0002_02_000001.tokens
2018-04-19 17:29:12,875 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
Container container_1524172890453_0002_02_000001 transitioned from LOCALIZING
to SCHEDULED
2018-04-19 17:29:12,875 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler.ContainerScheduler:
Starting container [container_1524172890453_0002_02_000001]
2018-04-19 17:29:12,978 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
Container container_1524172890453_0002_02_000001 transitioned from SCHEDULED
to RUNNING
2018-04-19 17:29:12,978 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Starting resource-monitoring for container_1524172890453_0002_02_000001
2018-04-19 17:29:13,032 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor:
Shell execution returned exit code: 29. Privileged Execution Operation Stderr:
Invalid docker ro mount
'/tmp/hadoop-ubuntu/nm-local-dir/filecache:/tmp/hadoop-ubuntu/nm-local-dir/filecache',
realpath=/tmp/hadoop-ubuntu/nm-local-dir/filecache
Error constructing docker command, docker error code=13, error message='Invalid
docker read-only mount'
Stdout: main : command provided 4
main : run as user is ubuntu
main : requested yarn user is ubuntu
Creating script paths...
Creating local dirs...
Full command array for failed execution:
[/home/ubuntu/hadoop-3.1.0/bin/container-executor, ubuntu, ubuntu, 4,
application_1524172890453_0002, container_1524172890453_0002_02_000001,
/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1524172890453_0002/container_1524172890453_0002_02_000001,
/tmp/hadoop-ubuntu/nm-local-dir/nmPrivate/application_1524172890453_0002/container_1524172890453_0002_02_000001/launch_container.sh,
/tmp/hadoop-ubuntu/nm-local-dir/nmPrivate/application_1524172890453_0002/container_1524172890453_0002_02_000001/container_1524172890453_0002_02_000001.tokens,
/tmp/hadoop-ubuntu/nm-local-dir/nmPrivate/application_1524172890453_0002/container_1524172890453_0002_02_000001/container_1524172890453_0002_02_000001.pid,
/tmp/hadoop-ubuntu/nm-local-dir, /home/ubuntu/hadoop-3.1.0/logs/userlogs,
/tmp/hadoop-ubuntu/nm-docker-cmds/docker.container_1524172890453_0002_02_0000011911992431676582343.cmd,
cgroups=none]
2018-04-19 17:29:13,032 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime:
Launch container failed. Exception:
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException:
ExitCodeException exitCode=29: Invalid docker ro mount
'/tmp/hadoop-ubuntu/nm-local-dir/filecache:/tmp/hadoop-ubuntu/nm-local-dir/filecache',
realpath=/tmp/hadoop-ubuntu/nm-local-dir/filecache
Error constructing docker command, docker error code=13, error message='Invalid
docker read-only mount'
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:180)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(DockerLinuxContainerRuntime.java:897)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime.launchContainer(DelegatingLinuxContainerRuntime.java:141)
at
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:545)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:511)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:304)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: ExitCodeException exitCode=29: Invalid docker ro mount
'/tmp/hadoop-ubuntu/nm-local-dir/filecache:/tmp/hadoop-ubuntu/nm-local-dir/filecache',
realpath=/tmp/hadoop-ubuntu/nm-local-dir/filecache
Error constructing docker command, docker error code=13, error message='Invalid
docker read-only mount'
at org.apache.hadoop.util.Shell.runCommand(Shell.java:1009)
at org.apache.hadoop.util.Shell.run(Shell.java:902)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1227)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:152)
... 10 more
2018-04-19 17:29:13,034 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime:
Docker command used: run
cap-add=SYS_CHROOT,MKNOD,SETFCAP,SETPCAP,FSETID,CHOWN,AUDIT_WRITE,SETGID,NET_RAW,FOWNER,SETUID,DAC_OVERRIDE,KILL,NET_BIND_SERVICE
cap-drop=ALL detach=true docker-command=run
docker-config=/tmp/hadoop-ubuntu/nm-local-dir/nmPrivate/application_1524172890453_0002/container_1524172890453_0002_02_000001
group-add=1000,4,24,27,30,46,110,115,116,117 image=local/hadoop-ubuntu:latest
launch-command=bash,/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1524172890453_0002/container_1524172890453_0002_02_000001/launch_container.sh
name=container_1524172890453_0002_02_000001 net=host
ro-mounts=/tmp/hadoop-ubuntu/nm-local-dir/filecache:/tmp/hadoop-ubuntu/nm-local-dir/filecache,/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/filecache:/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/filecache
rw-mounts=/home/ubuntu/hadoop-3.1.0/logs/userlogs/application_1524172890453_0002/container_1524172890453_0002_02_000001:/home/ubuntu/hadoop-3.1.0/logs/userlogs/application_1524172890453_0002/container_1524172890453_0002_02_000001,/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1524172890453_0002:/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1524172890453_0002
user=1000:1000
workdir=/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1524172890453_0002/container_1524172890453_0002_02_000001
2018-04-19 17:29:13,034 WARN
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code
from container container_1524172890453_0002_02_000001 is : 29
2018-04-19 17:29:13,034 WARN
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exception
from container-launch with container ID: container_1524172890453_0002_02_000001
and exit code: 29
org.apache.hadoop.yarn.server.nodemanager.containermanager.runtime.ContainerExecutionException:
Launch container failed
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(DockerLinuxContainerRuntime.java:904)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime.launchContainer(DelegatingLinuxContainerRuntime.java:141)
at
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:545)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:511)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:304)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-04-19 17:29:13,035 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from
container-launch.
2018-04-19 17:29:13,035 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id:
container_1524172890453_0002_02_000001
2018-04-19 17:29:13,035 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 29
2018-04-19 17:29:13,035 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception message:
Invalid docker ro mount
'/tmp/hadoop-ubuntu/nm-local-dir/filecache:/tmp/hadoop-ubuntu/nm-local-dir/filecache',
realpath=/tmp/hadoop-ubuntu/nm-local-dir/filecache
2018-04-19 17:29:13,035 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Error constructing
docker command, docker error code=13, error message='Invalid docker read-only
mount'
2018-04-19 17:29:13,035 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
2018-04-19 17:29:13,035 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Shell output: main
: command provided 4
2018-04-19 17:29:13,035 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: main : run as user
is ubuntu
2018-04-19 17:29:13,035 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: main : requested
yarn user is ubuntu
2018-04-19 17:29:13,035 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Creating script
paths...
2018-04-19 17:29:13,035 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Creating local
dirs...
2018-04-19 17:29:13,036 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
Container launch failed : Container exited with a non-zero exit code 29.
2018-04-19 17:29:13,036 ERROR
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
Failed to get tail of the container's prelaunch error log file
java.io.FileNotFoundException: File
/home/ubuntu/hadoop-3.1.0/logs/userlogs/application_1524172890453_0002/container_1524172890453_0002_02_000001/prelaunch.err
does not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:641)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:930)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.handleContainerExitWithFailure(ContainerLaunch.java:589)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.handleContainerExitCode(ContainerLaunch.java:555)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:338)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-04-19 17:29:13,041 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
Container container_1524172890453_0002_02_000001 transitioned from RUNNING to
EXITED_WITH_FAILURE
2018-04-19 17:29:13,043 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
Cleaning up container container_1524172890453_0002_02_000001
2018-04-19 17:29:18,158 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
Could not get pid for container_1524172890453_0002_02_000001. Waited for 5000
ms.
2018-04-19 17:29:18,158 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
Unable to obtain pid, but docker container request detected. Attempting to
reap container container_1524172890453_0002_02_000001
2018-04-19 17:29:20,578 INFO
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Deleting
absolute path :
/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1524172890453_0002/container_1524172890453_0002_02_000001
2018-04-19 17:29:20,579 WARN
org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=ubuntu
OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE
DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE
APPID=application_1524172890453_0002
CONTAINERID=container_1524172890453_0002_02_000001
2018-04-19 17:29:20,588 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
Container container_1524172890453_0002_02_000001 transitioned from
EXITED_WITH_FAILURE to DONE
2018-04-19 17:29:20,588 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
Removing container_1524172890453_0002_02_000001 from application
application_1524172890453_0002
2018-04-19 17:29:20,589 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Stopping resource-monitoring for container_1524172890453_0002_02_000001
2018-04-19 17:29:20,589 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
event CONTAINER_STOP for appId application_1524172890453_0002
2018-04-19 17:29:21,600 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed
completed containers from NM context: [container_1524172890453_0002_02_000001]
2018-04-19 17:29:21,601 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
Application application_1524172890453_0002 transitioned from RUNNING to
APPLICATION_RESOURCES_CLEANINGUP
2018-04-19 17:29:21,602 INFO
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Deleting
absolute path :
/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1524172890453_0002
2018-04-19 17:29:21,602 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
event APPLICATION_STOP for appId application_1524172890453_0002
2018-04-19 17:29:21,602 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
Application application_1524172890453_0002 transitioned from
APPLICATION_RESOURCES_CLEANINGUP to FINISHED
2018-04-19 17:29:21,602 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler:
Scheduling Log Deletion for application: application_1524172890453_0002, with
delay of 10800 seconds
2018-04-19 17:31:34,197 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
Cache Size Before Clean: 0, Total Deleted: 0, Public Deleted: 0, Private
Deleted: 0
{code}
> Docker container run_time
> -------------------------
>
> Key: YARN-8181
> URL: https://issues.apache.org/jira/browse/YARN-8181
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Seyyed Ahmad Javadi
> Priority: Major
>
> Hi All,
> I want to use docker container run time but could not solve the facing
> problem. I am following the guide below and the NM log is as follows. I can
> not see any docker containers to be created. It works when I use default LCE.
> Please also find how I submit a job at the end as well.
> Do you have any guide on how can I make Docker rum_time works?
> May you please let me know how can use LCE binary to make sure my docker
> setup is correct?
> I confirmed that "docker run" works fine. I really like this developing
> feature and would like to contribute to it. Many thanks in advance.
> [https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/DockerContainers.html]
> {code:java}
> NM LOG:
> ...
> 2018-04-19 11:49:24,568 INFO SecurityLogger.org.apache.hadoop.ipc.Server:
> Auth successful for appattempt_1524151293356_0005_000001 (auth:SIMPLE)
> 2018-04-19 11:49:24,580 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> Start request for container_1524151293356_0005_01_000001 by user ubuntu
> 2018-04-19 11:49:24,584 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> Creating a new application reference for app application_1524151293356_0005
> 2018-04-19 11:49:24,584 INFO
> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=ubuntu
> IP=130.245.127.176 OPERATION=Start Container Request
> TARGET=ContainerManageImpl RESULT=SUCCESS
> APPID=application_1524151293356_0005
> CONTAINERID=container_1524151293356_0005_01_000001
> 2018-04-19 11:49:24,585 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Application application_1524151293356_0005 transitioned from NEW to INITING
> 2018-04-19 11:49:24,585 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Adding container_1524151293356_0005_01_000001 to application
> application_1524151293356_0005
> 2018-04-19 11:49:24,585 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Application application_1524151293356_0005 transitioned from INITING to
> RUNNING
> 2018-04-19 11:49:24,588 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
> Container container_1524151293356_0005_01_000001 transitioned from NEW to
> LOCALIZING
> 2018-04-19 11:49:24,588 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
> event CONTAINER_INIT for appId application_1524151293356_0005
> 2018-04-19 11:49:24,589 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Created localizer for container_1524151293356_0005_01_000001
> 2018-04-19 11:49:24,616 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Writing credentials to the nmPrivate file
> /tmp/hadoop-ubuntu/nm-local-dir/nmPrivate/container_1524151293356_0005_01_000001.tokens
> 2018-04-19 11:49:28,090 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
> Container container_1524151293356_0005_01_000001 transitioned from
> LOCALIZING to SCHEDULED
> 2018-04-19 11:49:28,090 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler.ContainerScheduler:
> Starting container [container_1524151293356_0005_01_000001]
> 2018-04-19 11:49:28,212 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
> Container container_1524151293356_0005_01_000001 transitioned from SCHEDULED
> to RUNNING
> 2018-04-19 11:49:28,212 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Starting resource-monitoring for container_1524151293356_0005_01_000001
> 2018-04-19 11:49:29,401 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
> Container container_1524151293356_0005_01_000001 succeeded
> 2018-04-19 11:49:29,401 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
> Container container_1524151293356_0005_01_000001 transitioned from RUNNING
> to EXITED_WITH_SUCCESS
> 2018-04-19 11:49:29,401 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
> Cleaning up container container_1524151293356_0005_01_000001
> 2018-04-19 11:49:29,520 INFO
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Removing
> Docker container : container_1524151293356_0005_01_000001
> 2018-04-19 11:49:34,517 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
> Could not get pid for container_1524151293356_0005_01_000001. Waited for
> 5000 ms.
> 2018-04-19 11:49:34,517 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
> Unable to obtain pid, but docker container request detected. Attempting to
> reap container container_1524151293356_0005_01_000001
> 2018-04-19 11:49:36,927 INFO
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Deleting
> absolute path :
> /tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1524151293356_0005/container_1524151293356_0005_01_000001
> 2018-04-19 11:49:36,928 INFO
> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=ubuntu
> OPERATION=Container Finished - Succeeded TARGET=ContainerImpl
> RESULT=SUCCESS APPID=application_1524151293356_0005
> CONTAINERID=container_1524151293356_0005_01_000001
> 2018-04-19 11:49:36,929 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
> Container container_1524151293356_0005_01_000001 transitioned from
> EXITED_WITH_SUCCESS to DONE
> 2018-04-19 11:49:36,938 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Removing container_1524151293356_0005_01_000001 from application
> application_1524151293356_0005
> 2018-04-19 11:49:36,938 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Stopping resource-monitoring for container_1524151293356_0005_01_000001
> 2018-04-19 11:49:36,938 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
> event CONTAINER_STOP for appId application_1524151293356_0005
> 2018-04-19 11:49:37,941 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed
> completed containers from NM context: [container_1524151293356_0005_01_000001]
> 2018-04-19 11:49:50,966 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Application application_1524151293356_0005 transitioned from RUNNING to
> APPLICATION_RESOURCES_CLEANINGUP
> 2018-04-19 11:49:50,967 INFO
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Deleting
> absolute path :
> /tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1524151293356_0005
> 2018-04-19 11:49:50,967 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
> event APPLICATION_STOP for appId application_1524151293356_0005
> 2018-04-19 11:49:50,967 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Application application_1524151293356_0005 transitioned from
> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
> 2018-04-19 11:49:50,967 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler:
> Scheduling Log Deletion for application: application_1524151293356_0005,
> with delay of 10800 seconds
> {code}
> {code:java}
> vars="YARN_CONTAINER_RUNTIME_TYPE=docker,YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=hadoop-ubuntu:latest"
> #vars="YARN_CONTAINER_RUNTIME_TYPE=docker,YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=hadoop-ubuntu:latest,YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE=false,YARN_CONTAINER_RUNTIME_DOCKER_CONTAINER_NETWORK=host"
> #vars="YARN_CONTAINER_RUNTIME_TYPE=default"
> hadoop jar
> /home/ubuntu/hadoop-3.1.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.0.jar
> pi -Dyarn.app.mapreduce.am.env=$vars -Dmapreduce.map.env=$vars
> -Dmapreduce.reduce.env=$vars 2 10
> {code}
> {code:java}
> Number of Maps = 2
> Samples per Map = 10
> Wrote input for Map #0
> Wrote input for Map #1
> Starting Job
> 2018-04-19 11:49:22,786 INFO client.RMProxy: Connecting to ResourceManager at
> bay1-vm1/130.245.127.176:8032
> 2018-04-19 11:49:23,435 INFO mapreduce.JobResourceUploader: Disabling Erasure
> Coding for path:
> /tmp/hadoop-yarn/staging/ubuntu/.staging/job_1524151293356_0005
> 2018-04-19 11:49:23,601 INFO input.FileInputFormat: Total input files to
> process : 2
> 2018-04-19 11:49:23,756 INFO mapreduce.JobSubmitter: number of splits:2
> 2018-04-19 11:49:23,824 INFO Configuration.deprecation:
> yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead,
> use yarn.system-metrics-publisher.enabled
> 2018-04-19 11:49:24,015 INFO mapreduce.JobSubmitter: Submitting tokens for
> job: job_1524151293356_0005
> 2018-04-19 11:49:24,017 INFO mapreduce.JobSubmitter: Executing with tokens: []
> 2018-04-19 11:49:24,262 INFO conf.Configuration: resource-types.xml not found
> 2018-04-19 11:49:24,262 INFO resource.ResourceUtils: Unable to find
> 'resource-types.xml'.
> 2018-04-19 11:49:24,350 INFO impl.YarnClientImpl: Submitted application
> application_1524151293356_0005
> 2018-04-19 11:49:24,398 INFO mapreduce.Job: The url to track the job:
> http://bay1-vm1:8088/proxy/application_1524151293356_0005/
> 2018-04-19 11:49:24,399 INFO mapreduce.Job: Running job:
> job_1524151293356_0005
> 2018-04-19 11:49:50,658 INFO mapreduce.Job: Job job_1524151293356_0005
> running in uber mode : false
> 2018-04-19 11:49:50,660 INFO mapreduce.Job: map 0% reduce 0%
> 2018-04-19 11:49:50,676 INFO mapreduce.Job: Job job_1524151293356_0005 failed
> with state FAILED due to: Application application_1524151293356_0005 failed 2
> times due to AM Container for appattempt_1524151293356_0005_000002 exited
> with exitCode: 0
> Failing this attempt.Diagnostics: For more detailed output, check the
> application tracking page:
> http://bay1-vm1:8088/cluster/app/application_1524151293356_0005 Then click on
> links to logs of each attempt.
> . Failing the application.
> 2018-04-19 11:49:50,702 INFO mapreduce.Job: Counters: 0
> Job job_1524151293356_0005 failed!
> runtime in seconds: 34
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
