Thank you for your reply.
The problem is fixed.
Since the Hadoop version in use is 2.6.0, I diff the
LinuxContainerExecutor.java to the most updated one.
And I replace ‘,’ to ‘%’ in startLocalizer(…) and launchContainer(…) method
respectively, like the following code block:
“
command.addAll(Arrays.asList(containerExecutorExe,
runAsUser,
user,
Integer.toString(Commands.INITIALIZE_CONTAINER.getValue()),
appId,
nmPrivateContainerTokensPath.toUri().getPath().toString(),
StringUtils.join(",", localDirs), -->
StringUtils.join("%", localDirs),
StringUtils.join(",", logDirs))); -->
StringUtils.join("%", logDirs)));
”.
The mr pi finishes.
It seems that the ‘,’ would fail the execution of running container-executor.
Regards,
Reid Chan
From: 萝卜丝炒饭 <[email protected]>
Date: Tuesday, 7 February 2017 at 2:31 PM
To: 陈浩骏 <[email protected]>, "[email protected]"
<[email protected]>
Subject: Re: Unable to localize container, when using LinuxContainerExecutor
have you created the parent folder, such as /data1/yarn
---Original---
From: "陈浩骏"<[email protected]>
Date: 2017/2/7 00:43:47
To: "[email protected]"<[email protected]>;
Subject: Unable to localize container, when using LinuxContainerExecutor
Hi,
I tried to run secure containers in cluster, and followed the instructions
from
http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/SecureMode.html#LinuxContainerExecutor
After that, I ran the mapreduce example pi and got the following exception:
2017-02-06 17:46:24,018 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: main : command
provided 0
2017-02-06 17:46:24,018 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: main : run as user
is hadoop
2017-02-06 17:46:24,018 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: main : requested
yarn user is hadoop
2017-02-06 17:46:24,019 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Failed to create
directory /data1/yarn/nm,/data2/yarn/nm,/data3/yarn/nm/usercache/hadoop - No
such file or directory
2017-02-06 17:46:24,019 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
Localizer failed
java.io.IOException: Application application_1486374303689_0001 initialization
failed (exitCode=20) with output: main : command provided 0
main : run as user is hadoop
main : requested yarn user is hadoop
Failed to create directory
/data1/yarn/nm,/data2/yarn/nm,/data3/yarn/nm/usercache/hadoop - No such file or
directory
at
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.startLocalizer(LinuxContainerExecutor.java:241)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1088)
Caused by: ExitCodeException exitCode=20:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.startLocalizer(LinuxContainerExecutor.java:232)
... 1 more
Here is the container-executor.cfg under /etc/hadoop owned by root
(configurations are same to the yarn-site.xml):
yarn.nodemanager.local-dirs=/data1/yarn/nm,/data2/yarn/nm,/data3/yarn/nm
yarn.nodemanager.log-dirs=/data1/hadoop/logs/container
yarn.nodemanager.linux-container-executor.group=hadoop
banned.users=yarn
min.user.id=500
allowed.system.users=hadoop,dm
Hope to get some helps.
Regards,
Reid Chan.
