[
https://issues.apache.org/jira/browse/YARN-316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated YARN-316:
-------------------------------
Attachment: YARN-316-branch-trunk-win.1.patch
The attached patch is a port of the strategy used for branch-1-win MapReduce in
HADOOP-8899. On Windows, we build a temporary jar that contains only a
manifest. The manifest's Class-Path attribute references all of the classpath
entries. The Class-Path attribute is not subject to the Windows command line
length limitation, so it can grow as large as needed.
Within the Class-Path attribute, environment variables are not expanded
automatically and wildcards are not supported. The patch includes code to
evaluate environment variables and resolve wildcard patterns before writing the
Class-Path attribute, so Windows can still use classpath entries like
%HADOOP_HOME%/share/hadoop/common/*.
While working on this, I discovered a few related problems for Windows
compatibility. {{RawLocalFileSystem#setPermission}} used an extra leading '0'
in a chmod command. This was harmless on Linux, bu winutils rejects it, so I
removed it. Several lines of code in {{DefaultContainerExecutor}} and
{{ContainerLaunch}} called {{Path.toUri().getPath()}} instead of
{{Path.toString()}}. {{Path.toString()}} is needed to get the expected
behavior for handling a Windows path with a drive spec: it removes the leading
'/' before the drive spec.
I also discovered a problem across the YARN test suites on Windows. The tests
tend to create very long container working directories deep in the Maven build
hierarchy. cmd.exe cannot execute a script at an absolute path longer than 260
characters. This was causing numerous tests to fail while trying to run
default_container_executor.cmd. To resolve this, I shortened the directories
by switching calls to {{Class#getName}} to {{Class#getSimpleName}}. Then, I
created a symlink at a shorter path in temporary storage, targeting the full
path to the working directory. Tests then access the working directory through
the short symlink to avoid hitting the path length limitation. (See changes in
{{MiniMRClientClusterFactory}} and {{MiniYARNCluster}}.)
{{org.apache.hadoop.mapred.TestClusterMapReduceTestCase}} is an example of a
test suite that was failing on Windows, but it passes now with this patch.
> YARN container launch may exceed maximum Windows command line length due to
> long classpath
> ------------------------------------------------------------------------------------------
>
> Key: YARN-316
> URL: https://issues.apache.org/jira/browse/YARN-316
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 3.0.0, trunk-win
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Attachments: YARN-316-branch-trunk-win.1.patch
>
>
> On Windows, a command line longer than 8192 characters will fail. This can
> cause YARN container launch to fail on Windows if the classpath argument
> exceeds this limit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
