[
https://issues.apache.org/jira/browse/YARN-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13632068#comment-13632068
]
Chris Nauroth commented on YARN-358:
------------------------------------
{quote}
@Chris, are we just talking about the command line or does this affect
environment variables too?
{quote}
Yes, the length limitation that I mentioned here impacts both the command line
and environment variables.
Here is a bit more detail. Technically, the maximum length of a Windows
environment variable is 32,767. However, the nodemanager uses cmd scripts to
launch containers, and those cmd scripts contain {{@set}} commands to set
environment variables. (See
{{ContainerLaunch#WindowsShellScriptBuilder#env}}.) These {{@set}} commands
are subject to the cmd length limitation just like any other command, which is
8,191.
The technique that we used to package the classpath into an intermediate
manifest jar is just a workaround for the one specific case, not a
general-purpose mechanism that's useful for other long environment variables.
{quote}
Should this be a YARN feature or is it better to hand this off to the
application logic to handle correct launching of a container on a particular OS
type?
{quote}
This overlaps somewhat with comments I just added on YARN-445, so I'll just
summarize briefly here. It looks like there is a lot of difference in process
management between OS's. I don't think YARN can anticipate all of the
variations and handle them completely. I believe it would be beneficial for
the YARN API to support a mapping of platform/OS to unique
ContainerLaunchContext. The AM could use this to specify different process
management commands on different OS's, but without requiring the AM to know
whether a specific node is running Unix, Windows, or anything else. This has
the side benefit of laying groundwork for potential support of heterogeneous
clusters (nodemanagers running different OS's).
If the above approach makes sense, then it probably means that there is too
much JVM-specific launch logic in the nodemanager code right now. We could
refactor that back to the MR AM or helper libraries reusable by multiple AM
implementations.
> bundle container classpath in temporary jar on all platforms, not just Windows
> ------------------------------------------------------------------------------
>
> Key: YARN-358
> URL: https://issues.apache.org/jira/browse/YARN-358
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: trunk-win
> Reporter: Chris Nauroth
>
> Currently, a Windows-specific code path bundles the classpath into a
> temporary jar with a manifest to work around command line length limitations.
> This code path does not need to be Windows-specific. We can use the same
> approach on all platforms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
