[ https://issues.apache.org/jira/browse/YARN-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13632068#comment-13632068 ]
Chris Nauroth commented on YARN-358: ------------------------------------ {quote} @Chris, are we just talking about the command line or does this affect environment variables too? {quote} Yes, the length limitation that I mentioned here impacts both the command line and environment variables. Here is a bit more detail. Technically, the maximum length of a Windows environment variable is 32,767. However, the nodemanager uses cmd scripts to launch containers, and those cmd scripts contain {{@set}} commands to set environment variables. (See {{ContainerLaunch#WindowsShellScriptBuilder#env}}.) These {{@set}} commands are subject to the cmd length limitation just like any other command, which is 8,191. The technique that we used to package the classpath into an intermediate manifest jar is just a workaround for the one specific case, not a general-purpose mechanism that's useful for other long environment variables. {quote} Should this be a YARN feature or is it better to hand this off to the application logic to handle correct launching of a container on a particular OS type? {quote} This overlaps somewhat with comments I just added on YARN-445, so I'll just summarize briefly here. It looks like there is a lot of difference in process management between OS's. I don't think YARN can anticipate all of the variations and handle them completely. I believe it would be beneficial for the YARN API to support a mapping of platform/OS to unique ContainerLaunchContext. The AM could use this to specify different process management commands on different OS's, but without requiring the AM to know whether a specific node is running Unix, Windows, or anything else. This has the side benefit of laying groundwork for potential support of heterogeneous clusters (nodemanagers running different OS's). If the above approach makes sense, then it probably means that there is too much JVM-specific launch logic in the nodemanager code right now. We could refactor that back to the MR AM or helper libraries reusable by multiple AM implementations. > bundle container classpath in temporary jar on all platforms, not just Windows > ------------------------------------------------------------------------------ > > Key: YARN-358 > URL: https://issues.apache.org/jira/browse/YARN-358 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager > Affects Versions: trunk-win > Reporter: Chris Nauroth > > Currently, a Windows-specific code path bundles the classpath into a > temporary jar with a manifest to work around command line length limitations. > This code path does not need to be Windows-specific. We can use the same > approach on all platforms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira