Chris Nauroth commented on YARN-3685:

[~vinodkv], thanks for the notification.  I was not aware of this design goal 
at the time of YARN-316.

Perhaps it's possible to move the classpath jar generation to the MR client or 
AM.  It's not immediately obvious to me which of those 2 choices is better.  
We'd need to change the manifest to use relative paths in the Class-Path 
attribute instead of absolute paths.  (The client and AM are not aware of the 
exact layout of the NodeManager's {{yarn.nodemanager.local-dirs}}, so the 
client can't predict the absolute paths at time of container launch.)

There is one piece of logic that I don't see how to handle though.  Some 
classpath entries are defined in terms of environment variables.  These 
environment variables are expanded at the NodeManager via the container launch 
scripts.  This was true of Linux even before YARN-316, so in that sense, YARN 
did already have some classpath logic indirectly.  Environment variables cannot 
be used inside a manifest's Class-Path, so for Windows, NodeManager expands the 
environment variables before populating Class-Path.  It would be incorrect to 
do the environment variable expansion at the MR client, because it might be 
running with different configuration than the NodeManager.  I suppose if the AM 
did the expansion, then that would work in most cases, but it creates an 
assumption that the AM container is running with configuration that matches all 
NodeManagers in the cluster.  I don't believe that assumption exists today.

If we do move classpath handling out of the NodeManager, then it would be a 
backwards-incompatible change, and so it could not be shipped in the 2.x 
release line.

> NodeManager unnecessarily knows about classpath-jars due to Windows 
> limitations
> -------------------------------------------------------------------------------
>                 Key: YARN-3685
>                 URL: https://issues.apache.org/jira/browse/YARN-3685
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Vinod Kumar Vavilapalli
> Found this while looking at cleaning up ContainerExecutor via YARN-3648, 
> making it a sub-task.
> YARN *should not* know about classpaths. Our original design modeled around 
> this. But when we added windows suppport, due to classpath issues, we ended 
> up breaking this abstraction via YARN-316. We should clean this up.

This message was sent by Atlassian JIRA

Reply via email to