[
https://issues.apache.org/jira/browse/YARN-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14559309#comment-14559309
]
Vinod Kumar Vavilapalli commented on YARN-3685:
-----------------------------------------------
bq. Perhaps it's possible to move the classpath jar generation to the MR client
or AM. It's not immediately obvious to me which of those 2 choices is better.
For AM container, client is the right place. For the rest of the tasks, AM is.
bq. We'd need to change the manifest to use relative paths in the Class-Path
attribute instead of absolute paths. (The client and AM are not aware of the
exact layout of the NodeManager's yarn.nodemanager.local-dirs, so the client
can't predict the absolute paths at time of container launch.)
I think this was one of the chief issues in the original patches - we need to
investigate if manifest file can have relative paths or not. Otherwise, it's
ugly but we can still get YARN to replace some sort of markers only in specific
files like the manifest.
bq. Some classpath entries are defined in terms of environment variables. These
environment variables are expanded at the NodeManager via the container launch
scripts. This was true of Linux even before YARN-316, so in that sense, YARN
did already have some classpath logic indirectly.
Which ones are these?
bq. If we do move classpath handling out of the NodeManager, then it would be a
backwards-incompatible change, and so it could not be shipped in the 2.x
release line.
Not clear this is true or not. Have to see the final solution/patch to
realistically reason about this.
> NodeManager unnecessarily knows about classpath-jars due to Windows
> limitations
> -------------------------------------------------------------------------------
>
> Key: YARN-3685
> URL: https://issues.apache.org/jira/browse/YARN-3685
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
>
> Found this while looking at cleaning up ContainerExecutor via YARN-3648,
> making it a sub-task.
> YARN *should not* know about classpaths. Our original design modeled around
> this. But when we added windows suppport, due to classpath issues, we ended
> up breaking this abstraction via YARN-316. We should clean this up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
