[
https://issues.apache.org/jira/browse/YARN-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15104873#comment-15104873
]
Xuan Gong commented on YARN-4577:
---------------------------------
Thanks for the comments, [~sjlee0]
bq. "how important is it to support non-local classpaths"
It is important to support non-local classpath, especially HDFS classpath. It
is one of the requirement for this feature. Of course, the changes are not
trivial. I think that supporting HDFS could be one of the improvement for the
ApplicationClassLoader if we are planning to do it. If ApplicationClassLoader
supports it in future, we could replace it. But i still prefer to do it here
since it is part of the requirement.
bq. and "Regarding setting the URLStreamHandlerFactory, you can call
URL.setURLStreamHandlerFactory() at most once on a JVM, and any attempt to set
it again within the same process will throw an error:"
Good point. Added a static method to call it in AuxService.java. But I can not
find a better way to solve it. Any better suggestions ?
bq. other types of classloading
Actually, I do not even need a parent classloader here. For me, if the user
provided a specific classpath for the aux-service, it is user's responsibility
to make sure the provided jar file includes everything, includes the
dependency. And when we initiate the related aux-service, we only look for the
specific classpath. If the aux-service can not be initiated successfully with
the specific classpath, it should throw an exception instead of trying to load
it from parent classloader.
bq. unit test: l.366: I'm quite confused by the comment and the code
This unit test is used to test my previous comment.
> Enable aux services to have their own custom classpath/jar file
> ---------------------------------------------------------------
>
> Key: YARN-4577
> URL: https://issues.apache.org/jira/browse/YARN-4577
> Project: Hadoop YARN
> Issue Type: Improvement
> Affects Versions: 2.8.0
> Reporter: Xuan Gong
> Assignee: Xuan Gong
> Attachments: YARN-4577.1.patch, YARN-4577.2.patch, YARN-4577.3.patch,
> YARN-4577.3.rebase.patch, YARN-4577.4.patch
>
>
> Right now, users have to add their jars to the NM classpath directly, thus
> put them on the system classloader. But if multiple versions of the plugin
> are present on the classpath, there is no control over which version actually
> gets loaded. Or if there are any conflicts between the dependencies
> introduced by the auxiliary service and the NM itself, they can break the NM,
> the auxiliary service, or both.
> The solution could be: to instantiate aux services using a classloader that
> is different from the system classloader.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
