[
https://issues.apache.org/jira/browse/YARN-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15240658#comment-15240658
]
Vinod Kumar Vavilapalli commented on YARN-4577:
-----------------------------------------------
Just caught up with the wall of comments.
+1 in general for the ApplicationClassloader based solution. Aux-services was
always a hack since Chris Douglas and I originally introduced it, the better
solution is to move these services as first-class apps on top of YARN, but we
are where we are.
[~sjlee0],
bq. For example, if the aux service code depends on another class property
(owned by the aux service) in the configuration, that will be invoked via
Configuration.getClass(), and it will still use the system classloader to load
that class. Then it's very likely that you'll get a ClassNotFoundException.
Sangjin, you may be missing one important thing here - unlike in the MapReduce
case, there is no shared Configuration object between NodeManager and the
specific aux-service implementation here. We simply do not pass in any
configuration anywhere as part of the AuxService APIs - so this entire thread
of reasoning about getClass() is no long a problem? If needed, we can document
advising against adding Conf as part of future API changes.
bq. The thread context classloader represents another similar problem. The
moment the aux service code hits a code path that does Class.forName() that
loads classes via the thread context classloader, and it needs to load an aux
service-related class (that is not present in the main NM classpath), you will
get a ClassNotFoundException.
In addition to wrapping aux-service API calls under a class-loader, wouldn't it
suffice to simply have NM make all aux-services API calls in a separate thread
whose ContextClassLoader is changed to be another custom one that resolves both
System classes as well as AuxServices classes?
> Enable aux services to have their own custom classpath/jar file
> ---------------------------------------------------------------
>
> Key: YARN-4577
> URL: https://issues.apache.org/jira/browse/YARN-4577
> Project: Hadoop YARN
> Issue Type: Improvement
> Affects Versions: 2.8.0
> Reporter: Xuan Gong
> Assignee: Xuan Gong
> Attachments: YARN-4577.1.patch, YARN-4577.2.patch,
> YARN-4577.20160119.1.patch, YARN-4577.20160204.patch, YARN-4577.3.patch,
> YARN-4577.3.rebase.patch, YARN-4577.4.patch
>
>
> Right now, users have to add their jars to the NM classpath directly, thus
> put them on the system classloader. But if multiple versions of the plugin
> are present on the classpath, there is no control over which version actually
> gets loaded. Or if there are any conflicts between the dependencies
> introduced by the auxiliary service and the NM itself, they can break the NM,
> the auxiliary service, or both.
> The solution could be: to instantiate aux services using a classloader that
> is different from the system classloader.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
