[
https://issues.apache.org/jira/browse/YARN-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15145099#comment-15145099
]
Sangjin Lee commented on YARN-4577:
-----------------------------------
I think I need to clarify a little more what needs to be done with the custom
classloader. The custom classloader for a given aux service must be created
only once and used throughout for any code that exercises that aux service's
code.
I don't think you need to wrap creating the aux service with a
{{callWithClassLoader()}} call. That's really not necessary. What you do need
is, you need to intercept *subsequent* calls on the aux service and wrap them
with {{callWithClassLoader()}}. And when you do, you'd need to provide that
exact custom classloader instance you created when you created the aux service.
Otherwise, things like {{ClassCastException}} can ensue.
So *all* overridable calls to the public methods like
{{initializeApplication()}}, {{stopApplication()}}, {{getMetaData()}},
{{initializeContainer()}}, {{stopContainer()}}, as well as service lifecycle
methods such as {{serviceStart()}}, {{serviceInit()}}, and {{serviceStop()}}
should be intercepted.
Now, intercepting the aux-service-specific methods is easier as there are
explicit places where they are invoked. What's more difficult is the service
lifecycle methods as they are invoked from {{AuxServices}} only indirectly.
I can see 2 approaches to this (there may be more). First, we could consider
adding a wrapper class that can do this within the wrapper code. It might be
something like
{code}
class AuxiliaryServiceWithCustomClassLoader extends AuxiliaryService {
private final AuxiliaryService wrapped;
private final ClassLoader customClassLoader;
public AuxiliaryServiceWithCustomClassLoader(AuxiliaryService s, ClassLoader
cl) {
this.wrapped = s;
this.customClassLoader = cl;
}
...
@Override
protected void serviceStart() throws Exception {
callWithCustomClassLoader(wrapped.serviceStart());
}
...
@Override
public void initializeApplication(ApplicationInitializationContext context) {
callWithCustomClassLoader(wrapper.initializeApplication(context));
}
...
private callWithCustomClassLoader(...) {
}
}
{code}
The other approach may be to abandon dealing with the
{{Configuration.getClass()}} and the thread context classloader scenario, and
*requiring* any aux service that wants to use a custom classpath *NOT* to do
anything that will trigger {{Configuration.getClass()}} or the use of the TCCL,
either directly or indirectly. Then we don't need to do all this wrapping
business.
What I'm not sure of is how practical this requirement would be for those aux
services. Note that this has to be true even directly.
I also need to point out one potential risk with this wrapping. One potential
risk is that the configuration is a shared object. If one aux service resets
the configuration classloader to invoke its code, any other NM code that's
running concurrently could be impacted potentially. For example, if two aux
services have a race (on separate threads) setting the configuration class,
then we could have a nasty problem.
Could you consider the implications of these options? Perhaps requiring these
aux services not to do any {{Configuration.getClass()}} might not be that
unreasonable...
> Enable aux services to have their own custom classpath/jar file
> ---------------------------------------------------------------
>
> Key: YARN-4577
> URL: https://issues.apache.org/jira/browse/YARN-4577
> Project: Hadoop YARN
> Issue Type: Improvement
> Affects Versions: 2.8.0
> Reporter: Xuan Gong
> Assignee: Xuan Gong
> Attachments: YARN-4577.1.patch, YARN-4577.2.patch,
> YARN-4577.20160119.1.patch, YARN-4577.20160204.patch, YARN-4577.3.patch,
> YARN-4577.3.rebase.patch, YARN-4577.4.patch
>
>
> Right now, users have to add their jars to the NM classpath directly, thus
> put them on the system classloader. But if multiple versions of the plugin
> are present on the classpath, there is no control over which version actually
> gets loaded. Or if there are any conflicts between the dependencies
> introduced by the auxiliary service and the NM itself, they can break the NM,
> the auxiliary service, or both.
> The solution could be: to instantiate aux services using a classloader that
> is different from the system classloader.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
