[
https://issues.apache.org/jira/browse/MAPREDUCE-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13924092#comment-13924092
]
Sangjin Lee commented on MAPREDUCE-5751:
----------------------------------------
IMHO, making classloader isolation work at all times without fail is
unfortunately pretty difficult, most because Java allows leaky ways of loading
classes and it's difficult to always enforce the rules without breaking user
code.
One concrete scenario where the problem I mentioned above could happen without
making jetty system classes is as follows. You're probably familiar with how it
can happen (if so I apologize), but let me put it down still.
Suppose we set the job classloader just prior to starting jobs. And suppose the
app master decides to start a new jetty-based server *after* that point in
time. Normally all the jetty classes would have been loaded (including the
configuration class). But suppose it triggers a code path that wasn't exercised
yet which uses the *context classloader* (getContextClassLoader()). This would
now return the job classloader. If that class calls code like this, you would
get a ClassCastException:
{code}
ClassLoader cl = Thread.currentThread().getContextClassLoader();
// Foo was loaded by the app classloader previously
// FooImpl will be loaded by a different classloader
// this line will throw a ClassCastException
Foo foo = (Foo)Class.forName("org.jetty.foo.impl.FooImpl", cl); // or
(Foo)cl.loadClass("org.jetty.foo.impl.FooImpl")
{code}
For the problem to surface, it would need to meet all of the following
conditions:
- its classes are loaded through the app classloader (i.e. the code path is
outside the user task code path)
- it uses the context classloader *after* it has been set to the job classloader
- it tries to load another class (of its own) using that context classloader
- it tries to cast it to its own type (interface or base class) that's already
been loaded by the app classloader
It is pretty hard to get into all of these. But it does happen, and the above
code pattern for jetty was one. Another library that I know of that uses the
context classloader is jaxb (already included in the system classes). So by
including jetty in the system classes we gain just a bit more safety.
This is bit of trial and error. But we do know what hadoop may do outside the
user code, and hopefully this list does not need to grow.
My 2 cents...
> MR app master fails to start in some cases if mapreduce.job.classloader is
> true
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-5751
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5751
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 2.2.0
> Reporter: Sangjin Lee
> Assignee: Sangjin Lee
> Attachments: mapreduce-5751.patch
>
>
> If mapreduce.job.classloader is set to true, and the MR client includes a
> jetty jar in its libjars or job jar, the MR app master fails to start. A
> typical stack trace we get is as follows:
> {noformat}
> java.lang.ClassCastException: org.mortbay.jetty.webapp.WebInfConfiguration
> cannot be cast to org.mortbay.jetty.webapp.Configuration
> at
> org.mortbay.jetty.webapp.WebAppContext.loadConfigurations(WebAppContext.java:890)
> at
> org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:462)
> at
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
> at
> org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
> at
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
> at
> org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:224)
> at
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
> at org.apache.hadoop.http.HttpServer.start(HttpServer.java:676)
> at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:208)
> at
> org.apache.hadoop.mapreduce.v2.app.client.MRClientService.start(MRClientService.java:151)
> at
> org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1040)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1307)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1303)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1259)
> {noformat}
> This happens because as part of the MR app master start the jetty classes are
> loaded normally through the app classloader, but WebAppContext tries to load
> the specific Configuration class via the thread context classloader (which
> had been set to the user job classloader).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
