[
https://issues.apache.org/jira/browse/MAPREDUCE-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13924217#comment-13924217
]
Jason Lowe commented on MAPREDUCE-5751:
---------------------------------------
Yeah, that makes sense. I know we can't make this bulletproof, rather I am
thinking along the lines of which default is more likely to work for users. If
a user-provided jetty is not likely to break then I'm thinking we shouldn't
preclude it by default as that means the user doesn't get the jetty they expect.
I thought by initializing the job classloader after we've started services that
we should have avoided our jetty instance from trying to load the user-provided
jetty classes. Since we are going to start the jetty services before setting
the job classloader, I thought those jetty threads won't have the job
classloader as their context classloader and therefore not be affected by any
user-provided jetty jars. The AM is the only MR framework piece that uses
jetty within a job, and we should know exactly what it will do and when it will
init relative to user code.
I doubt many users will be starting jetty in their OutputCommitter code that
runs in the AM, and therefore it seemed likely that mixing the two jetty
versions and using the job classloader feature would work as intended. The AM
would get Hadoop's jetty version and tasks would get the user's version. If
it's likely to work then I don't see a need to preclude it by default and make
most users discover and change the system classes property to get what they
want in the majority of the cases.
Now if the user code starts up jetty inside the AM (i.e.: within their
OutputCommitter which seems weird but who knows) or if it's an uber job then we
probably have issues there.
Anyway I'm far from a classloader or jetty expert. If you guys agree that
jetty should be in the system classes by default for MR jobs then that's
probably a good thing. It's definitely the safer thing from the framework
perspective since we know we'll get what the framework wants (barring
user.classpath.first shenanigans). I just didn't want to preclude by default
users from using a different jetty version than Hadoop's dependency if it is
likely to work for most cases.
> MR app master fails to start in some cases if mapreduce.job.classloader is
> true
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-5751
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5751
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 2.2.0
> Reporter: Sangjin Lee
> Assignee: Sangjin Lee
> Attachments: mapreduce-5751.patch
>
>
> If mapreduce.job.classloader is set to true, and the MR client includes a
> jetty jar in its libjars or job jar, the MR app master fails to start. A
> typical stack trace we get is as follows:
> {noformat}
> java.lang.ClassCastException: org.mortbay.jetty.webapp.WebInfConfiguration
> cannot be cast to org.mortbay.jetty.webapp.Configuration
> at
> org.mortbay.jetty.webapp.WebAppContext.loadConfigurations(WebAppContext.java:890)
> at
> org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:462)
> at
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
> at
> org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
> at
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
> at
> org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:224)
> at
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
> at org.apache.hadoop.http.HttpServer.start(HttpServer.java:676)
> at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:208)
> at
> org.apache.hadoop.mapreduce.v2.app.client.MRClientService.start(MRClientService.java:151)
> at
> org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1040)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1307)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1303)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1259)
> {noformat}
> This happens because as part of the MR app master start the jetty classes are
> loaded normally through the app classloader, but WebAppContext tries to load
> the specific Configuration class via the thread context classloader (which
> had been set to the user job classloader).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
