[
https://issues.apache.org/jira/browse/MAPREDUCE-5957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14062285#comment-14062285
]
Jason Lowe commented on MAPREDUCE-5957:
---------------------------------------
Thanks Sangjin for the report and the patch. I'm pretty sure the test failures
are unrelated as I've seen other recent JIRAs also complain about the same set
of test failures.
I think the proposed solution is reasonable and would probably work in most
cases but will it cover all possible cases? In particular I'm thinking of a
case where an output format/committer creates threads in its constructor or
does reflection on its own with the current TCCL to load other classes. In
those cases don't we need to have the TCCL set as well? I'm not a fan of
flip-flopping the classloader around, but I'm wondering if we really should do
that to cover whatever crazy stuff user code might try to do.
> AM throws ClassNotFoundException with job classloader enabled if custom
> output format/committer is used
> -------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-5957
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5957
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 2.4.0
> Reporter: Sangjin Lee
> Assignee: Sangjin Lee
> Attachments: MAPREDUCE-5957.patch, MAPREDUCE-5957.patch
>
>
> With the job classloader enabled, the MR AM throws ClassNotFoundException if
> a custom output format class is specified.
> {noformat}
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> com.foo.test.TestOutputFormat not found
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:473)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:374)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1459)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1456)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1389)
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> Class com.foo.test.TestOutputFormat not found
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:469)
> ... 8 more
> Caused by: java.lang.ClassNotFoundException: Class
> com.foo.test.TestOutputFormat not found
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
> ... 10 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
