CompositeRecordReader: ClassLoader set in JobConf is not passed onto
WrappedRecordReaders
-----------------------------------------------------------------------------------------
Key: HADOOP-4975
URL: https://issues.apache.org/jira/browse/HADOOP-4975
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Affects Versions: 0.19.0
Reporter: Jingkei Ly
I am using a custom ClassLoader which I set in my JobConf via setClassLoader().
The ClassLoader is loaded key and value classes which are required to read
records from SequenceFiles that were written out in a previous MapReduce job.
However, I am getting a ClassNotFoundException when using the
CompositeInputFormat to create a RecordReader to read these SequenceFiles from
HDFS. It occurs when the SequenceFile.Reader tries to create an instance of the
Key/Value classes, presumably because the class loader SequenceFile.Reader is
using is not the one I set with JobConf.setClassLoader. Below is an example of
the stack trace I get:
{code}
Caused by: java.io.IOException: WritableName can't load class
at org.apache.hadoop.io.WritableName.getClass(WritableName.java:73)
at
org.apache.hadoop.io.SequenceFile$Reader.getKeyClass(SequenceFile.java:1596)
... 33 more
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.MyWritableClass
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)
at org.apache.hadoop.io.WritableName.getClass(WritableName.java:71)
... 34 more
{code}
I'll attach a unit test that can demonstrate this more clearly....
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
