[
https://issues.apache.org/jira/browse/AVRO-1103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423101#comment-13423101
]
Steven Willis commented on AVRO-1103:
-------------------------------------
I just came from AVRO-1123 which addresses a subset of this issue. I'm still
having problems with avro when my generated class can only be found via an
alternate ClassLoader. This patch should fix my issues. I'd appreciate it if
this jira isn't closed, but the patch applied.
A simple way to test the issue I'm having is to put a generated avro class in a
jar along with another class which tries to read from an avro file containing
records of your class, then run:
$ hadoop jar myjar.jar MainClass input_file.avro
You'll get a generic back from the reader instead of your generated class
because RunJar uses an alternate ClassLoader to run the jar. However if you run:
$ HADOOP_CLASSPATH=myjar.jar hadoop jar myjar.jar MainClass input_file.avro
Then you'll get what you expect because now the jar containing the avro class
is also accessible via the standard ClassLoader since it's now on the classpath.
> New AvroDeserializer should Locate Appropriate Classloader
> ----------------------------------------------------------
>
> Key: AVRO-1103
> URL: https://issues.apache.org/jira/browse/AVRO-1103
> Project: Avro
> Issue Type: Improvement
> Components: java
> Affects Versions: 1.7.0
> Environment: Hadoop 0.23.1 with Avro jars replaced by 1.7 jars
> Specific data classes assembled into JAR with mapper/reducer
> Reporter: Jacob Metcalf
> Assignee: Doug Cutting
> Attachments: AVRO-1103-for 0.23.1.patch, AVRO-1103.patch,
> AVRO-1103.patch, AVRO-1103.patch, AvroCDH4.patch
>
>
> Continuing on from AVRO-873 I believe some more work needs to be done to get
> the MapReduce 2 APIs in Avro 1.7 working with Hadoop 0.23. Since it revolves
> around classloaders it is complex to present a unit test which fails so I
> will explain the problem:
> - By default SpecificDatumReader will use the classloader it was loaded from
> to find a Specific class to deserialize into.
> - In earlier versions of Hadoop e.g. 0.20.2 Avro was not included so
> typically you would bundle Avro into your job jar along with the Specific
> classes so they would be on the same classpath.
>
> - However later versions of Hadoop such as 0.23 ship with Avro. Thus you find
> that the SpecificData.class.getClassloader() is typically a parent loader
> which just contains Hadoop components.
> - Thus when SpecificData goes to construct a Specific class from the schema
> it cannot locate it and silently defaults to creating a GenericData.
> In AVRO-873 an additional constructor was added to SpecificData to force it
> to use a different classloader. Thus to extend this fix to the new MR2 APIs:
> - AvroDeserializer could attempt to instantiate the class using
> Class.forName() and from this get the appropriate Classloader and pass this
> into the constructor of SpecificDatumReader.
> - Line 2771 of SpecificData.java is:
> bq. Class c = SpecificData.get().getClass(schema);
> - This would need to be changed to:
> bq. Class c = this.getClass(schema);
> I have raised this in the mail groups here:
> http://search-hadoop.com/m/wVUf1aLCwd/classloader/v=threaded so apologies if
> this is already being thought about.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
