[
https://issues.apache.org/jira/browse/HADOOP-12527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15384287#comment-15384287
]
Sean Busbey commented on HADOOP-12527:
--------------------------------------
This ends up being related to a similar request in HBase (and probably Spark),
esp if we're talking about going to 1.8.z. (which I'd prefer).
HBase being on 1.7.6 is probably an accident and not in any release. FWIW, I'll
be trying to update HBase to match the version shipping with whatever version
of Hadoop we default to when HBase 2.0.0 RCs come around (I hope that will be a
Hadoop 3 of some stripe).
As I mentioned on that HBase JIRA, the two relevant breaking changes in Avro
1.8 AFAICT are AVRO-1502 and AVRO-997. I believe I fixed Hive ages ago wrt
AVRO-997. I believe HBase's (currently unreleased) Spark SQL-over-Avro-in-HBase
code is currently incompatible with AVRO-997 (but should be fixable). I haven't
examined any other Spark use.
> Upgrade Avro dependency to 1.7.7 or later
> -----------------------------------------
>
> Key: HADOOP-12527
> URL: https://issues.apache.org/jira/browse/HADOOP-12527
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Affects Versions: 2.7.1
> Reporter: Jonathan Kelly
>
> Hadoop has depended upon Avro 1.7.4 for a couple of years now (see
> HADOOP-9672), but Apache Spark depends upon what is currently the latest
> version of Avro (1.7.7).
> This can cause issues if Spark is configured to include the full Hadoop
> classpath, as the classpath would then contain both Avro 1.7.4 and 1.7.7,
> with the 1.7.4 classes possibly winning depending on ordering. Here is an
> example of this issue:
> http://stackoverflow.com/questions/33159254/avro-error-on-aws-emr/33403111#33403111
> Would it be possible to upgrade Hadoop's Avro dependency to 1.7.7 now?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
