[
https://issues.apache.org/jira/browse/SPARK-24771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16579213#comment-16579213
]
Marcelo Vanzin edited comment on SPARK-24771 at 8/14/18 3:46 AM:
-----------------------------------------------------------------
The main problem pointed out in the original attempt is AVRO-1502; it means
that people with code generated by Avro 1.7 might run into problems if Spark
ships with Avro 1.8; that basically amounts to a binary compatibility issue,
which we always try not to break in minor releases.
It may be that it only applies in some specific situations and it may be
acceptable to release note it. But it would be nice to be sure, since this
affects existing users of Avro - including the Flume streaming connector which
is still available in Spark 2.4...
(Edit: the flume connector re-compiles the avro interface during the Spark
build, but it would be nice to check that the flume libraries themselves don't
use avro for anything. And we still should check how much other users of avro
would be affected - e.g. people who use avro directly in RDD code.)
was (Author: vanzin):
The main problem pointed out in the original attempt is AVRO-1502; it means
that people with code generated by Avro 1.7 might run into problems if Spark
ships with Avro 1.8; that basically amounts to a binary compatibility issue,
which we always try not to break in minor releases.
It may be that it only applies in some specific situations and it may be
acceptable to release note it. But it would be nice to be sure, since this
affects existing users of Avro - including the Flume streaming connector which
is still available in Spark 2.4...
> Upgrade AVRO version from 1.7.7 to 1.8
> --------------------------------------
>
> Key: SPARK-24771
> URL: https://issues.apache.org/jira/browse/SPARK-24771
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 2.4.0
> Reporter: Gengliang Wang
> Assignee: Gengliang Wang
> Priority: Major
> Fix For: 2.4.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
