[
https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15583356#comment-15583356
]
Xuefu Zhang commented on HIVE-14029:
------------------------------------
[~spena], Keeping b/c is a good thing in general. Before we take the effort
(which seems a lot) to do it, I think we should clearly understand and define
what b/c is in this case. Spark is rapidly releasing w/o much b/c in mind. So
far, Hive on Spark has once depended on Spark 1.2, 1.3, 1.4, 1.5, and 1.6. I'm
not sure what versions of Spark Hive has been released with, but one thing is
clear, Spark isn't b/c between these releases. Before Spark community has a
good sense of keeping b/c in their APIs, it's going to be very hard and
burdensome for Hive to maintain support for different Spark releases, not to
mention the library dependency issues we have had.
I'm okay to start thinking of a shim layer to support multiple versions of
Spark, but it sounds daunting to me due to the dynamics of Spark project.
> Update Spark version to 2.0.0
> -----------------------------
>
> Key: HIVE-14029
> URL: https://issues.apache.org/jira/browse/HIVE-14029
> Project: Hive
> Issue Type: Bug
> Reporter: Ferdinand Xu
> Assignee: Ferdinand Xu
> Labels: Incompatible, TODOC2.2
> Fix For: 2.2.0
>
> Attachments: HIVE-14029.1.patch, HIVE-14029.2.patch,
> HIVE-14029.3.patch, HIVE-14029.4.patch, HIVE-14029.5.patch,
> HIVE-14029.6.patch, HIVE-14029.7.patch, HIVE-14029.8.patch, HIVE-14029.patch
>
>
> There are quite some new optimizations in Spark 2.0.0. We need to bump up
> Spark to 2.0.0 to benefit those performance improvements.
> To update Spark version to 2.0.0, the following changes are required:
> * Spark API updates:
> ** SparkShuffler#call return Iterator instead of Iterable
> ** SparkListener -> JavaSparkListener
> ** InputMetrics constructor doesn’t accept readMethod
> ** Method remoteBlocksFetched and localBlocksFetched in ShuffleReadMetrics
> return long type instead of integer
> * Dependency upgrade:
> ** Jackson: 2.4.2 -> 2.6.5
> ** Netty version: 4.0.23.Final -> 4.0.29.Final
> ** Scala binary version: 2.10 -> 2.11
> ** Scala version: 2.10.4 -> 2.11.8
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
