[
https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15525416#comment-15525416
]
Rui Li commented on HIVE-14029:
-------------------------------
I tried the 5th patch locally. The jersey 2 stuff won't be pulled to lib, or
the hive-exec jar. I also tried to identify the minimum set. I managed to run
some simple queries (spark on yarn) with only {{scala-library, spark-core,
spark-network-common, spark-network-shuffle}}. However, if we want to support
local mode, we need more jars added to hive's lib, including the jersey 2. Then
we may have conflict problem. Other than that, I think the 5th patch is enough
for us (although I think we should exclude jersey 2 instead of just
javax.ws.rs).
If we still want to go the way as the 6th, 7th patches, maybe we can look at
how we handle the guava conflict in the pom of spark-client and do something
similar.
> Update Spark version to 2.0.0
> -----------------------------
>
> Key: HIVE-14029
> URL: https://issues.apache.org/jira/browse/HIVE-14029
> Project: Hive
> Issue Type: Bug
> Reporter: Ferdinand Xu
> Assignee: Ferdinand Xu
> Attachments: HIVE-14029.1.patch, HIVE-14029.2.patch,
> HIVE-14029.3.patch, HIVE-14029.4.patch, HIVE-14029.5.patch,
> HIVE-14029.6.patch, HIVE-14029.7.patch, HIVE-14029.8.patch, HIVE-14029.patch
>
>
> There are quite some new optimizations in Spark 2.0.0. We need to bump up
> Spark to 2.0.0 to benefit those performance improvements.
> To update Spark version to 2.0.0, the following changes are required:
> * Spark API updates:
> ** SparkShuffler#call return Iterator instead of Iterable
> ** SparkListener -> JavaSparkListener
> ** InputMetrics constructor doesn’t accept readMethod
> ** Method remoteBlocksFetched and localBlocksFetched in ShuffleReadMetrics
> return long type instead of integer
> * Dependency upgrade:
> ** Jackson: 2.4.2 -> 2.6.5
> ** Netty version: 4.0.23.Final -> 4.0.29.Final
> ** Scala binary version: 2.10 -> 2.11
> ** Scala version: 2.10.4 -> 2.11.8
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)