[ 
https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15525416#comment-15525416
 ] 

Rui Li commented on HIVE-14029:
-------------------------------

I tried the 5th patch locally. The jersey 2 stuff won't be pulled to lib, or 
the hive-exec jar. I also tried to identify the minimum set. I managed to run 
some simple queries (spark on yarn) with only {{scala-library, spark-core, 
spark-network-common, spark-network-shuffle}}. However, if we want to support 
local mode, we need more jars added to hive's lib, including the jersey 2. Then 
we may have conflict problem. Other than that, I think the 5th patch is enough 
for us (although I think we should exclude jersey 2 instead of just 
javax.ws.rs).

If we still want to go the way as the 6th, 7th patches, maybe we can look at 
how we handle the guava conflict in the pom of spark-client and do something 
similar.

> Update Spark version to 2.0.0
> -----------------------------
>
>                 Key: HIVE-14029
>                 URL: https://issues.apache.org/jira/browse/HIVE-14029
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Ferdinand Xu
>            Assignee: Ferdinand Xu
>         Attachments: HIVE-14029.1.patch, HIVE-14029.2.patch, 
> HIVE-14029.3.patch, HIVE-14029.4.patch, HIVE-14029.5.patch, 
> HIVE-14029.6.patch, HIVE-14029.7.patch, HIVE-14029.8.patch, HIVE-14029.patch
>
>
> There are quite some new optimizations in Spark 2.0.0. We need to bump up 
> Spark to 2.0.0 to benefit those performance improvements.
> To update Spark version to 2.0.0, the following changes are required:
> * Spark API updates:
> ** SparkShuffler#call return Iterator instead of Iterable
> ** SparkListener -> JavaSparkListener
> ** InputMetrics constructor doesn’t accept readMethod
> ** Method remoteBlocksFetched and localBlocksFetched in ShuffleReadMetrics 
> return long type instead of integer
> * Dependency upgrade:
> ** Jackson: 2.4.2 -> 2.6.5
> ** Netty version: 4.0.23.Final -> 4.0.29.Final
> ** Scala binary version: 2.10 -> 2.11
> ** Scala version: 2.10.4 -> 2.11.8



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to