[
https://issues.apache.org/jira/browse/HIVE-20032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16554502#comment-16554502
]
Sahil Takiar commented on HIVE-20032:
-------------------------------------
I originally thought that {{\-\-jars}} would add the specified jars to the
executor and driver class path, but apparently thats not the case. According to
https://stackoverflow.com/questions/37132559/add-jars-to-a-spark-job-spark-submit?answertab=votes#tab-top
you have to add the jars manually. It seems that {{\-\-jars}} will just copy
the jar file to the local fs of each executor (similar to MapReduce's
DistributedCache), but the jar has to be explicitly added to the driver /
executor classpath.
I think HIVE-15104 works because there isn't any Hive code that is calling a
class inside {{hive-kryo-registrator}}. It's only the Spark code. If you look
at the code in Spark's {{KryoSerializer}} you see that Spark sets up some
special classloader before calling the class specified in
{{spark.kryo.registrator}}.
> Don't serialize hashCode for repartitionAndSortWithinPartitions
> ---------------------------------------------------------------
>
> Key: HIVE-20032
> URL: https://issues.apache.org/jira/browse/HIVE-20032
> Project: Hive
> Issue Type: Improvement
> Components: Spark
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
> Priority: Major
> Attachments: HIVE-20032.1.patch, HIVE-20032.2.patch,
> HIVE-20032.3.patch, HIVE-20032.4.patch, HIVE-20032.5.patch,
> HIVE-20032.6.patch, HIVE-20032.7.patch, HIVE-20032.8.patch, HIVE-20032.9.patch
>
>
> Follow up on HIVE-15104, if we don't enable RDD cacheing or groupByShuffles,
> then we don't need to serialize the hashCode when shuffling data in HoS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
