[
https://issues.apache.org/jira/browse/HIVE-20032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16554502#comment-16554502
]
Sahil Takiar edited comment on HIVE-20032 at 7/24/18 4:43 PM:
--------------------------------------------------------------
I originally thought that {{\-\-jars}} would add the specified jars to the
executor and driver class path, but apparently thats not the case. According to
https://stackoverflow.com/questions/37132559/add-jars-to-a-spark-job-spark-submit?answertab=votes#tab-top
you have to add the jars manually. It seems that {{\-\-jars}} will just copy
the jar file to the local fs of each executor (similar to MapReduce's
DistributedCache), but the jar has to be explicitly added to the driver /
executor classpath.
I think HIVE-15104 works because there isn't any Hive code that is calling a
class inside {{hive-kryo-registrator}}. It's only the Spark code. If you look
at the code in Spark's {{KryoSerializer}} you see that Spark sets up some
special classloader before calling the class specified in
{{spark.kryo.registrator}}.
{quote} driver extra class path only takes jars on local FS. So will that be a
problem for cluster mode {quote} The {{\-\-jars}} option copies the jar to the
local fs of the driver and executor, it gets copied to {{$PWD}}
was (Author: stakiar):
I originally thought that {{\-\-jars}} would add the specified jars to the
executor and driver class path, but apparently thats not the case. According to
https://stackoverflow.com/questions/37132559/add-jars-to-a-spark-job-spark-submit?answertab=votes#tab-top
you have to add the jars manually. It seems that {{\-\-jars}} will just copy
the jar file to the local fs of each executor (similar to MapReduce's
DistributedCache), but the jar has to be explicitly added to the driver /
executor classpath.
I think HIVE-15104 works because there isn't any Hive code that is calling a
class inside {{hive-kryo-registrator}}. It's only the Spark code. If you look
at the code in Spark's {{KryoSerializer}} you see that Spark sets up some
special classloader before calling the class specified in
{{spark.kryo.registrator}}.
> Don't serialize hashCode for repartitionAndSortWithinPartitions
> ---------------------------------------------------------------
>
> Key: HIVE-20032
> URL: https://issues.apache.org/jira/browse/HIVE-20032
> Project: Hive
> Issue Type: Improvement
> Components: Spark
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
> Priority: Major
> Attachments: HIVE-20032.1.patch, HIVE-20032.2.patch,
> HIVE-20032.3.patch, HIVE-20032.4.patch, HIVE-20032.5.patch,
> HIVE-20032.6.patch, HIVE-20032.7.patch, HIVE-20032.8.patch, HIVE-20032.9.patch
>
>
> Follow up on HIVE-15104, if we don't enable RDD cacheing or groupByShuffles,
> then we don't need to serialize the hashCode when shuffling data in HoS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
