[
https://issues.apache.org/jira/browse/HIVE-20032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541076#comment-16541076
]
Rui Li commented on HIVE-20032:
-------------------------------
Hi [~stakiar], thanks for working on this. There're other cases where RDD is
cached, e.g. parallel order by. So you need to serialize the hash code in all
these cases (maybe multi-insert is another one).
Having separate SerDe for caching and shuffling would be good. But I guess that
needs help from Spark side. And btw, have you run benchmarks to get the
improvements of this change?
> Don't serialize hashCode when groupByShuffle and RDD cacheing is disabled
> -------------------------------------------------------------------------
>
> Key: HIVE-20032
> URL: https://issues.apache.org/jira/browse/HIVE-20032
> Project: Hive
> Issue Type: Improvement
> Components: Spark
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
> Priority: Major
> Attachments: HIVE-20032.1.patch, HIVE-20032.2.patch,
> HIVE-20032.3.patch
>
>
> Follow up on HIVE-15104, if we don't enable RDD cacheing or groupByShuffles,
> then we don't need to serialize the hashCode when shuffling data in HoS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
