[
https://issues.apache.org/jira/browse/HIVE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15836414#comment-15836414
]
Xuefu Zhang commented on HIVE-15580:
------------------------------------
Hi [~Ferd] and [~dapengsun], I'm wondering if you guys could help measure the
performance impact of the patch here? We at Uber don't have a dedicated
environment, so getting accurate measurement is challenging. It would be great
if you guys can help. Based on the result, we may have some followup work to
do. Thanks.
> Eliminate unbounded memory usage for orderBy and groupBy in Hive on Spark
> -------------------------------------------------------------------------
>
> Key: HIVE-15580
> URL: https://issues.apache.org/jira/browse/HIVE-15580
> Project: Hive
> Issue Type: Improvement
> Components: Spark
> Reporter: Xuefu Zhang
> Assignee: Xuefu Zhang
> Fix For: 2.2.0
>
> Attachments: HIVE-15580.1.patch, HIVE-15580.1.patch,
> HIVE-15580.2.patch, HIVE-15580.2.patch, HIVE-15580.3.patch,
> HIVE-15580.4.patch, HIVE-15580.5.patch, HIVE-15580.patch
>
>
> Currently, orderBy (sortBy) and groupBy in Hive on Spark uses unbounded
> memory. For orderBy, Hive accumulates key groups using ArrayList (described
> in HIVE-15527). For groupBy, Hive currently uses Spark's groupByKey operator,
> which has a shortcoming of not being able to spill to disk within a key
> group. Thus, for large key group, memory usage is also unbounded.
> It's likely that this will impact performance. We will profile and optimize
> afterwards. We could also make this change configurable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
