[
https://issues.apache.org/jira/browse/HIVE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-15580:
-------------------------------
Description: Currently, orderBy (sortBy) and groupBy in Hive on Spark uses
unbounded memory. For orderBy, Hive accumulates key groups using ArrayList
(described in HIVE-15527). For groupBy, Hive currently uses Spark's groupByKey
operator, which has a shortcoming of not being able to spill to disk within a
key group. Thus, for large key group, memory usage is also unbounded.
> Eliminate unbounded memory usage for orderBy and groupBy in Hive on Spark
> -------------------------------------------------------------------------
>
> Key: HIVE-15580
> URL: https://issues.apache.org/jira/browse/HIVE-15580
> Project: Hive
> Issue Type: Improvement
> Components: Spark
> Reporter: Xuefu Zhang
> Assignee: Xuefu Zhang
> Attachments: HIVE-15580.1.patch, HIVE-15580.1.patch,
> HIVE-15580.2.patch, HIVE-15580.2.patch, HIVE-15580.3.patch,
> HIVE-15580.4.patch, HIVE-15580.5.patch, HIVE-15580.patch
>
>
> Currently, orderBy (sortBy) and groupBy in Hive on Spark uses unbounded
> memory. For orderBy, Hive accumulates key groups using ArrayList (described
> in HIVE-15527). For groupBy, Hive currently uses Spark's groupByKey operator,
> which has a shortcoming of not being able to spill to disk within a key
> group. Thus, for large key group, memory usage is also unbounded.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
