[
https://issues.apache.org/jira/browse/HIVE-7685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151792#comment-14151792
]
Brock Noland edited comment on HIVE-7685 at 9/29/14 3:31 PM:
-------------------------------------------------------------
Hi Dong,
Ok, thank you for the investigation. I think we can either put the parquet
memory manager in Parquet or add API's to expose the information required to
implement the memory manager in Hive. Either approach is fine by me, we can
take this work up in PARQUET-108.
Brock
was (Author: brocknoland):
Hi Dong,
Ok, thank you for the investigation. I think we can either put the parquet
memory manager in Parquet or add API's to expose the information required to
implement the memory manager in HIve. Either approach is fine by me, we can
take this work up in PARQUET-108.
Brock
> Parquet memory manager
> ----------------------
>
> Key: HIVE-7685
> URL: https://issues.apache.org/jira/browse/HIVE-7685
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
> Reporter: Brock Noland
>
> Similar to HIVE-4248, Parquet tries to write large very large "row groups".
> This causes Hive to run out of memory during dynamic partitions when a
> reducer may have many Parquet files open at a given time.
> As such, we should implement a memory manager which ensures that we don't run
> out of memory due to writing too many row groups within a single JVM.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
