[
https://issues.apache.org/jira/browse/PARQUET-1767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brian Mwambazi updated PARQUET-1767:
------------------------------------
Summary: InternalParquetRecordWriter doesn't immediately limit current row
group to threshold (was: InternalParquetRecordWriter doesn't immediately limit
current row group to thres)
> InternalParquetRecordWriter doesn't immediately limit current row group to
> threshold
> ------------------------------------------------------------------------------------
>
> Key: PARQUET-1767
> URL: https://issues.apache.org/jira/browse/PARQUET-1767
> Project: Parquet
> Issue Type: Bug
> Components: parquet-mr
> Reporter: Brian Mwambazi
> Priority: Minor
>
> The MemoryManager adjust the row group size threshold of writers when the
> allocated memory pool fills up.
> *Problem*: However InternalParquetRecordWriter only re-adjusts the row group
> size on the next flush meaning they still use the old size.
> This opens up a possibility of getting an OOM error if all writers are
> started at relatively the same time and progress in tandem(I saw this when
> investigating failing jobs while writing to disk in Spark)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
