[
https://issues.apache.org/jira/browse/PARQUET-1767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated PARQUET-1767:
------------------------------------
Labels: pull-request-available (was: )
> InternalParquetRecordWriter doesn't immediately limit current row group to
> threshold
> ------------------------------------------------------------------------------------
>
> Key: PARQUET-1767
> URL: https://issues.apache.org/jira/browse/PARQUET-1767
> Project: Parquet
> Issue Type: Bug
> Components: parquet-mr
> Reporter: Brian Mwambazi
> Priority: Minor
> Labels: pull-request-available
>
> The MemoryManager adjust the row group size threshold of writers when the
> allocated memory pool fills up.
> *Problem*: However InternalParquetRecordWriter only re-adjusts the row group
> size on the next flush meaning they still use the old size.
> This opens up a possibility of getting an OOM error if all writers are
> started at relatively the same time and progress in tandem(I saw this when
> investigating failing jobs while writing to disk in Spark)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
