Brian Mwambazi created PARQUET-1768:
---------------------------------------
Summary: InternalParquetRecordWriter doesn't immediately limit
current row group to thres
Key: PARQUET-1768
URL: https://issues.apache.org/jira/browse/PARQUET-1768
Project: Parquet
Issue Type: Bug
Components: parquet-mr
Reporter: Brian Mwambazi
The MemoryManager adjust the row group size threshold of writers when the
allocated memory pool fills up.
*Problem*: However InternalParquetRecordWriter only re-adjusts the row group
size on the next flush meaning they still use the old size.
This opens up a possibility of getting an OOM error if all writers are started
at relatively the same time and progress in tandem(I saw this when
investigating failing jobs while writing to disk in Spark)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
