Vitalii Diravka created PARQUET-1006:
----------------------------------------
Summary: ColumnChunkPageWriter uses only heap memory.
Key: PARQUET-1006
URL: https://issues.apache.org/jira/browse/PARQUET-1006
Project: Parquet
Issue Type: Bug
Components: parquet-mr
Affects Versions: 1.8.0
Reporter: Vitalii Diravka
Fix For: 1.9.0
After PARQUET-160 was resolved, ColumnChunkPageWriter started using
ConcatenatingByteArrayCollector. There are all data is collected in the List of
byte[], before writing the page. No way to use direct memory for allocating
buffers. ByteBufferAllocator is present in the ColumnChunkPageWriter class, but
never used.
Using of java heap space in some cases can cause OOM exceptions or GC's
overhead.
ByteBufferAllocator should be used in the ConcatenatingByteArrayCollector or
OutputStream classes.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
