John Jenkins created PARQUET-852:
------------------------------------
Summary: Slowly ramp up sizes of byte[] in
ByteBasedBitPackingEncoder
Key: PARQUET-852
URL: https://issues.apache.org/jira/browse/PARQUET-852
Project: Parquet
Issue Type: Improvement
Reporter: John Jenkins
Priority: Minor
The current allocation policy for ByteBasedBitPackingEncoder is to allocate
64KB * #bits up-front. As similarly observed in [PARQUET-580], this can lead to
significant memory overheads for high-fanout scenarios (many columns and/or
open files, in my case using BooleanPlainValuesWriter).
As done in [PARQUET-585], I'll follow up with a PR that starts with a smaller
buffer and works its way up to a max.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
