[
https://issues.apache.org/jira/browse/PARQUET-852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julien Le Dem resolved PARQUET-852.
-----------------------------------
Resolution: Fixed
Fix Version/s: 1.10.0
Issue resolved by pull request 401
[https://github.com/apache/parquet-mr/pull/401]
> Slowly ramp up sizes of byte[] in ByteBasedBitPackingEncoder
> ------------------------------------------------------------
>
> Key: PARQUET-852
> URL: https://issues.apache.org/jira/browse/PARQUET-852
> Project: Parquet
> Issue Type: Improvement
> Reporter: John Jenkins
> Priority: Minor
> Fix For: 1.10.0
>
>
> The current allocation policy for ByteBasedBitPackingEncoder is to allocate
> 64KB * #bits up-front. As similarly observed in [PARQUET-580], this can lead
> to significant memory overheads for high-fanout scenarios (many columns
> and/or open files, in my case using BooleanPlainValuesWriter).
> As done in [PARQUET-585], I'll follow up with a PR that starts with a smaller
> buffer and works its way up to a max.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
