[ https://issues.apache.org/jira/browse/PARQUET-852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Julien Le Dem resolved PARQUET-852. ----------------------------------- Resolution: Fixed Fix Version/s: 1.10.0 Issue resolved by pull request 401 [https://github.com/apache/parquet-mr/pull/401] > Slowly ramp up sizes of byte[] in ByteBasedBitPackingEncoder > ------------------------------------------------------------ > > Key: PARQUET-852 > URL: https://issues.apache.org/jira/browse/PARQUET-852 > Project: Parquet > Issue Type: Improvement > Reporter: John Jenkins > Priority: Minor > Fix For: 1.10.0 > > > The current allocation policy for ByteBasedBitPackingEncoder is to allocate > 64KB * #bits up-front. As similarly observed in [PARQUET-580], this can lead > to significant memory overheads for high-fanout scenarios (many columns > and/or open files, in my case using BooleanPlainValuesWriter). > As done in [PARQUET-585], I'll follow up with a PR that starts with a smaller > buffer and works its way up to a max. -- This message was sent by Atlassian JIRA (v6.3.15#6346)