[
https://issues.apache.org/jira/browse/PARQUET-852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15848702#comment-15848702
]
John Jenkins commented on PARQUET-852:
--------------------------------------
PRs at https://github.com/apache/parquet-mr/pull/401 and
https://github.com/apache/parquet-mr/pull/402 for 1.8.x. Same patch applies
cleanly on both.
> Slowly ramp up sizes of byte[] in ByteBasedBitPackingEncoder
> ------------------------------------------------------------
>
> Key: PARQUET-852
> URL: https://issues.apache.org/jira/browse/PARQUET-852
> Project: Parquet
> Issue Type: Improvement
> Reporter: John Jenkins
> Priority: Minor
>
> The current allocation policy for ByteBasedBitPackingEncoder is to allocate
> 64KB * #bits up-front. As similarly observed in [PARQUET-580], this can lead
> to significant memory overheads for high-fanout scenarios (many columns
> and/or open files, in my case using BooleanPlainValuesWriter).
> As done in [PARQUET-585], I'll follow up with a PR that starts with a smaller
> buffer and works its way up to a max.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
