[
https://issues.apache.org/jira/browse/IMPALA-8710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880137#comment-16880137
]
Daniel Becker commented on IMPALA-8710:
---------------------------------------
The problem was the following:
The benchmarks showed that trying to read 64 bits instead of 32 bits when
possible produced better results. The strange behaviour with odd bit widths
between 9 and 16 does not occur if we try to read 64 bits.
The buffer overflow occurs if we are not reading a full batch of 32 values but
up to 31 values because in that case the buffer may be shorter than the code
expects. By introducing a template parameter indicating whether it is a full
batch read, we can keep the efficiency in the full batch case and avoid the
buffer overflow in the "up to 31 values" case by falling back to reading 32
bits. The benchmarks show that the performance is about the same as the
original for bit widths 0-12 and better afterwards.
> Increase allowed bit width to 64 for bit packing
> ------------------------------------------------
>
> Key: IMPALA-8710
> URL: https://issues.apache.org/jira/browse/IMPALA-8710
> Project: IMPALA
> Issue Type: Sub-task
> Components: Backend
> Reporter: Daniel Becker
> Assignee: Daniel Becker
> Priority: Major
>
> Increase the allowed bit width for bit packing and bit unpacking to 64
> bits. This is needed to support Parquet Delta Encoding.
>
> Also add new methods to BitWriter and BatchedBitReader handling Uleb and
> ZigZag integers for 64 bits.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
