[
https://issues.apache.org/jira/browse/LUCENE-5721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014707#comment-14014707
]
Adrien Grand commented on LUCENE-5721:
--------------------------------------
I'm afraid the integer average could trigger some worst-cases space-wise. For
example let's imagine that half your binary doc values have a length of 4 and
the other half has a length of 5: the average length is 4.5 and we use a
monotonic writer to store offsets.
With an integer average you need to approximate the length with 4 or 5. This
means that your maximum delta will be 0.5 * BLOCK_SIZE = 8192 with the current
encoding, that is 14 bits per value (13 + 1 for zig-zag). With a float average
there is a worst-case that typically happens if a block stores all 4s first and
all 5s afterwards (max delta occurs at BLOCK_SIZE / 2 and is then 0.5 *
BLOCK_SIZE / 2 = 4096) but if your 4s and 5s and more evenly distributed
inside a block (which I expect to be quite likely), then you would only need a
few bits per value.
> Monotonic packed could maybe be faster
> --------------------------------------
>
> Key: LUCENE-5721
> URL: https://issues.apache.org/jira/browse/LUCENE-5721
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Robert Muir
> Assignee: Adrien Grand
>
> This compression is used in lucene for monotonically increasing offsets, e.g.
> stored fields index, dv BINARY/SORTED_SET offsets, OrdinalMap (used for
> merging and faceting dv) and so on.
> Today this stores a +/- deviation from an expected line of y=mx + b, where b
> is the minValue for the block and m is the average delta from the previous
> value. Because it can be negative, we have to do some additional work to
> zigzag-decode.
> Can we just instead waste a bit for every value explicitly (lower the
> minValue by the min delta) so that deltas are always positive and we can have
> a simpler decode? Maybe If we do this, the new guy should assert that values
> are actually monotic at write-time. The current one supports "mostly monotic"
> but do we really need that flexibility anywhere? If so it could always be
> kept...
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
