vishnuprakaz opened a new pull request, #16880:
URL: https://github.com/apache/iceberg/pull/16880
### Rationale for this change
For each column, Iceberg records the smallest and largest value in a file
so queries can
skip files that can't contain a match. Long values are shortened to 16
bytes to save space
but when shortening the **largest** value, it has to round **up**, so the
stored "max" stays
at least as big as the real maximum. (If it rounded down, the stored max
could be smaller
than a real value, which breaks file skipping.)
For `BINARY` values inside a Variant column, this rounds the wrong way:
the upper bound is
shortened with `truncateBinaryMin` (rounds down) instead of
`truncateBinaryMax` (rounds up).
So the stored max can come out **smaller than a value actually in the
file**, and a query like
`binaryColumn > X` may decide the file can't match and skip it silently
returning wrong
(missing) results. The lower bound is correct.
This also disagrees with the code around it: the `STRING` branch uses
`truncateStringMax`,
the lower-bound method uses `truncateBinaryMin`, and the non-variant
`ParquetMetrics.truncateUpperBound` uses `truncateBinaryMax`. Introduced
in #12496.
### What changes are included in this PR?
Use `truncateBinaryMax` for the BINARY upper bound. One-line change.
### Are these changes tested?
Yes. `TestVariantMetrics.testShreddedBinaryBoundsTruncation` writes a
binary Variant value
longer than 16 bytes and checks the upper bound rounds up. Fails before
the fix (`…0F10`,
smaller than the value), passes after. Existing tests only used a 4-byte
value (under the
16-byte threshold), so the path was untested.
### Are there any user-facing changes?
No API changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
