schenksj opened a new pull request, #12331:
URL: https://github.com/apache/arrow-rs/pull/12331
# Which issue does this PR close?
- Closes #12330.
# Rationale for this change
The DELTA_LENGTH_BYTE_ARRAY and DELTA_BYTE_ARRAY decoders size their length,
prefix length and
suffix length vectors from the count in each DELTA_BINARY_PACKED header
without checking it, so a
malformed page can make the reader allocate gigabytes, or panic with a
capacity overflow, before
it reads any value. Malformed input should return a `ParquetError`.
# What changes are included in this PR?
- `DeltaBitPackDecoder::checked_values_left(max_values)` (crate-private)
returns the declared count
only if it is at most `max_values` and at most what the remaining bytes
could encode. Each block
after the first value needs at least a min delta byte and one bit width
byte per mini block. The
decoder already requires these bytes when it reads the values. If a check
fails, it returns an
error. A zero block size is refused there too, instead of dividing by it.
- `BitReader::bytes_left` (crate-private) supports the byte check.
- Every decoder that allocates from a length stream count uses it, with the
page's value count as
`max_values`:
- `encodings::decoding::DeltaLengthByteArrayDecoder::set_data` and
`DeltaByteArrayDecoder::set_data` use the `num_values` they are given.
- The arrow readers now pass `num_values.unwrap_or(num_levels)`, as the
PLAIN decoders do. That
is the non-null count of a v2 page and the level count otherwise, an
upper bound on the
entries in each length stream. These readers are
`ByteArrayDecoderDeltaLength`,
`ByteArrayDecoderDelta`, `ByteViewArrayDecoderDeltaLength`,
`ByteViewArrayDecoderDelta` and the
fixed length byte array `ValueDecoder`, through the crate-private
`arrow::decoder::DeltaByteArrayDecoder::new(data, num_values)`.
Conforming pages decode as before. Each check only refuses a count that the
decoder would fail on
later, or that holds more entries than the page has values.
The DELTA_BINARY_PACKED decoder used for INT32/INT64 values is unchanged. It
sizes nothing from the
count.
# Are these changes tested?
Yes. The new tests feed each decoder a length, prefix or suffix stream that
declares 2^31, 2^60 or
one more value than the page holds, or more values than its bytes could
encode. Each case must
return an error. The decoders tested are the legacy decoders, the byte
array, byte view and fixed
length byte array readers, and the DELTA_BYTE_ARRAY suffix stream. The tests
also check that a
conforming page decodes unchanged. That includes a v1 page whose level count
includes nulls, and a
fixed length byte array DELTA_BYTE_ARRAY round trip through `ArrowWriter`
with v1 and v2 pages.
Without the fix:
- The legacy decoder test is killed for running out of memory, under a 4 GiB
cap, when it resizes
to 2^31 entries.
- The arrow reader tests take the 2^31 count as given, reserve 8 GiB and
fail later with
`EOF: Not enough data to decode 'min_delta'`.
- The 2^60 count panics with a capacity overflow.
`cargo test -p parquet --tests`, clippy with `-D warnings` on all targets,
and `cargo fmt --check`
pass.
# Are there any user-facing changes?
No public API change; the changed constructors and methods are
crate-private. Malformed pages that
used to abort or panic now return a `ParquetError`.
# AI usage disclosure
This change was prepared with an AI coding assistant; tests, clippy and fmt
pass as described above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]