schenksj opened a new pull request, #12331:
URL: https://github.com/apache/arrow-rs/pull/12331

   # Which issue does this PR close?
   
   - Closes #12330.
   
   # Rationale for this change
   
   The DELTA_LENGTH_BYTE_ARRAY and DELTA_BYTE_ARRAY decoders size their length, 
prefix length and
   suffix length vectors from the count in each DELTA_BINARY_PACKED header 
without checking it, so a
   malformed page can make the reader allocate gigabytes, or panic with a 
capacity overflow, before
   it reads any value. Malformed input should return a `ParquetError`.
   
   # What changes are included in this PR?
   
   - `DeltaBitPackDecoder::checked_values_left(max_values)` (crate-private) 
returns the declared count
     only if it is at most `max_values` and at most what the remaining bytes 
could encode. Each block
     after the first value needs at least a min delta byte and one bit width 
byte per mini block. The
     decoder already requires these bytes when it reads the values. If a check 
fails, it returns an
     error. A zero block size is refused there too, instead of dividing by it.
   - `BitReader::bytes_left` (crate-private) supports the byte check.
   - Every decoder that allocates from a length stream count uses it, with the 
page's value count as
     `max_values`:
     - `encodings::decoding::DeltaLengthByteArrayDecoder::set_data` and
       `DeltaByteArrayDecoder::set_data` use the `num_values` they are given.
     - The arrow readers now pass `num_values.unwrap_or(num_levels)`, as the 
PLAIN decoders do. That
       is the non-null count of a v2 page and the level count otherwise, an 
upper bound on the
       entries in each length stream. These readers are 
`ByteArrayDecoderDeltaLength`,
       `ByteArrayDecoderDelta`, `ByteViewArrayDecoderDeltaLength`, 
`ByteViewArrayDecoderDelta` and the
       fixed length byte array `ValueDecoder`, through the crate-private
       `arrow::decoder::DeltaByteArrayDecoder::new(data, num_values)`.
   
   Conforming pages decode as before. Each check only refuses a count that the 
decoder would fail on
   later, or that holds more entries than the page has values.
   
   The DELTA_BINARY_PACKED decoder used for INT32/INT64 values is unchanged. It 
sizes nothing from the
   count.
   
   # Are these changes tested?
   
   Yes. The new tests feed each decoder a length, prefix or suffix stream that 
declares 2^31, 2^60 or
   one more value than the page holds, or more values than its bytes could 
encode. Each case must
   return an error. The decoders tested are the legacy decoders, the byte 
array, byte view and fixed
   length byte array readers, and the DELTA_BYTE_ARRAY suffix stream. The tests 
also check that a
   conforming page decodes unchanged. That includes a v1 page whose level count 
includes nulls, and a
   fixed length byte array DELTA_BYTE_ARRAY round trip through `ArrowWriter` 
with v1 and v2 pages.
   
   Without the fix:
   - The legacy decoder test is killed for running out of memory, under a 4 GiB 
cap, when it resizes
     to 2^31 entries.
   - The arrow reader tests take the 2^31 count as given, reserve 8 GiB and 
fail later with
     `EOF: Not enough data to decode 'min_delta'`.
   - The 2^60 count panics with a capacity overflow.
   
   `cargo test -p parquet --tests`, clippy with `-D warnings` on all targets, 
and `cargo fmt --check`
   pass.
   
   # Are there any user-facing changes?
   
   No public API change; the changed constructors and methods are 
crate-private. Malformed pages that
   used to abort or panic now return a `ParquetError`.
   
   # AI usage disclosure
   
   This change was prepared with an AI coding assistant; tests, clippy and fmt 
pass as described above.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to