schenksj opened a new issue, #12330:
URL: https://github.com/apache/arrow-rs/issues/12330
**Describe the bug**
The DELTA_LENGTH_BYTE_ARRAY and DELTA_BYTE_ARRAY decoders read the value
count from each
DELTA_BINARY_PACKED length header (lengths, prefix lengths, suffix lengths)
and size a `Vec<i32>`
from it before reading any lengths:
- `encodings::decoding::DeltaLengthByteArrayDecoder::set_data`
(`self.lengths.resize(num_lengths, 0)`)
- `encodings::decoding::DeltaByteArrayDecoder::set_data`
(`self.prefix_lengths.resize(num_prefixes, 0)`)
- `arrow::decoder::DeltaByteArrayDecoder::new` (`vec![0; num_prefix]`,
`vec![0; num_suffix]`), used by
the byte array, byte view and fixed length byte array readers
- `arrow::array_reader::byte_array::ByteArrayDecoderDeltaLength::new` and
`byte_view_array::ByteViewArrayDecoderDeltaLength::new` (`vec![0; values]`)
The count is not checked against the page's value count or against the
page's size. A page of a
few bytes declaring 2^60 values panics with a capacity overflow. A page
declaring 2^31 values asks
for 8 GiB, which aborts the process or gets it killed when memory is
limited, before the decoder
finds there is no data.
**To Reproduce**
A DELTA_LENGTH_BYTE_ARRAY page whose data is only a DELTA_BINARY_PACKED
header (block size 128,
4 mini blocks, total count `1 << 60`, first value 0), read through
`ByteArrayColumnValueDecoder`
with `set_data(Encoding::DELTA_LENGTH_BYTE_ARRAY, page, 4, Some(4))`.
**Expected behavior**
A `ParquetError` naming the declared and allowed counts, before anything is
sized from the count.
**Additional context**
Present in 59.3.1, 60.1.0 and `main`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]