pmoust opened a new pull request, #11419:
URL: https://github.com/apache/arrow-rs/pull/11419
# Which issue does this PR close?
Closes #11418.
# Rationale for this change
Decoding a Parquet footer reserves a `Vec` sized from the declared
row-group count before reading any row group:
```rust
let mut rg_vec = Vec::with_capacity(list_ident.size as usize);
```
`list_ident.size` comes straight off the wire and `RowGroupMetaData` is a
large struct, so a footer of a few bytes that declares a huge row-group
count asks for `count * size_of::<RowGroupMetaData>()` up front, before a
single row group is read. On a host under memory pressure that allocation
aborts the process.
PR #10979 bounded `read_thrift_vec` by the remaining input. The row-group
list is decoded inline in `parquet_metadata_from_bytes` and does not go
through that helper, so the bound did not reach it. The other footer lists
(schema elements, column chunks, sorting columns, key/values, column
orders) do go through `read_thrift_vec`. This is the one that did not.
# What changes are included in this PR?
- Apply the same remaining-input bound inline at the row-group list: each
row group is at least one byte on the wire, so a declared count larger
than the remaining input is malformed and is rejected before the
reserve.
- A test (`row_group_count_exceeding_remaining_input_is_rejected`) builds a
valid footer prefix and declares an over-sized `row_groups` list, then
asserts the decode reports the bound. Without the bound the same input
instead fails later with `Required field columns is missing`, so the
test is shown failing against the unfixed code.
# Are these changes tested?
Yes. The new test passes with the bound and fails without it, as described
above.
# Are there any user-facing changes?
A malformed footer that declares more row groups than its bytes can hold
now returns a `Parquet error` describing the bound, rather than attempting
a large allocation. This was found while hardening the Parquet
external-table reader in [Ravel](https://github.com/NOFireAI/ravel).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
