seeker89 opened a new issue, #11186:
URL: https://github.com/apache/arrow-rs/issues/11186
**Is your feature request related to a problem or challenge? Please describe
what you are trying to do.**
#10979 bounds Thrift compact-protocol list sizes by the remaining input
before reserving capacity. It landed for 60.0.0, but the 59.x line doesn't have
it, and DataFusion 55 (the current release) depends on `parquet ^59.2`.
On 59.3.0, a Parquet footer that declares a huge list count (for example two
billion schema elements) makes the metadata decoder call `Vec::with_capacity`
for roughly 200 GB before it reads the elements. On Linux the allocation fails
and the process aborts ("memory allocation of 206158430112 bytes failed"), so a
corrupted or crafted file can take down a service that only tries to read its
metadata. On macOS the same file returns an ordinary error, because the
untouched reservation succeeds.
**Describe the solution you'd like**
Cherry-pick 7e3b403ac6481493b169c6f83144070f6352026b onto `59_maintenance`
and release 59.3.1. The commit applies cleanly to the `59.3.0` tag (one file,
`parquet/src/parquet_thrift.rs`).
**Describe alternatives you've considered**
Downstream users on DataFusion 55 can use `[patch.crates-io]` with a fork of
the 59.3.0 tag plus this commit, which is what we're doing for now, but a
59.3.1 release would let everyone on DataFusion 55 pick up the fix.
**Additional context**
Found by a regression test that opens a one-row Parquet file whose footer
list count was edited to an impossible value; it aborts on Linux with 59.3.0
and passes with the backport applied.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
