M-Tesla opened a new pull request, #10979: URL: https://github.com/apache/arrow-rs/pull/10979
# Which issue does this PR close? - Closes #10920. # Rationale for this change `read_thrift_vec` reserved `Vec` capacity from the compact list header's declared element count before reading any element. A 23-byte footer whose `schema` list claimed ~109 million `SchemaElement`s asked for ~10 GB. Allocation failure aborts the process; `ParquetMetaDataReader` never returns an error. Each list element occupies at least one byte on the wire, so the declared size can be bounded by the bytes remaining in the metadata slice. That slice is already length-limited by the footer; this check is the level below. # What changes are included in this PR? - `ThriftCompactInputProtocol::remaining_bytes` reports unread length for slice-backed input (`ThriftSliceInputProtocol`). Stream-backed input (`ThriftReadInputProtocol`) leaves it unset. - `read_thrift_vec` returns `ParquetError::General` when the declared size exceeds remaining input, and only then calls `Vec::with_capacity`. No public API change. # Are these changes tested? Unit tests in `parquet_thrift`: - well-formed 2-element `i32` list still decodes - 14-element header with no payload returns an error - declared size 109_002_364 with two leftover bytes returns an error (the schema-list case from the issue) `cargo test -p parquet --lib parquet_thrift` and `cargo clippy -p parquet --lib --tests --all-features -- -D warnings`. # Are there any user-facing changes? Malformed footers that previously aborted now return `ParquetError`. Valid files are unchanged. # AI Disclosure Assisted draft of the remaining-bytes check and regression tests. Reviewed against the compact protocol list encoding and verified with the tests above. Made with [Cursor](https://cursor.com) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
