geoHeil opened a new issue, #12030: URL: https://github.com/apache/arrow-rs/issues/12030
**Describe the bug** `arrow-ipc` sizes its decompression buffer from the 8-byte uncompressed length the sender writes in front of each compressed body buffer, before it has decompressed a byte. A peer that declares `i64::MAX` makes the reader request that much memory, and the allocation failure aborts the process. `panic = "unwind"` and `catch_unwind` do not help, because an allocation failure is not a panic. On `main` (e2c48ef0dd83fa464d643e25f8e9f9177ca7316e), `arrow-ipc/src/compression.rs`: - `decompress_lz4` (l.312): `Vec::with_capacity(decompressed_size)`. - `decompress_zstd` (l.372-374): `zstd::bulk::Decompressor::decompress(input, decompressed_size)`. In zstd 0.14.3, that reserves the frame's recorded content size when the frame has one. Otherwise it reserves `decompressed_size` (`Vec::with_capacity`), unless zstd's `experimental` feature supplies a bound. zstd-rs master (c64aa2854c, unreleased) turns that failure into an error. Either way, the reservation is still sized by values the sender chose. `decompressed_size` is the sender's value after `usize::try_from` (l.242). Nothing bounds it by the body length. #10959 limited how far lz4's `read_to_end` reads, and #11881 bounds-checked buffer offsets and lengths. Neither changes this up-front reservation. #10734 is the same pattern in `parquet`'s `SerializedPageReader`. **To Reproduce** Decode a record batch message whose body buffer is LZ4_FRAME-compressed and whose 8-byte length prefix is `i64::MAX`, for example a Flight `DoExchange` frame from a peer. On 59.2.0 the process ends with: ``` memory allocation of 9223372036854775807 bytes failed (test aborted with signal 6: SIGABRT) ``` Only the lz4 case was run. The zstd case above is read from source: a frame written without a content size would take the same route. **Expected behavior** `read_record_batch` returns `ArrowError::IpcError`, and never asks the allocator for more than the input can expand to. **Additional context** Possible fixes, smallest first: 1. Cap the reservation by what the input can produce, for example `decompressed_size.min(input.len().saturating_mul(N))` for a codec bound `N`, and let the `Vec` grow from there. The existing length check after decompression (l.281) still refuses a mismatch. 2. Refuse a declared length above a limit before allocating. The limit could be a `RecordBatchDecoder` / `IpcReadOptions` setting, or one derived from the field node lengths. 3. For zstd, compare the declared length with the frame's own content size (`zstd_safe::get_frame_content_size`) when the frame records one. This reaches any Flight server or client, and any service that decodes IPC from a peer. We can open a PR with hand-built-message tests for both codecs if one of these directions is acceptable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
