geoHeil opened a new pull request, #12077: URL: https://github.com/apache/arrow-rs/pull/12077
# Which issue does this PR close? - Closes #12030 # Rationale for this change The reader sizes the decompression buffer from the uncompressed length the sender writes in front of each compressed body buffer, before it decompresses anything. If a peer declares `i64::MAX` on an LZ4 buffer, the reader asks for that much memory and the allocation failure aborts the process. ZSTD takes the same path for a frame that records no content size. Any Flight client or server, or other service decoding IPC from a peer, is affected. # What changes are included in this PR? `CompressionCodec::decompress` now returns `ArrowError::IpcError`, before allocating, when the declared size is more than the input could expand to. That limit is `input.len()` times a per-codec maximum expansion ratio: - LZ4: 255. Each match-length continuation byte adds at most 255 output bytes. - ZSTD: 128 KiB / 4. The smallest block that produces output is a 4-byte RLE block, and no block produces more than 128 KiB. The decoded length must already equal the declared size, so a size the input cannot reach was already invalid. No valid input is now refused. The check only moves that refusal ahead of the allocation. This is not a memory budget. A sender can still make the reader reserve up to the ratio times the compressed buffer length. A configurable limit would be a separate change. For ZSTD frames that record a content size, zstd already sizes the buffer from that value and refuses one above the capacity, so this change adds no ZSTD-specific check for them. # Are these changes tested? Yes. Two tests in `compression.rs`, one per codec, declare `usize::MAX` for a small compressed input and expect the new error. The ZSTD test writes its frame without a content size. A fork-only run with this check removed shows both tests panicking with `capacity overflow`, which is the unbounded reservation. With the check, both pass: https://github.com/geoHeil/arrow-rs/actions/runs/41818438271 Fork CI on this branch: https://github.com/geoHeil/arrow-rs/actions/runs/41818647307 # Are there any significant user-facing changes? No. AI disclosure: the change, the tests and this description were generated with an AI coding assistant (Claude Code). It derived the two expansion ratios from the LZ4 block format and the ZSTD format specification. It checked zstd 0.14.3's `bulk::Decompressor::decompress` to confirm that only frames without a recorded content size reserve the full capacity. It confirmed that both tests fail without the check, in the fork run linked above. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
