azecevic000 opened a new pull request, #10959: URL: https://github.com/apache/arrow-rs/pull/10959
# Rationale for this change Compressed Arrow IPC buffers advertise their uncompressed length in an 8-byte prefix. `decompress_to_buffer` interprets `0` as empty, `-1` as uncompressed, and a positive value as the expected decompressed length. For a positive value, IPC decompression already requires an exact match: the common codec path rejects a returned buffer whose length differs from the advertised length. The codecs previously differed in how they used this value while decompressing. ZSTD passes it as the output capacity to an API that returns an error if the decompressed data exceeds that capacity. LZ4, however, used it only as the initial `Vec` capacity before an unbounded `read_to_end`, allowing Arrow's output `Vec` to grow beyond the advertised length before the final check. This change gives both codecs the same output-size invariants: - No more than the advertised number of decompressed bytes are accumulated in the output `Vec`. - Successful decompression returns exactly the advertised number of bytes. Without the LZ4 bound, an untrusted compressed stream could cause Arrow's output `Vec` to grow beyond an accepted advertised length before decompression eventually failed. With the bound, the potentially unbounded output accumulation is limited to the advertised length. Downstream IPC consumers may also need to inspect the advertised length before decompression so that they can reject an unreasonable value before allocating the output buffer. Exposing `read_uncompressed_size` lets them do so without duplicating Arrow's interpretation of the IPC prefix. # What changes are included in this PR? - Publicly re-export `read_uncompressed_size` from `arrow-ipc` and document the `-1`, `0`, and positive-length semantics. - Limit the Arrow-owned LZ4 output buffer to the advertised length while decompressing, and reject output that exceeds it without growing that buffer. - Correct the existing length-mismatch error message to refer to the decompressed length. The advertised length remains the exact required output length on successful return and bounds the number of decompressed bytes accumulated in Arrow's output `Vec`. It does not bound the compressed input, allocator overhead, or codec working memory. # Are these changes tested? No new tests are introduced. Existing tests cover successful LZ4 and ZSTD round trips and rejection of prefixes shorter than 8 bytes. The new LZ4 over-limit rejection is not directly covered. # Are there any user-facing changes? Yes. `arrow_ipc::read_uncompressed_size` is a new public, non-breaking API that lets callers inspect the IPC compression prefix before decompression. Malformed LZ4 input whose decompressed output exceeds its advertised length is now rejected before Arrow's output buffer grows beyond that length. Valid IPC input is unaffected. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
