This is an automated email from the ASF dual-hosted git repository.
Jefffrey pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-rs.git
The following commit(s) were added to refs/heads/main by this push:
new 1d9a792e69 Limit LZ4 buffer to declared decompressed size (#10959)
1d9a792e69 is described below
commit 1d9a792e6947347d3498144ea1a6fdfb94a4ab7c
Author: Aleksandar Zecevic <[email protected]>
AuthorDate: Wed Sep 23 05:45:44 2026 +0200
Limit LZ4 buffer to declared decompressed size (#10959)
# Rationale for this change
Compressed Arrow IPC buffers advertise their uncompressed length in an
8-byte prefix. `decompress_to_buffer` interprets `0` as empty, `-1` as
uncompressed, and a positive value as the expected decompressed length.
For a positive value, IPC decompression already requires an exact match:
the common codec path rejects a returned buffer whose length differs
from the advertised length.
The codecs previously differed in how they used this value while
decompressing. ZSTD passes it as the output capacity to an API that
returns an error if the decompressed data exceeds that capacity. LZ4,
however, used it only as the initial `Vec` capacity before an unbounded
`read_to_end`, allowing Arrow's output `Vec` to grow beyond the
advertised length before the final check.
This change gives both codecs the same output-size invariants:
- No more than the advertised number of decompressed bytes are
accumulated in the output `Vec`.
- Successful decompression returns exactly the advertised number of
bytes.
Without the LZ4 bound, an untrusted compressed stream could cause
Arrow's output `Vec` to grow beyond an accepted advertised length before
decompression eventually failed. With the bound, the potentially
unbounded output accumulation is limited to the advertised length.
# What changes are included in this PR?
- Limit the Arrow-owned LZ4 output buffer to the advertised length while
decompressing, and reject output that exceeds it without growing that
buffer.
The advertised length remains the exact required output length on
successful return and bounds the number of decompressed bytes
accumulated in Arrow's output `Vec`.
# Are these changes tested?
Added a test that verifies that initial buffer size overflow results in
a direct error, instead of an unbounded increase of capacity.
# Are there any user-facing changes?
Malformed LZ4 input whose decompressed output exceeds its advertised
length is now rejected before Arrow's output buffer grows beyond that
length. Valid IPC input is unaffected.
---
arrow-ipc/src/compression.rs | 35 +++++++++++++++++++++++++++++++++--
1 file changed, 33 insertions(+), 2 deletions(-)
diff --git a/arrow-ipc/src/compression.rs b/arrow-ipc/src/compression.rs
index 59e10f7698..494b0e0b46 100644
--- a/arrow-ipc/src/compression.rs
+++ b/arrow-ipc/src/compression.rs
@@ -280,7 +280,7 @@ impl CompressionCodec {
};
if ret.len() != decompressed_size {
return Err(ArrowError::IpcError(format!(
- "Expected compressed length of {decompressed_size} got {}",
+ "Expected decompressed length of {decompressed_size} got {}",
ret.len()
)));
}
@@ -310,7 +310,18 @@ fn compress_lz4(_input: &[u8], _output: &mut Vec<u8>) ->
Result<(), ArrowError>
fn decompress_lz4(input: &[u8], decompressed_size: usize) -> Result<Vec<u8>,
ArrowError> {
use std::io::Read;
let mut output = Vec::with_capacity(decompressed_size);
- lz4_flex::frame::FrameDecoder::new(input).read_to_end(&mut output)?;
+ let mut decoder = lz4_flex::frame::FrameDecoder::new(input);
+ decoder
+ .by_ref()
+ .take(decompressed_size as u64)
+ .read_to_end(&mut output)?;
+
+ // Probe without growing `output` to reject data exceeding the advertised
size.
+ if decoder.read(&mut [0])? != 0 {
+ return Err(ArrowError::IpcError(format!(
+ "LZ4 decompressed buffer exceeds advertised size of
{decompressed_size}"
+ )));
+ }
Ok(output)
}
@@ -413,6 +424,26 @@ mod tests {
assert_eq!(input_bytes, result.as_slice());
}
+ #[test]
+ #[cfg(feature = "lz4")]
+ fn test_lz4_decompression_rejects_output_exceeding_advertised_size() {
+ let input_bytes = b"hello lz4";
+ let codec = super::CompressionCodec::Lz4Frame;
+ let mut compressed = Vec::new();
+ codec
+ .compress(input_bytes, &mut compressed, &mut Default::default())
+ .unwrap();
+
+ let err = codec
+ .decompress(&compressed, input_bytes.len() - 1, &mut
Default::default())
+ .expect_err("output larger than the advertised size should fail");
+
+ assert!(
+ err.to_string().contains("exceeds advertised size"),
+ "unexpected error: {err}"
+ );
+ }
+
#[test]
#[cfg(feature = "zstd")]
fn test_zstd_compression() {
