mzabaluev opened a new issue, #11129:
URL: https://github.com/apache/arrow-rs/issues/11129
**Describe your question / problem**
`decompress_lz4` constructs a new `lz4_flex::frame::FrameDecoder` for every
buffer it decodes:
```rust
#[cfg(feature = "lz4")]
fn decompress_lz4(input: &[u8], decompressed_size: usize) -> Result<Vec<u8>,
ArrowError> {
use std::io::Read;
let mut output = Vec::with_capacity(decompressed_size);
lz4_flex::frame::FrameDecoder::new(input).read_to_end(&mut output)?;
Ok(output)
}
```
A `FrameDecoder` starts with empty internal `src`/`dst` vectors and grows
them on first use via
```rust
// lz4_flex 0.13.1, src/frame/decompress.rs:443
fn vec_resize_and_get_mut(v: &mut Vec<u8>, start: usize, end: usize) -> &mut
[u8] {
v.resize(end, 0)
...
```
`Vec::resize` zero-fills, and the buffer is then immediately overwritten in
full by
`read_exact`.
**Additional context**
`decompress()` already threads a `&mut DecompressionContext` to both codec
arms, and the zstd
arm uses it to hold a reusable decompressor:
```rust
let ret = match self {
CompressionCodec::Lz4Frame => decompress_lz4(input, decompressed_size)?,
// context unused
CompressionCodec::Zstd(_) => decompress_zstd(input, decompressed_size,
context)?, // context used
};
```
`DecompressionContext`'s own doc comment states the intent:
> ... between subsequent decompression calls to avoid the performance
overhead of initialising
> a new context for every decompression.
and its only field today is `#[cfg(feature = "zstd")] decompressor`. So the
mechanism to fix
this already exists and is already plumbed to the call site, but the LZ4 arm
does
not use it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
