thinkharderdev commented on code in PR #9400:
URL: https://github.com/apache/arrow-rs/pull/9400#discussion_r2799056699
##########
parquet/src/compression.rs:
##########
@@ -503,20 +503,27 @@ pub use lz4_codec::*;
#[cfg(any(feature = "zstd", test))]
mod zstd_codec {
- use std::io::{self, Write};
-
use crate::compression::{Codec, ZstdLevel};
use crate::errors::Result;
/// Codec for Zstandard compression algorithm.
+ ///
+ /// Uses `zstd::bulk` API with reusable compressor/decompressor contexts
+ /// to avoid the overhead of reinitializing contexts for each operation.
pub struct ZSTDCodec {
- level: ZstdLevel,
+ compressor: zstd::bulk::Compressor<'static>,
+ decompressor: zstd::bulk::Decompressor<'static>,
Review Comment:
Would it be possible to do even better than this and initialize a
thread-local compressor?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
