Bukhtawar opened a new issue, #12250:
URL: https://github.com/apache/arrow-rs/issues/12250

   **Which part is this question about**
   
   `parquet` crate, compression (`parquet/src/compression.rs`), 58.3.0. `main` 
as of 2026-11-14 (`39ae0f7c35`) differs only by #11858, which makes the 
contexts lazy; the per-codec ownership described here is unchanged.
   
   **Describe the bug**
   
   `create_codec` builds one `ZSTDCodec` per column chunk writer (and per 
column chunk reader), and each `ZSTDCodec` owns its own zstd contexts:
   
   ```rust
   pub struct ZSTDCodec {
       compressor: zstd::bulk::Compressor<'static>,
       decompressor: zstd::bulk::Decompressor<'static>,
   }
   ```
   
   zstd allocates a context's workspace (`ZSTD_CCtx`: window, hash and chain 
tables, sequence store, entropy buffers; sized by the level and by the pledged 
input size, up to a few MB per context at level 3 on large pages) with the C 
allocator on the first `compress`, and keeps it until the codec is dropped, 
i.e. until the column writer closes. A writer of a wide schema therefore holds 
one live compressor per ZSTD column for its whole life, and a process that 
holds several such writers at once (parallel sorted chunks, a k-way merge, one 
writer per shard) holds `writers x columns` of them. On the read side a 
`ParquetRecordBatchReader` over many columns holds `readers x columns` 
decompressors the same way.
   
   This memory is invisible to Rust-side accounting (it is not a `Vec`, a 
`Bytes`, or an arrow buffer), and when the application uses a different global 
allocator (jemalloc, mimalloc) it does not appear in that allocator's 
statistics either; it sits in glibc's arenas.
   
   We hit it in a search engine writing 2,003-column Parquet files (1,400 
string columns at the default ZSTD level 3): with 8 writers open on a node, 
glibc's arenas grew from 2.4 GB to 25.8 GB of address space (only 2.6 GB of it 
free, so live allocations) and the process gained ~7 GB of resident anonymous 
memory when the writers flushed. The same run with the string columns on LZ4 
stayed flat at 2.4 GB. Netty/Arrow-Java, JVM NMT and glibc retention were all 
ruled out before we found the zstd contexts.
   
   #11858 (lazy creation) removes the decompressor from writers and the 
compressor from readers, which helps, but the context that is actually used 
still scales with `writers x columns` / `readers x columns`.
   
   **To Reproduce**
   
   Standalone, system allocator, only `parquet` and `arrow`. N writers x C 
`Utf8` columns at ZSTD level 3; each writer writes and flushes one row group 
and is kept open (as the writers of a merge or of parallel sorted chunks are). 
The read phase opens R readers over the same files.
   
   ```rust
   use std::sync::Arc;
   use arrow::array::{ArrayRef, StringArray};
   use arrow::datatypes::{DataType, Field, Schema};
   use arrow::record_batch::RecordBatch;
   use parquet::arrow::ArrowWriter;
   use parquet::basic::{Compression, ZstdLevel};
   use parquet::file::properties::WriterProperties;
   
   fn anon_mb() -> u64 { // Linux: anonymous resident memory of this process
       let s = std::fs::read_to_string("/proc/self/smaps_rollup").unwrap();
       s.lines().find(|l| l.starts_with("Anonymous:")).and_then(|l| 
l.split_whitespace().nth(1))
           .and_then(|v| v.parse::<u64>().ok()).unwrap_or(0) / 1024
   }
   
   fn main() {
       let (cols, writers, rows) = (1400usize, 8usize, 2_000usize);
       let schema = Arc::new(Schema::new(
           (0..cols).map(|c| Field::new(format!("k{c}"), DataType::Utf8, 
true)).collect::<Vec<_>>(),
       ));
       let batch = RecordBatch::try_new(schema.clone(), (0..cols).map(|c| {
           Arc::new(StringArray::from_iter((0..rows).map(|r| ((r * 31 + c) % 7 
== 0).then(|| format!("v{}-{}", c, r % 97))))) as ArrayRef
       }).collect()).unwrap();
       let props = WriterProperties::builder()
           
.set_compression(Compression::ZSTD(ZstdLevel::try_new(3).unwrap())).build();
       let before = anon_mb();
       let mut open = Vec::new();
       for _ in 0..writers {
           let mut w = ArrowWriter::try_new(Vec::new(), schema.clone(), 
Some(props.clone())).unwrap();
           w.write(&batch).unwrap();
           w.flush().unwrap();          // compresses every column: one 
ZSTD_CCtx per column stays allocated
           open.push(w);
       }
       println!("anonymous memory held by {writers} open writers x {cols} 
columns: {} MB", anon_mb() - before);
       for w in open { w.close().unwrap(); }
   }
   ```
   
   Measured (arm64 Linux, glibc, same harness, only the `parquet` source 
differs; 1,400 columns, 2,000 rows per row group):
   
   | build | 8 open writers | 30 open readers |
   |---|---|---|
   | 58.3.0 as released | +114 MB | +1,578 MB |
   | 58.3.0 + #11858 (lazy contexts) | +35 MB | +1,405 MB |
   | 58.3.0 + one context per thread (proposed) | +35 MB | **+308 MB** |
   
   At 2,000 rows per row group the compressor workspaces are small, so #11858 
already covers the writer side here; the 2,003-column engine run above uses 
~100k-row groups, where the per-column `ZSTD_CCtx` grows to MBs and accounts 
for the ~7 GB. I will add the 100k-row numbers from the same harness in a 
comment. The read side is the clearer case even at small pages: 30 readers x 
1,400 columns hold 1.6 GB of decompressor contexts, 0.3 GB with one per thread.
   
   **Expected behavior**
   
   Memory held by compression contexts should not scale with `writers x 
columns` or `readers x columns`. A zstd context is used by one thread for one 
call at a time, and `ZSTD_compress2` resets the session and applies the 
context's parameters on every call, so a reused context produces the same bytes 
as a fresh one.
   
   **Proposed fix**
   
   Keep one compressor per thread and per level (levels can differ between 
columns), and one decompressor per thread, created on first use; `ZSTDCodec` 
keeps only its level:
   
   ```rust
   thread_local! {
       static COMPRESSORS: RefCell<Vec<(i32, zstd::bulk::Compressor<'static>)>> 
= const { RefCell::new(Vec::new()) };
       static DECOMPRESSOR: RefCell<Option<zstd::bulk::Decompressor<'static>>> 
= const { RefCell::new(None) };
   }
   pub struct ZSTDCodec { level: i32 }
   // compress(): find or create the (level, compressor) pair for this thread, 
then compress
   // decompress(): get_or_insert the thread's decompressor, then decompress
   ```
   
   Output bytes are identical: we compared files written by the unmodified 
58.3.0 and by the patched crate over a sorted writer that compresses chunks, 
decompresses and recompresses them in a k-way merge, with dictionary and plain 
pages; the files are byte-identical (`cmp`). Unit tests compare the codec's 
output with a fresh `bulk::Compressor` at levels 1, 3, 9 and 19 on inputs from 
empty to 3 MiB, interleaved across threads.
   
   A thread-local keeps one context alive per worker thread for the thread's 
lifetime (a few MB per level), bounded by the thread count rather than the 
column count. Alternatives: a per-call context (pays the creation per page, 
which #11858 rightly avoids) or a pool owned by `WriterProperties` / 
`ReadOptions`. We run the thread-local variant as a patch on 58.3.0 and would 
be glad to send it as a PR if the approach is acceptable.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to