Bukhtawar opened a new issue, #12250:
URL: https://github.com/apache/arrow-rs/issues/12250
**Which part is this question about**
`parquet` crate, compression (`parquet/src/compression.rs`), 58.3.0. `main`
as of 2026-11-14 (`39ae0f7c35`) differs only by #11858, which makes the
contexts lazy; the per-codec ownership described here is unchanged.
**Describe the bug**
`create_codec` builds one `ZSTDCodec` per column chunk writer (and per
column chunk reader), and each `ZSTDCodec` owns its own zstd contexts:
```rust
pub struct ZSTDCodec {
compressor: zstd::bulk::Compressor<'static>,
decompressor: zstd::bulk::Decompressor<'static>,
}
```
zstd allocates a context's workspace (`ZSTD_CCtx`: window, hash and chain
tables, sequence store, entropy buffers; sized by the level and by the pledged
input size, up to a few MB per context at level 3 on large pages) with the C
allocator on the first `compress`, and keeps it until the codec is dropped,
i.e. until the column writer closes. A writer of a wide schema therefore holds
one live compressor per ZSTD column for its whole life, and a process that
holds several such writers at once (parallel sorted chunks, a k-way merge, one
writer per shard) holds `writers x columns` of them. On the read side a
`ParquetRecordBatchReader` over many columns holds `readers x columns`
decompressors the same way.
This memory is invisible to Rust-side accounting (it is not a `Vec`, a
`Bytes`, or an arrow buffer), and when the application uses a different global
allocator (jemalloc, mimalloc) it does not appear in that allocator's
statistics either; it sits in glibc's arenas.
We hit it in a search engine writing 2,003-column Parquet files (1,400
string columns at the default ZSTD level 3): with 8 writers open on a node,
glibc's arenas grew from 2.4 GB to 25.8 GB of address space (only 2.6 GB of it
free, so live allocations) and the process gained ~7 GB of resident anonymous
memory when the writers flushed. The same run with the string columns on LZ4
stayed flat at 2.4 GB. Netty/Arrow-Java, JVM NMT and glibc retention were all
ruled out before we found the zstd contexts.
#11858 (lazy creation) removes the decompressor from writers and the
compressor from readers, which helps, but the context that is actually used
still scales with `writers x columns` / `readers x columns`.
**To Reproduce**
Standalone, system allocator, only `parquet` and `arrow`. N writers x C
`Utf8` columns at ZSTD level 3; each writer writes and flushes one row group
and is kept open (as the writers of a merge or of parallel sorted chunks are).
The read phase opens R readers over the same files.
```rust
use std::sync::Arc;
use arrow::array::{ArrayRef, StringArray};
use arrow::datatypes::{DataType, Field, Schema};
use arrow::record_batch::RecordBatch;
use parquet::arrow::ArrowWriter;
use parquet::basic::{Compression, ZstdLevel};
use parquet::file::properties::WriterProperties;
fn anon_mb() -> u64 { // Linux: anonymous resident memory of this process
let s = std::fs::read_to_string("/proc/self/smaps_rollup").unwrap();
s.lines().find(|l| l.starts_with("Anonymous:")).and_then(|l|
l.split_whitespace().nth(1))
.and_then(|v| v.parse::<u64>().ok()).unwrap_or(0) / 1024
}
fn main() {
let (cols, writers, rows) = (1400usize, 8usize, 2_000usize);
let schema = Arc::new(Schema::new(
(0..cols).map(|c| Field::new(format!("k{c}"), DataType::Utf8,
true)).collect::<Vec<_>>(),
));
let batch = RecordBatch::try_new(schema.clone(), (0..cols).map(|c| {
Arc::new(StringArray::from_iter((0..rows).map(|r| ((r * 31 + c) % 7
== 0).then(|| format!("v{}-{}", c, r % 97))))) as ArrayRef
}).collect()).unwrap();
let props = WriterProperties::builder()
.set_compression(Compression::ZSTD(ZstdLevel::try_new(3).unwrap())).build();
let before = anon_mb();
let mut open = Vec::new();
for _ in 0..writers {
let mut w = ArrowWriter::try_new(Vec::new(), schema.clone(),
Some(props.clone())).unwrap();
w.write(&batch).unwrap();
w.flush().unwrap(); // compresses every column: one
ZSTD_CCtx per column stays allocated
open.push(w);
}
println!("anonymous memory held by {writers} open writers x {cols}
columns: {} MB", anon_mb() - before);
for w in open { w.close().unwrap(); }
}
```
Measured (arm64 Linux, glibc, same harness, only the `parquet` source
differs; 1,400 columns, 2,000 rows per row group):
| build | 8 open writers | 30 open readers |
|---|---|---|
| 58.3.0 as released | +114 MB | +1,578 MB |
| 58.3.0 + #11858 (lazy contexts) | +35 MB | +1,405 MB |
| 58.3.0 + one context per thread (proposed) | +35 MB | **+308 MB** |
At 2,000 rows per row group the compressor workspaces are small, so #11858
already covers the writer side here; the 2,003-column engine run above uses
~100k-row groups, where the per-column `ZSTD_CCtx` grows to MBs and accounts
for the ~7 GB. I will add the 100k-row numbers from the same harness in a
comment. The read side is the clearer case even at small pages: 30 readers x
1,400 columns hold 1.6 GB of decompressor contexts, 0.3 GB with one per thread.
**Expected behavior**
Memory held by compression contexts should not scale with `writers x
columns` or `readers x columns`. A zstd context is used by one thread for one
call at a time, and `ZSTD_compress2` resets the session and applies the
context's parameters on every call, so a reused context produces the same bytes
as a fresh one.
**Proposed fix**
Keep one compressor per thread and per level (levels can differ between
columns), and one decompressor per thread, created on first use; `ZSTDCodec`
keeps only its level:
```rust
thread_local! {
static COMPRESSORS: RefCell<Vec<(i32, zstd::bulk::Compressor<'static>)>>
= const { RefCell::new(Vec::new()) };
static DECOMPRESSOR: RefCell<Option<zstd::bulk::Decompressor<'static>>>
= const { RefCell::new(None) };
}
pub struct ZSTDCodec { level: i32 }
// compress(): find or create the (level, compressor) pair for this thread,
then compress
// decompress(): get_or_insert the thread's decompressor, then decompress
```
Output bytes are identical: we compared files written by the unmodified
58.3.0 and by the patched crate over a sorted writer that compresses chunks,
decompresses and recompresses them in a k-way merge, with dictionary and plain
pages; the files are byte-identical (`cmp`). Unit tests compare the codec's
output with a fresh `bulk::Compressor` at levels 1, 3, 9 and 19 on inputs from
empty to 3 MiB, interleaved across threads.
A thread-local keeps one context alive per worker thread for the thread's
lifetime (a few MB per level), bounded by the thread count rather than the
column count. Alternatives: a per-call context (pays the creation per page,
which #11858 rightly avoids) or a pool owned by `WriterProperties` /
`ReadOptions`. We run the thread-local variant as a patch on 58.3.0 and would
be glad to send it as a PR if the approach is acceptable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]