aptro opened a new pull request, #12294: URL: https://github.com/apache/arrow-rs/pull/12294
# Which issue does this PR close? - Closes #12250. # Rationale for this change `create_codec` builds one `ZSTDCodec` per column chunk, and each codec owns its zstd contexts. After #11858 they are created lazily, but a reader still creates (and frees) one decompression context per column chunk it reads, and a writer holds one compression context per column. For a query engine reading many small column chunks, context creation is a measurable share of CPU: in our service (DataFusion over zstd Parquet), a uprobe counted ~582k `ZSTD_createDCtx` calls in 110 s on 59.3.2, about 11% of query CPU. # What changes are included in this PR? As proposed in #12250: one decompressor per thread and one compressor per (thread, level), created on first use; `ZSTDCodec` keeps only its level. A zstd bulk call resets its session, so a reused context produces the same bytes as a fresh one. A re-entrant call (not possible today) falls back to a fresh context rather than panicking. # Are these changes tested? New test `test_codec_zstd_shared_contexts_match_fresh_contexts`: four threads, codecs at levels 1/3/9/19 interleaved, inputs from 0 B to 3 MiB; compressed bytes must equal a fresh `bulk::Compressor`'s, and round-trip. (Negative control: making the lookup ignore the level fails it.) Measured on 59.3.2 + this patch, same machine, alternated rounds: - `ZSTD_createDCtx`: 581,990 -> 852 per 110 s run; `ZSTD_createCCtx`: 583,418 -> 2,603. - Query CPU per poll, small reads (8 columns): 9.63 -> 7.88 ms (-18%); large reads: 81 -> 57 ms (-30%). Latency p50 down in every row. # Are there any significant user-facing changes? No API change. Contexts now live for the thread's lifetime (bounded by threads x levels used, not by columns). # AI usage disclosure The change, the test and this description were drafted with Claude Code (an AI coding assistant), following the fix proposed in #12250; the measurements above were run with its help on our own deployment. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
