Bukhtawar commented on issue #12250: URL: https://github.com/apache/arrow-rs/issues/12250#issuecomment-6869768095
Follow-up with larger row groups: the same repro with `ROWS=100000` (1 400 `Utf8` columns, ZSTD level 3, 8 writers each holding one flushed row group, then 30 readers each holding a `ParquetRecordBatchReader` over one of those files after its first batch). Same machine (arm64 Linux, glibc 2.26), same harness, system allocator; only the `parquet` source differs. Anonymous memory delta, MB: | parquet build | 8 open writers | 30 open readers | |---|---|---| | 58.3.0 (eager `Compressor` + `Decompressor` per codec) | 354 | 1 712 | | 58.3.0 + #11858 (both created lazily, per codec) | 326 | 1 519 | | 58.3.0 + one compressor per thread and level, one decompressor per thread | 131 | 472 | For comparison, the 2 000-row run from the issue body: | parquet build | 8 open writers | 30 open readers | |---|---|---| | 58.3.0 | 114 | 1 578 | | 58.3.0 + #11858 | 35 | 1 405 | | 58.3.0 + per-thread contexts | 35 | 308 | With larger pages, lazy creation stops helping writers much: what is left is one compression context per column that has compressed a page, which the per-thread variant collapses to one per thread (the remaining writer delta is the writers' own page buffers). On the read side the per-column decompressors dominate at both sizes; sharing one per thread removes most of it. The files written by the per-thread variant were byte-identical (`cmp`) to those written by unmodified 58.3.0, for a sorted writer whose k-way merge decompresses and recompresses its chunks, with dictionary and plain pages. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
