Bukhtawar commented on issue #12250:
URL: https://github.com/apache/arrow-rs/issues/12250#issuecomment-6869768095

   Follow-up with larger row groups: the same repro with `ROWS=100000` (1 400 
`Utf8` columns, ZSTD level 3, 8 writers each holding one flushed row group, 
then 30 readers each holding a `ParquetRecordBatchReader` over one of those 
files after its first batch). Same machine (arm64 Linux, glibc 2.26), same 
harness, system allocator; only the `parquet` source differs. Anonymous memory 
delta, MB:
   
   | parquet build | 8 open writers | 30 open readers |
   |---|---|---|
   | 58.3.0 (eager `Compressor` + `Decompressor` per codec) | 354 | 1 712 |
   | 58.3.0 + #11858 (both created lazily, per codec) | 326 | 1 519 |
   | 58.3.0 + one compressor per thread and level, one decompressor per thread 
| 131 | 472 |
   
   For comparison, the 2 000-row run from the issue body:
   
   | parquet build | 8 open writers | 30 open readers |
   |---|---|---|
   | 58.3.0 | 114 | 1 578 |
   | 58.3.0 + #11858 | 35 | 1 405 |
   | 58.3.0 + per-thread contexts | 35 | 308 |
   
   With larger pages, lazy creation stops helping writers much: what is left is 
one compression context per column that has compressed a page, which the 
per-thread variant collapses to one per thread (the remaining writer delta is 
the writers' own page buffers). On the read side the per-column decompressors 
dominate at both sizes; sharing one per thread removes most of it.
   
   The files written by the per-thread variant were byte-identical (`cmp`) to 
those written by unmodified 58.3.0, for a sorted writer whose k-way merge 
decompresses and recompresses its chunks, with dictionary and plain pages.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to