mapleFU commented on issue #10448: URL: https://github.com/apache/arrow-rs/issues/10448#issuecomment-5114072845
> Does that mean that https://github.com/apache/arrow-rs/pull/10020 made writing faster but use more memory? No, emmm, with shrink_to_fit in DataPageV1, it introduce less copying, since the desting buffer size is equal to adding sub-buffers. In DataPageV2, it would be "unshrinked buffer". Actually, it doesn't matter previously, before https://github.com/apache/arrow-rs/pull/10345 , the behavior is "use zstd::bound for temp buffer, final buffer might be small", but after that, the compress buffer might be large ( Since the `output_buf.reserve(len)` is called ) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
