alamb commented on issue #10448: URL: https://github.com/apache/arrow-rs/issues/10448#issuecomment-5113597055
I am not very familiar with how the V2 datapages are written / buffered The downside of using `shrink_to_fit`, as @mapleFU says, is that it will likely both do a new allocation and copy the data into a smaller allocation. This seems like a classic tradeoff between resource types (CPU and Memory) It seems like an ideal solution would be to properly size the buffer before hand before compression, but we probably don't know what that size would be. If we find that `shrink_to_fit` slows down writing, I suggest 1. Add a flag to the writer properties, something like `optimize_for_memory` 2. Call `shrink_to_fit` if that flag is enabled That way users can make the tradeoff between CPU and Memory that is most appropriate for their systems -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
