mapleFU commented on issue #10448: URL: https://github.com/apache/arrow-rs/issues/10448#issuecomment-5112271030
> Is a mismatched frame size from actual decompressed size common enough to matter? I guess I'm suffering from high RSS problem after this. A `GenericColumnWriter` would keeps `data_pages`, when compression ratio is high for something like zstd level = 10 and the text cases, the compression bound would be huge, and these memory would not being released for a long time, causing much higher memory usage For DataPageV1 this method is free, but when using DataPageV2 for text types, this will introducing memory overheads -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
