HippoBaro commented on code in PR #9697:
URL: https://github.com/apache/arrow-rs/pull/9697#discussion_r3083332852
##########
parquet/src/arrow/push_decoder/reader_builder/mod.rs:
##########
@@ -610,6 +617,12 @@ impl RowGroupReaderBuilder {
&mut self.buffers,
)?;
+ // All data for this row group has been extracted into the
+ // InMemoryRowGroup. Release physical buffers up to the end
+ // of this row group so streaming IO can reclaim memory.
+ self.buffers
Review Comment:
Thank to you both, I wasn't aware of this! This certainly changes things. My
watermark approach was nice and optimal, but this makes it fundamentally
flawed. Oh well, back to the drawing board!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
