etseidl commented on code in PR #10520:
URL: https://github.com/apache/arrow-rs/pull/10520#discussion_r4273616556
##########
parquet/src/column/writer/mod.rs:
##########
@@ -1985,6 +1988,14 @@ fn has_dictionary_support(kind: Type) -> bool {
}
}
+/// Calls [`Vec::shrink_to_fit`] only when spare capacity exceeds 10% of the
length,
+/// so we don't pay for a reallocation and copy that saves just a few bytes.
+fn shrink_to_fit_if_wasteful(buf: &mut Vec<u8>) {
+ if buf.capacity() - buf.len() > buf.len() / 10 {
Review Comment:
Thanks @vedjaw, this is looking better. One thought is that in addition to a
percent savings, we should also make sure the amount saved will exceed a page
of memory; otherwise we're going to wind up spending CPU time and not saving
any actual physical memory. How about something like
```rust
let spare = buf.capacity() - buf.len();
if spare > buf.len() / 10 && spare > 4 * 1024 { .. }
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
