mapleFU commented on issue #10448:
URL: https://github.com/apache/arrow-rs/issues/10448#issuecomment-5114142269
```mermaid
flowchart TD
subgraph V1["WriterVersion::PARQUET_1_0 (DataPage V1)"]
A1["Vec::with_capacity(uncompressed_size)<br/>cap =
uncompressed_size"]
A2["cmpr.compress(...)<br/>len = compressed_size (usually « cap)"]
A3["compressed_buf.shrink_to_fit()<br/>cap == len ✅"]
A4["buffer.into() → Bytes::from(Vec)"]
A5{"len == cap ?"}
A6["into_boxed_slice()<br/>zero waste, compact ✅"]
A1 --> A2 --> A3 --> A4 --> A5
A5 -- "yes" --> A6
end
subgraph V2["WriterVersion::PARQUET_2_0 (DataPage V2)"]
B1["let mut buffer = vec![] (rep/def levels)"]
B2["cmpr.compress(&values, &mut buffer)"]
B3["inside
ZSTDCodec::compress:<br/>reserve(compress_bound(input))<br/>cap grows to
WORST-CASE bound"]
B4["only compressed_size bytes written<br/>len « cap ❌ (no shrink)"]
B5["buffer.into() → Bytes::from(Vec)"]
B6{"len == cap ?"}
B7["Shared { buf, cap }<br/>keeps the WHOLE over-allocated cap ❌"]
B1 --> B2 --> B3 --> B4 --> B5 --> B6
B6 -- "no" --> B7
end
A6 --> OK["CompressedPage → Bytes is compact"]
B7 --> BAD["CompressedPage → Bytes carries wasted capacity"]
style A3 fill:#d4edda,stroke:#28a745
style A6 fill:#d4edda,stroke:#28a745
style OK fill:#d4edda,stroke:#28a745
style B4 fill:#f8d7da,stroke:#dc3545
style B7 fill:#f8d7da,stroke:#dc3545
style BAD fill:#f8d7da,stroke:#dc3545
```
The key point reviewers may have missed is **what `Bytes::from(Vec<u8>)`
does**
([bytes-1.12.0/src/bytes.rs](https://docs.rs/bytes/latest/src/bytes/bytes.rs.html)):
```rust
impl From<Vec<u8>> for Bytes {
fn from(vec: Vec<u8>) -> Bytes {
// ...
// Avoid an extra allocation if possible.
if len == cap {
return Bytes::from(vec.into_boxed_slice()); // compact
}
// otherwise: keep the original ptr + cap as-is
let shared = Box::new(Shared { buf: ptr, cap, /* ... */ });
// ...
}
}
```
So `.into()` only compacts when `len == cap`. V1 guarantees that via
`shrink_to_fit()`; V2 does not, so the worst-case `compress_bound`
over-allocation is **carried by the resulting `Bytes` for its entire lifetime**.
Whether this actually costs memory depends on the page's lifetime:
```mermaid
flowchart LR
P["V2 CompressedPage<br/>(Bytes with wasted cap)"]
Q{"dictionary encoded<br/>& not deferred?"}
P --> Q
Q -- "no" --> W["write_data_page →
write_page<br/>sink.write_all(page.data())<br/>copied out, then dropped<br/>⚠️
transient waste only"]
Q -- "yes" --> BUF["push_back
into<br/>VecDeque<CompressedPage><br/>buffered until close()"]
BUF --> ACC["waste accumulates:<br/>Σ per-page over-allocation<br/>❌
real, sustained memory growth"]
style ACC fill:#f8d7da,stroke:#dc3545
style BUF fill:#fff3cd,stroke:#ffc107
```
**Summary:** in the buffered (dictionary) path, every queued V2 page holds
its own `compress_bound`-sized over-allocation until `close()`, and nothing
ever shrinks it. The fix is a single `buffer.shrink_to_fit()` before
`buffer.into()` on the V2 branch, matching V1 (same one-memcpy tradeoff). Note
`write_dictionary_page` has the same pattern too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
