alamb commented on code in PR #10432:
URL: https://github.com/apache/arrow-rs/pull/10432#discussion_r4039679231
##########
parquet/src/encodings/rle.rs:
##########
@@ -65,11 +65,11 @@ pub struct RleEncoder {
// Underlying writer which holds an internal buffer.
bit_writer: BitWriter,
- // Buffered values for bit-packed runs.
- buffered_values: [u64; BIT_PACK_GROUP_SIZE],
-
- // Number of current buffered values. Must be less than
BIT_PACK_GROUP_SIZE.
- num_buffered_values: usize,
+ // Values of the current in-progress bit-packed run, including the
partially
+ // filled trailing group. Only materialized into `bit_writer` when the run
+ // closes, so the whole run can be packed in one vectorised `put_batch`
call.
+ // Bounded by `MAX_GROUPS_PER_BIT_PACKED_RUN * BIT_PACK_GROUP_SIZE` values.
+ pending_values: Vec<u64>,
Review Comment:
I think the bound is always 512 values or less (so this would be at most a
4KB buffer)
Did you consider just leaving as a fixed size buffer to avoid an allocation
/ indirection?
Something like
```rust
buffered_values: [u64; MAX_GROUPS_PER_BIT_PACKED_RUN*BIT_PACK_GROUP_SIZE],
```
🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
