Rich-T-kid commented on PR #10383:
URL: https://github.com/apache/arrow-rs/pull/10383#issuecomment-5017572359
```rust
fn compare_column_sizes(batch: &RecordBatch) {
let schema = batch.schema();
for (i, col) in batch.columns().iter().enumerate() {
let name = schema.field(i).name();
let data = col.to_data();
let slice_size = data.get_slice_memory_size().unwrap();
let buffer_size = col.get_buffer_memory_size();
if slice_size != buffer_size {
println!(
"[{name}] data_type={:?} slice_size={slice_size}
buffer_size={buffer_size} diff={}",
data.data_type(),
buffer_size as i64 - slice_size as i64
);
} else {
println!(
"[{name}] data_type={:?} sizes match ({slice_size})",
data.data_type()
);
}
}
}
```
is resulting in
```rust
[s1] data_type=Utf8 slice_size=14336 buffer_size=20544 diff=6208
[i1] data_type=Int16 sizes match (2048)
[s2] data_type=Utf8 slice_size=10240 buffer_size=12352 diff=2112
[i2] data_type=Int64 sizes match (8192)
```
the calculations for the string arrays are not accurate. I could fix the
breaking test `encode::tests::flight_data_size_even` by making the allowed
coverage larger but I think updating `get_slice_memory_size` to be more
accurate is better
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
