Abdallah-Afifi opened a new issue, #11985:
URL: https://github.com/apache/arrow-rs/issues/11985
### Describe the bug
On high-entropy floating point data, ALP produces a larger column than
`PLAIN` does. There is no fallback to storing values unencoded when the
encoding does not pay off, so the expansion reaches the file.
### Reproduction
8192 `f64` values with a random mantissa in `[1, 2)`:
```rust
let values: Vec<f64> = (0..8 * 1024)
.map(|i| {
let x = (i as u64).wrapping_mul(0x9E3779B97F4A7C15);
f64::from_bits(0x3FF0000000000000 | (x >> 12))
})
.collect();
```
Written through `ArrowWriter` with `set_dictionary_enabled(false)`, one
column, total file bytes:
| compression | `Encoding::ALP` | `Encoding::PLAIN` | ALP / PLAIN |
| --- | --- | --- | --- |
| `UNCOMPRESSED` | 91055 | 65972 | 1.380x |
| `SNAPPY` | 84317 | 65976 | 1.278x |
| `ZSTD` (default level) | 68713 | 61607 | 1.115x |
The raw payload is 65536 bytes, so uncompressed ALP is 1.389x the size of
the values themselves.
Zstd recovers most of it but not all. At the encoder level the page is 90619
bytes against a 65536-byte payload.
### Why it happens
Nearly every value becomes an exception, and each exception costs its full
width plus a position, on top of the packed placeholder. Choosing better
parameters does not rescue it: an exhaustive per-vector search over every
`(exponent, factor)` pair the format allows still gives 82013 bytes, so even
optimal ALP expands this input by about 25%.
### What I would expect
Something equivalent to checking the encoded size against the unencoded size
and keeping the smaller, so that ALP never makes a column bigger. I do not know
whether that belongs in the encoder or in the writer's encoding selection,
which is why this is a report rather than a PR.
Noticed while working on #10646, which measures how close the parameter
sampling lands to exhaustive search. @x-at-01 mentioned in
https://github.com/apache/arrow-rs/issues/10646#issuecomment-5972810221 having
added exactly this kind of raw fallback in their own implementation, so the
remedy looks established.
### Component(s)
Parquet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
