Abdallah-Afifi opened a new pull request, #11984: URL: https://github.com/apache/arrow-rs/pull/11984
# Which issue does this PR close? - Closes #10646. # Rationale for this change The ALP encoder picks `(exponent, factor)` pairs from samples rather than from the full data, at two levels: `build_preset` looks at 8 vectors per page and 256 of the 1024 values in each, then each vector chooses among at most 5 surviving candidates instead of all 66 (`f32`) or 190 (`f64`). Nothing checked how close that lands to searching every pair. # What changes are included in this PR? Tests only, no change to the encoder. `per_vector_exhaustive_size` is a lower bound: for every vector independently, the best pair over all the format allows, chosen with full knowledge of the vector. The heuristic is then compared against it. # Are these changes tested? That is the whole change. Results on 8-vector pages: | input | heuristic vs exhaustive | | --- | --- | | two decimals | 1.003x | | six decimals | 1.000x | | scale changes halfway through the page | 1.000x | | scale alternates per vector | 1.002x | | scale changes only in the last vector | 1.000x | | high entropy | 1.105x | So sampling is within 0.3% on structured data, including the inputs I expected to mislead it — a page whose scale changes after the sampled vectors, and one that alternates scale per vector. I also expected the 1024/256 sampling stride to alias against period-4 data and it made no measurable difference, so that test isn't included. The one real gap is high-entropy floats, pinned as a regression case at 1.105x. The mechanism: `build_preset` returns the single pair `(16, 0)` while every vector's true optimum is `(0, 0)`, and because the preset holds only one candidate `select_params` returns it without evaluating anything, so no per-vector correction is possible. Bounds are ratios rather than byte counts, so they still pass if the heuristic improves. Encoding is deterministic, so the inputs reproduce exactly. `cargo test --workspace` passes (7514 tests), with `cargo fmt --all --check`, `cargo clippy --workspace --all-targets --all-features -- -D warnings` and `typos` clean. Separately, while measuring this I found that ALP expands this high-entropy input rather than compressing it, which survives the compression codec. Filed separately rather than folded in here. --- Per the AI policy in CONTRIBUTING.md: I used Claude Code for these tests and the measurements. I've reviewed them and confirmed the numbers myself. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
