Abdallah-Afifi commented on issue #10646: URL: https://github.com/apache/arrow-rs/issues/10646#issuecomment-6498836760
Opened #11984 for this. Summary of what it measured, against a per-vector exhaustive search over every `(exponent, factor)` pair the format allows: sampling lands within 0.3% on structured data, including a page whose scale changes after the sampled vectors and one that alternates scale per vector. I also expected the 1024/256 sampling stride to alias against period-4 data and it made no measurable difference. The one real gap is high-entropy floats at 1.105x, pinned as a regression case. `build_preset` returns the single pair `(16, 0)` there while every vector's true optimum is `(0, 0)`, and since the preset holds one candidate `select_params` returns it without evaluating anything, so no per-vector correction can recover. While measuring it I also found that ALP expands this input rather than compressing it, which survives the compression codec — filed as #11985 rather than folded in here, since the fix is a different question. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
