x-at-01 commented on issue #10646: URL: https://github.com/apache/arrow-rs/issues/10646#issuecomment-5516664838
Hi, on the question of validating the ALP sampling heuristic: I evaluated this while building [fastalp](https://github.com/webc-site/wedb_embed/tree/main/fastalp) (also on [crates.io](https://crates.io/crates/fastalp)), comparing the 32-sample heuristic against an exhaustive search of all 135 parameter pairs across the 31 ALP paper datasets (253 KB total payload). A few practical observations from my validation: 1. Compression ratio parity: The 32-sample heuristic found the optimal or near-optimal exponent pair on 30 out of 31 datasets, yielding an aggregate 2.29x compression ratio, identical to full-space search. 2. Pathological inputs and expansion: On high-entropy or irregular datasets (such as air_sensor_f, poi_lat, and poi_lon), ALP can produce high exception counts where the exception dictionary exceeds the saved bitpacking bytes, resulting in negative compression (down to 0.51x in the reference implementation). Adding a lightweight size check to fall back to uncompressed raw bytes with a small header prevents this expansion entirely. 3. Throughput: With lower-bound pruning and an early exit threshold for low bit-widths, sampling overhead drops to approximately 200 ns, allowing compression to run at 6.1 to 6.5 GB/s on f64 arrays and decompression at 55 to 77 GB/s. Sharing these data points in case they help with establishing test cases and validation criteria for the Parquet implementation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
