GitHub user x-at-01 created a discussion: Sharing benchmark observations on decimal floating-point compression in IoT time-series: comparing integer mapping with Gorilla/Chimp
In IoT time-series systems, sensor measurements (environmental conditions, smart grid voltage, machinery vibration, vehicular telemetry) are fundamentally captured with finite decimal precision. Conventional floating-point compression algorithms (such as Gorilla, Chimp, and Elf) rely on bitwise XOR differences between IEEE 754 representations. However, small variations in real-world measurements often flip mantissa bits unpredictably, which limits compression ratio and impedes SIMD parallelism due to sequential bitstream decoding. Over the past months, I have been working on [fastalp](https://github.com/webc-site/wedb_embed/tree/main/fastalp) (available on [crates.io](https://crates.io/crates/fastalp)), an implementation of the Adaptive Lossless Floating-Point (ALP) compression algorithm. Instead of bitwise XOR on floating-point representations, it samples the data block to determine the optimal decimal exponent e, scales values into exact integers, and compresses them with Frame-of-Reference (FOR) bit-packing. Key architectural characteristics: - Decimal integer mapping: Transforms floats into compact integers losslessly, storing rare non-exact values in an auxiliary exception vector. - Vectorized SIMD un-bitpacking: Because encoded integers reside at uniform bit widths within each vector block, decoding unpacks directly into CPU registers without bit-by-bit branch loops, reaching 55 to 77 GB/s decompression throughput. - Zero-allocation API: Provides compress_into and decompress_into methods to eliminate heap churn on the read/write paths. - Built-in fallback: Automatically falls back to raw storage when high-entropy noise floats are encountered, guaranteeing zero negative expansion. Benchmark results measured on standard time-series datasets (weather telemetry, financial prices, disk metrics) with 1,000 double-precision values per batch: | Algorithm | Compressed Size (bits/val) | Compression Latency (µs / 1000 vals) | Decompression Latency (µs / 1000 vals) | Decompression Throughput | |---|---|---|---|---| | fastalp | 16.34 | 2.255 | 0.423 | 2364 Mval/s | | Chimp128 | 17.29 | 8.631 | 9.270 | 108 Mval/s | | Patas | 21.51 | 3.820 | 5.140 | 195 Mval/s | | Gorilla | 52.70 | 6.042 | 5.920 | 169 Mval/s | In these measurements, fastalp achieved 16.34 bits/val while decompressing 1,000 values in 0.423 µs (423 ns), corresponding to 2.36 billion values per second on a single core. I hope these benchmark measurements and architectural details provide useful context for time-series encoding evaluations and future codec research in IoTDB.  GitHub link: https://github.com/apache/iotdb/discussions/18581 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
