houseme opened a new issue, #659: URL: https://github.com/apache/avro-rs/issues/659
## Motivation The Rust Snappy codec currently uses `crc32fast` only for the Avro block checksum: - on compression, it computes CRC32 over the uncompressed block and appends the big-endian 4-byte checksum; - on decompression, it computes CRC32 over the decoded block and compares it with that trailer. That makes this a narrow, low-risk replacement candidate: there are only two call sites, both operate on a contiguous byte slice, and neither needs streaming/incremental hashing state. ## Why `crc-fast` `crc-fast` 1.10.0 provides a specialized `crc32_iso_hdlc(&[u8]) -> u32` helper. CRC-32/ISO-HDLC is the standard CRC32 variant used by `crc32fast` and by Avro Snappy block checksums. The standard check vector `b"123456789" -> 0xcbf43926` matches the expected algorithm. The upstream crate is also built around SIMD acceleration: - acceleration paths for `aarch64`, `x86_64`, and `x86`; - safe table-based fallback for other targets; - one-shot helpers for common algorithms, avoiding the extra hasher object/update/finalize flow used in the current code. For this repository's actual call pattern, a local microbenchmark on `Darwin arm64` (`Mac17,6`, `rustc 1.98.1`) compared the current `crc32fast::Hasher::new(); update(); finalize()` flow with `crc_fast::crc32_iso_hdlc(data)`. The checksum was asserted equal for every case. | Block size | crc32fast | crc-fast | Speedup | | ---: | ---: | ---: | ---: | | 128 B | 12.12 GiB/s | 21.02 GiB/s | 1.73x | | 1 KiB | 10.93 GiB/s | 70.03 GiB/s | 6.41x | | 16 KiB | 22.56 GiB/s | 81.98 GiB/s | 3.63x | | 1 MiB | 33.27 GiB/s | 101.22 GiB/s | 3.04x | | 16 MiB | 33.42 GiB/s | 97.76 GiB/s | 2.93x | This matters for Snappy because CRC32 is paid on every compressed block in both write and read paths. The larger the uncompressed block, the more directly the checksum cost shows up next to Snappy's own encode/decode work. ## Trade-offs The main trade-off is MSRV: `crc-fast` 1.10.0 declares `rust-version = "1.89"`, while this workspace currently declares and tests `1.88.0`. A correct replacement therefore needs to bump the workspace MSRV and the CI MSRV entry to `1.89.0`. Dependency-wise, using `crc-fast` with `default-features = false, features = ["std"]` avoids its default `ffi`/`panic-handler` features while enabling the std-backed runtime SIMD detection needed for the performance win. This adds its `std` dependency chain (`digest 0.10`, `crypto-common 0.1`, `generic-array`, and `spin`) and removes `crc32fast`. ## Suggested change - Replace the optional Snappy dependency `crc32fast` with `crc-fast = "1.10.0"`. - Use `crc_fast::crc32_iso_hdlc(...)` at the two Snappy checksum call sites. - Add a regression test that verifies the Snappy trailer for `b"123456789"` is the big-endian CRC-32/ISO-HDLC check value `0xcbf43926`. - Bump workspace and CI MSRV from `1.88.0` to `1.89.0`. ## Local validation - `cargo check -p apache-avro --features snappy` - `cargo test -p apache-avro --features snappy,derive snappy` - `cargo fmt --all --check` - `cargo build --all-features --all-targets` - `cargo check -p apache-avro --target aarch64-unknown-linux-gnu --features snappy` Not completed locally: - `wasm32-unknown-unknown` CI target validation. Installing the target locally stalled while downloading `rust-std`, so this should still be watched in CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
