phongn opened a new pull request, #13257: URL: https://github.com/apache/trafficserver/pull/13257
## Add lz4 and zstd compression support to the CLFUS RAM cache ### Summary This adds two modern compression backends to the CLFUS RAM cache, as optional build-time dependencies: - **lz4** (`proxy.config.cache.ram_cache.compress: 4`) — a replacement for fastlz: strictly better compression ratio at substantially higher throughput. - **zstd** (`proxy.config.cache.ram_cache.compress: 5`, level 3) — a replacement for libz/deflate-6: comparable ratio at roughly 10× the compression speed and 3× the decompression speed. The existing fastlz/libz/liblzma backends are unchanged and remain valid for their config values; the docs now recommend lz4 over fastlz and zstd over libz. Benchmarks (lzbench, silesia XML corpus, Xeon Gold 6338, one thread): | Method | Compress | Decompress | Final size | |---|---|---|---| | fastlz | 452 MB/s | 913 MB/s | ~26% | | **lz4** | **727 MB/s** | **3458 MB/s** | **~23%** | | libz-6 | 54 MB/s | 536 MB/s | ~13% | | **zstd-3** | **508 MB/s** | **1690 MB/s** | **~12%** | ### Implementation notes - lz4 uses the one-shot `LZ4_compress_default` / `LZ4_decompress_safe` API. - zstd uses per-thread reusable contexts (`thread_local` `ZSTD_CCtx`/`ZSTD_DCtx`) with `ZSTD_compress2` and a sticky compression level, avoiding a context allocation per call — relevant since decompression sits on the cache-hit path. This requires zstd ≥ 1.4.0 (the first release with the advanced one-shot API stable); the version floor is enforced in `find_package`. - Misconfiguration is caught at startup: configuring a backend that wasn't compiled in is `Fatal`, matching the existing liblzma behavior. - New `cmake/FindLZ4.cmake` and `cmake/FindZSTD.cmake` modules. zstd detection previously used `find_package(zstd CONFIG)` only, which fails on distributions that don't ship `zstd-config.cmake`; it now resolves via the module (the `zstd::zstd` alias shim is kept for builds using `CMAKE_FIND_PACKAGE_PREFER_CONFIG`). - `traffic_layout info` now reports `TS_HAS_LZ4` and the lz4 build/runtime versions, mirroring zstd. ### Testing The `RamCacheCLFUS` class definition moved from the `.cc` into a new private header so unit tests can drive `compress_entries()` synchronously. A new Catch2 test (`test_RamCacheCLFUS`) does store → compress → read-back roundtrips across **all five backends**, asserting byte-for-byte equality and the expected `RAM_HIT_COMPRESS_*` state, plus incompressible-fallback and small-payload cases. No prior test verified compression data integrity for any backend. ### Drive-by fixes - The liblzma compress path allocated its output buffer at `e->len` instead of `lzma_stream_buffer_bound(e->len)`, causing encode failures (and spurious "incompressible" marking) for data that didn't shrink; it now matches the other backends, with the existing `REQUIRED_COMPRESSION`/`REQUIRED_SHRINK` thresholds deciding what to keep. - Removed a stale unconditional "libz not available for RAM cache compression" warning that fired every second when `compress: 2` was configured — zlib is a required dependency and always available. ### CI / packaging `liblz4-dev` / `lz4-devel` added to the deb and yum CI images. The Fedora CI image will need `lz4-devel` added for build coverage of the new backend (zstd-devel is already present). ### Future work Compression is currently implemented inside CLFUS only. A follow-up refactor will extract it into a shared layer so all RAM cache algorithms can use it; the new unit test is structured to migrate there. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
