Thanks to everyone who participated, and to Christian for these detailed notes!
Meeting recording <https://www.youtube.com/watch?v=8ugj9V6hTR4> - Update on compression (Rich) - I promise I’ll stop eventually - Design review requested for PR #13244 zstd “early abort” <https://github.com/openzfs/zfs/pull/13244> - Goal was more or less “make higher zstd levels more generally useful”. They currently aren’t in scenarios with mixed compressibility data due to high CPU cost on incompressible parts. - Approach: use cheap compression algorithms as heuristics to determine whether a large block (>= 128k) is compressible. Only use the high zstd level if heuristic matches. Currently, heuristics are LZ4 first, then zstd-1. - Benchmark results: - The Pi 4 compression chart on incompressible data is pretty…drastic <https://user-images.githubusercontent.com/214141/160515419-51de7625-8e4f-4913-a317-fc2e439f54d5.png>, going from over 2 hours to 15 minutes to recv a 41 GB dataset at zstd-15. (It’s probably even more drastic for 18, but that might take over a day to get the baseline for…) - The Ryzen 5900X chart isn’t bad either <https://user-images.githubusercontent.com/214141/159585202-fa6d0f3d-8606-463c-a669-e5bde187d1e0.png>, 4+ minutes to under 1 at zstd-18. - Peak losses observed in compression savings were under 100 MB across a 41 GB fairly incompressible and 54 GB very compressible dataset, which seems more than acceptable to me - and there’s an easy tunable if you’d like to not use it. - Regarding correctness: Since it doesn’t change the compression parameters, it doesn’t produce different compressed results, either the same compressed results or uncompressed. - Could use reviewers, not that it’s a particularly invasive change (I have an even less invasive version, just want to benchmark the additional overhead of it isn’t noticeable) - Could have a dataset parameter to control whether it’s tried or not, if desired - Probably also works with LZ4+gzip-1 on gzip, though I haven’t tried the experiment yet. - Aspects discussed during the meeting that need follow-up: - Add kstats to keep track of heuristic’s decisions. Will help characterize different datasets and determine when this is useful. - Need to determine the worst-case CPU-time-saved/compression-lost ratio due to this feature. - Synthetic benchmark where we always run lz4, zstd-1, and zstd-4. (zstd-4 is the first level at which this mechanism kicks in.) - A pathological dataset where lz4 and zstd-1 heuristic match, but the data is not zstd-4 compressible. - Consider publishing the scripts that create the datasets, run the benchmarks, and post-process results. - (Below this is just negative results, feel free to skip if people aren’t interested in why updating LZ4/zstd/gzip don’t look like wins currently) - LZ4 update: - Branch works, lz4_version property lets you swap between 1.9.3 and legacy at will per dataset - Want to do more stress testing, add more tests, teach ztest to randomly swap that property while doing its other games, but… - Not convinced the compressor’s a win, really, performance or savings wise - 5900X incompressible chart <https://www.dropbox.com/s/2jun8ryhr53dsi1/lz4%201M%20lowcomp.png?dl=0> - 5900X compressible chart <https://www.dropbox.com/s/4weiu2zhrgaofns/lz4%201M%20highcomp.png?dl=0> - No feature flag needed, older LZ4 happily decompresses it, even with a version number hidden past where the LZ4 size “header” thinks the stream ends to avoid breaking older readers - Send/recv prints an annoying message about not knowing the lz4_version property if you send -p it to someone who doesn’t know it, but it doesn’t error out - PR: If I find a compelling reason to use the new version - zstd update - Branch works, swapping compressor with zstd_version property - Plays nice with the above lz4_version property too - Benchmarking 1.5.0/1.5.1/1.5.2/1.5.0+(SSE|NEON)/1.5.2+(SSE|NEON) continues - Preliminary results not very compelling: - (“s” versions built with SSE2 optimizations and kernel_fpu dance) - 5900X incompressible chart <https://www.dropbox.com/s/f4ft7qjlpnmlfoc/zstd%205900X%20incompressible%201M.png?dl=0> - 5900X compressible chart <https://www.dropbox.com/s/b2o1f2a2xj7th6k/zstd%205900X%20compressible%201M.png?dl=0> - Same thing about not needing a feature flag as LZ4 - though zstd already had a version in its header field, so no fun games needed - PR: same as LZ4, if I find a use case it’s an improvement for or a flaw in my approach that results in better performance - gzip update - Kidding… - …but I really did experiment with it. - Tried benchmarking a few different zlib forks instead of Linux builtin, none were compellingly different so far - Quick and dirty graphs of zlib-ng, which was the most different, here <https://github.com/openzfs/zfs/issues/13245> - Seems like at best maybe the decompressor might be worth examining, and the SIMD improvements don’t seem to make much difference for our use cases. - Questions about tests like refquota_008_neg that are giving false fails after the write inflation change PR #12773 <https://github.com/openzfs/zfs/pull/12773> - Solution: make the test lower the dirty data max tunable. - Christian: questions on TBD PR that makes ZTS detect if a test does not properly restore tunable values. - test-runner.py is an internal interface, zfs-tests.sh is the external interface of the test suite. Ok to change / cut down test-runner.py functionality that is no longer in use. - If we can detect the tunable change, can’t we just reset them and continue running tests? Maybe, if we assume any tunable can be written can be written in any order. But a test that doesn’t restore tunable values properly is a bug. So, just fix any broken tests once in this PR, then this problem should be solved going forward. On Mon, Mar 28, 2022 at 3:38 PM Matthew Ahrens <mahr...@delphix.com> wrote: > The next OpenZFS Leadership meeting will be held tomorrow, March 29th, 1pm > Pacific time. > Everyone is welcome to attend and participate, and we will try to keep the > meeting on agenda and on time. The meetings will be held online via Zoom, > and recorded and posted to the website and YouTube after the meeting. > > For more information and details on how to attend, as well as notes and > video from the previous meeting, please see the agenda document: > > > https://docs.google.com/document/d/1w2jv2XVYFmBVvG1EGf-9A5HBVsjAYoLIFZAnWHhV-BM/edit > > --matt > ------------------------------------------ openzfs: openzfs-developer Permalink: https://openzfs.topicbox.com/groups/developer/Tde9acb4c64be171e-Mf5669337fa4729706acd4beb Delivery options: https://openzfs.topicbox.com/groups/developer/subscription
