apurtell commented on pull request #3244: URL: https://github.com/apache/hbase/pull/3244#issuecomment-844624849
SNAPPY or ZSTD are recommended, all other options provided for comparison. (LZMA is included as a sanity check that indeed an expensive algorithm really is expensive.) When using SNAPPY or ZSTD we derive a performance benefit due to reduced IO for the large values in the test case. Microbenchmarks are collected with [this change](https://gist.github.com/apurtell/596310d08b5ad75cd9677466d36360e4). Statistics are collected over the lifetime of the regionserver and are dumped at end of test at shutdown. Statistics are updated under synchronization but this is done in a way that excludes that overhead from measurement. The normal patch does not contain either the instrumentation or the synchronization point. Nanoseconds are converted to milliseconds for the table. Mode | WALs aggregate size | WALs aggregate size difference | WAL writer append time (ms avg) -- | -- | -- | -- Default | 5,117,369,553 | - | 0.290 (stdev 0.328) Compression enabled, value compression not enabled | 5,002,683,600 | (2.241%) | 0.372 (stddev 0.336) ~~Compression enabled, value compression enabled, v1 patch, Deflate (best speed)~~ | ~~1,209,947,515~~ | ~~(76.4%)~~ | ~~12.694 (stddev 8.48)~~ Compression enabled, value compression enabled, v2 patch, algorithm=SNAPPY | 1,616,387,702 | (68.4%) | 0.027 (stddev 0.204) Compression enabled, value compression enabled, v2 patch, algorithm=ZSTD (best speed) | 1,149,008,133 | (77.55%) | 0.043 (stddev 0.195) Compression enabled, value compression enabled, v2 patch, algorithm=ZSTD (default) | 1,089,241,811 | (78.7%) | 0.056 (stdev 0.310) Compression enabled, value compression enabled, v2 patch, algorithm=ZSTD (best compression) | 941,452,655 | (81.2%) | 0.231 (stddev 1.11) _Options below not recommended._ | - | - | - Compression enabled, value compression enabled, v2 patch, algorithm=GZ | 1,082,414,015 | (78.9%) | 0.267 (stddev 1.325) Compression enabled, value compression enabled, v2 patch, algorithm=LZMA (level 1) | 1,013,951,637 | (80.2%) | 2.157 (stddev 3.302) Compression enabled, value compression enabled, v2 patch, algorithm=LZMA (default) | 940,884,618 | (81.7%) | 4.739 (stdev 8.609) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
