merlimat opened a new pull request #5390: Use pure-java Air-Compressor instead of JNI based libraries URL: https://github.com/apache/pulsar/pull/5390 ### Motivation Right now we're using JNI based libraries to perform data compression. These libraries are do have an overhead in terms of size (7Mb out of 20Mb of Pulsar-Client lib) and are incurring the JNI overhead which is typically measurable when compressing many small payloads. We can replaces compression libraries for LZ4, ZStd and Snappy with AirCompressor (https://github.com/airlift/aircompressor), which is a pure Java compression library used by Presto. ### Microbenchmarks Microbenchmark code is available at https://github.com/merlimat/compression-benchmark The results are on-par with the JNI version in most cases. Results: ``` Benchmark (provider) (size) Mode Cnt Score Error Units CompressDecompress.compress Zip 10b thrpt 214.639 ops/ms CompressDecompress.compress Zip 1kb thrpt 40.054 ops/ms CompressDecompress.compress Zip 10kb thrpt 4.898 ops/ms CompressDecompress.compress Zip 64kb thrpt 1.416 ops/ms CompressDecompress.compress JNI-LZ4 10b thrpt 2307.323 ops/ms CompressDecompress.compress JNI-LZ4 1kb thrpt 496.654 ops/ms CompressDecompress.compress JNI-LZ4 10kb thrpt 81.646 ops/ms CompressDecompress.compress JNI-LZ4 64kb thrpt 20.605 ops/ms CompressDecompress.compress JNI-ZStd 10b thrpt 1635.891 ops/ms CompressDecompress.compress JNI-ZStd 1kb thrpt 98.864 ops/ms CompressDecompress.compress JNI-ZStd 10kb thrpt 21.072 ops/ms CompressDecompress.compress JNI-ZStd 64kb thrpt 6.316 ops/ms CompressDecompress.compress JNI-Snappy 10b thrpt 2198.521 ops/ms CompressDecompress.compress JNI-Snappy 1kb thrpt 264.103 ops/ms CompressDecompress.compress JNI-Snappy 10kb thrpt 27.095 ops/ms CompressDecompress.compress JNI-Snappy 64kb thrpt 7.961 ops/ms CompressDecompress.compress AC-Snappy 10b thrpt 7542.674 ops/ms CompressDecompress.compress AC-Snappy 1kb thrpt 374.631 ops/ms CompressDecompress.compress AC-Snappy 10kb thrpt 54.042 ops/ms CompressDecompress.compress AC-Snappy 64kb thrpt 16.932 ops/ms CompressDecompress.compress AC-LZ4 10b thrpt 9360.128 ops/ms CompressDecompress.compress AC-LZ4 1kb thrpt 374.507 ops/ms CompressDecompress.compress AC-LZ4 10kb thrpt 50.168 ops/ms CompressDecompress.compress AC-LZ4 64kb thrpt 12.202 ops/ms CompressDecompress.compress AC-ZStd 10b thrpt 3 641.855 ops/ms CompressDecompress.compress AC-ZStd 1kb thrpt 3 40.996 ops/ms CompressDecompress.compress AC-ZStd 10kb thrpt 3 9.626 ops/ms CompressDecompress.compress AC-ZStd 64kb thrpt 3 2.979 ops/ms CompressDecompress.decompress Zip 10b thrpt 1856.131 ops/ms CompressDecompress.decompress Zip 1kb thrpt 164.930 ops/ms CompressDecompress.decompress Zip 10kb thrpt 35.547 ops/ms CompressDecompress.decompress Zip 64kb thrpt 7.846 ops/ms CompressDecompress.decompress JNI-LZ4 10b thrpt 4127.293 ops/ms CompressDecompress.decompress JNI-LZ4 1kb thrpt 2003.209 ops/ms CompressDecompress.decompress JNI-LZ4 10kb thrpt 326.834 ops/ms CompressDecompress.decompress JNI-LZ4 64kb thrpt 66.849 ops/ms CompressDecompress.decompress JNI-ZStd 10b thrpt 1003.361 ops/ms CompressDecompress.decompress JNI-ZStd 1kb thrpt 156.012 ops/ms CompressDecompress.decompress JNI-ZStd 10kb thrpt 48.902 ops/ms CompressDecompress.decompress JNI-ZStd 64kb thrpt 19.477 ops/ms CompressDecompress.decompress JNI-Snappy 10b thrpt 3564.673 ops/ms CompressDecompress.decompress JNI-Snappy 1kb thrpt 884.649 ops/ms CompressDecompress.decompress JNI-Snappy 10kb thrpt 123.879 ops/ms CompressDecompress.decompress JNI-Snappy 64kb thrpt 24.231 ops/ms CompressDecompress.decompress AC-Snappy 10b thrpt 8733.916 ops/ms CompressDecompress.decompress AC-Snappy 1kb thrpt 830.710 ops/ms CompressDecompress.decompress AC-Snappy 10kb thrpt 118.719 ops/ms CompressDecompress.decompress AC-Snappy 64kb thrpt 26.784 ops/ms CompressDecompress.decompress AC-LZ4 10b thrpt 9422.680 ops/ms CompressDecompress.decompress AC-LZ4 1kb thrpt 1746.522 ops/ms CompressDecompress.decompress AC-LZ4 10kb thrpt 228.481 ops/ms CompressDecompress.decompress AC-LZ4 64kb thrpt 56.751 ops/ms CompressDecompress.decompress AC-ZStd 10b thrpt 6789.080 ops/ms CompressDecompress.decompress AC-ZStd 1kb thrpt 194.514 ops/ms CompressDecompress.decompress AC-ZStd 10kb thrpt 42.434 ops/ms CompressDecompress.decompress AC-ZStd 64kb thrpt 15.292 ops/ms ``` https://docs.google.com/spreadsheets/d/18ntnyxiQY3VedYeywoum9JXV97f-G9RL7xD-T6th7tA/edit#gid=153785868 #### Compression     #### Decompression    
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
