[
https://issues.apache.org/jira/browse/HADOOP-7657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170733#comment-13170733
]
Binglin Chang commented on HADOOP-7657:
---------------------------------------
testdata/alice29.txt :
ZLIB: [b 1M] bytes 152089 -> 54404 35.8% comp 12.4 MB/s uncomp 189.3 MB/s
LZ4: [b 1M] bytes 152089 -> 90735 59.7% comp 188.3 MB/s uncomp 1055.6 MB/s
SNAPPY: [b 4M] bytes 152089 -> 90965 59.8% comp 133.0 MB/s uncomp 594.3 MB/s
testdata/asyoulik.txt :
ZLIB: [b 1M] bytes 125179 -> 48897 39.1% comp 11.3 MB/s uncomp 177.4 MB/s
LZ4: [b 1M] bytes 125179 -> 79026 63.1% comp 180.7 MB/s uncomp 1062.6 MB/s
SNAPPY: [b 4M] bytes 125179 -> 80207 64.1% comp 130.0 MB/s uncomp 568.6 MB/s
testdata/baddata1.snappy :
ZLIB: [b 1M] bytes 27512 -> 22941 83.4% comp 24.0 MB/s uncomp 113.8 MB/s
LZ4: [b 1M] bytes 27512 -> 26486 96.3% comp 500.4 MB/s uncomp 3963.1 MB/s
SNAPPY: [b 4M] bytes 27512 -> 26675 97.0% comp 387.4 MB/s uncomp 3445.4 MB/s
testdata/baddata2.snappy :
ZLIB: [b 1M] bytes 27483 -> 23023 83.8% comp 24.1 MB/s uncomp 113.0 MB/s
LZ4: [b 1M] bytes 27483 -> 26633 96.9% comp 549.6 MB/s uncomp 4310.0 MB/s
SNAPPY: [b 4M] bytes 27483 -> 26724 97.2% comp 423.3 MB/s uncomp 3647.4 MB/s
testdata/baddata3.snappy :
ZLIB: [b 1M] bytes 28384 -> 23718 83.6% comp 23.9 MB/s uncomp 112.6 MB/s
LZ4: [b 1M] bytes 28384 -> 27529 97.0% comp 542.0 MB/s uncomp 4256.5 MB/s
SNAPPY: [b 4M] bytes 28384 -> 27476 96.8% comp 342.9 MB/s uncomp 3005.0 MB/s
testdata/cp.html :
ZLIB: [b 1M] bytes 24603 -> 7961 32.4% comp 27.8 MB/s uncomp 198.8 MB/s
LZ4: [b 1M] bytes 24603 -> 12039 48.9% comp 327.8 MB/s uncomp 1561.8 MB/s
SNAPPY: [b 4M] bytes 24603 -> 11838 48.1% comp 195.6 MB/s uncomp 1002.1 MB/s
testdata/fields.c :
ZLIB: [b 1M] bytes 11150 -> 3122 28.0% comp 29.0 MB/s uncomp 237.6 MB/s
LZ4: [b 1M] bytes 11150 -> 5285 47.4% comp 453.1 MB/s uncomp 1356.9 MB/s
SNAPPY: [b 4M] bytes 11150 -> 4728 42.4% comp 224.1 MB/s uncomp 990.1 MB/s
testdata/geo.protodata :
ZLIB: [b 1M] bytes 118588 -> 15131 12.8% comp 56.0 MB/s uncomp 396.9 MB/s
LZ4: [b 1M] bytes 118588 -> 19862 16.7% comp 627.9 MB/s uncomp 2526.5 MB/s
SNAPPY: [b 4M] bytes 118588 -> 27459 23.2% comp 422.5 MB/s uncomp 1640.4 MB/s
testdata/grammar.lsp :
ZLIB: [b 1M] bytes 3721 -> 1222 32.8% comp 30.9 MB/s uncomp 208.9 MB/s
LZ4: [b 1M] bytes 3721 -> 1947 52.3% comp 461.7 MB/s uncomp 1556.7 MB/s
SNAPPY: [b 4M] bytes 3721 -> 1800 48.4% comp 281.0 MB/s uncomp 1118.8 MB/s
testdata/house.jpg :
ZLIB: [b 1M] bytes 126958 -> 126513 99.6% comp 24.7 MB/s uncomp 224.6 MB/s
LZ4: [b 1M] bytes 126958 -> 127272 100.2% comp 2339.8 MB/s uncomp 7389.2
MB/s
SNAPPY: [b 4M] bytes 126958 -> 126803 99.9% comp 2866.3 MB/s uncomp 14627.9
MB/s
testdata/html :
ZLIB: [b 1M] bytes 102400 -> 13699 13.4% comp 44.6 MB/s uncomp 356.4 MB/s
LZ4: [b 1M] bytes 102400 -> 21835 21.3% comp 501.6 MB/s uncomp 1990.6 MB/s
SNAPPY: [b 4M] bytes 102400 -> 24140 23.6% comp 356.6 MB/s uncomp 1381.3 MB/s
testdata/html_x_4 :
ZLIB: [b 1M] bytes 409600 -> 53367 13.0% comp 41.4 MB/s uncomp 356.1 MB/s
LZ4: [b 1M] bytes 409600 -> 85864 21.0% comp 507.6 MB/s uncomp 1962.6 MB/s
SNAPPY: [b 4M] bytes 409600 -> 96472 23.6% comp 350.9 MB/s uncomp 1346.3 MB/s
testdata/kennedy.xls :
ZLIB: [b 1M] bytes 1029744 -> 203992 19.8% comp 18.8 MB/s uncomp 305.2 MB/s
LZ4: [b 1M] bytes 1029744 -> 375713 36.5% comp 416.4 MB/s uncomp 1665.8
MB/s
SNAPPY: [b 4M] bytes 1029744 -> 425735 41.3% comp 304.5 MB/s uncomp 879.3 MB/s
testdata/kppkn.gtb :
ZLIB: [b 1M] bytes 184320 -> 38751 21.0% comp 8.1 MB/s uncomp 248.7 MB/s
LZ4: [b 1M] bytes 184320 -> 84897 46.1% comp 325.8 MB/s uncomp 933.0 MB/s
SNAPPY: [b 4M] bytes 184320 -> 70535 38.3% comp 190.9 MB/s uncomp 690.2 MB/s
testdata/lcet10.txt :
ZLIB: [b 1M] bytes 426754 -> 144904 34.0% comp 12.8 MB/s uncomp 196.0 MB/s
LZ4: [b 1M] bytes 426754 -> 237316 55.6% comp 191.6 MB/s uncomp 1050.0 MB/s
SNAPPY: [b 4M] bytes 426754 -> 243710 57.1% comp 138.7 MB/s uncomp 621.7 MB/s
testdata/mapreduce-osdi-1.pdf :
ZLIB: [b 1M] bytes 94330 -> 74928 79.4% comp 28.8 MB/s uncomp 181.6 MB/s
LZ4: [b 1M] bytes 94330 -> 77760 82.4% comp 794.6 MB/s uncomp 4056.2 MB/s
SNAPPY: [b 4M] bytes 94330 -> 77477 82.1% comp 761.5 MB/s uncomp 3864.8 MB/s
testdata/plrabn12.txt :
ZLIB: [b 1M] bytes 481861 -> 195261 40.5% comp 9.6 MB/s uncomp 175.4 MB/s
LZ4: [b 1M] bytes 481861 -> 323373 67.1% comp 172.4 MB/s uncomp 997.3 MB/s
SNAPPY: [b 4M] bytes 481861 -> 329339 68.3% comp 122.3 MB/s uncomp 518.8 MB/s
testdata/ptt5 :
ZLIB: [b 1M] bytes 513216 -> 56465 11.0% comp 31.9 MB/s uncomp 331.6 MB/s
LZ4: [b 1M] bytes 513216 -> 86327 16.8% comp 659.1 MB/s uncomp 1793.4 MB/s
SNAPPY: [b 4M] bytes 513216 -> 93455 18.2% comp 488.2 MB/s uncomp 1342.0 MB/s
testdata/sum :
ZLIB: [b 1M] bytes 38240 -> 12990 34.0% comp 14.8 MB/s uncomp 186.6 MB/s
LZ4: [b 1M] bytes 38240 -> 18916 49.5% comp 317.7 MB/s uncomp 1582.3 MB/s
SNAPPY: [b 4M] bytes 38240 -> 19837 51.9% comp 195.5 MB/s uncomp 853.2 MB/s
testdata/urls.10K :
ZLIB: [b 1M] bytes 702087 -> 222613 31.7% comp 23.9 MB/s uncomp 207.0 MB/s
LZ4: [b 1M] bytes 702087 -> 329450 46.9% comp 306.6 MB/s uncomp 1448.9 MB/s
SNAPPY: [b 4M] bytes 702087 -> 357267 50.9% comp 209.3 MB/s uncomp 948.3 MB/s
testdata/xargs.1 :
ZLIB: [b 1M] bytes 4227 -> 1736 41.1% comp 31.0 MB/s uncomp 197.2 MB/s
LZ4: [b 1M] bytes 4227 -> 2671 63.2% comp 401.6 MB/s uncomp 1459.2 MB/s
SNAPPY: [b 4M] bytes 4227 -> 2509 59.4% comp 237.4 MB/s uncomp 950.0 MB/s
The speed really rocks...
> Add support for LZ4 compression
> -------------------------------
>
> Key: HADOOP-7657
> URL: https://issues.apache.org/jira/browse/HADOOP-7657
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Mr Bsd
> Labels: compression
>
> According to several benchmark sites, LZ4 seems to overtake other fast
> compression algorithms, especially in the decompression speed area. The
> interface is also trivial to integrate
> (http://code.google.com/p/lz4/source/browse/trunk/lz4.h) and there is no
> license issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
