Re: [RFB] add LZ4 compression method to btrfs
You're right about the broken links, my bad. The first link related to a few statements by Mark Ruijter, copy/pasted below : I just looked into it and it appears to be promising. On average the speeds appear to be 48% faster then snappy. This is amazing since snappy has proved to be 30% faster then QuickLZ which once was the fastest compression protocol. == http://www.lessfs.com/wordpress/?p=684#comments I did not test LZ4 on high end hardware yet. However even on my laptop it is clear that LZ4 does outperform snappy. With the hardware being the bottleneck LZ4 still manages to speed things up by 2~5%. Most likely the difference will be larger when fast hardware is used == http://www.lessfs.com/wordpress/?p=688 There is no precise benchmark though. Just this sentence : I will post the exact performance numbers on low and high end hardware after testing has finished. So maybe someone has to ask Mark about these results. The second link initially linked to Hadoop, which recently decided to integrate LZ4 compression : == https://issues.apache.org/jira/browse/HADOOP-7657 However, since then, i've discovered the website of the developper in charge of this patch (Binglin Chang), and he has come nice comparison figures available. The main summary is copy/pasted below : Perf.RawCompressionLZ4Block size: 64K Compress: 428M/s Decompress: 681M/s( 1584M/s) ratio: 43.0% - Total Perf.RawCompressionSnappy Block size: 64K Compress: 400M/s Decompress: 442M/s( 979M/s) ratio: 45.2% - Total == https://github.com/decster/jnicompressions/blob/master/README.md -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFB] add LZ4 compression method to btrfs
On Tue, Feb 14, 2012 at 1:47 PM, Hugo Chevrain hugochevr...@gmail.com wrote: Are you sure about these figures ? the difference seems too large. It's almost unbelievable. -- You should not, Mark Ruijter found the same for LessFS (http://www.lessfs.com/wordpress/? p=688) and there is also such finding into an Hadoop thread (https://scribe.twitter.com/#!/otisg/status/148848850914902016) The first link only shows results, not data. The second link is dead and just shows a dead twitter page, in two browsers. Science isn't hard folks! Just post the raw numbers so people can verify the results. Cheers, Auke -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFB] add LZ4 compression method to btrfs
The second link is broken, just remove scribe. from it. https://twitter.com/#!/otisg/status/148848850914902016 It causes a cross-domain error, not sure why he could see it though. On Wednesday, February 15, 2012 9:23:37 AM, Kok, Auke-jan H wrote: On Tue, Feb 14, 2012 at 1:47 PM, Hugo Chevrainhugochevr...@gmail.com wrote: Are you sure about these figures ? the difference seems too large. It's almost unbelievable. -- You should not, Mark Ruijter found the same for LessFS (http://www.lessfs.com/wordpress/? p=688) and there is also such finding into an Hadoop thread (https://scribe.twitter.com/#!/otisg/status/148848850914902016) The first link only shows results, not data. The second link is dead and just shows a dead twitter page, in two browsers. Science isn't hard folks! Just post the raw numbers so people can verify the results. Cheers, Auke -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFB] add LZ4 compression method to btrfs
Silesia corpus (avg of 10 runs), AMD bulldozer box, 12G ram, 1Ghz cpu: lz4 = 739860 us ( 286 MB/s) 195930 us (1081 MB/s) 211957760 - 101630873 47.9% snappy 1.0.4 = 1050 ms ( 201 MB/s) 248 ms ( 853 MB/s) 211957760 - 104739310 49.4% snappy-c = 940111 us ( 225 MB/s) 299690 us ( 707 MB/s) 211957760 - 131060567 61.8% lzo 2.06 1x_1= 739421 us ( 286 MB/s) 436542 us ( 485 MB/s) 211957760 - 100576151 47.5% Silesia corpus (avg of 10 runs), Nehalem X7560, 2.3Ghz cpu: lz4 = 624170 us ( 339 MB/s) 200622 us (1056 MB/s) 211957760 - 101630873 47.9% snappy 1.0.4 = 1047 ms ( 202 MB/s) 265 ms ( 797 MB/s) 211957760 - 104739310 49.4% snappy-c = 836415 us ( 253 MB/s) 300567 us ( 705 MB/s) 211957760 - 131060567 61.8% lzo 2.06 1x_1= 639305 us ( 331 MB/s) 470840 us ( 450 MB/s) 211957760 - 100576151 47.5% Are you sure about these figures ? the difference seems too large. It's almost unbelievable. -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFB] add LZ4 compression method to btrfs
Markus Lindberg marcuslindb...@gmail.com writes: Are you sure about these figures ? the difference seems too large. It's almost unbelievable. Yes, my benchmarks totally disagree with them. In my tests lz4 is generally slower than snappy-c. -Andi -- a...@linux.intel.com -- Speaking for myself only -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFB] add LZ4 compression method to btrfs
Are you sure about these figures ? the difference seems too large. It's almost unbelievable. -- You should not, Mark Ruijter found the same for LessFS (http://www.lessfs.com/wordpress/? p=688) and there is also such finding into an Hadoop thread (https://scribe.twitter.com/#!/otisg/status/148848850914902016) -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[RFB] add LZ4 compression method to btrfs
Hi, so here it is, LZ4 compression method inside btrfs. The patchset is based on top of current Chris' for-linus + Andi's snappy implementation + the fixes from Li Zefan. Passes xfstests and stresstests. I haven't measured performance on wide range of hardware or workloads, rather wanted to publish the patches before I get distracted again. I'd like to ask anyone willing and able to test this to share your results with us. At least an example from standalone benchmarks of snappy-c/snappy(upstream)/lzo/lz4: Silesia corpus (avg of 10 runs), AMD bulldozer box, 12G ram, 1Ghz cpu: lz4 = 739860 us ( 286 MB/s) 195930 us (1081 MB/s) 211957760 - 101630873 47.9% snappy 1.0.4 = 1050 ms ( 201 MB/s) 248 ms ( 853 MB/s) 211957760 - 104739310 49.4% snappy-c = 940111 us ( 225 MB/s) 299690 us ( 707 MB/s) 211957760 - 131060567 61.8% lzo 2.06 1x_1= 739421 us ( 286 MB/s) 436542 us ( 485 MB/s) 211957760 - 100576151 47.5% Silesia corpus (avg of 10 runs), Nehalem X7560, 2.3Ghz cpu: lz4 = 624170 us ( 339 MB/s) 200622 us (1056 MB/s) 211957760 - 101630873 47.9% snappy 1.0.4 = 1047 ms ( 202 MB/s) 265 ms ( 797 MB/s) 211957760 - 104739310 49.4% snappy-c = 836415 us ( 253 MB/s) 300567 us ( 705 MB/s) 211957760 - 131060567 61.8% lzo 2.06 1x_1= 639305 us ( 331 MB/s) 470840 us ( 450 MB/s) 211957760 - 100576151 47.5% * snappy 1.0.4 svn r58 * snappy-c as Andi sent it to mailinglist * lzo 2.0.6 1x_1 variant * lz4 r55 (r54 + bugfix in the hash table entry type) * compiled by gcc 4.7, -O2 pullable from: git://repo.or.cz/linux-2.6/btrfs-unstable.git dev/compression-squad david -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
