[Bacula-users] Bacula Compression - other then GZIP
Hi folks... I need to know if the Compression flag on FileSet must be gzip or I can use another compress program... I want use bzip2 to compress my file, because I thing bzip2 is more efficient... Thanks for any help... Regards Gilberto Nunes Ferreira TI Selbetti Gestão de Documentos Telefone: +55 (47) 3441-6004 Celular: +55 (47) 8861-6672 -- SOLARIS 10 is the OS for Data Centers - provides features such as DTrace, Predictive Self Healing and Award Winning ZFS. Get Solaris 10 NOW http://p.sf.net/sfu/solaris-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression - other then GZIP
On 10.02.2010 / 09:05:19 -0200, Gilberto Nunes wrote: Hi folks... I need to know if the Compression flag on FileSet must be gzip or I can use another compress program... I want use bzip2 to compress my file, because I thing bzip2 is more efficient... or even Parallel BZIP2, see http://compression.ca/pbzip2/ Thanks. -- SOLARIS 10 is the OS for Data Centers - provides features such as DTrace, Predictive Self Healing and Award Winning ZFS. Get Solaris 10 NOW http://p.sf.net/sfu/solaris-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression - other then GZIP
On 02/10/10 06:05, Gilberto Nunes wrote: Hi folks... I need to know if the Compression flag on FileSet must be gzip or I can use another compress program... I want use bzip2 to compress my file, because I thing bzip2 is more efficient... It is true that bzip2 is more efficient than gzip, but it is also slower and very much more CPU-intensive. These are things to keep in mind. gzip may not be the best compression out there, but it is fast. -- Phil Stracchino, CDK#2 DoD#299792458 ICBM: 43.5607, -71.355 ala...@caerllewys.net ala...@metrocast.net p...@co.ordinate.org Renaissance Man, Unix ronin, Perl hacker, Free Stater It's not the years, it's the mileage. -- SOLARIS 10 is the OS for Data Centers - provides features such as DTrace, Predictive Self Healing and Award Winning ZFS. Get Solaris 10 NOW http://p.sf.net/sfu/solaris-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression - other then GZIP
On Wed, Feb 10, 2010 at 09:05:19AM -0200, Gilberto Nunes wrote: (...) gzip or I can use another compress program... No. I want use bzip2 to compress my file, because I thing bzip2 is more efficient... Really? $ du -m /tmp/foo.iso 625 /tmp/foo.iso $ gzip -c /tmp/foo.iso | dd bs=64K /dev/null 0+34388 records in 0+34388 records out 563405802 bytes (563 MB) copied, 64.9428 s, 8.7 MB/s $ bzip2 -c /tmp/foo.iso | dd bs=64K /dev/null 0+137488 records in 0+137488 records out 563150276 bytes (563 MB) copied, 445.201 s, 1.3 MB/s 255526 bytes less while six times slower.. Petar Bogdanovic -- SOLARIS 10 is the OS for Data Centers - provides features such as DTrace, Predictive Self Healing and Award Winning ZFS. Get Solaris 10 NOW http://p.sf.net/sfu/solaris-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression - other then GZIP
On 2/10/2010 8:16 AM, Petar Bogdanovic wrote: I want use bzip2 to compress my file, because I thing bzip2 is more efficient... Really? $ du -m /tmp/foo.iso 625 /tmp/foo.iso $ gzip -c/tmp/foo.iso | dd bs=64K/dev/null 0+34388 records in 0+34388 records out 563405802 bytes (563 MB) copied, 64.9428 s, 8.7 MB/s $ bzip2 -c/tmp/foo.iso | dd bs=64K/dev/null 0+137488 records in 0+137488 records out 563150276 bytes (563 MB) copied, 445.201 s, 1.3 MB/s 255526 bytes less while six times slower.. Petar Bogdanovic This is extremely dependent on the contents of foo.iso. I don't think its a good test because you are only seeing 10% compression either way. There is a good chance that much of the data within your ISO is already compressed. When using data which is typically more compressible (text and other data that is not already compressed), the resulting size of something compressed with bzip2 can be much smaller than when compressed using gzip. It's true that it is much slower, but if he's talking about it being more efficient in terms of disk space used, then he is correct. -- SOLARIS 10 is the OS for Data Centers - provides features such as DTrace, Predictive Self Healing and Award Winning ZFS. Get Solaris 10 NOW http://p.sf.net/sfu/solaris-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression - other then GZIP
On 2010Feb10 8:50 AM, Steve Polyack wrote: On 2/10/2010 8:16 AM, Petar Bogdanovic wrote: I want use bzip2 to compress my file, because I thing bzip2 is more efficient... Really? [...] 255526 bytes less while six times slower.. This is extremely dependent on the contents of foo.iso. I don't think its a good test because you are only seeing 10% compression either way. There is a good chance that much of the data within your ISO is already compressed. When using data which is typically more compressible (text and other data that is not already compressed), the resulting size of something compressed with bzip2 can be much smaller than when compressed using gzip. It's true that it is much slower, but if he's talking about it being more efficient in terms of disk space used, then he is correct. xz/lzma is another consideration. At moderate compression levels, lzma seems to be about the same or slightly faster than bzip2 with a little better compression. At lower compression levels it seems like it's about as fast as gzip while compressing noticeably farther - at least in the small amount of testing I've done so far with the xz implementation of lzma compression. (The small amount of testing I've done so far suggests to me that xz with a compression level of 1 runs about as fast as gzip4 with compression at or better than gzip7, approaching bzip2 for some types of files. Cranking up to xz 6 or 7 runs a bit faster than bzip2 default but tends to give better compression.) -- SOLARIS 10 is the OS for Data Centers - provides features such as DTrace, Predictive Self Healing and Award Winning ZFS. Get Solaris 10 NOW http://p.sf.net/sfu/solaris-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression - other then GZIP
On 2/10/2010 10:36 AM, Sean M Clark wrote: On 2010Feb10 8:50 AM, Steve Polyack wrote: On 2/10/2010 8:16 AM, Petar Bogdanovic wrote: I want use bzip2 to compress my file, because I thing bzip2 is more efficient... Really? [...] 255526 bytes less while six times slower.. This is extremely dependent on the contents of foo.iso. I don't think its a good test because you are only seeing 10% compression either way. There is a good chance that much of the data within your ISO is already compressed. When using data which is typically more compressible (text and other data that is not already compressed), the resulting size of something compressed with bzip2 can be much smaller than when compressed using gzip. It's true that it is much slower, but if he's talking about it being more efficient in terms of disk space used, then he is correct. xz/lzma is another consideration. At moderate compression levels, lzma seems to be about the same or slightly faster than bzip2 with a little better compression. At lower compression levels it seems like it's about as fast as gzip while compressing noticeably farther - at least in the small amount of testing I've done so far with the xz implementation of lzma compression. (The small amount of testing I've done so far suggests to me that xz with a compression level of 1 runs about as fast as gzip4 with compression at or better than gzip7, approaching bzip2 for some types of files. Cranking up to xz 6 or 7 runs a bit faster than bzip2 default but tends to give better compression.) On the other side of the spectrum, LZO/LZO2 compression is available which greatly favors compression speed while still providing a decent compression ratio. I'd like to see these algorithms make their way into Bacula, but their doesn't seem to be much interest in doing so. I suppose its understandable, as GZIP is fairly flexible. -- SOLARIS 10 is the OS for Data Centers - provides features such as DTrace, Predictive Self Healing and Award Winning ZFS. Get Solaris 10 NOW http://p.sf.net/sfu/solaris-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression - other then GZIP
From: Sean M Clark smcl...@tamu.edu xz/lzma is another consideration. At moderate compression levels, lzma seems to be about the same or slightly faster than bzip2 with a little better compression. At lower compression levels it seems like it's about as fast as gzip while compressing noticeably farther - at least in the small amount of testing I've done so far with the xz implementation of lzma compression. (The small amount of testing I've done so far suggests to me that xz with a compression level of 1 runs about as fast as gzip4 with compression at or better than gzip7, approaching bzip2 for some types of files. Cranking up to xz 6 or 7 runs a bit faster than bzip2 default but tends to give better compression.) Judjing by the following becnhmarks, lzma seems quite resource hungry... http://tukaani.org/lzma/benchmarks.html JD -- SOLARIS 10 is the OS for Data Centers - provides features such as DTrace, Predictive Self Healing and Award Winning ZFS. Get Solaris 10 NOW http://p.sf.net/sfu/solaris-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression - other than GZIP {xz/lzma]
On 2010Feb10 10:31 AM, John Doe wrote: From: Sean M Clark smcl...@tamu.edu xz/lzma is another consideration. At moderate compression levels, lzma seems to be about the same or slightly faster than bzip2 with a little better compression. At lower compression levels it seems like it's about as fast as gzip while compressing noticeably farther - at least in the small amount of testing I've done so far with the xz implementation of lzma compression. [...] Judjing by the following becnhmarks, lzma seems quite resource hungry... http://tukaani.org/lzma/benchmarks.html Hmmm, those results more or less reflect what I remember from the testing I did. I don't remember the difference in compression speed between xz and bzip2 being quite as high as this, but that could either be due to xz being more efficient than lzmash and/or my own faulty memory. I note that lzma -2 tended to compress better than bzip2 could manage at any setting, and faster than default bzip2. I had forgotten about the much larger memory usage of xz, though in a modern context the amount still looks pretty trivial (even at the default setting it requires less than 90MB [the me of 5 years ago would be appalled to see me describe 90MB as trivial, but still...). lzma -2 only requires 12M in those results. Wouldn't necessarily bother with lzma compression on a tiny NAS box with only 32-64MB RAM in it, but I think it'd be a useful option on a real computer. I have no idea what would be involved in adding additional compression options to bacula-fd/bacula-sd, though. -- SOLARIS 10 is the OS for Data Centers - provides features such as DTrace, Predictive Self Healing and Award Winning ZFS. Get Solaris 10 NOW http://p.sf.net/sfu/solaris-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression - other then GZIP
On 02/10/10 10:36, Sean M Clark wrote: xz/lzma is another consideration. At moderate compression levels, lzma seems to be about the same or slightly faster than bzip2 with a little better compression. At lower compression levels it seems like it's about as fast as gzip while compressing noticeably farther - at least in the small amount of testing I've done so far with the xz implementation of lzma compression. I was going to mention xz myself. I just completed some rather more extensive tests. I'm using three example test files here. The first, a 590MB ISO of Windows XP Pro SP3, contains a large amount of already-compressed data, and can be expected to compress poorly. The second, an 8.5MB stripped ELF 32-bit LSB executable, can probably be expected to compress moderately well. The third, a ebook resaved in text format, isabout 1.5MB of English ASCII text and should compress very well. I'm compressing each with gzip default options, gzip -9, bzip2, xz default options, and xz -7. (The xz man page notes that compression settings above 7 are not recommended unless absolute maximum compression is necessary due to time and memory usage.) First, the WinXP ISO (whitespace adjusted for clarity): babylon5:alaric:~:10 $ ls -l winxp.iso -rw-r- 1 alaric users 617754624 Feb 10 10:24 winxp.iso babylon5:alaric:~:11 $ time gzip -c winxp.iso | dd bs=64K /dev/null 0+35022 records in 0+35022 records out 573799160 bytes (574 MB) copied, 78.782 s, 7.3 MB/s real1m18.935s user0m53.804s sys 0m4.357s compression: 7.12% compression/time: 0.0901 babylon5:alaric:~:12 $ time gzip -9 -c winxp.iso | dd bs=64K /dev/null 0+35013 records in 0+35013 records out 573652786 bytes (574 MB) copied, 111.185 s, 5.2 MB/s real1m51.207s user1m11.860s sys 0m4.905s compression: 7.14% compression/time: 0.0643 babylon5:alaric:~:13 $ time bzip2 -c winxp.iso | dd bs=64K /dev/null 0+140444 records in 0+140444 records out 575258513 bytes (575 MB) copied, 808.258 s, 712 kB/s real13m28.370s user10m11.257s sys 0m6.221s compression: 6.88% compression/time: 0.0085 babylon5:alaric:~:14 $ time xz -c winxp.iso | dd bs=64K /dev/null 0+69111 records in 0+69111 records out 566328660 bytes (566 MB) copied, 1395.3 s, 406 kB/s real23m15.341s user17m39.189s sys 0m9.664s compression: 8.43% compression/time: 0.0060 babylon5:alaric:~:15 $ time xz -7 -c winxp.iso | dd bs=64K /dev/null 0+69040 records in 0+69040 records out 565609576 bytes (566 MB) copied, 1512.2 s, 374 kB/s real25m12.247s user19m7.363s sys 0m10.943s compression: 8.45% compression/time: 0.0055 With this poorly compressible data, both gzip and gzip -9 yield better compression than bzip2, with roughly an order of magnitude higher throughput and lower CPU usage. The best compression on this file, by a hair, is achieved by xz -7, with default xz only 0.02% behind but taking 8% less time. The worst compression of 6.88% is bzip2, but it takes around half the time xz takes to do it, resulting in an actual compression/time score 50% better than xz. gzip achieves about 1.3% less compression than xz and about 0.25% better than bzip2, but does it 7 to 10 times faster than bzip2 and 12 to 20 times faster than xz. The best compression per unit time score is achieved by default gzip. The worst, xz -7, is an order of magnitude worse than gzip -9 in compression/time and achieves only 1.29% additional compression. Next, the ELF executable. babylon5:alaric:~:21 $ ls -l mplayer -rwxr-x--- 1 alaric users 8485168 Feb 10 12:04 mplayer babylon5:alaric:~:22 $ time gzip -c mplayer | dd bs=64K /dev/null 0+230 records in 0+230 records out 3752190 bytes (3.8 MB) copied, 1.26176 s, 3.0 MB/s real0m1.266s user0m1.032s sys 0m0.055s compression: 55.8% compression/time: 44.075 babylon5:alaric:~:23 $ time gzip -9 -c mplayer | dd bs=64K /dev/null 0+228 records in 0+228 records out 3734027 bytes (3.7 MB) copied, 2.76918 s, 1.3 MB/s real0m2.779s user0m2.119s sys 0m0.054s compression: 56% compression/time: 20.173 babylon5:alaric:~:24 $ time bzip2 -c mplayer | dd bs=64K /dev/null 0+880 records in 0+880 records out 3603587 bytes (3.6 MB) copied, 6.41314 s, 562 kB/s real0m6.426s user0m5.128s sys 0m0.050s compression: 57.5% compression/time: 8.948 babylon5:alaric:~:25 $ time xz -c mplayer | dd bs=64K /dev/null 0+362 records in 0+362 records out 2964084 bytes (3.0 MB) copied, 21.0693 s, 141 kB/s real0m21.098s user0m15.434s sys 0m0.316s compression: 65% compression/time: 3.081 babylon5:alaric:~:26 $ time xz -7 -c mplayer | dd bs=64K /dev/null 0+362 records in 0+362 records out 2964084 bytes (3.0 MB) copied, 19.8819 s, 149 kB/s real0m19.913s user0m15.347s sys 0m0.301s compression: 65% compression/time: 3.264 This is not all that dissimilar a picture. Interestingly, here, default xz and xz -7 achieve identical compression, but xz -7 accomplishes it slightly over a second faster. Both
