Re: [Bacula-users] bacula disk based compression strategy
On 08/09/2010 12:39 AM, Joseph Dickson wrote: Greetings.. I'm using Bacula in an all-disk based environment, using a disk based autochanger script. I'm wondering if anyone has come up with a magic recipe for dealing with the compression issues that are inherent with this type of strategy.. What I'm talking about mostly is that bacula-fd performance with software compression on is pretty slow, which I think I've read can be attributed to the single threaded compression algorithm that is being used.. I would much rather stream the data from the hosts uncompressed and then compress it once it reaches the SD, since my backup server tends to be fairly beefy compared to many of my hosts.. What have other people done to combat this issue? Is it worth my time investigating some of the virtual tape library software out there for Linux, which presumably implements reasonable compression on the virtual tape lib side? Or are people perhaps using a transparent compression filesystem with good results? Just trying to get a feel for what people have found works.. need to reduce the window to backup each host (i.e. up the throughput from the 2-3MB/sec I'm getting) but don't want to sacrifice the space of doing completely without compression :) Thanks! Joe Hi Joe, During the time the SD compression possibility exist, I'm using compression also on certain windows server. Some tests have prove to me that using a GZIP2 param in fileset is sufficient to obtain a good ratio between speed and compression. The normal GZIP set gzip to it's default compression level 6. Which consume more cpu (and time for a 1 ou 0.5% gain on total bytes ) This is what I obtain for a low-end server Win2K3 2GB Ram (P4 Dualcore 3.Ghz) with 2 sata raid soft harddrive. Client: chesne-fd 3.0.2 (18Jul09) Linux,Cross-compile,Win32 FileSet:FileSet-Chesne-Full 2008-05-11 15:52:17 Pool: Pool_FileWeeklyCHESNE (From Job resource) Catalog:MyCatalog (From Client resource) Storage:Store_FSCHESNE_WEEK (From Job resource) Scheduled time: 07-Aug-2010 18:00:00 Start time: 07-Aug-2010 18:01:46 End time: 07-Aug-2010 18:31:01 Elapsed time: 29 mins 15 secs Priority: 9 FD Files Written: 75,110 SD Files Written: 75,110 FD Bytes Written: 9,931,147,048 (9.931 GB) SD Bytes Written: 9,945,142,186 (9.945 GB) Rate: 5658.8 KB/s Software Compression: 46.3 % VSS:yes Encryption: no Accurate: no But it really depend on what type of data you have. My data on linux compress only at 10% and typical speed drop from a 40MB/s to 18MB/s with a gzip2 level. -- Bruno Friedmann br...@ioda-net.ch Ioda-Net Sàrl www.ioda-net.ch openSUSE Member User www.ioda.net/r/osu Blog www.ioda.net/r/blog fsfe fellowship www.fsfe.org (bruno.friedmann (at) fsfe.org ) tigerfoot on irc GPG KEY : D5C9B751C4653227 -- This SF.net email is sponsored by Make an app they can't live without Enter the BlackBerry Developer Challenge http://p.sf.net/sfu/RIM-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] bacula disk based compression strategy
Greetings.. I'm using Bacula in an all-disk based environment, using a disk based autochanger script. I'm wondering if anyone has come up with a magic recipe for dealing with the compression issues that are inherent with this type of strategy.. What I'm talking about mostly is that bacula-fd performance with software compression on is pretty slow, which I think I've read can be attributed to the single threaded compression algorithm that is being used.. I would much rather stream the data from the hosts uncompressed and then compress it once it reaches the SD, since my backup server tends to be fairly beefy compared to many of my hosts.. What have other people done to combat this issue? Is it worth my time investigating some of the virtual tape library software out there for Linux, which presumably implements reasonable compression on the virtual tape lib side? Or are people perhaps using a transparent compression filesystem with good results? Just trying to get a feel for what people have found works.. need to reduce the window to backup each host (i.e. up the throughput from the 2-3MB/sec I'm getting) but don't want to sacrifice the space of doing completely without compression :) Thanks! Joe -- This SF.net email is sponsored by Make an app they can't live without Enter the BlackBerry Developer Challenge http://p.sf.net/sfu/RIM-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula disk based compression strategy
I’m using Bacula in an all-disk based environment, using a disk based autochanger script. I’m wondering if anyone has come up with a magic recipe for dealing with the compression issues that are inherent with this type of strategy.. What I’m talking about mostly is that bacula-fd performance with software compression on is pretty slow, which I think I’ve read can be attributed to the single threaded compression algorithm that is being used.. I would much rather stream the data from the hosts uncompressed and then compress it once it reaches the SD, since my backup server tends to be fairly beefy compared to many of my hosts.. What have other people done to combat this issue? Is it worth my time investigating some of the virtual tape library software out there for Linux, which presumably implements reasonable compression on the virtual tape lib side? Or are people perhaps using a transparent compression filesystem with good results? Just trying to get a feel for what people have found works.. need to reduce the window to backup each host (i.e. up the throughput from the 2-3MB/sec I’m getting) but don’t want to sacrifice the space of doing completely without compression J I was thinking on doing this on the filesystem level and lazy when the server was not busy. I would like to see in linux what has been available for 10+ years with ntfs the ability to turn on compression on a per file basis at any time. There are several options for doing filesystem compression under linux but most of these are experimental. I have not spent much more thought about this because I mainly use tape for backup at work. At home I do use disk but that happens at 3AM and hopefully I am not still using the machine at that time.. John -- This SF.net email is sponsored by Make an app they can't live without Enter the BlackBerry Developer Challenge http://p.sf.net/sfu/RIM-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
