Re: [Bacula-users] bacula compression -- LTO4
Hello,
You might be interested in checking this thread:
https://sourceforge.net/p/bacula/mailman/message/36386670/
A perl script is mentioned there:
https://github.com/hreinecke/sg3_utils/issues/18 which can provide you
with e.g. daily reports of raw space remaining on tapes.
Thanks,
Adam
On 21/01/19 08:44, krashoverr...@free.fr wrote:
Hey there!
New question for you, about job compression (on tapes)
I'm running Bacula 7.4.4 server, with a 5.2.6 client (and PG db)
My client has 1.3 To to backup, i'm having an LTO4 drive, meaning 800Go native,
1.6To compressed, so I've set my pool to have a Maximum Volume Bytes to 1600G
I've read on Bacula docs that for tapes, it's better not to configure
compression in FileSet, and leave the hardware part do the compression
So what I did (as docs said) is to
- cat /sys/class/scsi_tape/nst0/default_compression which has "1" value
+
root@server:~# tapeinfo -f /dev/nst0
Product Type: Tape Drive
Vendor ID: 'TANDBERG'
Product ID: 'LTO-4 HH'
Revision: 'U519'
Attached Changer API: No
SerialNumber: 'HU1023AMW9'
MinBlock: 1
MaxBlock: 16777215
SCSI ID: 1
SCSI LUN: 0
Ready: yes
BufferedMode: yes
Medium Type: Not Loaded
Density Code: 0x46
BlockSize: 0
DataCompEnabled: yes
DataCompCapable: yes
DataDeCompEnabled: yes
CompType: 0x1
DeCompType: 0x1
BOP: yes
Block Position: 0
Partition 0 Remaining Kbytes: 800226
Partition 0 Size in Kbytes: 800226
ActivePartition: 0
EarlyWarningSize: 0
NumPartitions: 0
MaxPartitions: 0
So the compression seems to be enabled here
The fact is, when I run my backup, only ~1.1 To is used on the tape before
being Full
+-++---+-+---+--+--+-+--+---+---+-+---+
| mediaid | volumename | volstatus | enabled | volbytes | volfiles |
volretention | recycle | slot | inchanger | mediatype | lastwritten |
expiresin |
+-++---+-+---+--+--+-+--+---+---+-+---+
| 2 | | Full | 1 | 1,090,410,854,400 |1,090 |
2,592,000 | 1 |0 | 0 | LTO-4 | 2019-01-21 03:25:59 |
2,569,770 |
+-++---+-+---+--+--+-+--+---+---+-+---+
On the error mail, I can see that
21-Jan 03:26 server-sd JobId 3: End of medium on Volume "x"
Bytes=1,090,410,854,400 Blocks=16,902,449 at 21-Jan-2019 03:26.
The number of blocks is almost the same than in the tapeinfo command.
Is the "problem" coming from here ?
Anything I can configure to have more that 1.1To used ?
Thank you!
---
Pool {
Name = QUOT
Pool Type = Backup
Volume Retention = 30 days
Storage = Lecteur_LTO4
Recycle = yes
AutoPrune = yes
RecyclePool = QUOT
Maximum Volume Bytes = 1600G
}
Storage {
Name = Lecteur_LTO4
Password = --
Address = server
SDPort = 9103
Device = Lecteur_LTO4
Media Type = LTO-4
}
Device {
Name = Lecteur_LTO4
Archive Device = /dev/nst0
Media Type = LTO-4
LabelMedia = no
Random Access = no
AutomaticMount = yes
RemovableMedia = no
AlwaysOpen = yes
}
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula compression -- LTO4
On Mon, Jan 21, 2019, at 09:44, krashoverr...@free.fr wrote: > My client has 1.3 To to backup, i'm having an LTO4 drive, meaning 800Go > native, 1.6To compressed, so I've set my pool to have a Maximum Volume > Bytes to 1600G 800 GB is the real capacity of an LTO-4 cartridge. The 1,6 TB "compressed capacity" is purely a marketing number based on the unfounded assumption that your data can be compressed to 50%. Compression depends heavily on the type of data. Much redundancy (eg. logfiles) -> much compression Little redundancy (eg. videos) -> little or no compression > The fact is, when I run my backup, only ~1.1 To is used on the tape > before being Full This confirms that compression is active (otherwise it would be full after 800 GB) and it is probably all you can get out of compression with your data. -- Tilman Schmidt til...@imap.cc ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] bacula compression -- LTO4
Hey there!
New question for you, about job compression (on tapes)
I'm running Bacula 7.4.4 server, with a 5.2.6 client (and PG db)
My client has 1.3 To to backup, i'm having an LTO4 drive, meaning 800Go native,
1.6To compressed, so I've set my pool to have a Maximum Volume Bytes to 1600G
I've read on Bacula docs that for tapes, it's better not to configure
compression in FileSet, and leave the hardware part do the compression
So what I did (as docs said) is to
- cat /sys/class/scsi_tape/nst0/default_compression which has "1" value
+
root@server:~# tapeinfo -f /dev/nst0
Product Type: Tape Drive
Vendor ID: 'TANDBERG'
Product ID: 'LTO-4 HH'
Revision: 'U519'
Attached Changer API: No
SerialNumber: 'HU1023AMW9'
MinBlock: 1
MaxBlock: 16777215
SCSI ID: 1
SCSI LUN: 0
Ready: yes
BufferedMode: yes
Medium Type: Not Loaded
Density Code: 0x46
BlockSize: 0
DataCompEnabled: yes
DataCompCapable: yes
DataDeCompEnabled: yes
CompType: 0x1
DeCompType: 0x1
BOP: yes
Block Position: 0
Partition 0 Remaining Kbytes: 800226
Partition 0 Size in Kbytes: 800226
ActivePartition: 0
EarlyWarningSize: 0
NumPartitions: 0
MaxPartitions: 0
So the compression seems to be enabled here
The fact is, when I run my backup, only ~1.1 To is used on the tape before
being Full
+-++---+-+---+--+--+-+--+---+---+-+---+
| mediaid | volumename | volstatus | enabled | volbytes | volfiles |
volretention | recycle | slot | inchanger | mediatype | lastwritten |
expiresin |
+-++---+-+---+--+--+-+--+---+---+-+---+
| 2 | | Full | 1 | 1,090,410,854,400 |1,090 |
2,592,000 | 1 |0 | 0 | LTO-4 | 2019-01-21 03:25:59 |
2,569,770 |
+-++---+-+---+--+--+-+--+---+---+-+---+
On the error mail, I can see that
21-Jan 03:26 server-sd JobId 3: End of medium on Volume "x"
Bytes=1,090,410,854,400 Blocks=16,902,449 at 21-Jan-2019 03:26.
The number of blocks is almost the same than in the tapeinfo command.
Is the "problem" coming from here ?
Anything I can configure to have more that 1.1To used ?
Thank you!
---
Pool {
Name = QUOT
Pool Type = Backup
Volume Retention = 30 days
Storage = Lecteur_LTO4
Recycle = yes
AutoPrune = yes
RecyclePool = QUOT
Maximum Volume Bytes = 1600G
}
Storage {
Name = Lecteur_LTO4
Password = --
Address = server
SDPort = 9103
Device = Lecteur_LTO4
Media Type = LTO-4
}
Device {
Name = Lecteur_LTO4
Archive Device = /dev/nst0
Media Type = LTO-4
LabelMedia = no
Random Access = no
AutomaticMount = yes
RemovableMedia = no
AlwaysOpen = yes
}
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression on Windows
I'm currently using this Fileset in my windows servers and it's working fine
(bacula v. 2.4.4).
FileSet {
Name = "XXXFileset"
Enable VSS = yes
Include {
Options {
signature = MD5
compression=GZIP
}
File = "C:/"
}
}
Maybe GZIP9 isn't working?
On Jueves, 6 de Mayo de 2010 10:00:14 Carlo Filippetto wrote:
> [--]
>
> > Can you be a little more specific about "don't worl"? What doesn't
> > work? Compression? Backups?
> >
> > If you're not getting compression and you're asking about that, from the
> > fragmentary bits of configuration you've posted above you appear to have
> > compression both turned on and turned off. This probably isn't helping.
>
> The bck is good, but there's no compression, on Windows, on-linux there is
>
> I can't see where the compression in turned off... can you help me?
>
> my actual FileSet:
>
> FileSet {
> Name = FS-test-windows
> Enable VSS = yes
> Ignore FileSet Changes = no
> Include {
> File = C:/Programmi/Test
> Options {
> compression = GZIP9 #aggiungo compressione massima
> signature = MD5 #aggiungo MD5
> fstype = ntfs
> }
> }
> }
>
> ---
> --- ___
> Bacula-users mailing list
> Bacula-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>
--
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression on Windows
[--]
> Can you be a little more specific about "don't worl"? What doesn't
> work? Compression? Backups?
>
> If you're not getting compression and you're asking about that, from the
> fragmentary bits of configuration you've posted above you appear to have
> compression both turned on and turned off. This probably isn't helping.
>
The bck is good, but there's no compression, on Windows, on-linux there is
I can't see where the compression in turned off... can you help me?
my actual FileSet:
FileSet {
Name = FS-test-windows
Enable VSS = yes
Ignore FileSet Changes = no
Include {
File = C:/Programmi/Test
Options {
compression = GZIP9 #aggiungo compressione massima
signature = MD5 #aggiungo MD5
fstype = ntfs
}
}
}
--
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression on Windows
2010/5/4 Uwe Schuerkamp :
> On Mon, May 03, 2010 at 04:35:20PM -0400, Phil Stracchino wrote:
>> On 05/03/10 06:02, Carlo Filippetto wrote:
>> > Hi all,
>> > I have a problem,
>> > I had installed bacula 5.0.1. on Winows client, and work fine, but I
>> > have no compression
>> >
>> > Software Compression: None
>> >
>> > while
>> >
>> > FileSet {
>> > Name = FS-test-windows
>> > Enable VSS = yes
>> > Ignore FileSet Changes = no
>> > Include {
>> > File = C:/Programmi/Test
>> > Options {
>> > compression = GZIP
>> > signature = MD5
>> > }
>> > }
>> > }
>> >
>
>
> Hm, I've never seen Option *inside* an Include block, maybe that's the
> problem unless that's a new feature in 5.x?
>
I use this configuration since 1 hear with bacula 2.4.1 and on linux
it works well, but Linux is now Windows, luckily :-)
I will try to change it, but in witch way??
as this:
FileSet {
Name = FS-test-windows
Enable VSS = yes
Ignore FileSet Changes = no
Include {
File = C:/Programmi/Test
}
Options {
compression = GZIP
signature = MD5
}
}
Thank's
--
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression on Windows
On Mon, May 03, 2010 at 04:35:20PM -0400, Phil Stracchino wrote:
> On 05/03/10 06:02, Carlo Filippetto wrote:
> > Hi all,
> > I have a problem,
> > I had installed bacula 5.0.1. on Winows client, and work fine, but I
> > have no compression
> >
> > Software Compression: None
> >
> > while
> >
> > FileSet {
> > Name = FS-test-windows
> > Enable VSS = yes
> > Ignore FileSet Changes = no
> > Include {
> > File = C:/Programmi/Test
> > Options {
> > compression = GZIP
> > signature = MD5
> > }
> > }
> > }
> >
Hm, I've never seen Option *inside* an Include block, maybe that's the
problem unless that's a new feature in 5.x?
All the best, Uwe
> >
> > I read that is the same fd that make this compression.. why it don't work??
>
> Can you be a little more specific about "don't worl"? What doesn't
> work? Compression? Backups?
>
> If you're not getting compression and you're asking about that, from the
> fragmentary bits of configuration you've posted above you appear to have
> compression both turned on and turned off. This probably isn't helping.
>
>
> --
> Phil Stracchino, CDK#2 DoD#299792458 ICBM: 43.5607, -71.355
> ala...@caerllewys.net ala...@metrocast.net p...@co.ordinate.org
> Renaissance Man, Unix ronin, Perl hacker, Free Stater
> It's not the years, it's the mileage.
>
> --
> ___
> Bacula-users mailing list
> Bacula-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bacula-users
--
uwe.schuerk...@nionex.net fon: [+49] 5242.91 - 4740, fax:-69 72
Hauptsitz: Avenwedder Str. 55, D-33311 Gütersloh, Germany
Registergericht Gütersloh HRB 4196, Geschäftsführer: H. Gosewehr, D. Suda
NIONEX ist ein Unternehmen der DirectGroup Germany www.directgroupgermany.de
--
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression on Windows
On 05/03/10 06:02, Carlo Filippetto wrote:
> Hi all,
> I have a problem,
> I had installed bacula 5.0.1. on Winows client, and work fine, but I
> have no compression
>
> Software Compression: None
>
> while
>
> FileSet {
> Name = FS-test-windows
> Enable VSS = yes
> Ignore FileSet Changes = no
> Include {
> File = C:/Programmi/Test
> Options {
> compression = GZIP
> signature = MD5
> }
> }
> }
>
>
> I read that is the same fd that make this compression.. why it don't work??
Can you be a little more specific about "don't worl"? What doesn't
work? Compression? Backups?
If you're not getting compression and you're asking about that, from the
fragmentary bits of configuration you've posted above you appear to have
compression both turned on and turned off. This probably isn't helping.
--
Phil Stracchino, CDK#2 DoD#299792458 ICBM: 43.5607, -71.355
ala...@caerllewys.net ala...@metrocast.net p...@co.ordinate.org
Renaissance Man, Unix ronin, Perl hacker, Free Stater
It's not the years, it's the mileage.
--
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression on Windows
On 05/03/10 15:08, Jari Fredriksson wrote:
> On 3.5.2010 13:02, Carlo Filippetto wrote:
>> Hi all,
>> I have a problem,
>> I had installed bacula 5.0.1. on Winows client, and work fine, but I
>> have no compression
>>
>> Software Compression: None
>>
>> while
>>
>> FileSet {
>> Name = FS-test-windows
>> Enable VSS = yes
>> Ignore FileSet Changes = no
>> Include {
>> File = C:/Programmi/Test
>> Options {
>> compression = GZIP
>> signature = MD5
>> }
>> }
>> }
>>
>>
>> I read that is the same fd that make this compression.. why it don't work??
>>
>
> Because it does not backup anything.
>
> The FileSet should be as follows, if I'm not mistaken:
>
> FileSet {
> Name = FS-test-windows
> Enable VSS = yes
> Ignore FileSet Changes = no
> Include {
> Options {
> compression = GZIP
> signature = MD5
> }
> }
> File = C:/Programmi/Test
> }
Actually, THIS FileSet will not back up anything, because it does not
include anything. File (and related) directives go within the Include{}
and Exclude{} directives. That one isn't.
--
Phil Stracchino, CDK#2 DoD#299792458 ICBM: 43.5607, -71.355
ala...@caerllewys.net ala...@metrocast.net p...@co.ordinate.org
Renaissance Man, Unix ronin, Perl hacker, Free Stater
It's not the years, it's the mileage.
--
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression on Windows
On 3.5.2010 13:02, Carlo Filippetto wrote:
> Hi all,
> I have a problem,
> I had installed bacula 5.0.1. on Winows client, and work fine, but I
> have no compression
>
> Software Compression: None
>
> while
>
> FileSet {
> Name = FS-test-windows
> Enable VSS = yes
> Ignore FileSet Changes = no
> Include {
> File = C:/Programmi/Test
> Options {
> compression = GZIP
> signature = MD5
> }
> }
> }
>
>
> I read that is the same fd that make this compression.. why it don't work??
>
Because it does not backup anything.
The FileSet should be as follows, if I'm not mistaken:
FileSet {
Name = FS-test-windows
Enable VSS = yes
Ignore FileSet Changes = no
Include {
Options {
compression = GZIP
signature = MD5
}
}
File = C:/Programmi/Test
}
--
http://www.iki.fi/jarif/
He draweth out the thread of his verbosity finer than the staple of his
argument.
-- William Shakespeare, "Love's Labour's Lost"
signature.asc
Description: OpenPGP digital signature
--
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Bacula Compression on Windows
Hi all,
I have a problem,
I had installed bacula 5.0.1. on Winows client, and work fine, but I
have no compression
Software Compression: None
while
FileSet {
Name = FS-test-windows
Enable VSS = yes
Ignore FileSet Changes = no
Include {
File = C:/Programmi/Test
Options {
compression = GZIP
signature = MD5
}
}
}
I read that is the same fd that make this compression.. why it don't work??
CIAO
---
Carlo Filippetto
--
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression - other then GZIP
On 02/10/10 10:36, Sean M Clark wrote: > xz/lzma is another consideration. At moderate compression levels, lzma > seems to be about the same or slightly faster than bzip2 with a little > better compression. At lower compression levels it seems like it's > about as fast as gzip while compressing noticeably farther - at least > in the small amount of testing I've done so far with the "xz" > implementation of lzma compression. I was going to mention xz myself. I just completed some rather more extensive tests. I'm using three example test files here. The first, a 590MB ISO of Windows XP Pro SP3, contains a large amount of already-compressed data, and can be expected to compress poorly. The second, an 8.5MB stripped ELF 32-bit LSB executable, can probably be expected to compress moderately well. The third, a ebook resaved in text format, isabout 1.5MB of English ASCII text and should compress very well. I'm compressing each with gzip default options, gzip -9, bzip2, xz default options, and xz -7. (The xz man page notes that compression settings above 7 are not recommended unless absolute maximum compression is necessary due to time and memory usage.) First, the WinXP ISO (whitespace adjusted for clarity): babylon5:alaric:~:10 $ ls -l winxp.iso -rw-r- 1 alaric users 617754624 Feb 10 10:24 winxp.iso babylon5:alaric:~:11 $ time gzip -c < winxp.iso | dd bs=64K >/dev/null 0+35022 records in 0+35022 records out 573799160 bytes (574 MB) copied, 78.782 s, 7.3 MB/s real1m18.935s user0m53.804s sys 0m4.357s compression: 7.12% compression/time: 0.0901 babylon5:alaric:~:12 $ time gzip -9 -c < winxp.iso | dd bs=64K >/dev/null 0+35013 records in 0+35013 records out 573652786 bytes (574 MB) copied, 111.185 s, 5.2 MB/s real1m51.207s user1m11.860s sys 0m4.905s compression: 7.14% compression/time: 0.0643 babylon5:alaric:~:13 $ time bzip2 -c < winxp.iso | dd bs=64K >/dev/null 0+140444 records in 0+140444 records out 575258513 bytes (575 MB) copied, 808.258 s, 712 kB/s real13m28.370s user10m11.257s sys 0m6.221s compression: 6.88% compression/time: 0.0085 babylon5:alaric:~:14 $ time xz -c < winxp.iso | dd bs=64K >/dev/null 0+69111 records in 0+69111 records out 566328660 bytes (566 MB) copied, 1395.3 s, 406 kB/s real23m15.341s user17m39.189s sys 0m9.664s compression: 8.43% compression/time: 0.0060 babylon5:alaric:~:15 $ time xz -7 -c < winxp.iso | dd bs=64K >/dev/null 0+69040 records in 0+69040 records out 565609576 bytes (566 MB) copied, 1512.2 s, 374 kB/s real25m12.247s user19m7.363s sys 0m10.943s compression: 8.45% compression/time: 0.0055 With this poorly compressible data, both gzip and gzip -9 yield better compression than bzip2, with roughly an order of magnitude higher throughput and lower CPU usage. The best compression on this file, by a hair, is achieved by xz -7, with default xz only 0.02% behind but taking 8% less time. The worst compression of 6.88% is bzip2, but it takes around half the time xz takes to do it, resulting in an actual compression/time score 50% better than xz. gzip achieves about 1.3% less compression than xz and about 0.25% better than bzip2, but does it 7 to 10 times faster than bzip2 and 12 to 20 times faster than xz. The best compression per unit time score is achieved by default gzip. The worst, xz -7, is an order of magnitude worse than gzip -9 in compression/time and achieves only 1.29% additional compression. Next, the ELF executable. babylon5:alaric:~:21 $ ls -l mplayer -rwxr-x--- 1 alaric users 8485168 Feb 10 12:04 mplayer babylon5:alaric:~:22 $ time gzip -c < mplayer | dd bs=64K >/dev/null 0+230 records in 0+230 records out 3752190 bytes (3.8 MB) copied, 1.26176 s, 3.0 MB/s real0m1.266s user0m1.032s sys 0m0.055s compression: 55.8% compression/time: 44.075 babylon5:alaric:~:23 $ time gzip -9 -c < mplayer | dd bs=64K >/dev/null 0+228 records in 0+228 records out 3734027 bytes (3.7 MB) copied, 2.76918 s, 1.3 MB/s real0m2.779s user0m2.119s sys 0m0.054s compression: 56% compression/time: 20.173 babylon5:alaric:~:24 $ time bzip2 -c < mplayer | dd bs=64K >/dev/null 0+880 records in 0+880 records out 3603587 bytes (3.6 MB) copied, 6.41314 s, 562 kB/s real0m6.426s user0m5.128s sys 0m0.050s compression: 57.5% compression/time: 8.948 babylon5:alaric:~:25 $ time xz -c < mplayer | dd bs=64K >/dev/null 0+362 records in 0+362 records out 2964084 bytes (3.0 MB) copied, 21.0693 s, 141 kB/s real0m21.098s user0m15.434s sys 0m0.316s compression: 65% compression/time: 3.081 babylon5:alaric:~:26 $ time xz -7 -c < mplayer | dd bs=64K >/dev/null 0+362 records in 0+362 records out 2964084 bytes (3.0 MB) copied, 19.8819 s, 149 kB/s real0m19.913s user0m15.347s sys 0m0.301s compression: 65% compression/time: 3.264 This is not all that dissimilar a picture. Interestingly, here, default xz and xz -7 achieve identical compression, but xz -7 accomplishes it slightly o
Re: [Bacula-users] Bacula Compression - other than GZIP {xz/lzma]
On 2010Feb10 10:31 AM, John Doe wrote: > From: Sean M Clark >> xz/lzma is another consideration. At moderate compression levels, lzma >> seems to be about the same or slightly faster than bzip2 with a little >> better compression. At lower compression levels it seems like it's >> about as fast as gzip while compressing noticeably farther - at least >> in the small amount of testing I've done so far with the "xz" >> implementation of lzma compression. >>[...] > Judjing by the following becnhmarks, lzma seems quite resource hungry... > http://tukaani.org/lzma/benchmarks.html Hmmm, those results more or less reflect what I remember from the testing I did. I don't remember the difference in compression speed between xz and bzip2 being quite as high as this, but that could either be due to xz being more efficient than "lzmash" and/or my own faulty memory. I note that lzma -2 tended to compress better than bzip2 could manage at any setting, and faster than "default" bzip2. I had forgotten about the much larger memory usage of xz, though in a modern context the amount still looks pretty trivial (even at the default setting it requires less than 90MB [the "me" of 5 years ago would be appalled to see me describe "90MB" as "trivial", but still...). lzma -2 only requires 12M in those results. Wouldn't necessarily bother with lzma compression on a tiny NAS box with only 32-64MB RAM in it, but I think it'd be a useful option on a "real" computer. I have no idea what would be involved in adding additional compression options to bacula-fd/bacula-sd, though. -- SOLARIS 10 is the OS for Data Centers - provides features such as DTrace, Predictive Self Healing and Award Winning ZFS. Get Solaris 10 NOW http://p.sf.net/sfu/solaris-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression - other then GZIP
From: Sean M Clark > xz/lzma is another consideration. At moderate compression levels, lzma > seems to be about the same or slightly faster than bzip2 with a little > better compression. At lower compression levels it seems like it's > about as fast as gzip while compressing noticeably farther - at least > in the small amount of testing I've done so far with the "xz" > implementation of lzma compression. > > (The small amount of testing I've done so far suggests to me that xz > with a compression level of 1 runs about as fast as gzip4 with > compression at or better than gzip7, approaching bzip2 for some types of > files. Cranking up to xz 6 or 7 runs a bit faster than bzip2 default > but tends to give better compression.) Judjing by the following becnhmarks, lzma seems quite resource hungry... http://tukaani.org/lzma/benchmarks.html JD -- SOLARIS 10 is the OS for Data Centers - provides features such as DTrace, Predictive Self Healing and Award Winning ZFS. Get Solaris 10 NOW http://p.sf.net/sfu/solaris-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression - other then GZIP
On 2/10/2010 10:36 AM, Sean M Clark wrote: > On 2010Feb10 8:50 AM, Steve Polyack wrote: > >> On 2/10/2010 8:16 AM, Petar Bogdanovic wrote: >> I want use bzip2 to compress my file, because I thing bzip2 is more efficient... >>> Really? >>> > [...] > >>> 255526 bytes less while six times slower.. >>> >>> >> This is extremely dependent on the contents of foo.iso. I don't think >> its a good test because you are only seeing 10% compression either way. >> There is a good chance that much of the data within your ISO is already >> compressed. When using data which is typically more compressible (text >> and other data that is not already compressed), the resulting size of >> something compressed with bzip2 can be much smaller than when compressed >> using gzip. It's true that it is much slower, but if he's talking about >> it being more efficient in terms of disk space used, then he is correct. >> > xz/lzma is another consideration. At moderate compression levels, lzma > seems to be about the same or slightly faster than bzip2 with a little > better compression. At lower compression levels it seems like it's > about as fast as gzip while compressing noticeably farther - at least > in the small amount of testing I've done so far with the "xz" > implementation of lzma compression. > > (The small amount of testing I've done so far suggests to me that xz > with a compression level of 1 runs about as fast as gzip4 with > compression at or better than gzip7, approaching bzip2 for some types of > files. Cranking up to xz 6 or 7 runs a bit faster than bzip2 default > but tends to give better compression.) > > On the other side of the spectrum, LZO/LZO2 compression is available which greatly favors compression speed while still providing a decent compression ratio. I'd like to see these algorithms make their way into Bacula, but their doesn't seem to be much interest in doing so. I suppose its understandable, as GZIP is fairly flexible. -- SOLARIS 10 is the OS for Data Centers - provides features such as DTrace, Predictive Self Healing and Award Winning ZFS. Get Solaris 10 NOW http://p.sf.net/sfu/solaris-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression - other then GZIP
On 2010Feb10 8:50 AM, Steve Polyack wrote: > On 2/10/2010 8:16 AM, Petar Bogdanovic wrote: >>> I want use bzip2 to compress my file, because I thing bzip2 is more >>> efficient... >>> >> Really? [...] >> 255526 bytes less while six times slower.. >> > This is extremely dependent on the contents of foo.iso. I don't think > its a good test because you are only seeing 10% compression either way. > There is a good chance that much of the data within your ISO is already > compressed. When using data which is typically more compressible (text > and other data that is not already compressed), the resulting size of > something compressed with bzip2 can be much smaller than when compressed > using gzip. It's true that it is much slower, but if he's talking about > it being more efficient in terms of disk space used, then he is correct. xz/lzma is another consideration. At moderate compression levels, lzma seems to be about the same or slightly faster than bzip2 with a little better compression. At lower compression levels it seems like it's about as fast as gzip while compressing noticeably farther - at least in the small amount of testing I've done so far with the "xz" implementation of lzma compression. (The small amount of testing I've done so far suggests to me that xz with a compression level of 1 runs about as fast as gzip4 with compression at or better than gzip7, approaching bzip2 for some types of files. Cranking up to xz 6 or 7 runs a bit faster than bzip2 default but tends to give better compression.) -- SOLARIS 10 is the OS for Data Centers - provides features such as DTrace, Predictive Self Healing and Award Winning ZFS. Get Solaris 10 NOW http://p.sf.net/sfu/solaris-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression - other then GZIP
On 2/10/2010 8:16 AM, Petar Bogdanovic wrote: >> I want use bzip2 to compress my file, because I thing bzip2 is more >> efficient... >> > Really? > > $ du -m /tmp/foo.iso > 625 /tmp/foo.iso > $ gzip -c/dev/null > 0+34388 records in > 0+34388 records out > 563405802 bytes (563 MB) copied, 64.9428 s, 8.7 MB/s > $ bzip2 -c/dev/null > 0+137488 records in > 0+137488 records out > 563150276 bytes (563 MB) copied, 445.201 s, 1.3 MB/s > > 255526 bytes less while six times slower.. > > Petar Bogdanovic > > This is extremely dependent on the contents of foo.iso. I don't think its a good test because you are only seeing 10% compression either way. There is a good chance that much of the data within your ISO is already compressed. When using data which is typically more compressible (text and other data that is not already compressed), the resulting size of something compressed with bzip2 can be much smaller than when compressed using gzip. It's true that it is much slower, but if he's talking about it being more efficient in terms of disk space used, then he is correct. -- SOLARIS 10 is the OS for Data Centers - provides features such as DTrace, Predictive Self Healing and Award Winning ZFS. Get Solaris 10 NOW http://p.sf.net/sfu/solaris-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression - other then GZIP
On Wed, Feb 10, 2010 at 09:05:19AM -0200, Gilberto Nunes wrote: > > (...) gzip or I can use another compress program... No. > I want use bzip2 to compress my file, because I thing bzip2 is more > efficient... Really? $ du -m /tmp/foo.iso 625 /tmp/foo.iso $ gzip -c /dev/null 0+34388 records in 0+34388 records out 563405802 bytes (563 MB) copied, 64.9428 s, 8.7 MB/s $ bzip2 -c /dev/null 0+137488 records in 0+137488 records out 563150276 bytes (563 MB) copied, 445.201 s, 1.3 MB/s 255526 bytes less while six times slower.. Petar Bogdanovic -- SOLARIS 10 is the OS for Data Centers - provides features such as DTrace, Predictive Self Healing and Award Winning ZFS. Get Solaris 10 NOW http://p.sf.net/sfu/solaris-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression - other then GZIP
On 02/10/10 06:05, Gilberto Nunes wrote: > Hi folks... > > I need to know if the Compression flag on FileSet must be gzip or I can > use another compress program... > > I want use bzip2 to compress my file, because I thing bzip2 is more > efficient... It is true that bzip2 is more efficient than gzip, but it is also slower and very much more CPU-intensive. These are things to keep in mind. gzip may not be the best compression out there, but it is fast. -- Phil Stracchino, CDK#2 DoD#299792458 ICBM: 43.5607, -71.355 ala...@caerllewys.net ala...@metrocast.net p...@co.ordinate.org Renaissance Man, Unix ronin, Perl hacker, Free Stater It's not the years, it's the mileage. -- SOLARIS 10 is the OS for Data Centers - provides features such as DTrace, Predictive Self Healing and Award Winning ZFS. Get Solaris 10 NOW http://p.sf.net/sfu/solaris-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression - other then GZIP
On 10.02.2010 / 09:05:19 -0200, Gilberto Nunes wrote: > Hi folks... > > I need to know if the Compression flag on FileSet must be gzip or I can > use another compress program... > > I want use bzip2 to compress my file, because I thing bzip2 is more > efficient... or even Parallel BZIP2, see http://compression.ca/pbzip2/ Thanks. -- SOLARIS 10 is the OS for Data Centers - provides features such as DTrace, Predictive Self Healing and Award Winning ZFS. Get Solaris 10 NOW http://p.sf.net/sfu/solaris-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Bacula Compression - other then GZIP
Hi folks... I need to know if the Compression flag on FileSet must be gzip or I can use another compress program... I want use bzip2 to compress my file, because I thing bzip2 is more efficient... Thanks for any help... Regards Gilberto Nunes Ferreira TI Selbetti Gestão de Documentos Telefone: +55 (47) 3441-6004 Celular: +55 (47) 8861-6672 -- SOLARIS 10 is the OS for Data Centers - provides features such as DTrace, Predictive Self Healing and Award Winning ZFS. Get Solaris 10 NOW http://p.sf.net/sfu/solaris-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression
On Mon, Jul 20, 2009 at 11:43 AM, Victor Hugo dos Santos wrote: > On Mon, Jul 20, 2009 at 10:57 AM, John Drescher wrote: >> On Mon, Jul 20, 2009 at 10:37 AM, Victor Hugo dos >> Santos wrote: > > [...] > >> You are not using the same exact dataset (145GB versus 140GB source >> data ) in both tests so its meaningless that GZIP6 is better than >> GZIP9. > > yes.. one report is a little more big that other, but are the same > files in distinct dates and "only" 5GB of extra data !!! > It looks like the 5 GB of extra data is not as compressible as the rest of the 140GB. > > IMHO, serves of base of comparative. > > In the compression rate (point more important) of this 3 reports, the > difference between GZIP6 and GZIP9 is of 0.1% I would bet that if you did the test again with the 145GB but with the default GZIP6 there would be more than a .1 difference and it would be GZIP6 that would be the one with the worse compression rate. > but elapsed time is extremely distinct. > That is expected and why bacula defaults to GZIP6 instead of GZIP9 or some other compression. John -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/Challenge ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression
On Mon, Jul 20, 2009 at 10:57 AM, John Drescher wrote: > On Mon, Jul 20, 2009 at 10:37 AM, Victor Hugo dos > Santos wrote: [...] > You are not using the same exact dataset (145GB versus 140GB source > data ) in both tests so its meaningless that GZIP6 is better than > GZIP9. yes.. one report is a little more big that other, but are the same files in distinct dates and "only" 5GB of extra data !!! IMHO, serves of base of comparative. In the compression rate (point more important) of this 3 reports, the difference between GZIP6 and GZIP9 is of 0.1% but elapsed time is extremely distinct. bye -- -- Victor Hugo dos Santos Linux Counter #224399 -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/Challenge ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression
On Mon, Jul 20, 2009 at 10:37 AM, Victor Hugo dos Santos wrote: > On Fri, Jul 17, 2009 at 6:53 PM, John Drescher wrote: >> On Fri, Jul 17, 2009 at 5:59 PM, Simone S. >> Santiago wrote: >>> Hey co-workers, >>> >>> I wonder if it is possible improve the Bacula compression? >>> I am using "compression = GZIP" but sometimes it compress only 15% of >>> all volume. >>> >>> Note: My backup is FILE Type. >>> >> Try: >> >> compression=GZIP9 >> >> http://www.bacula.org/en/rel-manual/Configuring_Director.html#SECTION00147 >> >> Note that this will take much longer to compress however. > > > .. GZIP9 no represent exactly best compression method.. > > I send one or two mail to list, about this theme and I ran various > tests with distinct values to GZIP and basically this are the results: > > > > Reporte de hoy (con GZIP=5 y 140GB) > > Scheduled time: 17-Mar-2009 04:30:00 > Start time: 17-Mar-2009 04:51:20 > End time: 17-Mar-2009 05:44:24 > Elapsed time: 53 mins 4 secs > FD Bytes Written: 18,119,898,810 (18.11 GB) > SD Bytes Written: 18,119,923,547 (18.11 GB) > Rate: 5690.9 KB/s > Software Compression: 87.9 % > > > > Reporte de ayer (con GZIP=6 y 140GB) > > Scheduled time: 16-Mar-2009 04:30:00 > Start time: 16-Mar-2009 04:35:28 > End time: 16-Mar-2009 05:55:55 > Elapsed time: 1 hour 20 mins 27 secs > FD Bytes Written: 17,570,761,480 (17.57 GB) > SD Bytes Written: 17,570,786,217 (17.57 GB) > Rate: 3640.1 KB/s > Software Compression: 88.2 % > > > > Reporte del domingo pasado (con GZIP=9 y 145GB) > > Scheduled time: 08-Mar-2009 04:30:00 > Start time: 08-Mar-2009 04:30:27 > End time: 08-Mar-2009 08:34:00 > Elapsed time: 4 hours 3 mins 33 secs > Priority: 10 > FD Bytes Written: 18,452,613,999 (18.45 GB) > SD Bytes Written: 18,458,242,303 (18.45 GB) > Rate: 1262.8 KB/s > Software Compression: 88.1 % > > > > well.. as you can see, for me, the GZIP=6 (default) is the bester > method...because is more faster and compress better that GZIP9. > > bye > You are not using the same exact dataset (145GB versus 140GB source data ) in both tests so its meaningless that GZIP6 is better than GZIP9. John -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/Challenge ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression
On Mon, Jul 20, 2009 at 10:57 AM, John Drescher wrote: > On Mon, Jul 20, 2009 at 10:37 AM, Victor Hugo dos > Santos wrote: >> On Fri, Jul 17, 2009 at 6:53 PM, John Drescher wrote: >>> On Fri, Jul 17, 2009 at 5:59 PM, Simone S. >>> Santiago wrote: Hey co-workers, I wonder if it is possible improve the Bacula compression? I am using "compression = GZIP" but sometimes it compress only 15% of all volume. Note: My backup is FILE Type. >>> Try: >>> >>> compression=GZIP9 >>> >>> http://www.bacula.org/en/rel-manual/Configuring_Director.html#SECTION00147 >>> >>> Note that this will take much longer to compress however. >> >> >> .. GZIP9 no represent exactly best compression method.. >> >> I send one or two mail to list, about this theme and I ran various >> tests with distinct values to GZIP and basically this are the results: >> >> >> >> Reporte de hoy (con GZIP=5 y 140GB) >> >> Scheduled time: 17-Mar-2009 04:30:00 >> Start time: 17-Mar-2009 04:51:20 >> End time: 17-Mar-2009 05:44:24 >> Elapsed time: 53 mins 4 secs >> FD Bytes Written: 18,119,898,810 (18.11 GB) >> SD Bytes Written: 18,119,923,547 (18.11 GB) >> Rate: 5690.9 KB/s >> Software Compression: 87.9 % >> >> >> >> Reporte de ayer (con GZIP=6 y 140GB) >> >> Scheduled time: 16-Mar-2009 04:30:00 >> Start time: 16-Mar-2009 04:35:28 >> End time: 16-Mar-2009 05:55:55 >> Elapsed time: 1 hour 20 mins 27 secs >> FD Bytes Written: 17,570,761,480 (17.57 GB) >> SD Bytes Written: 17,570,786,217 (17.57 GB) >> Rate: 3640.1 KB/s >> Software Compression: 88.2 % >> >> >> >> Reporte del domingo pasado (con GZIP=9 y 145GB) >> >> Scheduled time: 08-Mar-2009 04:30:00 >> Start time: 08-Mar-2009 04:30:27 >> End time: 08-Mar-2009 08:34:00 >> Elapsed time: 4 hours 3 mins 33 secs >> Priority: 10 >> FD Bytes Written: 18,452,613,999 (18.45 GB) >> SD Bytes Written: 18,458,242,303 (18.45 GB) >> Rate: 1262.8 KB/s >> Software Compression: 88.1 % >> >> >> >> well.. as you can see, for me, the GZIP=6 (default) is the bester >> method...because is more faster and compress better that GZIP9. >> >> bye >> > > You are not using the same exact dataset (145GB versus 140GB source > data ) in both tests so its meaningless that GZIP6 is better than > GZIP9. > Remember every compression method is data dependent otherwise you could keep compressing data down to a few bytes... John -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/Challenge ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression
On Sun, Jul 19, 2009 at 5:46 AM, Gavin McCullagh wrote:
> Hi,
[...]
> Another useful feature might be a wildcard "excludefromcompression"
> directive so you could say:
>
> excludefromcompression {
> pattern = *.zip
> pattern = *.gz
> pattern = *.mp3
> pattern = *.mpeg
> pattern = *.flv
> }
this is a very good recommendation..
I think that this options for exclude zip, gz, mp3 and other
compressed files should be for default in bacula configuration.
or not ??
salu2
--
--
Victor Hugo dos Santos
Linux Counter #224399
--
Enter the BlackBerry Developer Challenge
This is your chance to win up to $100,000 in prizes! For a limited time,
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize
details at: http://p.sf.net/sfu/Challenge
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression
On Fri, Jul 17, 2009 at 6:53 PM, John Drescher wrote: > On Fri, Jul 17, 2009 at 5:59 PM, Simone S. > Santiago wrote: >> Hey co-workers, >> >> I wonder if it is possible improve the Bacula compression? >> I am using "compression = GZIP" but sometimes it compress only 15% of >> all volume. >> >> Note: My backup is FILE Type. >> > Try: > > compression=GZIP9 > > http://www.bacula.org/en/rel-manual/Configuring_Director.html#SECTION00147 > > Note that this will take much longer to compress however. .. GZIP9 no represent exactly best compression method.. I send one or two mail to list, about this theme and I ran various tests with distinct values to GZIP and basically this are the results: Reporte de hoy (con GZIP=5 y 140GB) Scheduled time: 17-Mar-2009 04:30:00 Start time: 17-Mar-2009 04:51:20 End time: 17-Mar-2009 05:44:24 Elapsed time: 53 mins 4 secs FD Bytes Written: 18,119,898,810 (18.11 GB) SD Bytes Written: 18,119,923,547 (18.11 GB) Rate: 5690.9 KB/s Software Compression: 87.9 % Reporte de ayer (con GZIP=6 y 140GB) Scheduled time: 16-Mar-2009 04:30:00 Start time: 16-Mar-2009 04:35:28 End time: 16-Mar-2009 05:55:55 Elapsed time: 1 hour 20 mins 27 secs FD Bytes Written: 17,570,761,480 (17.57 GB) SD Bytes Written: 17,570,786,217 (17.57 GB) Rate: 3640.1 KB/s Software Compression: 88.2 % Reporte del domingo pasado (con GZIP=9 y 145GB) Scheduled time: 08-Mar-2009 04:30:00 Start time: 08-Mar-2009 04:30:27 End time: 08-Mar-2009 08:34:00 Elapsed time: 4 hours 3 mins 33 secs Priority: 10 FD Bytes Written: 18,452,613,999 (18.45 GB) SD Bytes Written: 18,458,242,303 (18.45 GB) Rate: 1262.8 KB/s Software Compression: 88.1 % well.. as you can see, for me, the GZIP=6 (default) is the bester method...because is more faster and compress better that GZIP9. bye -- -- Victor Hugo dos Santos Linux Counter #224399 -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/Challenge ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression
Thank you, John. best rds, Simone John Drescher escreveu: On Fri, Jul 17, 2009 at 5:59 PM, Simone S. Santiago wrote: Hey co-workers, I wonder if it is possible improve the Bacula compression? I am using "compression = GZIP" but sometimes it compress only 15% of all volume. Note: My backup is FILE Type. Try: compression=GZIP9 http://www.bacula.org/en/rel-manual/Configuring_Director.html#SECTION00147 Note that this will take much longer to compress however. John -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/Challenge___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression
> On Sun, 19 Jul 2009 15:54:35 +0100, Gavin McCullagh said:
>
> Hi,
>
> On Sun, 19 Jul 2009, Bruno Friedmann wrote:
>
> > > Another useful feature might be a wildcard "excludefromcompression"
> > > directive so you could say:
>
> > > This might avoid wasting time trying to compress the unccompressible while
> > > gaining the compression ratio on those files which gain from it?
> >
> > Wouah, that would be nice, is this already working with recent bacula
> > version ?
>
> Not that I've heard of. I was just thinking out loud.
You can do it already using multiple Options clauses:
Options {
# Files that we don't want to compress.
signature = MD5
wild = *.gz
wild = *.Z
wild = *.mp3
}
Options {
# Default clause to compress all other files
signature = MD5
compression = GZIP
}
Remember to put all other options (such as signature) into both clauses.
__Martin
--
Enter the BlackBerry Developer Challenge
This is your chance to win up to $100,000 in prizes! For a limited time,
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize
details at: http://p.sf.net/sfu/Challenge
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression
Hi, On Sun, 19 Jul 2009, Bruno Friedmann wrote: > Most of the data are compressible ( exchange server storage, and Navision > Database ) at a 75% rate with gzip2 > we have 78% with gzip6 but it double easyly the time need to obtain it. So > sometime it doesn't help to try to do big compression. > > Is it more clear ? Sorry, yeah I was thinking about it after and realised that I was thinking of the wrong definition of compression ratio (ie it's how much it reduced /by/ not /to/ so 78% is better than 72%). My mistake. > > Another useful feature might be a wildcard "excludefromcompression" > > directive so you could say: > > This might avoid wasting time trying to compress the unccompressible while > > gaining the compression ratio on those files which gain from it? > > Wouah, that would be nice, is this already working with recent bacula version > ? Not that I've heard of. I was just thinking out loud. > > On the other hand, perhaps it would complicate the job or the volume layout > > too much to mix compressed with uncompressed data? > > Raah your last comment tend to me to believe, bacula should be capable of > excluding from compression. > If it's not already asked, could you put a request for enhancement be sure I > will vote to it. I can. I wonder is there someone on this list who could advise us if this is practical though? Gavin -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/Challenge ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression
Gavin McCullagh wrote:
> Hi,
>
> On Sat, 18 Jul 2009, Bruno Friedmann wrote:
>
>> Some time ago, I've made some tests on a customer site.
>> They have plenty data that could be compressed ( a 75% ratio )
>>
>> With GZIP ( which is equal to gzip default level 6 ) we loose hours of
>> compression to obtain finally only a 78% ratio
>> compared to the gzip2 I actually use.
>
> Just so I understand, are you're saying you get 78% compression with gzip6,
> but
> 75% of your data is compressible or 75% compression is possible. Where
> does the 75% come from?
Most of the data are compressible ( exchange server storage, and Navision
Database ) at a 75% rate with gzip2
we have 78% with gzip6 but it double easyly the time need to obtain it. So
sometime it doesn't help to try to do big compression.
Is it more clear ?
>
>> Waht would be cool is to have multi-threaded bacula-fd using lzma
>> compression :-))
>
> Another useful feature might be a wildcard "excludefromcompression"
> directive so you could say:
>
> excludefromcompression {
> pattern = *.zip
> pattern = *.gz
> pattern = *.mp3
> pattern = *.mpeg
> pattern = *.flv
> }
>
> This might avoid wasting time trying to compress the unccompressible while
> gaining the compression ratio on those files which gain from it?
Wouah, that would be nice, is this already working with recent bacula version ?
>
> On the other hand, perhaps it would complicate the job or the volume layout
> too much to mix compressed with uncompressed data?
>
> Gavin
>
Raah your last comment tend to me to believe, bacula should be capable of
excluding from compression.
If it's not already asked, could you put a request for enhancement be sure I
will vote to it.
>
> --
> Enter the BlackBerry Developer Challenge
> This is your chance to win up to $100,000 in prizes! For a limited time,
> vendors submitting new applications to BlackBerry App World(TM) will have
> the opportunity to enter the BlackBerry Developer Challenge. See full prize
> details at: http://p.sf.net/sfu/Challenge
> ___
> Bacula-users mailing list
> Bacula-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bacula-users
--
Bruno Friedmann
Ioda-Net Sàrl
2830 Vellerat - Switzerland
Tél : ++41 32 435 7171
Fax : ++41 32 435 7172
gsm : ++41 78 802 6760
www.ioda-net.ch
Centre de Formation et de Coaching En Ligne
www.cfcel.com
--
Enter the BlackBerry Developer Challenge
This is your chance to win up to $100,000 in prizes! For a limited time,
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize
details at: http://p.sf.net/sfu/Challenge
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression
Hi,
On Sat, 18 Jul 2009, Bruno Friedmann wrote:
> Some time ago, I've made some tests on a customer site.
> They have plenty data that could be compressed ( a 75% ratio )
>
> With GZIP ( which is equal to gzip default level 6 ) we loose hours of
> compression to obtain finally only a 78% ratio
> compared to the gzip2 I actually use.
Just so I understand, are you're saying you get 78% compression with gzip6, but
75% of your data is compressible or 75% compression is possible. Where
does the 75% come from?
> Waht would be cool is to have multi-threaded bacula-fd using lzma compression
> :-))
Another useful feature might be a wildcard "excludefromcompression"
directive so you could say:
excludefromcompression {
pattern = *.zip
pattern = *.gz
pattern = *.mp3
pattern = *.mpeg
pattern = *.flv
}
This might avoid wasting time trying to compress the unccompressible while
gaining the compression ratio on those files which gain from it?
On the other hand, perhaps it would complicate the job or the volume layout
too much to mix compressed with uncompressed data?
Gavin
--
Enter the BlackBerry Developer Challenge
This is your chance to win up to $100,000 in prizes! For a limited time,
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize
details at: http://p.sf.net/sfu/Challenge
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression
John Drescher wrote: > On Fri, Jul 17, 2009 at 5:59 PM, Simone S. > Santiago wrote: >> Hey co-workers, >> >> I wonder if it is possible improve the Bacula compression? >> I am using "compression = GZIP" but sometimes it compress only 15% of >> all volume. >> >> Note: My backup is FILE Type. >> > Try: > > compression=GZIP9 > > http://www.bacula.org/en/rel-manual/Configuring_Director.html#SECTION00147 > > > Note that this will take much longer to compress however. > > John > Some time ago, I've made some tests on a customer site. They have plenty data that could be compressed ( a 75% ratio ) With GZIP ( which is equal to gzip default level 6 ) we loose hours of compression to obtain finally only a 78% ratio compared to the gzip2 I actually use. Waht would be cool is to have multi-threaded bacula-fd using lzma compression :-)) -- Bruno Friedmann -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/Challenge ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression
On Fri, Jul 17, 2009 at 5:59 PM, Simone S. Santiago wrote: > Hey co-workers, > > I wonder if it is possible improve the Bacula compression? > I am using "compression = GZIP" but sometimes it compress only 15% of > all volume. > > Note: My backup is FILE Type. > Try: compression=GZIP9 http://www.bacula.org/en/rel-manual/Configuring_Director.html#SECTION00147 Note that this will take much longer to compress however. John -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/Challenge ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Bacula Compression
Hey co-workers, I wonder if it is possible improve the Bacula compression? I am using "compression = GZIP" but sometimes it compress only 15% of all volume. Note: My backup is FILE Type. Best Regards, Simone -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/Challenge ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression
Compression is handled by the FD, so you will find all the related compression code there. However, you will also need to patch the DIR and the SD to recognize the new compression. I was looking into this before, and I probably will again, but the compression code isn't really simple so adding a new compression algorithm would involve lots of new code. This would include a special handler to determine what kind of compression is being used (as right now it checks for whether or not GZIP is on). Another problem you will probably encounter is that the encryption and compression code are not separated in any sensible way. IIRC bacula has to have a separate chunk of code for uncompressed unencrypted, compressed unencrypted, compressed encrypted, and uncompressed encrypted backup modes. There is a lot of room for improvement here. Cheers, -Michael - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression
Oh yes, it would be nice.. Ok, so you're telling me that I need to patch the code myself in order to make other compression methods working? The problem is that this network is composed of SOME HUNDREDS of server with different distributions (redhat, fedora, debian) so I wish I could found something pre-made. Ok, I'm not a programmer, but I'll try... Now, this is a developer-question. I know, there is another list for developers, but I'm not subscribed to it. I'll do, if I'll decide to work on this "patch". For the Backup jobs, I *think* compression is done by File Daemon. Does someone know how does it works for restoring jobs? Thanks a lot for your answers Diego On Thu, 2007-12-27 at 07:04 +0100, Bruno Friedmann wrote: > Diego Roccia wrote: > > Hi all, > > a question: is there a way to implement other compression methods (like > > lzo) in bacula? I found an old patch for bacula 1.3.something are > > there any news about this? > > > > thanks in advance > > Diego > > > > > One thing that should be a big big advantage would be to have some compressor > multi-threaded ready to take advantage of Dual > Core and multiproc ... - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression
Diego Roccia wrote: > Hi all, > a question: is there a way to implement other compression methods (like > lzo) in bacula? I found an old patch for bacula 1.3.something are > there any news about this? > > thanks in advance > Diego > > One thing that should be a big big advantage would be to have some compressor multi-threaded ready to take advantage of Dual Core and multiproc ... -- Bruno Friedmann [EMAIL PROTECTED] Ioda-Net Sàrl - www.ioda-net.ch 2830 Vellerat - Switzerland Tél : ++41 32 435 7171 Fax : ++41 32 435 7172 gsm : ++41 78 802 6760 C'est Facile et Cool d'Évoluer en ligne : www.cfcel.com - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression
I think that LZMA would definitely be a nice addition to bacula (which is used by the 7z format) to compress the data. -Michael - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Compression
The compression project was abandoned, but in theory it should not be hard to create. GZIP is OK, but there are better things out there now like rk, rar and 7z Diego Roccia wrote: > Hi all, > a question: is there a way to implement other compression methods (like > lzo) in bacula? I found an old patch for bacula 1.3.something are > there any news about this? > > thanks in advance > Diego > > > - > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2005. > http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ > ___ > Bacula-users mailing list > Bacula-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bacula-users > - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Bacula Compression
Hi all, a question: is there a way to implement other compression methods (like lzo) in bacula? I found an old patch for bacula 1.3.something are there any news about this? thanks in advance Diego - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Bacula compression problem and mutli-tapes troubles
Hi everybody, I got 2 problems with my bacula configuration : I got 41,xxx,xxx kbytes of data to backup (got this number with estimate command in bacula console), but when a full backup has finished It take 2 tapes with 38,xxx,xxx kbytes on one tape and 22,xxx,xxx in the other. I precise that during the backup, when the first tape is full, i receive an error message of bacula that say « 09-Nov 06:40 zeus-sd: Job NightlySave.2005-11-08_23.00.00 waiting. Cannot find any appendable volumes. », so i make a end_of_backup.sh then I insert the second tape and make a mount in bacula console, and the backup goes on and continue at the end i got this : 08-Nov 23:00 zeus-dir: Start Backup JobId 141, Job=NightlySave.2005-11- 08_23.00.00 08-Nov 23:00 zeus-sd: Volume "mardi" previously written, moving to end of data. 08-Nov 23:01 zeus-sd: Ready to append to end of Volume "mardi" at file=35. 08-Nov 23:37 zeus-sd: End of Volume "mardi" at 37:9075 on device /dev/tape. Write of 64512 bytes got -1. 08-Nov 23:39 zeus-sd: Re-read of last block succeeded. 08-Nov 23:39 zeus-sd: End of medium on Volume "mardi" Bytes=36,463,486,387 Blocks=565,221 at 08-Nov-2005 23:39. 08-Nov 23:40 zeus-dir: Pruned 2 Jobs on Volume "mardi" from catalog. 08-Nov 23:40 zeus-sd: Job NightlySave.2005-11-08_23.00.00 waiting. Cannot find any appendable volumes. Please use the "label" command to create a new Volume for: Storage: tape Media type: tape Pool: TuesdayPool 09-Nov 00:40 zeus-sd: Job NightlySave.2005-11-08_23.00.00 waiting. Cannot find any appendable volumes. Please use the "label" command to create a new Volume for: Storage: tape Media type: tape Pool: TuesdayPool 09-Nov 02:40 zeus-sd: Job NightlySave.2005-11-08_23.00.00 waiting. Cannot find any appendable volumes. Please use the "label" command to create a new Volume for: Storage: tape Media type: tape Pool: TuesdayPool 09-Nov 06:40 zeus-sd: Job NightlySave.2005-11-08_23.00.00 waiting. Cannot find any appendable volumes. Please use the "label" command to create a new Volume for: Storage: tape Media type: tape Pool: TuesdayPool 09-Nov 11:13 zeus-sd: Someone woke me up, but I cannot find any appendable volumes for Job=NightlySave.2005-11-08_23.00.00. 09-Nov 11:13 zeus-sd: Job NightlySave.2005-11-08_23.00.00 waiting. Cannot find any appendable volumes. Please use the "label" command to create a new Volume for: Storage: tape Media type: tape Pool: TuesdayPool 09-Nov 11:18 zeus-sd: Wrote label to prelabeled Volume "mardi-2" on device "/dev/tape" 09-Nov 11:18 zeus-sd: New volume "mardi-2" mounted on device /dev/tape at 09- Nov-2005 11:18. 09-Nov 15:52 zeus-dir: Bacula 1.36.2 (28Feb05): 09-Nov-2005 15:52:31 JobId: 141 Job:NightlySave.2005-11-08_23.00.00 Backup Level: Full Client: zeus-fd FileSet:"Full Set" 2005-07-19 23:00:02 Pool: "TuesdayPool" Storage:"tape" Start time: 08-Nov-2005 23:00:02 End time: 09-Nov-2005 15:52:31 FD Files Written: 147,951 SD Files Written: 147,951 FD Bytes Written: 28,485,097,692 SD Bytes Written: 28,512,925,826 Rate: 468.9 KB/s Software Compression: 31.6 % Volume name(s): mardi-2|mardi Volume Session Id: 15 Volume Session Time:1129909905 Last Volume Bytes: 25,963,120,632 Non-fatal FD errors:0 SD Errors: 0 FD termination status: OK SD termination status: OK Termination:Backup OK 09-Nov 15:52 zeus-dir: Begin pruning Jobs. 09-Nov 15:52 zeus-dir: No Jobs found to prune. 09-Nov 15:52 zeus-dir: Begin pruning Files. 09-Nov 15:55 zeus-dir: Pruned 147,135 Files from 9 Jobs for client zeus-fd from catalog. 09-Nov 15:55 zeus-dir: End auto prune. My First question is : Why 2 tapes for 41 Go (without compression) !? My tapes are DLT IV 40/80 Gb… Sometimes, if i don’t insert the second tape of a backup at time, Bacula put the missing tapes in error, so i have to delete volume, make a mt –t /dev/tape rewind && mt –t /dev/tape weof and label agin this tape and insert in the pool… why ? why Bacula puts it in error ? Thanks for your help ! Here my list media *list media Pool: MondayPool +-++---++--+--+- +--+---+---+-+ | MediaId | VolumeName | VolStatus | VolBytes | VolFiles | VolRetention | Recycle | Slot | InChanger | MediaType | LastWritten | +-++---++--+--+- +--+---+---+-+ | 41 | lundi | Full | 36,470,203,213 |
