[Bacula-users] BigInt hit max value in mysql... what now?
Hi folks, over the weekend, our mysql bacula db seems to have hit the maximum possible auto_increment counter for an unsigned int(10) as I'm getting duplicate key errors on insert's into the File table. Is there a way to increase the maximum value or do I need to rebuild the bacula db from the last working dump?`The file table is around 15G at the moment, we have 67657255 files with a total db size around 23G. will alter table File change FileId FileId unsigned int(20) do anything useful? Thanks in advance for your comments, Uwe -- NIONEX ist ein Unternehmen der DirectGroup Germany www.directgroupgermany.de -- Sell apps to millions through the Intel(R) Atom(Tm) Developer Program Be part of this innovative community and reach millions of netbook users worldwide. Take advantage of special opportunities to increase revenue and speed time-to-market. Join now, and jumpstart your future. http://p.sf.net/sfu/intel-atom-d2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Verify differences: SHA1 sum doesn't match but it should
'Paul Mather' wrote: On Aug 28, 2010, at 7:12 AM, Steve Costaras wrote: Could be due to a transient error (transmission or wild/torn read at time of calculation). I see this a lot with integrity checking of files here (50TiB of storage). Only way to get around this now is to do a known-good sha1/md5 hash of data (2-3 reads of the file make sure that they all match and that the file is not corrupted) save that as a baseline and then when doing reads/compares if one fails do another re-read and see if the first one was in error and compare that with your baseline. This is one reason why I'm switching to the new generation of sas drives that have ioecc checks on READS not just writes to help cut down on some of this. Corruption does occur as well and is more probable with the higher the capacity of the drive. Ideally you would have a drive that would do ioecc on reads, plus using T10 PI extensions (DIX/DIF) from drive to controller up to your file system layer. It won't always prevent it by itself but would allow if you have a raid setup to do some self-healing when a drive reports a non transient (i.e. corrupted sector of data). However the T10 PI extensions are only on sas/fc drives (520/528 byte blocks) and so far as I can tell only the new LSI hba's support a small subset of this (no hardware raid controllers I can find) and have not seen any support up to the OS/filesystem level. SATA is not included at all as the T13 group opted not to include it in the spec. You could also stick with your current hardware and use a file system that emphasises end-to-end data integrity like ZFS. ZFS checksums at many levels, and has a don't trust the hardware mentality. It can detect silent data corruption and automatically self-heal where redundancy permits. ZFS also supports pool scrubbing---akin to the patrol reading of many RAID controllers---for proactive detection of silent data corruption. With drive capacities becoming very large, the probability of an unrecoverable read becomes very high. This becomes very significant even in redundant storage systems because a drive failure necessitates a lengthy rebuild period during which the storage array lacks any redundancy (in the case of RAID-5). It is for this reason that RAID-6 (ZFS raidz2) is becoming de rigeur for many-terabyte arrays using large drives, and, specifically, the reason ZFS garnered its triple-parity raidz3 pool type (in ZFS pool version 17). Have you ever tried scrubbing a 40+ TB pool ? I believe Btrfs intends to bring many ZFS features to Linux. Cheers, Paul. -- Sell apps to millions through the Intel(R) Atom(Tm) Developer Program Be part of this innovative community and reach millions of netbook users worldwide. Take advantage of special opportunities to increase revenue and speed time-to-market. Join now, and jumpstart your future. http://p.sf.net/sfu/intel-atom-d2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- Med venlig hilsen / Best Regards Henrik Johansen hen...@scannet.dk Tlf. 75 53 35 00 ScanNet Group A/S ScanNet -- Sell apps to millions through the Intel(R) Atom(Tm) Developer Program Be part of this innovative community and reach millions of netbook users worldwide. Take advantage of special opportunities to increase revenue and speed time-to-market. Join now, and jumpstart your future. http://p.sf.net/sfu/intel-atom-d2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Verify differences: SHA1 sum doesn't match but it should
A little mis-quoted there: On 2010-08-30 02:59, Henrik Johansen wrote: On Aug 28, 2010, at 7:12 AM, Steve Costaras wrote: Could be due to a transient error (transmission or wild/torn read at time of calculation). I see this a lot with integrity checking of files here (50TiB of storage). Only way to get around this now is to do a known-good sha1/md5 hash of data (2-3 reads of the file make sure that they all match and that the file is not corrupted) save that as a baseline and then when doing reads/compares if one fails do another re-read and see if the first one was in error and compare that with your baseline. This is one reason why I'm switching to the new generation of sas drives that have ioecc checks on READS not just writes to help cut down on some of this. Corruption does occur as well and is more probable with the higher the capacity of the drive. Ideally you would have a drive that would do ioecc on reads, plus using T10 PI extensions (DIX/DIF) from drive to controller up to your file system layer. It won't always prevent it by itself but would allow if you have a raid setup to do some self-healing when a drive reports a non transient (i.e. corrupted sector of data). However the T10 PI extensions are only on sas/fc drives (520/528 byte blocks) and so far as I can tell only the new LSI hba's support a small subset of this (no hardware raid controllers I can find) and have not seen any support up to the OS/filesystem level. SATA is not included at all as the T13 group opted not to include it in the spec. You could also stick with your current hardware and use a file system that emphasises end-to-end data integrity like ZFS. ZFS checksums at many levels, and has a don't trust the hardware mentality. It can detect silent data corruption and automatically self-heal where redundancy permits. 'Paul Mather' wrote: ZFS also supports pool scrubbing---akin to the patrol reading of many RAID controllers---for proactive detection of silent data corruption. With drive capacities becoming very large, the probability of an unrecoverable read becomes very high. This becomes very significant even in redundant storage systems because a drive failure necessitates a lengthy rebuild period during which the storage array lacks any redundancy (in the case of RAID-5). It is for this reason that RAID-6 (ZFS raidz2) is becoming de rigeur for many-terabyte arrays using large drives, and, specifically, the reason ZFS garnered its triple-parity raidz3 pool type (in ZFS pool version 17). On 2010-08-30 02:59, Henrik Johansen wrote: Have you ever tried scrubbing a 40+ TB pool ? If the question was to me, then yes, I have but with the comment that I am working with SANs and otherwise redundant luns/disks that I run ZFS on top of.So the availability portion of the disk subsystem is pretty stable already. I use ZFS mainly to check/verify data integrity as well as for volume management functions.For performance reasons I am mainly using mirroring. When pool sizes get large 50, 100, or more TiB the problem is the time it takes to do a scrub and the cpu i/o costs are high. For ~50TiB I would say you would want to have a subsystem that is capable of 2-3GiB/s. And then increase that in proportion with larger sets. Even then it takes a toll on a system that the primary job is NOT disk integrity but to run X application. -- Sell apps to millions through the Intel(R) Atom(Tm) Developer Program Be part of this innovative community and reach millions of netbook users worldwide. Take advantage of special opportunities to increase revenue and speed time-to-market. Join now, and jumpstart your future. http://p.sf.net/sfu/intel-atom-d2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] BigInt hit max value in mysql... what now?
On Mon, Aug 30, 2010 at 08:45:50AM +0200, Uwe Schuerkamp wrote: Hi folks, over the weekend, our mysql bacula db seems to have hit the maximum possible auto_increment counter for an unsigned int(10) as I'm getting duplicate key errors on insert's into the File table. Is there a way to increase the maximum value or do I need to rebuild the bacula db from the last working dump?`The file table is around 15G at the moment, we have 67657255 files with a total db size around 23G. will alter table File change FileId FileId unsigned int(20) Hi folks, I was able to extend the value range by issuing ALTER TABLE File CHANGE COLUMN FileId FileId bigint unsigned; I should have mentioned I'm running bacula 3.x on that server, bacula 5.x seems to use a bigint type value by default. All the best, Uwe -- NIONEX ist ein Unternehmen der DirectGroup Germany www.directgroupgermany.de -- Sell apps to millions through the Intel(R) Atom(Tm) Developer Program Be part of this innovative community and reach millions of netbook users worldwide. Take advantage of special opportunities to increase revenue and speed time-to-market. Join now, and jumpstart your future. http://p.sf.net/sfu/intel-atom-d2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] BigInt hit max value in mysql... what now?
Hi folks, looks like extending the value range did not fix the error entirely. When running an incr. backup, I receive the following error message: Fatal error: Can't fill File table Query failed: INSERT INTO File (FileIndex, JobId, PathId, FilenameId, LStat, MD5)SELECT batch.FileIndex, batch.JobId, Path.PathId, Filename.FilenameId,batch.LStat, batch.MD5 FROM batch JOIN Path ON (batch.Path = Path.Path) JOIN Filename ON (batch.Name = Filename.Name): ERR=Duplicate entry '0' for key 1 I've tried deleting those rows with 0 from File (1 Row), but apparently that did not fix the problem. Will I need to upgrade to bacula 5.0 in order to resolve the issue? Is 3.x limited to 32bit auto_inc counters or something? All the best, Uwe -- NIONEX ist ein Unternehmen der DirectGroup Germany www.directgroupgermany.de -- Sell apps to millions through the Intel(R) Atom(Tm) Developer Program Be part of this innovative community and reach millions of netbook users worldwide. Take advantage of special opportunities to increase revenue and speed time-to-market. Join now, and jumpstart your future. http://p.sf.net/sfu/intel-atom-d2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Verify differences: SHA1 sum doesn't match but it should
'Steve Costaras' wrote: A little mis-quoted there: On 2010-08-30 02:59, Henrik Johansen wrote: On Aug 28, 2010, at 7:12 AM, Steve Costaras wrote: Could be due to a transient error (transmission or wild/torn read at time of calculation). I see this a lot with integrity checking of files here (50TiB of storage). Only way to get around this now is to do a known-good sha1/md5 hash of data (2-3 reads of the file make sure that they all match and that the file is not corrupted) save that as a baseline and then when doing reads/compares if one fails do another re-read and see if the first one was in error and compare that with your baseline. This is one reason why I'm switching to the new generation of sas drives that have ioecc checks on READS not just writes to help cut down on some of this. Corruption does occur as well and is more probable with the higher the capacity of the drive. Ideally you would have a drive that would do ioecc on reads, plus using T10 PI extensions (DIX/DIF) from drive to controller up to your file system layer. It won't always prevent it by itself but would allow if you have a raid setup to do some self-healing when a drive reports a non transient (i.e. corrupted sector of data). However the T10 PI extensions are only on sas/fc drives (520/528 byte blocks) and so far as I can tell only the new LSI hba's support a small subset of this (no hardware raid controllers I can find) and have not seen any support up to the OS/filesystem level. SATA is not included at all as the T13 group opted not to include it in the spec. You could also stick with your current hardware and use a file system that emphasises end-to-end data integrity like ZFS. ZFS checksums at many levels, and has a don't trust the hardware mentality. It can detect silent data corruption and automatically self-heal where redundancy permits. 'Paul Mather' wrote: ZFS also supports pool scrubbing---akin to the patrol reading of many RAID controllers---for proactive detection of silent data corruption. With drive capacities becoming very large, the probability of an unrecoverable read becomes very high. This becomes very significant even in redundant storage systems because a drive failure necessitates a lengthy rebuild period during which the storage array lacks any redundancy (in the case of RAID-5). It is for this reason that RAID-6 (ZFS raidz2) is becoming de rigeur for many-terabyte arrays using large drives, and, specifically, the reason ZFS garnered its triple-parity raidz3 pool type (in ZFS pool version 17). On 2010-08-30 02:59, Henrik Johansen wrote: Have you ever tried scrubbing a 40+ TB pool ? If the question was to me, then yes, I have but with the comment that I am working with SANs and otherwise redundant luns/disks that I run ZFS on top of.So the availability portion of the disk subsystem is pretty stable already. I use ZFS mainly to check/verify data integrity as well as for volume management functions.For performance reasons I am mainly using mirroring. When pool sizes get large 50, 100, or more TiB the problem is the time it takes to do a scrub and the cpu i/o costs are high. For ~50TiB I would say you would want to have a subsystem that is capable of 2-3GiB/s. And then increase that in proportion with larger sets. Even then it takes a toll on a system that the primary job is NOT disk integrity but to run X application. My point exactly. Scrubbing is problematic when dealing with large datasets that have a limited lifetime and / or a high changerate (eg D2D backups). There is a hefty cost associated with scrubbing both in terms of IOPS / CPU cyles and still you may not be able to cover your entire dataset before data in it gets removed or changed. Once a fix for CR6730306 is integrated it may become feasible to schedule scrubbing operations during off-hours though. Like most ZFS related stuff it all sounds (and looks) extremely easy but in reality it is not quite so simple. -- Sell apps to millions through the Intel(R) Atom(Tm) Developer Program Be part of this innovative community and reach millions of netbook users worldwide. Take advantage of special opportunities to increase revenue and speed time-to-market. Join now, and jumpstart your future. http://p.sf.net/sfu/intel-atom-d2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- Med venlig hilsen / Best Regards Henrik Johansen hen...@scannet.dk Tlf. 75 53 35 00 ScanNet Group A/S ScanNet -- Sell apps to millions through the Intel(R) Atom(Tm) Developer Program Be part of this innovative community and reach millions of netbook users worldwide. Take advantage of special opportunities to increase revenue and
[Bacula-users] Run backup job from command line in a non-interactive mode
Hi, Let's say that you had a problem with last scheduled job, and you need to backup tonight. Idon't want to create a new job on bacula-dir.conf only for use once. I'd love to add an entry on crontab file that does it. But I don't know how to do it (is it possible). For example I can run: echo st dir | bconsole But how should I do in order to run a backup job? Thanks in advance!! -- -- Sergio Belkin http://www.sergiobelkin.com Watch More TV http://sebelk.blogspot.com Sergio Belkin - -- Sell apps to millions through the Intel(R) Atom(Tm) Developer Program Be part of this innovative community and reach millions of netbook users worldwide. Take advantage of special opportunities to increase revenue and speed time-to-market. Join now, and jumpstart your future. http://p.sf.net/sfu/intel-atom-d2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] FileSet for Windows 7
Hallo! I'd like to kindly ask if someone could post a reasonable Windows 7 FileSet! Sincerely, Ralph -- Sell apps to millions through the Intel(R) Atom(Tm) Developer Program Be part of this innovative community and reach millions of netbook users worldwide. Take advantage of special opportunities to increase revenue and speed time-to-market. Join now, and jumpstart your future. http://p.sf.net/sfu/intel-atom-d2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Run backup job from command line in a non-interactive mode
Hello, *help run Command Description === === run Run a job Arguments: job=job-name client=client-name fileset=FileSet-name level=level-keyword storage=storage-namewhere=directory-prefix when=universal-time-specification comment=text yes When at a prompt, entering a period cancels the command. and then pipe it into bconsole, like you did with the status command. hth On Mon, 30 Aug 2010 08:48:48 -0300, Sergio Belkin seb...@gmail.com wrote: Hi, Let's say that you had a problem with last scheduled job, and you need to backup tonight. Idon't want to create a new job on bacula-dir.conf only for use once. I'd love to add an entry on crontab file that does it. But I don't know how to do it (is it possible). For example I can run: echo st dir | bconsole But how should I do in order to run a backup job? Thanks in advance!! -- Sell apps to millions through the Intel(R) Atom(Tm) Developer Program Be part of this innovative community and reach millions of netbook users worldwide. Take advantage of special opportunities to increase revenue and speed time-to-market. Join now, and jumpstart your future. http://p.sf.net/sfu/intel-atom-d2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Problem with offline clients
Thank you, the runbefore script solved the problem!! Truly appreciated. +-- |This was sent by informat...@dhul.es via Backup Central. |Forward SPAM to ab...@backupcentral.com. +-- -- Sell apps to millions through the Intel(R) Atom(Tm) Developer Program Be part of this innovative community and reach millions of netbook users worldwide. Take advantage of special opportunities to increase revenue and speed time-to-market. Join now, and jumpstart your future. http://p.sf.net/sfu/intel-atom-d2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Bacula VMs
This just came up on IRC. As an aid to newcomers, a ready-made VM may be useful. A potential user could download and run the VM, without having to go through the install process. If you use VMs (eg VMWare, VirtualBox), then I think this would be a great way for you to contribute to the project. And help to lower the barrier to entry. Sometimes, the hardest step can be installation and just getting it running. :) -- Dan Langille http://langille.org/ -- Sell apps to millions through the Intel(R) Atom(Tm) Developer Program Be part of this innovative community and reach millions of netbook users worldwide. Take advantage of special opportunities to increase revenue and speed time-to-market. Join now, and jumpstart your future. http://p.sf.net/sfu/intel-atom-d2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Verify differences: SHA1 sum doesn't match but it should
On Aug 30, 2010, at 6:41 AM, Henrik Johansen wrote: Like most ZFS related stuff it all sounds (and looks) extremely easy but in reality it is not quite so simple. Yes, but does ZFS makes things easier or harder? Silent data corruption won't go away just because your pool is large. :-) (But, this is all getting a bit off-topic for Bacula-users.) Cheers, Paul. -- Sell apps to millions through the Intel(R) Atom(Tm) Developer Program Be part of this innovative community and reach millions of netbook users worldwide. Take advantage of special opportunities to increase revenue and speed time-to-market. Join now, and jumpstart your future. http://p.sf.net/sfu/intel-atom-d2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Question about jobs and pools
Hello, At the moment i have the following pools: - FileServer-Pool-full - FileServer-Pool-Incr - FileServer-Pool-Diff - MailServer-Pool-fulll - MailServer-Pool-Incr - MailServer-Pool-Diff I have 7 tapes in my library and all of them were put in the Scratch pool. I have calculated that a full backup for the FileServer will comfortable fit in 2 tapes and for the MailServer 1 tape. Now, i would like the Incremental and Differential jobs for the FileServer to write to 1 tape and incremental and differential jobs for MailServer to write to another tape. This will end up with MailServer using 2 tapes and FileServer using 3 tapes. Leaving me 2 tapes for other backups. But it is not happening like that, each pool just pulls a new tape out of scratch and writes to it, if the job has run, it will append to it, but incremental and differential seem to not be able to coexists on the same tape. My question is, what can i do to have incremental and differential jobs to be written to 1 tape? After a week cycle i ended up with: Pool: FileServPool-Full-tape +-++---+-+- +--+--+-+--+---+--- +-+ | MediaId | VolumeName | VolStatus | Enabled | VolBytes| VolFiles | VolRetention | Recycle | Slot | InChanger | MediaType | LastWritten | +-++---+-+- +--+--+-+--+---+--- +-+ | 15 | CNH895 | Full | 1 | 165,219,038,208 | 166 |2,592,000 | 1 |1 | 1 | tape | 2010-08-25 14:31:09 | | 16 | CNH896 | Used | 1 | 139,932,721,152 | 140 |2,592,000 | 1 |2 | 1 | tape | 2010-08-25 18:27:05 | +-++---+-+- +--+--+-+--+---+--- +-+ Pool: FileServPool-Diff-tape +-++---+-+-+-- +--+-+--+---+--- +-+ | MediaId | VolumeName | VolStatus | Enabled | VolBytes| VolFiles | VolRetention | Recycle | Slot | InChanger | MediaType | LastWritten | +-++---+-+-+-- +--+-+--+---+--- +-+ | 20 | CNH900 | Append| 1 | 778,982,400 |1 |2,592,000 | 1 |6 | 1 | tape | 2010-08-29 18:14:51 | +-++---+-+-+-- +--+-+--+---+--- +-+ Pool: FileServPool-Incre-tape +-++---+-+-+-- +--+-+--+---+--- +-+ | MediaId | VolumeName | VolStatus | Enabled | VolBytes| VolFiles | VolRetention | Recycle | Slot | InChanger | MediaType | LastWritten | +-++---+-+-+-- +--+-+--+---+--- +-+ | 17 | CNH897 | Append| 1 | 801,239,040 |3 | 432,000 | 1 |3 | 1 | tape | 2010-08-27 20:14:59 | +-++---+-+-+-- +--+-+--+---+--- +-+ Pool: MailPool-Full-tape +-++---+-+- +--+--+-+--+---+--- +-+ | MediaId | VolumeName | VolStatus | Enabled | VolBytes| VolFiles | VolRetention | Recycle | Slot | InChanger | MediaType | LastWritten | +-++---+-+- +--+--+-+--+---+--- +-+ | 18 | CNH898 | Used | 1 | 113,341,132,800 | 114 |2,592,000 | 1 |4 | 1 | tape | 2010-08-25 23:50:11 | +-++---+-+- +--+--+-+--+---+--- +-+ Pool: MailPool-Incre-tape +-++---+-+-+-- +--+-+--+---+--- +-+ | MediaId | VolumeName | VolStatus | Enabled | VolBytes| VolFiles | VolRetention | Recycle | Slot | InChanger | MediaType | LastWritten | +-++---+-+-+-- +--+-+--+---+--- +-+ | 19 | CNH899 | Append| 1 |
Re: [Bacula-users] Verify differences: SHA1 sum doesn't match but it should
Steve Costaras stev...@chaven.com writes: Could be due to a transient error (transmission or wild/torn read at time of calculation). I see this a lot with integrity checking of files here (50TiB of storage). Only way to get around this now is to do a known-good sha1/md5 hash of data (2-3 reads of the file make sure that they all match and that the file is not corrupted) save that as a baseline and then when doing reads/compares if one fails do another re-read and see if the first one was in error and compare that with your baseline. This is one reason why I'm switching to the new generation of sas drives that have ioecc checks on READS not just writes to help cut down on some of this. Corruption does occur as well and is more probable with the higher the capacity of the drive. Ideally you would have a drive that would do ioecc on reads, plus using T10 PI extensions (DIX/DIF) from drive to controller up to your file system layer.It won't always prevent it by itself but would allow if you have a raid setup to do some self-healing when a drive reports a non transient (i.e. corrupted sector of data). First off, thanks for the answers. The thing is that I am well aware of the reliability problems of hard drives and I would love to use some advanced file system like ZFS or btrfs, but I am on Debian and I will stay on Debian. And btrfs is not mature enough to be used in production at the moment. The other thing is that I do not think that this is an issue of corruption of the data itself! As I said I checked the files against backups and MD5 sums supplied by Debian (several times and from cold cache) and the data seems to be OK. The executables that are reported by Bacula to have changed continue to work well and bug-free just as before. So I think this is a problem/bug with either the Postgresql database or Bacula, not with my hard drives. I just wonder how something like this could happen and how I could avoid this. I'm also not willing to do additional checksums with other programs (AIDE or similar) because they take _lots_ of time to run. With Bacula I get the checksums for free. I just want to use them to detect corruption on disk from time to time and because I use VirtualFull and want to know if my differential backups have missed something. So I still don't know how to proceed. Apart from that I will try to upgrade my director and sd to 5.0.2 as soon as Debian backports are available and see if the problem goes away. I will also re-run the DiskToCatalog after my next differential backup and see if something is different. Thanks, Tobias -- This SF.net Dev2Dev email is sponsored by: Show off your parallel programming skills. Enter the Intel(R) Threading Challenge 2010. http://p.sf.net/sfu/intel-thread-sfd ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Verify differences: SHA1 sum doesn't match but it should
'Paul Mather' wrote: On Aug 30, 2010, at 6:41 AM, Henrik Johansen wrote: Like most ZFS related stuff it all sounds (and looks) extremely easy but in reality it is not quite so simple. Yes, but does ZFS makes things easier or harder? It hides a lot of the complexity involved. In ZFS it is either super-easy or super-hard (once super-easy fails to apply). Silent data corruption won't go away just because your pool is large. :-) No, but a large pool combined with a high changerate renders scrubbing somewhat useless. (But, this is all getting a bit off-topic for Bacula-users.) Agreed. Cheers, Paul. -- Med venlig hilsen / Best Regards Henrik Johansen hen...@scannet.dk Tlf. 75 53 35 00 ScanNet Group A/S ScanNet -- This SF.net Dev2Dev email is sponsored by: Show off your parallel programming skills. Enter the Intel(R) Threading Challenge 2010. http://p.sf.net/sfu/intel-thread-sfd ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Verify differences: SHA1 sum doesn't match but it should
On 08/30/2010 06:35 PM, Tobias Brink wrote:
Steve Costaras stev...@chaven.com writes:
Could be due to a transient error (transmission or wild/torn read at
time of calculation). I see this a lot with integrity checking of
files here (50TiB of storage).
Only way to get around this now is to do a known-good sha1/md5 hash of
data (2-3 reads of the file make sure that they all match and that the
file is not corrupted) save that as a baseline and then when doing
reads/compares if one fails do another re-read and see if the first
one was in error and compare that with your baseline. This is one
reason why I'm switching to the new generation of sas drives that have
ioecc checks on READS not just writes to help cut down on some of
this.
Corruption does occur as well and is more probable with the higher the
capacity of the drive. Ideally you would have a drive that would
do ioecc on reads, plus using T10 PI extensions (DIX/DIF) from drive
to controller up to your file system layer.It won't always prevent
it by itself but would allow if you have a raid setup to do some
self-healing when a drive reports a non transient (i.e. corrupted
sector of data).
First off, thanks for the answers. The thing is that I am well aware of
the reliability problems of hard drives and I would love to use some
advanced file system like ZFS or btrfs, but I am on Debian and I will
stay on Debian. And btrfs is not mature enough to be used in production
at the moment. The other thing is that I do not think that this is an
issue of corruption of the data itself! As I said I checked the files
against backups and MD5 sums supplied by Debian (several times and from
cold cache) and the data seems to be OK. The executables that are
reported by Bacula to have changed continue to work well and bug-free
just as before.
So I think this is a problem/bug with either the Postgresql database or
Bacula, not with my hard drives. I just wonder how something like this
could happen and how I could avoid this. I'm also not willing to do
additional checksums with other programs (AIDE or similar) because they
take _lots_ of time to run. With Bacula I get the checksums for free.
I just want to use them to detect corruption on disk from time to time
and because I use VirtualFull and want to know if my differential
backups have missed something.
So I still don't know how to proceed. Apart from that I will try to
upgrade my director and sd to 5.0.2 as soon as Debian backports are
available and see if the problem goes away. I will also re-run the
DiskToCatalog after my next differential backup and see if something
is different.
Thanks,
Tobias
Tobias, I use this little python script to extract information which I used to
track duplicates files
(users are users :-)
Hope this could help you a bit to have inspiration and decode the lstat column.
(If I remember, someone has also do the same in pl/pgsql: check the archives
list)
#!/usr/bin/python
# -*- coding: utf-8 -*-
#
# call it with a jobid and pipe it to csv file
#
import sys
import time
import MySQLdb
jobid = sys.argv[1]
def base64_decode_lstat(record, position):
b64 = ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/
val = 0
size = record.split(' ')[position]
for i in range(len(size)):
val += (b64.find(size[i])) * (pow(64,(len(size)-i)-1))
return val
# Adjust localhost,username,passwd,dbname
db = MySQLdb.connect(host=localhost, user=bacula, passwd=bacula,
db=bacula)
db.set_character_set('utf8')
cursor = db.cursor()
cursor.execute('SET NAMES utf8;')
cursor.execute('SET CHARACTER SET utf8;')
cursor.execute('SET character_set_connection=utf8;')
cursor.execute(SELECT File.MD5 as cheksum, convert(Filename.Name using utf8)
as filename, convert(Path.Path using utf8) as
path, File.LStat as lstat\
FROM File, Filename, Path \
WHERE File.JobId = '%s' \
AND Filename.FilenameId = File.FilenameId \
AND Path.PathId = File.PathId\
ORDER BY
File.MD5,Filename.Name,Path.Path % jobid)
result = cursor.fetchall()
# no headerprint
'checksum;filename;path;lstat;gid;uid;bytes;blocksize;blocks_allocated;atime;mtime;ctime'
for record in result:
print '%s;%s;%s;%s;%s;%s;%s;%s;%s;%s;%s;%s' % (
record[0] , record[1] , record[2] , record[3] , \
base64_decode_lstat(record[3],5) , \
base64_decode_lstat(record[3],6) , \
base64_decode_lstat(record[3],7) , \
base64_decode_lstat(record[3],8) , \
base64_decode_lstat(record[3],9) , \
base64_decode_lstat(record[3],10) , \
base64_decode_lstat(record[3],11) , \
base64_decode_lstat(record[3],12) \
)
# no empty line at end print
--
Bruno Friedmann br...@ioda-net.ch
Ioda-Net Sàrl www.ioda-net.ch
openSUSE Member
User www.ioda.net/r/osu
Blog
Re: [Bacula-users] Bacula VMs
I assume you mean SD and Dir in a VM. I have been using Bacula in a qemu-kvm guest for some time now. Dir and database run beautifully in a VM. SD is a bit trickier. SD works well with disk storage. All works as expected even with USB disk, but only at USB 1 speeds. That is only due to qemu-kvm not having EHCI (USB 2.0) device emulation, however I hear that is forthcoming. The problem lies in using SCSI or SAS tape drives. Passing particular SCSI or SATA/SAS devices through to the VM is accomplished by passing through the PCI / PCIe controller hardware, rather than the drive itself. Setting up qemu-kvm to attach anything other than disk storage devices to the VM is more complex, and I'm not sure it would make it any easier. I've never tried with VMWare or VirtualBox, though. On 8/30/2010 10:50 AM, Dan Langille wrote: This just came up on IRC. As an aid to newcomers, a ready-made VM may be useful. A potential user could download and run the VM, without having to go through the install process. If you use VMs (eg VMWare, VirtualBox), then I think this would be a great way for you to contribute to the project. And help to lower the barrier to entry. Sometimes, the hardest step can be installation and just getting it running. :) -- This SF.net Dev2Dev email is sponsored by: Show off your parallel programming skills. Enter the Intel(R) Threading Challenge 2010. http://p.sf.net/sfu/intel-thread-sfd ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Duplicate job not allowed during copy job.
Bacula 5.0.3 Ubuntu 10.04 64bit MySQL
Since upgrading to 5.0.3 from 3.0.x I get the following duplicate job
error when running from a schedule. If I run the copy job manually, it
runs without error. The following was typed out by hand, there may be
some typos.
Brian-
25-Aug 09:02 marlin2-dir JobId 10782: The following six JobIds were
chosen to be copied: 10775,10776,10777,10778,10779,10781
25-Aug 09:02 marlin2-dir JobId 10782: Job queued. JobId=10783
25-Aug 09:02 marlin2-dir JobId 10782: Copying JobId 10783 started.
25-Aug 09:02 marlin2-dir JobId 10782: Job queued. JobId=10785
25-Aug 09:02 marlin2-dir JobId 10782: Copying JobId 10785 started.
25-Aug 09:02 marlin2-dir JobId 10782: Job queued. JobId=10785
25-Aug 09:02 marlin2-dir JobId 10782: Copying JobId 10785 started.
25-Aug 09:02 marlin2-dir JobId 10782: Job queued. JobId=10787
25-Aug 09:02 marlin2-dir JobId 10782: Copying JobId 10787 started.
25-Aug 09:02 marlin2-dir JobId 10782: Job queued. JobId=10789
25-Aug 09:02 marlin2-dir JobId 10782: Copying JobId 10789 started.
25-Aug 09:02 marlin2-dir JobId 10782: Job queued. JobId=10791
25-Aug 09:02 marlin2-dir JobId 10782: Copying JobId 10791 started.
25-Aug 09:02 marlin2-dir JobId 10783: Fatal Error: JobId 10782 already
running. Duplicate job not allowed.
25-Aug 09:02 marlin2-dir JobId 10783: Fatal Error: JobId 10785 already
running. Duplicate job not allowed.
25-Aug 09:02 marlin2-dir JobId 10783: Fatal Error: JobId 10791 already
running. Duplicate job not allowed.
25-Aug 09:02 marlin2-dir JobId 10783: Fatal Error: JobId 10787 already
running. Duplicate job not allowed.
25-Aug 09:02 marlin2-dir JobId 10783: Fatal Error: JobId 10789 already
running. Duplicate job not allowed.
## Disk Full Pool
Pool {
Name = Disk-Full
Pool Type= Backup
Storage = Disk
Volume Retention = 4 Weeks
Auto Prune = yes
Recycle = yes
Label Format = vol-
Maximum Jobs = 1
Maximum Volume Bytes = 4G
Scratch Pool = Disk-Scratch
Recycle Pool = Disk-Scratch
Volume Use Duration = 1 days
Action On Purge = Truncate
Next Pool= LTO3-Copy_Full
}
## LTO-3 Pool
Pool {
Name = LTO3-Copy_Full
Pool Type = Backup
Storage= LTO-3
Volume Retention = 16 Weeks
Auto Prune = yes
Recycle= yes
Volume Use Duration = 8 days
}
## Copy Job to LTO
Job {
Name = LTO3-Copy_Full
Type = Copy
Schedule = LTO3-Copy_Full
Client= None
File Set = None
Pool = Disk-Full
Messages = Standard
Maximum Concurrent Jobs = 1
Priority = 98
Write Bootstrap = /opt/bacula/bsr/%1-%j.bsr
Allow Duplicate Jobs = yes
Allow Higher Duplicates = yes
Cancel Queued Duplicate = no
Cancel Running Duplicates = no
Spool Attributes = yes
Selection Type= SQLQuery
Selection Pattern = SELECT DISTINCT Job.Job.Id FROM Job,Pool WHERE
Pool.Name = 'Disk-Full' AND Pool.PoolId = Job.PoolId AND Job.Type = 'B'
AND Job.JobStatus IN ('T', 'W') AND Job.jobBytes 0 AND Job.JobId NOT
IN (SELECT PriorJobId FROM Job WHERE Type IN ('B', 'C') AND
Job.JobStatus IN ('T', 'W') AND PriorJobId !=0) ORDER by Job.StartTime;
}
--
This SF.net Dev2Dev email is sponsored by:
Show off your parallel programming skills.
Enter the Intel(R) Threading Challenge 2010.
http://p.sf.net/sfu/intel-thread-sfd
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Bacula Developer's Conference
Hello, As you can see from the email below, we have been planning to hold a Bacula Developer's conference the 27th and 28th of September. Despite some early enthusiasm, there are currently not enough people signed up to hold the conference. However disappointing that is, I am hoping that there will be enough interest manifested by the end of this week. Real interest can be shown by sending a registration email to the address noted below. Best regards, Kern === Email sent 9 August == Hello, We will be holding the first Bacula Developer's conference during the last part of September and would enjoy meeting you and having you participate in the conference. Please see below for more information. For those of you who are planning to give a presentation, please contact me with the title and a brief abstract of your talk so that I can add it to the bacula.org web site page concerning the conference. Please see the details below. Best regards, Kern Bacula Developer's Conference 27-28 September 2010 Yverdon-les-Bains Switzerland Are you interested in: - Meeting the Bacula developers in person. - Learning how we maintain the source code. - Hearing presentations about Bacula from the developers. - Learning about and helping define the Bacula roadmap. - Giving feedback or ideas directly to the developers. - Learning more about Bacula Systems. - Hearing from other Bacula developers and users. If so, you should know that there will be a Bacula Developer's conference in Yverdon, Switzerland (about 1.5 hour by train from Geneva Airport) on the 27th and 28th of September. In addition, Bacula Systems will hold an Open House on Sunday the 26th beginning at noon where you can meet most of the Bacula Systems founders and key Bacula developers person. This conference will be most appropriate for developers and advanced Bacula users, but it is open and free for everyone interested in Bacula. If you are or have been a contributor to the Bacula project, the project may be able to sponsor part of your trip -- please ask. Sunday 26 Sept. Bacula Systems Open House. Visit the Bacula Systems offices and meet the Bacula developers and the Bacula Systems founders. Totally informal and optional. Day 1: Monday 27 September 2010 Presentations by Bacula developers and anyone else who would like to give a formal presentation (30 to 45 minutes) Tentative program: - Swisscom sharing our experiences with Bacula (SAP backup, ...) - DassIT new Bacula conf file GUI editor - Bacula in Brazil - Linux Bare Metal Recovery - Bacula development process - Bacula Roadmap - Bweb - How Bacula Systems supports the project -- Rob Morrison Day 2: Tuesday 28 September 2010 Birds of a feather meetings: Informal direct conversations with the developers, planning, organizing, ... - Using git - How patches are integrated - Regression testing - CDash regression dashboard - Release cycle - Roadmap discussion and your input Brain storming new backup strategies such as deduplication - How Bacula plugins work - Rpms and how to improve them ... Naturally, we will furnish plenty of beer and pizza and other goodies. Your input needed: We are still in the process of organizing the presentations and the time with the developers, so if you are planning to attend and you would like to make a formal presentation, or if you have some particular subjects that you would like us to address, please let us know as soon as possible. Registration: To register for this event, please send an email to: isabe...@baculasystems.com and include your: Name, Company, Telephone number, Mobile Telephone number, Date of arrival, Date of departure, and any special interests you have for the conference. In return, Isabelle will send you an email that gives detailed information on available hotels, our location, how to get here, and other useful information for the conference. -- This SF.net Dev2Dev email is sponsored by: Show off your parallel programming skills. Enter the Intel(R) Threading Challenge 2010. http://p.sf.net/sfu/intel-thread-sfd ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Run backup job from command line in a non-interactive mode
2010/8/30 Sebastian Bachmann m...@free-minds.net: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 mh you should add a yes to the end, not comment=yes hth On 30.08.10 21:32, Sergio Belkin wrote: 2010/8/30 m...@free-minds.net: when=universal-time-specification Thanks, I did the following: [r...@pepino ~]# echo run job=Backup_centeno_Martes-LTO3 client=centeno-fd fileset=dumpcenteno storage=LTO3 when=2010-08-30 21:00:00 comment=yes | bconsole Connecting to Director pepino:9101 1000 OK: pepino-dir Version: 2.4.3 (10 October 2008) Enter a period to cancel a command. run job=Backup_centeno_Martes-LTO3 client=centeno-fd fileset=dumpcenteno storage=LTO3 when=2010-08-30 21:00:00 comment=yes Using Catalog MyCatalog Invalid keyword: 21:00:00 I've tried quoting date but the error is the same, what am I doing wrong? Thanks in advance!! The error is still there :( -- -- Sergio Belkin http://www.sergiobelkin.com Watch More TV http://sebelk.blogspot.com Sergio Belkin - -- This SF.net Dev2Dev email is sponsored by: Show off your parallel programming skills. Enter the Intel(R) Threading Challenge 2010. http://p.sf.net/sfu/intel-thread-sfd ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Run backup job from command line in a non-interactive mode
2010/8/30 Sergio Belkin seb...@gmail.com: 2010/8/30 Sebastian Bachmann m...@free-minds.net: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 mh you should add a yes to the end, not comment=yes hth It's getting better echo run job=Backup_centeno_Martes-LTO3 client=spectrum-fd fileset=dumpcenteno storage=LTO3 when=2010-08-30_21:00:00 yes | bconsole It runs the job but tight now, I can set the hourly right. TIA -- -- Sergio Belkin http://www.sergiobelkin.com Watch More TV http://sebelk.blogspot.com Sergio Belkin - -- This SF.net Dev2Dev email is sponsored by: Show off your parallel programming skills. Enter the Intel(R) Threading Challenge 2010. http://p.sf.net/sfu/intel-thread-sfd ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Job waiting on mount request blocking other jobs
Hello, Recently we went through a following scenario: An external USB drive to which JobA was supposed to write became unusable. JobA requested a mount. Jobs B through Z that were scheduled to run after JobA did not run. As a result we lost two days of backups. How can we make sure that jobs whose destination drives are accessible do not get blocked by jobs whose destination drives are inaccessible? Thanks, Greg Golin smime.p7s Description: S/MIME cryptographic signature -- This SF.net Dev2Dev email is sponsored by: Show off your parallel programming skills. Enter the Intel(R) Threading Challenge 2010. http://p.sf.net/sfu/intel-thread-sfd___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula VMs
On 8/30/2010 1:00 PM, Josh Fisher wrote: On 8/30/2010 10:50 AM, Dan Langille wrote: This just came up on IRC. As an aid to newcomers, a ready-made VM may be useful. A potential user could download and run the VM, without having to go through the install process. If you use VMs (eg VMWare, VirtualBox), then I think this would be a great way for you to contribute to the project. And help to lower the barrier to entry. Sometimes, the hardest step can be installation and just getting it running. :) I assume you mean SD and Dir in a VM. And an FD. Everything a novice would need in order to play with Bacula. I have been using Bacula in a qemu-kvm guest for some time now. Dir and database run beautifully in a VM. SD is a bit trickier. SD works well with disk storage. All works as expected even with USB disk, but only at USB 1 speeds. That is only due to qemu-kvm not having EHCI (USB 2.0) device emulation, however I hear that is forthcoming. The problem lies in using SCSI or SAS tape drives. Passing particular SCSI or SATA/SAS devices through to the VM is accomplished by passing through the PCI / PCIe controller hardware, rather than the drive itself. Setting up qemu-kvm to attach anything other than disk storage devices to the VM is more complex, and I'm not sure it would make it any easier. I've never tried with VMWare or VirtualBox, though. I would expect any VM example to be doing disk-based backups. To be clear, I think we need snapshots of working functional VMs which a novice can download, install, run, and then play with Bacula. -- Dan Langille - http://langille.org/ -- This SF.net Dev2Dev email is sponsored by: Show off your parallel programming skills. Enter the Intel(R) Threading Challenge 2010. http://p.sf.net/sfu/intel-thread-sfd ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
