Re: [Bacula-users] What causes orphaned Path/Files in database?
Hi, 16.10.2008 04:14, Hemant Shah wrote: --- On Wed, 10/15/08, Arno Lehmann [EMAIL PROTECTED] wrote: ... Thanks for the explanation, I will setup a cron job to run dbcheck. I would be careful with a cron job... basically, as the dbcheck process can take a long time and locks the database, it can result in backups not running as scheduled. It's better, in my opinion, to just add the dbcheck to your list of things to do during maintenance periods. You usually don't need to run it more often than a few times a year, too, so if you choose to do that via cron, you should schedule it to a time where no backups run, and only run it monthly or even less often. Arno Hemant Shah E-mail: [EMAIL PROTECTED] - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- Arno Lehmann IT-Service Lehmann Sandstr. 6, 49080 Osnabrück www.its-lehmann.de - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] What causes orphaned Path/Files in database?
In the message dated: Thu, 16 Oct 2008 10:08:07 +0200, The pithy ruminations from Arno Lehmann on Re: [Bacula-users] What causes orphaned Path/Files in database? were: = Hi, = = 16.10.2008 04:14, Hemant Shah wrote: = = = --- On Wed, 10/15/08, Arno Lehmann [EMAIL PROTECTED] wrote: = ... = Thanks for the explanation, I will setup a cron job to run dbcheck. = = I would be careful with a cron job... basically, as the dbcheck = = process can take a long time and locks the database, it can result in = = backups not running as scheduled. A very good point. In our environment, dbcheck takes about 3 days to run. The catalog is about 21GB, and the server is a dual-proc 3.2GHz, 32-bit machine with 12GB of RAM, running bacula 1.38.11 and MySQL 5.?. = = It's better, in my opinion, to just add the dbcheck to your list of = = things to do during maintenance periods. You usually don't need to run = = it more often than a few times a year, too, so if you choose to do = = that via cron, you should schedule it to a time where no backups run, = = and only run it monthly or even less often. Hmmm... Do you know if the time to run dbcheck is proportional to the number of catalog entries (the database size), or the number of orphaned entries? If it's the former, than I'd agree that running dbcheck infrequently (during a maintenance interval) is a good idea. However, if the time to run dbcheck is proportional to the number of orphaned entries, then perhaps it should be run much more often. As far as running dbcheck when no backups are active, I'd suggest running it via the RunAfterJob directive in the BackupCatalog job. You could write a wrapper to dbcheck that looks for the presence of a file, such as: if [ -f /etc/run_dbcheck ] ; then dbcheck . rm -f /etc/run_dbcheck fi so that the RunAfterJob can execute after every backup, but the dbcheck is only executed if the /etc/run_dbcheck file was manually created in advance. Mark = = Arno = = Hemant Shah = E-mail: [EMAIL PROTECTED] = = [SNIP!] = = Arno Lehmann = IT-Service Lehmann = Sandstr. 6, 49080 Osnabr=FCck = www.its-lehmann.de [SNIP!] Mark Bergman voice: 215-662-7310 [EMAIL PROTECTED] fax: 215-614-0266 System Administrator Section of Biomedical Image Analysis Department of RadiologyUniversity of Pennsylvania PGP Key: https://www.rad.upenn.edu/sbia/bergman The information contained in this e-mail message is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail, and delete the original message. - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] What causes orphaned Path/Files in database?
On Thu, 16 Oct 2008, [EMAIL PROTECTED] wrote: In our environment, dbcheck takes about 3 days to run. The catalog is about 21GB, and the server is a dual-proc 3.2GHz, 32-bit machine with 12GB of RAM, running bacula 1.38.11 and MySQL 5.?. That sounds like rather a long time. One of my bacula environments has a current catalog size of about 12 GB, and on a 3.0 GHz Core2 box (two cores total), CentOS 5.2, dbcheck takes about an hour or so. Obviously this is a much faster machine than you are using, but it shouldb't make _that_ much difference. I'm doing a: CREATE INDEX file_tmp_pathid_idx ON File (PathId); CREATE INDEX file_tmp_filenameid_idx ON File (FilenameId); before the dbcheck, which produces an enormous speedup. Steve Steve Thompson E-mail: smt AT vgersoft DOT com Voyager Software LLC Web: http://www DOT vgersoft DOT com 39 Smugglers Path VSW Support: support AT vgersoft DOT com Ithaca, NY 14850 186,300 miles per second: it's not just a good idea, it's the law - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] What causes orphaned Path/Files in database?
--- On Thu, 10/16/08, Arno Lehmann [EMAIL PROTECTED] wrote: From: Arno Lehmann [EMAIL PROTECTED] Subject: Re: [Bacula-users] What causes orphaned Path/Files in database? To: baculausers bacula-users@lists.sourceforge.net Date: Thursday, October 16, 2008, 3:08 AM Hi, 16.10.2008 04:14, Hemant Shah wrote: --- On Wed, 10/15/08, Arno Lehmann [EMAIL PROTECTED] wrote: ... Thanks for the explanation, I will setup a cron job to run dbcheck. I would be careful with a cron job... basically, as the dbcheck process can take a long time and locks the database, it can result in backups not running as scheduled. It's better, in my opinion, to just add the dbcheck to your list of things to do during maintenance periods. You usually don't need to run it more often than a few times a year, too, so if you choose to do that via cron, you should schedule it to a time where no backups run, and only run it monthly or even less often. Arno Hemant Shah E-mail: [EMAIL PROTECTED] - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- Arno Lehmann IT-Service Lehmann Sandstr. 6, 49080 Osnabrück www.its-lehmann.de - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users I am going to run dbcheck once a month after my full backups. This is when I delete all the disk volumes more than month old. Hemant Shah E-mail: [EMAIL PROTECTED] - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] What causes orphaned Path/Files in database?
--- On Thu, 10/16/08, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: From: [EMAIL PROTECTED] [EMAIL PROTECTED] Subject: Re: [Bacula-users] What causes orphaned Path/Files in database? To: baculausers bacula-users@lists.sourceforge.net Date: Thursday, October 16, 2008, 11:53 AM In the message dated: Thu, 16 Oct 2008 10:08:07 +0200, The pithy ruminations from Arno Lehmann on Re: [Bacula-users] What causes orphaned Path/Files in database? were: = Hi, = = 16.10.2008 04:14, Hemant Shah wrote: = = = --- On Wed, 10/15/08, Arno Lehmann [EMAIL PROTECTED] wrote: = ... = Thanks for the explanation, I will setup a cron job to run dbcheck. = = I would be careful with a cron job... basically, as the dbcheck = = process can take a long time and locks the database, it can result in = = backups not running as scheduled. A very good point. In our environment, dbcheck takes about 3 days to run. The catalog is about 21GB, and the server is a dual-proc 3.2GHz, 32-bit machine with 12GB of RAM, running bacula 1.38.11 and MySQL 5.?. I am using PostgreSQL and database is 11GB. It only took 10-15 minutes to run dbcheck. The server has 2.0 GHz dual core Xeon and 4GB RAM. It runs 64-bit Fedora 9 and PostgreSQL is also 64-bit. = = It's better, in my opinion, to just add the dbcheck to your list of = = things to do during maintenance periods. You usually don't need to run = = it more often than a few times a year, too, so if you choose to do = = that via cron, you should schedule it to a time where no backups run, = = and only run it monthly or even less often. Hmmm... Do you know if the time to run dbcheck is proportional to the number of catalog entries (the database size), or the number of orphaned entries? If it's the former, than I'd agree that running dbcheck infrequently (during a maintenance interval) is a good idea. However, if the time to run dbcheck is proportional to the number of orphaned entries, then perhaps it should be run much more often. As far as running dbcheck when no backups are active, I'd suggest running it via the RunAfterJob directive in the BackupCatalog job. You could write a wrapper to dbcheck that looks for the presence of a file, such as: if [ -f /etc/run_dbcheck ] ; then dbcheck . rm -f /etc/run_dbcheck fi so that the RunAfterJob can execute after every backup, but the dbcheck is only executed if the /etc/run_dbcheck file was manually created in advance. Mark = = Arno = = Hemant Shah = E-mail: [EMAIL PROTECTED] = = [SNIP!] = = Arno Lehmann = IT-Service Lehmann = Sandstr. 6, 49080 Osnabr=FCck = www.its-lehmann.de [SNIP!] Mark Bergman voice: 215-662-7310 [EMAIL PROTECTED] fax: 215-614-0266 System Administrator Section of Biomedical Image Analysis Department of RadiologyUniversity of Pennsylvania PGP Key: https://www.rad.upenn.edu/sbia/bergman The information contained in this e-mail message is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail, and delete the original message. - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users Hemant Shah E-mail: [EMAIL PROTECTED] - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] What causes orphaned Path/Files in database?
A very good point. In our environment, dbcheck takes about 3 days to run. The catalog is about 21GB, and the server is a dual-proc 3.2GHz, 32-bit machine with 12GB of RAM, running bacula 1.38.11 and MySQL 5.?. How many rows are in your 'file' and 'path' tables? Would there be any sense in doing the orphan check for only a subset of entries at any given time? Eg every time db_check runs check only for certain prefixes, eg first backup check for files starting with 'a', then 'b' on the next backup and so on. Assuming the file is indexed there shouldn't be much overhead. I think there are around 64 valid filename characters, so 1/64th of 3 days would be an hour or two (depending on the distribution of filenames over the set of valid characters, which won't be uniform), which is probably more acceptable. If the dataset was much larger though you might look at checking for prefixes of (eg) 'aa - ae', then 'af - aj' etc, which would reduce the time required but would mean a longer cycle time, during which time orphans would increase in number. James - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] What causes orphaned Path/Files in database?
Hi, 15.10.2008 19:01, Hemant Shah wrote: Folks, I ran dbcheck command on bacula catalog and I it found following: Checking for Filenames with a trailing slash Found 0 bad Filename records. Checking for Paths without a trailing slash Found 0 bad Path records. Checking for duplicate Filename entries. Found 0 duplicate Filename records. Checking for duplicate Path entries. Found 0 duplicate Path records. Checking for orphaned JobMedia entries. Checking for orphaned File entries. This may take some time! Checking for orphaned Path entries. This may take some time! Found 5231 orphaned Path records. Checking for orphaned Filename entries. This may take some time! Found 35007 orphaned Filename records. Checking for orphaned FileSet entries. This takes some time! Found 0 orphaned FileSet records. Checking for orphaned Client entries. Found 0 orphaned Client records. Checking for orphaned Job entries. Found 0 orphaned Job records. Checking for Admin Job entries. Found 19 Admin Job records. Checking for Restore Job entries. Found 0 Restore Job records. What causes the orphaned paths and files? That happens because, when Bacula prunes jobs, it does not verify that a file name and path name record is still in use. For example, you have backed up /somedir/somefile. The file is pruned, but the path and the file name remain in the catalog. This is done because - as you probably noticed - finding these entries can take a very long time which we don't want to spend during regular, in-job, pruning. Also, the entries will be reused if, later, you back up /somedir/anotherfile and /home/user/somefile. Or, in short - you're not seeing errors, you just found out a reason why a scheduled dbcheck can help limiting the catalog size. Arno Hemant Shah E-mail: [EMAIL PROTECTED] - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- Arno Lehmann IT-Service Lehmann Sandstr. 6, 49080 Osnabrück www.its-lehmann.de - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] What causes orphaned Path/Files in database?
--- On Wed, 10/15/08, Arno Lehmann [EMAIL PROTECTED] wrote: From: Arno Lehmann [EMAIL PROTECTED] Subject: Re: [Bacula-users] What causes orphaned Path/Files in database? To: baculausers bacula-users@lists.sourceforge.net Date: Wednesday, October 15, 2008, 5:46 PM Hi, 15.10.2008 19:01, Hemant Shah wrote: Folks, I ran dbcheck command on bacula catalog and I it found following: Checking for Filenames with a trailing slash Found 0 bad Filename records. Checking for Paths without a trailing slash Found 0 bad Path records. Checking for duplicate Filename entries. Found 0 duplicate Filename records. Checking for duplicate Path entries. Found 0 duplicate Path records. Checking for orphaned JobMedia entries. Checking for orphaned File entries. This may take some time! Checking for orphaned Path entries. This may take some time! Found 5231 orphaned Path records. Checking for orphaned Filename entries. This may take some time! Found 35007 orphaned Filename records. Checking for orphaned FileSet entries. This takes some time! Found 0 orphaned FileSet records. Checking for orphaned Client entries. Found 0 orphaned Client records. Checking for orphaned Job entries. Found 0 orphaned Job records. Checking for Admin Job entries. Found 19 Admin Job records. Checking for Restore Job entries. Found 0 Restore Job records. What causes the orphaned paths and files? That happens because, when Bacula prunes jobs, it does not verify that a file name and path name record is still in use. For example, you have backed up /somedir/somefile. The file is pruned, but the path and the file name remain in the catalog. This is done because - as you probably noticed - finding these entries can take a very long time which we don't want to spend during regular, in-job, pruning. Also, the entries will be reused if, later, you back up /somedir/anotherfile and /home/user/somefile. Or, in short - you're not seeing errors, you just found out a reason why a scheduled dbcheck can help limiting the catalog size. Arno Hemant Shah E-mail: [EMAIL PROTECTED] - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- Arno Lehmann IT-Service Lehmann Sandstr. 6, 49080 Osnabrück www.its-lehmann.de - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users Thanks for the explanation, I will setup a cron job to run dbcheck. Hemant Shah E-mail: [EMAIL PROTECTED] - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] What causes this?
On Feb 1, 2007, at 8:57 AM, Richard White wrote: Can anyone tell me where to look for the cause of this error? 29-Jan 16:07 lbackup-dir: Start Backup JobId 1462, Job=ArcIMS.2007-01-29_16.07.46 29-Jan 18:07 lbackup-dir: ArcIMS.2007-01-29_16.07.46 Fatal error: Network error with FD during Backup: ERR=Connection reset by peer 29-Jan 18:07 lbackup-dir: ArcIMS.2007-01-29_16.07.46 Fatal error: No Job status returned from FD. That looks like exactly two hours... which to me suggests you try turning on Heartbeat Interval... http://www.bacula.org/rel-manual/Storage_Daemon_Configuratio.html#8741 http://www.bacula.org/rel-manual/Client_Fi_daemon_Configura.html#8485 Network connections dying from inactivity, perhaps. -- --Darien A. Hager [EMAIL PROTECTED] - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier. Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] What causes this?
Can anyone tell me where to look for the cause of this error? 29-Jan 16:07 lbackup-dir: Start Backup JobId 1462, Job=ArcIMS.2007-01-29_16.07.46 29-Jan 18:07 lbackup-dir: ArcIMS.2007-01-29_16.07.46 Fatal error: Network error with FD during Backup: ERR=Connection reset by peer 29-Jan 18:07 lbackup-dir: ArcIMS.2007-01-29_16.07.46 Fatal error: No Job status returned from FD. That looks like exactly two hours... which to me suggests you try turning on Heartbeat Interval... http://www.bacula.org/rel-manual/Storage_Daemon_Configuratio.html#8741 http://www.bacula.org/rel-manual/Client_Fi_daemon_Configura.html#8485 Network connections dying from inactivity, perhaps. Thanks, Darien. Alas, I have Bacula 1.36.3, and this is unsupported. I will soon upgrade. Incidentally, in what units are the time intervals? I.e., what will this mean: Heartbeat Interval = 100 -- Tks n rgds, Richard White CNE6 Network Engineer Mason County, Washington 360-427-5501 [EMAIL PROTECTED] - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier. Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] What causes this?
Oops. Worng. I was editing the director configuration. After a more careful reading, I learned that I should have been editing the storage daemon and file daemon configuration files. I have done so and will test. That worked! Thanks. -- Tks n rgds, Richard White CNE6 Network Engineer Mason County, Washington 360-427-5501 [EMAIL PROTECTED] - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier. Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
