[Bacula-users] Recycling of Missing File Volumes
Hi guys, Having a problem with volume recycling on removable file storage: It seems Bacula doesn't check whether a file volume actually exists on disk before recycling it. Is this intended and / or expected behaviour? I'm running Bacula 2.1.28 (including the director and SDs) on Windows Server 2003 machines. Please hassle me if you'd like me to send through the conf files. I'm backing up several clients to USB-connected external hard-disks (one disk for each day of the week). I'm seeing multiple volumes with the same file names appear on multiple disks. This is both a pain from the sense of identifying which volumes I might actually need, should disaster recovery be required, and in that it means I have to manually jump in and delete files once in a while, to avoid disks filling up. I've heard the tape-autochanger script(s) can be used in these kinds of situations, but am not terribly sure I want to go down this path, if it can be avoided. Is there a reason Bacula doesn't check for the existence of volumes (particularly those marked as being removable, as these are) before recycling them, or could this be considered a suggestion for future versions? Thanks for your time! -- Nick Withers email: [EMAIL PROTECTED] Web: http://www.nickwithers.com Mobile: +61 414 397 446 - This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now http://get.splunk.com/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] On-Site and Off-Site Backup Replicas... Wait For Copy Job?
On Sat, 10 Mar 2007 12:46:04 +0100 Mikael Kermorgant [EMAIL PROTECTED] wrote: How about using mirroring using raid1 ? (you'd probably have to buy a thirs 200gb). This way, you achieve data synchronisation easily, always have a local copy from which to run restores and you cycle between 2 disks to keep an offsite copy. Certainly a possibility - thanks for the suggestion! Regards, -- Mikael Kermorgant -- Nick Withers email: [EMAIL PROTECTED] Web: http://www.nickwithers.com Mobile: +61 414 397 446 - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] On-Site and Off-Site Backup Replicas... Wait For Copy Job?
On Wed, 7 Mar 2007 13:06:15 -0800 (PST)
Kel Raywood [EMAIL PROTECTED] wrote:
A few weeks ago there was a short thread on a similar theme. See
http://article.gmane.org/gmane.comp.sysutils.backup.bacula.general/31927
and other posts in the thread.
Hmmm... I think I had seen this one, but am glad you pointed it out
again, as I think I'm now much more receptive to the first idea.
In that thread, the on-site disk-volumes were written first and
then copied to tape for off-site protection.
However your requirement ...
I'd prefer to do the actual backup to the removable drives in the
first instance as these are the critical ones and I'd like the
job(s) to fail in the case of full removable-drives.
makes this significantly different.
It's really more of a desirable - I could handle the idea of a
separate archive job failing, if needs be.
I hadn't really put too much thought into the idea of having multiple
catalogs, but this could be quite dandy - I could have completely
different backup regimes for on-site and off-site stores, for
instance. Copy jobs'd sure be tops though! :-)
I guess since the off-site backups would only be used in an emergency
situation, and since a restore (using the main catalog) would ask
for the right volumes, which could then (or beforehand) easily
be restored from the off-site media to the on-site location this
could work quite well. I'd rather not be making the assumption that a
huge amount of storage will be available to restore the on-site
volumes to, but really that's not an unreasonable one anyway.
You said that you are all windows based. Does that include the
server running the storage daemon?
It does.
Using Linux, you might be able to achieve what you want using
bind mounts. Here is an outline of a basic scheme.
Create a directory for the on-site disk-volumes and bind it to the
location where the storage-daemon expects to find them.
mount --bind /on-site/bacula/volumes /var/lib/bacula/volumes
This should be in /etc/rc.d/rc.local or similar.
Your storage-daemon config would include:
Device {
...
Archive Device = /var/lib/bacula/volumes
Device Type = File;
LabelMedia = yes;
Random Access = Yes;
AutomaticMount = yes;
RemovableMedia = no;
AlwaysOpen = no;
...
}
Have a pre-flight (RunBefore) script in a job with a unique, lowest
Priority value, to ensure that it runs alone and first. It should
unbind the on-site storage and bind the removable-media.
Pre-flight script:
umount /var/lib/bacula/volumes
mount --bind /media/usbdisk/volumes /var/lib/bacula/volumes
Have a post-flight script in a job with a unique, highest
Priority value to ensure that it runs alone and last. It should
unbind the removable disk, copy the disk volumes to the on-site
storage, and bind that to the appropriate disk-volume directory.
Post-flight script:
umount /var/lib/bacula/volumes
rsync -a /media/usbdisk/volumes/ /on-site/bacula/volumes/
mount --bind /on-site/bacula/volumes /var/lib/bacula/volumes
Now the storage daemon will be able to restore from the on-site
copy of the volumes.
In the resource of the Pool to which the disk volumes are assigned,
you should have Maximum Volume Jobs = 1 to ensure that the
disk-volumes are closed after use. It won't be necessary to know
a-priori the names of the volumes since rsync will only copy new
volumes. You should also be dumping your catalogue and keeping
copies of your config files somewhere.
I haven't thought about issues with recycling but it should be
scriptable.
Windows does allow for drives to be mounted on directories, but they
have to be empty - which unfortunately somewhat destroys the
possibility of tricking Bacula in a similar way to this (I'd quite
happily have the removable drive(s) mounted on both G: (for
instance) and D:\Backups, then remove the D:\Backups mount to
perform a copy (assuming there's a scriptable way of doing this in
Windows), but the empty directory thing kills it.
Still - a nifty idea!
Hope this helps.
It did indeed - thanks very much.
Kel
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to
share your opinions on IT business topics through brief
surveys-and earn cash
http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV
___ Bacula-users
mailing list Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
--
Nick Withers
email: [EMAIL PROTECTED]
Web: http://www.nickwithers.com
Mobile: +61 414 397 446
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions
[Bacula-users] On-Site and Off-Site Backup Replicas... Wait For Copy Job?
G'day guys, Just trying to design a backup solution using Bacula for a small company I work for and would appreciate some help with a few issues. This email may be rather long, so certainly appreciate anyone taking the time to read it, let alone offer any insight they may have! The main problem I'm having is that I want backups both on-site (for restoring files users accidentally deleted and other relatively trivial matters) and off-site (for when the site gets stepped on by Godzilla). Methinks I'm after (upcoming?) copy job magic... :-) The company currently has two 200-odd GB USB-accessible HDDs and five 110-odd GB USB accessible HDDs. I'd like to avoid having to acquire any further hardware at this point and think that this should be enough to hold the required data anyway, at least following the scheme outlined below. My current idea runs like this: - A monthly full backup of each machine to one of the 200 GB drives (each machine uses it's own full backup pool) - This drive is then taken off-site and the other 200 GB drive put in its place for the next monthly full backup - Weekday night-run differential backups to one of the 110 GB drives (each machine uses it's own differential backup pool) - This drive is then taken off-site and the 110 GB drive for the next differential backup is put in its place This would mean that with the just the full backup and the previous day's differential backup drives from off-site, the previous day's state could be completely restored. Bet I've missed some really obviously nicer way of achieving this or something similar though! I don't believe that too much data will be changing on a daily basis, so hopefully the increasingly large differential backups throughout the month won't be a problem. Now I also want to be able to access the backups on-site, without having to drag in off-site backup drives. I'd prefer to do the actual backup to the removable drives in the first instance as these are the critical ones and I'd like the job(s) to fail in the case of full removable drives. I've thought of: - Copying the backup volumes from the removable drives to a local location following a backup. Problems / potential problems: - Have to know the names of the relevant volumes on the removable media - Would really like to be able to specify restoring from the relocated volumes in a nice manner, rather than those on the removable media - Migrating the volumes from the removable media to a local location following a backup. Problems / potential problems: - Would want to be able to easily use the removable-drive volumes if the local ones go AWOL (e.g., Godzilla...) - Would want matching volume names on local and removable locations so that volumes are easily identifiable - Would want volume recycling to occur on both locations I've attached (slightly sanitised) Director and Software Director config files for the current setup (very much alpha), in case this helps. Anyone have any ideas? Should I just hang on until copy job saves everything? Am I being profoundly stupid in one / many ways? By the way, the system's all-Windows and screaming along very nicely using 2.0.2 - huzzah! Any and all thoughts appreciated! -- Nick Withers email: [EMAIL PROTECTED] Web: http://www.nickwithers.com Mobile: +61 414 397 446 bacula-dir.conf Description: Binary data bacula-sd.conf Description: Binary data - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] On-Site and Off-Site Backup Replicas... Wait For Copy Job?
On Wed, 07 Mar 2007 08:36:44 -0700 Don MacArthur [EMAIL PROTECTED] wrote: Nick, I use a similar approach with one significant difference and a few small ones. I have have periodic (daily and weekly) pools and write (in parallel jobs for all the clients) all the backups to one volume to save disk space and reduce the management complexity. I thought about having one big (well, bigger) volume for each day's work, but am currently going with individual volumes for each host so that each backup can be performed simultaneously. Have I missed something here or would your parallel jobs not actually be parallel due to each having to wait for the volume to become available? This media goes off site. Then, I have a separate set of jobs at a lower priority, scheduled to run later, that write to another media for on site storage. This can be an SD resource on a local hard drive volume. I use a disk rack with a few TB, but anything with enough space will work. I have my volumes configured to limit the size of each (thanks to a post I saw from Kern) as they are used, and they get reused when the jobs expire. The priority thing keeps the second set of jobs from overlapping with the first set. I found that I got radically different (differential) results when both FD jobs were running on a client at the same time. I don't know how/why, but this seems to get me the results I want - two relatively similar copies of the same data. Wouldn't this create potential dramas with restores (assuming you're using the same catalog for both jobs)? I mean, couldn't a restore potentially demand both on-site and off-site volumes? All my on site jobs expire after 2 weeks. Off site is kept for 4 weeks (daily) or 52 weeks (weekly). Of course, you'll adjust retention for your needs. FWIW. Thanks! On Wed, 2007-03-07 at 22:40 +1100, Nick Withers wrote: G'day guys, Just trying to design a backup solution using Bacula for a small company I work for and would appreciate some help with a few issues. This email may be rather long, so certainly appreciate anyone taking the time to read it, let alone offer any insight they may have! The main problem I'm having is that I want backups both on-site (for restoring files users accidentally deleted and other relatively trivial matters) and off-site (for when the site gets stepped on by Godzilla). Methinks I'm after (upcoming?) copy job magic... :-) The company currently has two 200-odd GB USB-accessible HDDs and five 110-odd GB USB accessible HDDs. I'd like to avoid having to acquire any further hardware at this point and think that this should be enough to hold the required data anyway, at least following the scheme outlined below. My current idea runs like this: - A monthly full backup of each machine to one of the 200 GB drives (each machine uses it's own full backup pool) - This drive is then taken off-site and the other 200 GB drive put in its place for the next monthly full backup - Weekday night-run differential backups to one of the 110 GB drives (each machine uses it's own differential backup pool) - This drive is then taken off-site and the 110 GB drive for the next differential backup is put in its place This would mean that with the just the full backup and the previous day's differential backup drives from off-site, the previous day's state could be completely restored. Bet I've missed some really obviously nicer way of achieving this or something similar though! I don't believe that too much data will be changing on a daily basis, so hopefully the increasingly large differential backups throughout the month won't be a problem. Now I also want to be able to access the backups on-site, without having to drag in off-site backup drives. I'd prefer to do the actual backup to the removable drives in the first instance as these are the critical ones and I'd like the job(s) to fail in the case of full removable drives. I've thought of: - Copying the backup volumes from the removable drives to a local location following a backup. Problems / potential problems: - Have to know the names of the relevant volumes on the removable media - Would really like to be able to specify restoring from the relocated volumes in a nice manner, rather than those on the removable media - Migrating the volumes from the removable media to a local location following a backup. Problems / potential problems: - Would want to be able to easily use the removable-drive volumes if the local ones go AWOL (e.g., Godzilla...) - Would want matching volume names on local and removable locations so that volumes are easily identifiable - Would want volume recycling to occur on both locations I've attached (slightly sanitised) Director and Software Director config files for the current setup (very much alpha), in case
Re: [Bacula-users] On-Site and Off-Site Backup Replicas... Wait For Copy Job?
On Wed, 07 Mar 2007 10:31:27 -0600 Brad Larson [EMAIL PROTECTED] wrote: I currently have my on-site bacula performing backups to a large LVM partition (no tapes). Some extra stuff gets copied to that partion (ie. bacula db dump,etc...) and then gets rsync'd through a ssh connection over the internet to an off-site LVM. One server has a monthly full backup that is too large ( 120G) to transfer that way so I manually copy it to a usb-book type disk and manually transfer it to the off-site location. The dailys, diffs and all other fulls transfer fine (about two/four hours in the middle of the night) without any problems. Rsync cleans up unused files and has run hands-free every night. I have a complete mirror copy of my bacula storage well before work starts the next day. That'd be pretty dandy - a relatively elegant solution, methinks. Unfortunately I don't really think there's a data link available to any other location with sufficient capacity to transfer even the nightly backup data off-site... The two LVMs are about 1.2T in each location although your size may vary. Nick Withers wrote: G'day guys, Just trying to design a backup solution using Bacula for a small company I work for and would appreciate some help with a few issues. This email may be rather long, so certainly appreciate anyone taking the time to read it, let alone offer any insight they may have! The main problem I'm having is that I want backups both on-site (for restoring files users accidentally deleted and other relatively trivial matters) and off-site (for when the site gets stepped on by Godzilla). Methinks I'm after (upcoming?) copy job magic... :-) The company currently has two 200-odd GB USB-accessible HDDs and five 110-odd GB USB accessible HDDs. I'd like to avoid having to acquire any further hardware at this point and think that this should be enough to hold the required data anyway, at least following the scheme outlined below. My current idea runs like this: - A monthly full backup of each machine to one of the 200 GB drives (each machine uses it's own full backup pool) - This drive is then taken off-site and the other 200 GB drive put in its place for the next monthly full backup - Weekday night-run differential backups to one of the 110 GB drives (each machine uses it's own differential backup pool) - This drive is then taken off-site and the 110 GB drive for the next differential backup is put in its place This would mean that with the just the full backup and the previous day's differential backup drives from off-site, the previous day's state could be completely restored. Bet I've missed some really obviously nicer way of achieving this or something similar though! I don't believe that too much data will be changing on a daily basis, so hopefully the increasingly large differential backups throughout the month won't be a problem. Now I also want to be able to access the backups on-site, without having to drag in off-site backup drives. I'd prefer to do the actual backup to the removable drives in the first instance as these are the critical ones and I'd like the job(s) to fail in the case of full removable drives. I've thought of: - Copying the backup volumes from the removable drives to a local location following a backup. Problems / potential problems: - Have to know the names of the relevant volumes on the removable media - Would really like to be able to specify restoring from the relocated volumes in a nice manner, rather than those on the removable media - Migrating the volumes from the removable media to a local location following a backup. Problems / potential problems: - Would want to be able to easily use the removable-drive volumes if the local ones go AWOL (e.g., Godzilla...) - Would want matching volume names on local and removable locations so that volumes are easily identifiable - Would want volume recycling to occur on both locations I've attached (slightly sanitised) Director and Software Director config files for the current setup (very much alpha), in case this helps. Anyone have any ideas? Should I just hang on until copy job saves everything? Am I being profoundly stupid in one / many ways? By the way, the system's all-Windows and screaming along very nicely using 2.0.2 - huzzah! Any and all thoughts appreciated! - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV
