Re: [Bacula-users] Bacula disk to tape config
On Fri, 2011-09-30 at 18:08 -0400, Dan Langille wrote: On Sep 27, 2011, at 12:42 PM, René Moser wrote: On Tue, 2011-09-27 at 16:51 +0100, Gavin McCullagh wrote: It sounds like what you're looking for could be implemented as: 1. A normal Full/Differential/Incremental job to a disk-based storage device to happen during the day. 2. A Migrate job to tape from disk to happen during the night. Yep, http://www.bacula.org/manuals/en/concepts/concepts/Migration_Copy.html is what I was looking for. Key point: A migrate or copy job can involved exactly one SD. Therefore, you cannot copy between SD. Thus, your single SD involved in this must store both the disk and the tape backups. I am not sure if I understand your point. SD = Storage Device, I guess. I can not migrate between a file storage device, to a tape storage device? But http://www.bacula.org/manuals/en/concepts/concepts/Migration_Copy.html#SECTION00164 does exactly show this by example right? Thanks for explaining again :) -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2dcopy1 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula disk to tape config
On Mon, 10 Oct 2011 14:13:26 +0200 René Moser r...@adfinis.com wrote: [...] SD = Storage Device, I guess. I can not migrate between a file storage device, to a tape storage device? [...] SD stands for Storage Daemon--it's a program which manages one or more storage devices available on the host a particular SD instance runs on. What you have been told, I suppose, is that migrate jobs can't cross SDs which means you cannot migrate a backup jobs which storage media is managed by an SD running on one host to a storage media managed by an SD running on another host. The reason for this, I supposed, is that this would require SDs to transfer data over the wire between themselves while normally data only flows between FDs and SDs. -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2dcopy1 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula disk to tape config
The reason for this, I supposed, is that this would require SDs to transfer data over the wire between themselves while normally data only flows between FDs and SDs. That is correct. Storage Daemons do not currently communicate. John -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2dcopy1 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula disk to tape config
On Sep 27, 2011, at 12:42 PM, René Moser wrote: On Tue, 2011-09-27 at 16:51 +0100, Gavin McCullagh wrote: It sounds like what you're looking for could be implemented as: 1. A normal Full/Differential/Incremental job to a disk-based storage device to happen during the day. 2. A Migrate job to tape from disk to happen during the night. Yep, http://www.bacula.org/manuals/en/concepts/concepts/Migration_Copy.html is what I was looking for. Key point: A migrate or copy job can involved exactly one SD. Therefore, you cannot copy between SD. Thus, your single SD involved in this must store both the disk and the tape backups. -- Dan Langille - http://langille.org -- All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2dcopy2 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula disk to tape config
René Moser wrote: Hi We are currently using a proprietary backup solution, Which proprietary backup system? and we evaluation bacula to replace it. We have some 100 hosts to backup up. The current work flow is like: Are these hosts PCs? What do they do? How much changes on them each night? -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2dcopy1 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula disk to tape config
On Tue, 27 Sep 2011, René Moser wrote: We are currently using a proprietary backup solution, and we evaluation bacula to replace it. We have some 100 hosts to backup up. The current work flow is like: 1. backup server backups host files over working time to a disk volume on backup server. 2. During night, the disk volume is written to tape, verified, and the disk volume will be purged. The catalogue knows afterwards that the files have to be restored from tapes and no longer exists on disk volume. So during working day, the tape drive is accessible and we are able to restore files form the tapes. How can does be done with bacula? It sounds like what you're looking for could be implemented as: 1. A normal Full/Differential/Incremental job to a disk-based storage device to happen during the day. 2. A Migrate job to tape from disk to happen during the night. It is important to note in the above that at present the disk-based storage device and the tape device must be on the same Storage Daemon for the Migrate to work. This should achieve moreorless what you're looking for I think (though I must say that I haven't used migrate jobs personally). To be honest though, the above work-flow raises a lot of questions for me: 1. You're doing backups of your live servers _during_ working hours. This is the opposite of what most people ordinarily do. Does this not create load on the already busy servers? Are the backups you get always in a consistent state? 2. It is apparently not possible to backup all 100 hosts straight to tape, so you write them to disk first. Why is that? It sounds like you're manually doing Spooling, which Bacula has transparent support for. You could, in Bacula, set up a single job which copied the backup data to disk first and then as that continues, spool them out to tape. This would need mean the two jobs happening at the same time, so you'd have to do it at night -- or else the tape drives would not be available during the day for restores. 3. Do you use Incremental backups at present or full backups each day? Gavin -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2dcopy1 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula disk to tape config
On Tue, 2011-09-27 at 16:41 +0100, Alan Brown wrote: We are currently using a proprietary backup solution, Which proprietary backup system? BRU Backup Server We have some 100 hosts to backup up. The current work flow is like: Are these hosts PCs? What do they do? How much changes on them each night? Linux Servers (like Mailserver, Virtual Private Servers, Web servers, FTP Servers, etc.) We do differential backups. it really depends on the server how many files changed. The range is from a few MB to 2-3GB changes We do not backup every host during night, just a few per night on a day per week. -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2dcopy1 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula disk to tape config
On Tue, 2011-09-27 at 16:51 +0100, Gavin McCullagh wrote: It sounds like what you're looking for could be implemented as: 1. A normal Full/Differential/Incremental job to a disk-based storage device to happen during the day. 2. A Migrate job to tape from disk to happen during the night. Yep, http://www.bacula.org/manuals/en/concepts/concepts/Migration_Copy.html is what I was looking for. To be honest though, the above work-flow raises a lot of questions for me: 1. You're doing backups of your live servers _during_ working hours. This is the opposite of what most people ordinarily do. Does this not create load on the already busy servers? Are the backups you get always in a consistent state? Okay, it is not _really_ during day it is more like backups over night to disk and backup to tape should be finished in the morning. But this is just a detail. We did not have (yet) really (big) problems about consistency. But as you say these systems are live systems and if a we hit a bad timing, that is always a problem. 2. It is apparently not possible to backup all 100 hosts straight to tape, so you write them to disk first. Why is that? It sounds like you're manually doing Spooling, which Bacula has transparent support for. You could, in Bacula, set up a single job which copied the backup data to disk first and then as that continues, spool them out to tape. This would need mean the two jobs happening at the same time, so you'd have to do it at night -- or else the tape drives would not be available during the day for restores. This may be a limitation of the current backup software in use. It may be faster this way (we are using a tape changer). We also have had troubles with some tape drives and this way (backup first to disk) was an advantage so the backup was already on disk and we just had to fix the tape changer and backup the disk volume to tape. I am really open do to it the bacula way if there is a better workflow. 3. Do you use Incremental backups at present or full backups each day? Actually mostly differentials. -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2dcopy1 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula disk to tape config
On Tue, 2011-09-27 at 16:51 +0100, Gavin McCullagh wrote: 2. It is apparently not possible to backup all 100 hosts straight to tape, so you write them to disk first. Why is that? It sounds like you're manually doing Spooling, which Bacula has transparent support for. You could, in Bacula, set up a single job which copied the backup data to disk first and then as that continues, spool them out to tape. Do you have an example, how the spooling like you described can be implemented? -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2dcopy1 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula disk to tape config
On Tue, 27 Sep 2011, René Moser wrote: On Tue, 2011-09-27 at 16:51 +0100, Gavin McCullagh wrote: 2. It is apparently not possible to backup all 100 hosts straight to tape, so you write them to disk first. Why is that? It sounds like you're manually doing Spooling, which Bacula has transparent support for. You could, in Bacula, set up a single job which copied the backup data to disk first and then as that continues, spool them out to tape. Do you have an example, how the spooling like you described can be implemented? We're only starting to use tapes ourselves now, so this is somewhat theoretical still for me. However, this is where the documentation is and I'm pretty sure it's used by a good number of people who use tapes and the feature has been in Bacula for a long time: http://www.bacula.org/5.0.x-manuals/en/main/main/Data_Spooling.html Gavin -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2dcopy1 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula disk to tape config
Hi, On Tue, 27 Sep 2011, René Moser wrote: Okay, it is not _really_ during day it is more like backups over night to disk and backup to tape should be finished in the morning. But this is just a detail. We did not have (yet) really (big) problems about consistency. But as you say these systems are live systems and if a we hit a bad timing, that is always a problem. I'm not sure how your timings work out, but spooling might give you a shorter overall backup window -- so the tapes would free up quicker, which might allow you to run the entire process outside of work hours. It's difficult to tell from here though. This may be a limitation of the current backup software in use. It may be faster this way (we are using a tape changer). We also have had troubles with some tape drives and this way (backup first to disk) was an advantage so the backup was already on disk and we just had to fix the tape changer and backup the disk volume to tape. Not having extensively used tapes and spooling, I can't say if this will be workable here. You'd need to test. I am really open do to it the bacula way if there is a better workflow. I guess you'll need to do some testing and see what works best for you. It sounds somewhat promising anyway. 3. Do you use Incremental backups at present or full backups each day? Actually mostly differentials. Might incrementals be quicker in terms of limiting load on the live server? Another feature you might consider is the ability to run Virtual Full backups to consolidate an old full backup and subsequent incremental or differential backups into a new full backup. We do this using disk-based backups, but I'm not sure it will be so practical on tapes. You need to have two devices too, one to read from and one to write to which you may not have. Gavin -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2dcopy1 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
