[Bacula-users] Copying backup On the fly
Hello, I'm using bacula for servers but the problem is that both servers and where bacula runs are on the same room. I'd like to know how to implement a duplication of the data bacula stores in order to put a copy on another location (via internet). Of course it is not possible to copy simply the backup file because it is too big. I've seen that it is possible to set copy jobs but I don't know how to use it (and I don't know if this is made for what I want) Thanks for your help Sebastien VINOT -- Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Copying backup On the fly
I'm using bacula for servers but the problem is that both servers and where bacula runs are on the same room. I'd like to know how to implement a duplication of the data bacula stores in order to put a copy on another location (via internet). Of course it is not possible to copy simply the backup file because it is too big. One suggestion. Make the volumes smaller. 1 to 10GB should be fine. Then use rsync (possibly run from a cron job) to send the volumes offsite. John -- Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Copying backup On the fly
Hi. I am actually thinking about same thing at this time, and well... deduplicated replication over network under linux seems to be a little bit complicated task... But I have an idea that could solve this issue, however, I don't have it implemented yet... but the idea is to use DRBD and lessfs combination, DRBD for replication and lessfs for deduplication. Sadly lessfs is not production use ready, and DRBD is not well tested either :-( So I would expect a lot of problems with implementation of this solution... I guess I will find out soon myself :) However, developer of lessfs aims to support replication directly in lessfs (I hope he mean replication over network), so maybe I would ask him about his schedule plan, then we could wait till he implement replication. From: Sébastien VINOT [mailto:sebastien.vi...@logisphere.fr] Sent: Thursday, October 15, 2009 2:29 PM To: bacula-users@lists.sourceforge.net Subject: [Bacula-users] Copying backup On the fly Hello, I'm using bacula for servers but the problem is that both servers and where bacula runs are on the same room. I'd like to know how to implement a duplication of the data bacula stores in order to put a copy on another location (via internet). Of course it is not possible to copy simply the backup file because it is too big. I've seen that it is possible to set copy jobs but I don't know how to use it (and I don't know if this is made for what I want) Thanks for your help Sebastien VINOT -- Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Copying backup On the fly
This is not a solution at all, because Bacula every time you create new backup will create new virtual tapes, and since rsync deduplicate only files, not entire amount of data that is going to be transfared, this do not solve anything for us. Acctually I am not even sure if we would use LVM on botom and rsync those snapshots of a LVM volumes, I guess rsync do not compare hashes of all blocks of transfered but only first block hash with a first block, second with a second,... So again, rsync would not be help even in this case. I am confused how it does not solve the problem. I am talking about using rsync to mirror the volume files that bacula uses to store all of your backups. If you limit the size of these to a few GB, rsync will only be sending the volumes that have changed but you will still have a complete second copy of the bacula volumes offsite. John -- Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Copying backup On the fly
Yeah it will, but when new full backup is taken, you again copy entire operating system copy over network, even though you have already a few copy of that operating systém on a target server. So, yeah it will low the traffic, but not that much as a smarter solution could. :-) Every time you také full backup, all data will be transfared, every time you will do a incremental backup you will transfer all files those were changet from last backup even though there were only added few bytes to the end of a file, and well... Maybe that is good for you, but I would be much more grateful for something that would do completely deduplicated replication as you can get for example from NetApp :) Yeah, I agree that your solution is 1000 times simplier, but also it is less effective than a deduplicated replication, that I hope can get with lessfs + DRBD... And yeah, my idea works only in theory, your is really working. :-) -Original Message- From: John Drescher [mailto:dresche...@gmail.com] Sent: Thursday, October 15, 2009 4:08 PM To: Stopka m Marek; bacula-users Subject: Re: [Bacula-users] Copying backup On the fly This is not a solution at all, because Bacula every time you create new backup will create new virtual tapes, and since rsync deduplicate only files, not entire amount of data that is going to be transfared, this do not solve anything for us. Acctually I am not even sure if we would use LVM on botom and rsync those snapshots of a LVM volumes, I guess rsync do not compare hashes of all blocks of transfered but only first block hash with a first block, second with a second,... So again, rsync would not be help even in this case. I am confused how it does not solve the problem. I am talking about using rsync to mirror the volume files that bacula uses to store all of your backups. If you limit the size of these to a few GB, rsync will only be sending the volumes that have changed but you will still have a complete second copy of the bacula volumes offsite. John -- Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Copying backup On the fly
On Thu, 15 Oct 2009, John Drescher might have said: I'm using bacula for servers but the problem is that both servers and where bacula runs are on the same room. I'd like to know how to implement a duplication of the data bacula stores in order to put a copy on another location (via internet). Of course it is not possible to copy simply the backup file because it is too big. One suggestion. Make the volumes smaller. 1 to 10GB should be fine. Then use rsync (possibly run from a cron job) to send the volumes offsite. John My bacula setup is pulling the remote files to on disk volumes that are 5GB each. I have a final job that looks for what disk volumes have changed from the previous backup and writes those volumes to tape. I could use rsync(1) to copy the 5GB volumes to another server and did consider copying those disk volumes over the net to a remote location. I decided against this approach as the rsync(1) would consume my entire bandwidth. Not a bad solution for incrementals, but not good for fulls. Mike -- Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Copying backup On the fly
2009/10/15 marek.m.sto...@tieto.com: Yeah it will, but when new full backup is taken, you again copy entire operating system copy over network, even though you have already a few copy of that operating systém on a target server. So, yeah it will low the traffic, but not that much as a smarter solution could. :-) Every time you také full backup, all data will be transfared, every time you will do a incremental backup you will transfer all files those were changet from last backup even though there were only added few bytes to the end of a file, and well... Maybe that is good for you, but I would be much more grateful for something that would do completely deduplicated replication as you can get for example from NetApp :) Yeah, I agree that your solution is 1000 times simplier, but also it is less effective than a deduplicated replication, that I hope can get with lessfs + DRBD... And yeah, my idea works only in theory, your is really working. :-) -Original Message- From: John Drescher [mailto:dresche...@gmail.com] Sent: Thursday, October 15, 2009 4:08 PM To: Stopka m Marek; bacula-users Subject: Re: [Bacula-users] Copying backup On the fly This is not a solution at all, because Bacula every time you create new backup will create new virtual tapes, and since rsync deduplicate only files, not entire amount of data that is going to be transfared, this do not solve anything for us. Acctually I am not even sure if we would use LVM on botom and rsync those snapshots of a LVM volumes, I guess rsync do not compare hashes of all blocks of transfered but only first block hash with a first block, second with a second,... So again, rsync would not be help even in this case. I see thanks for the answer.. John -- Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Copying backup On the fly
On Thu, Oct 15, 2009 at 11:05 AM, John Drescher dresche...@gmail.com wrote: 2009/10/15 marek.m.sto...@tieto.com: Yeah it will, but when new full backup is taken, you again copy entire operating system copy over network, even though you have already a few copy of that operating systém on a target server. So, yeah it will low the traffic, but not that much as a smarter solution could. :-) Every time you také full backup, all data will be transfared, every time you will do a incremental backup you will transfer all files those were changet from last backup even though there were only added few bytes to the end of a file, and well... Maybe that is good for you, but I would be much more grateful for something that would do completely deduplicated replication as you can get for example from NetApp :) Yeah, I agree that your solution is 1000 times simplier, but also it is less effective than a deduplicated replication, that I hope can get with lessfs + DRBD... And yeah, my idea works only in theory, your is really working. :-) BTW, I think base jobs can help here. Although this is very new. John -- Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Copying backup On the fly
I may be missing something, but using smaller volume files should work for you. You would still need to transfer the entire job, but only the volumes that that job touched. So for example, instead of a 1G job appending to the end of a 10G volume, and having to send the entire 10G again, if you used smaller volumes, say 2G, then you would just send that 2G file. It sounds like you want (and I have not tried much yet) opensolaris and ZFS. You would take a snapshot, and send it to the remote server. marek.m.sto...@tieto.com wrote: Yeah it will, but when new full backup is taken, you again copy entire operating system copy over network, even though you have already a few copy of that operating systém on a target server. So, yeah it will low the traffic, but not that much as a smarter solution could. :-) Every time you také full backup, all data will be transfared, every time you will do a incremental backup you will transfer all files those were changet from last backup even though there were only added few bytes to the end of a file, and well... Maybe that is good for you, but I would be much more grateful for something that would do completely deduplicated replication as you can get for example from NetApp :) Yeah, I agree that your solution is 1000 times simplier, but also it is less effective than a deduplicated replication, that I hope can get with lessfs + DRBD... And yeah, my idea works only in theory, your is really working. :-) -Original Message- From: John Drescher [mailto:dresche...@gmail.com] Sent: Thursday, October 15, 2009 4:08 PM To: Stopka m Marek; bacula-users Subject: Re: [Bacula-users] Copying backup On the fly This is not a solution at all, because Bacula every time you create new backup will create new virtual tapes, and since rsync deduplicate only files, not entire amount of data that is going to be transfared, this do not solve anything for us. Acctually I am not even sure if we would use LVM on botom and rsync those snapshots of a LVM volumes, I guess rsync do not compare hashes of all blocks of transfered but only first block hash with a first block, second with a second,... So again, rsync would not be help even in this case. I am confused how it does not solve the problem. I am talking about using rsync to mirror the volume files that bacula uses to store all of your backups. If you limit the size of these to a few GB, rsync will only be sending the volumes that have changed but you will still have a complete second copy of the bacula volumes offsite. John -- Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Copying backup On the fly
You can throttle rsync with --bwlimit, unless you would need to use the entire bandwidth to finish in your time window. Mike Eggleston wrote: On Thu, 15 Oct 2009, John Drescher might have said: I'm using bacula for servers but the problem is that both servers and where bacula runs are on the same room. I'd like to know how to implement a duplication of the data bacula stores in order to put a copy on another location (via internet). Of course it is not possible to copy simply the backup file because it is too big. One suggestion. Make the volumes smaller. 1 to 10GB should be fine. Then use rsync (possibly run from a cron job) to send the volumes offsite. John My bacula setup is pulling the remote files to on disk volumes that are 5GB each. I have a final job that looks for what disk volumes have changed from the previous backup and writes those volumes to tape. I could use rsync(1) to copy the 5GB volumes to another server and did consider copying those disk volumes over the net to a remote location. I decided against this approach as the rsync(1) would consume my entire bandwidth. Not a bad solution for incrementals, but not good for fulls. Mike -- Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Copying backup On the fly
W dniu 15 października 2009 17:13 użytkownik Brian Debelius bdebel...@intelesyscorp.com napisał: I may be missing something, but using smaller volume files should work for you. You would still need to transfer the entire job, but only the volumes that that job touched. So for example, instead of a 1G job appending to the end of a 10G volume, and having to send the entire 10G again, if you used smaller volumes, say 2G, then you would just send that 2G file. Well, if You are making backup on disk, the best way is to setup volume pool(s) with one job per volume feature. Then You will get as many new or recycled volumes as jobs. So, finally You will replicate only last altered volumes with a size of made backups. bye -- Radosław Korzeniewski rados...@korzeniewski.net -- Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Copying backup On the fly
I'm thinking of a solution (maybe not so simple): - Before a job starts, a small script checks if the file exists and if it is a socket (assuming it is possible to know the file name at this time). If it is a real file (which means it is a new file, the script can replace the file by a socket - The FD sends data into a socket and a small script listens on the socket. For all data, it writes it at the end of the real file and can do the copy (or prepare it for a copy after all). Is it a possible solution ? Sebastien Sounds a little like a network raid 1. There are filesystems that can do this. http://www.drbd.org/ John -- Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Copying backup On the fly
Hi, 15.10.2009 21:23, Sébastien VINOT wrote: ... I'm thinking of a solution (maybe not so simple): - Before a job starts, a small script checks if the file exists and if it is a socket (assuming it is possible to know the file name at this time). If it is a real file (which means it is a new file, the script can replace the file by a socket - The FD sends data into a socket and a small script listens on the socket. For all data, it writes it at the end of the real file and can do the copy (or prepare it for a copy after all). Is it a possible solution ? Yup. Only you can simplify it a lot - have a look at the SD device resource in the manual, especially the Archive Device and Device Type sections... and carefully read the notes regarding the FIFO type. That should enable you to transparently capture the data flow, and store it both locally and pass it to a remote location. Still no deduplication, though... and managing volume handling and restores will be no fun at all. Arno Sebastien -- Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- Arno Lehmann IT-Service Lehmann Sandstr. 6, 49080 Osnabrück www.its-lehmann.de -- Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
