Re: [Bacula-users] Using Bittorrent as Backup Technology
On 2/7/2010 10:29 PM, mehma sarja wrote: Thanks to both of you for your thoughts. I am actually looking for anyone who has tried it or someone who has thought about trying it and dropped the idea. I understand 10 instances is very small - but I am not contemplating huge files either. Is the transfer latency something like filesize/number of nodes? Bacula can dump a daily differential and Bittorrent can take it from there. Databases will be more problematic. DRBD assumes, atleast that's what Google says, stable, decent bandwidth. My design takes network cutting out on a daily basis into account. We are talking the US postal service kind of a backup. It should work under adverse situations. DRBD assumes stable, decent bandwidth, but that is not to say it doesn't handle network problems well. DRBD is essentially RAID 1 over a network connection as opposed to SCSI, SAS, etc. If Bacula is writing to a DRBD device and the connection between DRBD nodes goes down, Bacula will never know the difference. When the network comes back up, the DRBD node on the other side will be rebuilt in the background, just as a raid array will be rebuilt when a failing drive is replaced. I also want to be able to plug another unit in and have it start backing up without any twiddling. I have been there on the rsync front and it is reliable and simple. I just think if a cleaning person bumps the power cord and then plugs it back in (at night) that the machine should right itself. I know it can be done with scripts and rsync. Why invent and maintain something when p2p apps have solved the problem already? For some reason, I still think Bittorrent could be a winner in the set it and forget it backup arena. Technically, this is not really a backup, it is actually backing up a backup. A poor man's disaster planning. DRBD is designed for high availability systems. It is a device driver, thus does not require a two step approach. It is already available and works as is without any changes to Bacula. To be clear, here is the amount of work I am willing to do with this technology: a. Get the hardware and connection installed b. Configure encrypted transfers among a limited number of nodes c. Configure the master directories which needs to be backed up for 3x redundancy THAT'S IT - any more work is getting into fiddling. Mehma === On Sun, Feb 7, 2010 at 1:25 AM, Wolfgang Denk w...@denx.de mailto:w...@denx.de wrote: Dear mehma sarja, In message ec5d34681002061636p308947abre20a0b3887454...@mail.gmail.com mailto:ec5d34681002061636p308947abre20a0b3887454...@mail.gmail.com you wrote: PROBLEM I am trying to solve off-site, on-disk backup problem in the Bay Area which is prone to earthquakes. QUESTION Is anyone out there using bittorrent as a backup tool? Hm... have you ever looked into DRBD instead? Bittorrent has not been designed for what you want to use it. DRBD has. I am aware that you can use many tools for things they have not specifically designed for, but I also believe that in most cases the tools that have been designed for a specific purpose are most efficient. Best regards, Wolfgang Denk -- DENX Software Engineering GmbH, MD: Wolfgang Denk Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: w...@denx.de mailto:w...@denx.de There is a time in the tides of men, Which, taken at its flood, leads on to success. On the other hand, don't count on it. - T. K. Lawson -- The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Using Bittorrent as Backup Technology
Dear mehma sarja, In message ec5d34681002061636p308947abre20a0b3887454...@mail.gmail.com you wrote: PROBLEM I am trying to solve off-site, on-disk backup problem in the Bay Area which is prone to earthquakes. QUESTION Is anyone out there using bittorrent as a backup tool? Hm... have you ever looked into DRBD instead? Bittorrent has not been designed for what you want to use it. DRBD has. I am aware that you can use many tools for things they have not specifically designed for, but I also believe that in most cases the tools that have been designed for a specific purpose are most efficient. Best regards, Wolfgang Denk -- DENX Software Engineering GmbH, MD: Wolfgang Denk Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: w...@denx.de There is a time in the tides of men, Which, taken at its flood, leads on to success. On the other hand, don't count on it. - T. K. Lawson -- The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Using Bittorrent as Backup Technology
Thanks to both of you for your thoughts. I am actually looking for anyone who has tried it or someone who has thought about trying it and dropped the idea. I understand 10 instances is very small - but I am not contemplating huge files either. Is the transfer latency something like filesize/number of nodes? Bacula can dump a daily differential and Bittorrent can take it from there. Databases will be more problematic. DRBD assumes, atleast that's what Google says, stable, decent bandwidth. My design takes network cutting out on a daily basis into account. We are talking the US postal service kind of a backup. It should work under adverse situations. I also want to be able to plug another unit in and have it start backing up without any twiddling. I have been there on the rsync front and it is reliable and simple. I just think if a cleaning person bumps the power cord and then plugs it back in (at night) that the machine should right itself. I know it can be done with scripts and rsync. Why invent and maintain something when p2p apps have solved the problem already? For some reason, I still think Bittorrent could be a winner in the set it and forget it backup arena. Technically, this is not really a backup, it is actually backing up a backup. A poor man's disaster planning. To be clear, here is the amount of work I am willing to do with this technology: a. Get the hardware and connection installed b. Configure encrypted transfers among a limited number of nodes c. Configure the master directories which needs to be backed up for 3x redundancy THAT'S IT - any more work is getting into fiddling. Mehma === On Sun, Feb 7, 2010 at 1:25 AM, Wolfgang Denk w...@denx.de wrote: Dear mehma sarja, In message ec5d34681002061636p308947abre20a0b3887454...@mail.gmail.com you wrote: PROBLEM I am trying to solve off-site, on-disk backup problem in the Bay Area which is prone to earthquakes. QUESTION Is anyone out there using bittorrent as a backup tool? Hm... have you ever looked into DRBD instead? Bittorrent has not been designed for what you want to use it. DRBD has. I am aware that you can use many tools for things they have not specifically designed for, but I also believe that in most cases the tools that have been designed for a specific purpose are most efficient. Best regards, Wolfgang Denk -- DENX Software Engineering GmbH, MD: Wolfgang Denk Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: w...@denx.de There is a time in the tides of men, Which, taken at its flood, leads on to success. On the other hand, don't count on it. - T. K. Lawson -- The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Using Bittorrent as Backup Technology
On 02/06/10 19:36, mehma sarja wrote: WHY POST THIS? I am posting this message for two reasons: a) Bacula and other enterprise backup tools do not particularly like unreliable bandwidth connections and a bittorrent-like technology fills the gap. The gap is that Bacula and other tools are making disk backups convenient. As people move away from tape, the disk-based systems are increasingly at risk from natural disasters and wear and tear over time. Thus increasing risk as compared to tape backups. Although they too are susceptible to wear and tear. Do you consider some sort of off-site as a natural cost of doing on-disk backups? b) If someone is doing it or thinking about using bittorrent technologies - I'd like to know how of your experiences (and config files) and what hardware you use. My first thoughts on this: (1) With a comparatively small number of hosts and clients, all with reasonably fast business-grade network connections, there are no advantages to using BitTorrent. BitTorrent is fast because it distributes the server load across a large number of peers to serve a large number of clients, which themselves re-seed to pick up the server load. You do not have that situation. BitTorrent is a good technology for software distribution, but it's not really applicable to the problem of network backup. (2) The problem of unreliable network connections can be trivially solved using rsync. Server A rsyncs changes in its data set every hour, say, to mirrors at sites B and C, which sites B and C then back up every night. Likewise, site B rsyncs *its* server data pool to mirrors at sites A and C, which they then back up. And this is a worst case assuming sires A, B and C have different data sets. If the three sites are hosting the same data, then they simply maintain nightly local backups of their own data, and rsync the changes between themselves as needed to keep themselves in sync. For that matter, if all three sites A, B and C have the same data, you need not even necessarily bother backing them up at all; you could have a master repository at a hardened facility somewhere geographically remote from all three, which has a master copy of the data and simply pushes all changes out to the slave sites. In short, your question isn't really a backup-software question so much as it is a distributed site architecture and data consistency question. -- Phil Stracchino, CDK#2 DoD#299792458 ICBM: 43.5607, -71.355 ala...@caerllewys.net ala...@metrocast.net p...@co.ordinate.org Renaissance Man, Unix ronin, Perl hacker, Free Stater It's not the years, it's the mileage. -- The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
