Re: [Bacula-users] Using Bittorrent as Backup Technology
On 2/7/2010 10:29 PM, mehma sarja wrote: Thanks to both of you for your thoughts. I am actually looking for anyone who has tried it or someone who has thought about trying it and dropped the idea. I understand 10 instances is very small - but I am not contemplating huge files either. Is the transfer latency something like filesize/number of nodes? Bacula can dump a daily differential and Bittorrent can take it from there. Databases will be more problematic. DRBD assumes, atleast that's what Google says, stable, decent bandwidth. My design takes network cutting out on a daily basis into account. We are talking the US postal service kind of a backup. It should work under adverse situations. DRBD assumes stable, decent bandwidth, but that is not to say it doesn't handle network problems well. DRBD is essentially RAID 1 over a network connection as opposed to SCSI, SAS, etc. If Bacula is writing to a DRBD device and the connection between DRBD nodes goes down, Bacula will never know the difference. When the network comes back up, the DRBD node on the other side will be rebuilt in the background, just as a raid array will be rebuilt when a failing drive is replaced. I also want to be able to plug another unit in and have it start backing up without any twiddling. I have been there on the rsync front and it is reliable and simple. I just think if a cleaning person bumps the power cord and then plugs it back in (at night) that the machine should right itself. I know it can be done with scripts and rsync. Why invent and maintain something when p2p apps have solved the problem already? For some reason, I still think Bittorrent could be a winner in the set it and forget it backup arena. Technically, this is not really a backup, it is actually backing up a backup. A poor man's disaster planning. DRBD is designed for high availability systems. It is a device driver, thus does not require a two step approach. It is already available and works as is without any changes to Bacula. To be clear, here is the amount of work I am willing to do with this technology: a. Get the hardware and connection installed b. Configure encrypted transfers among a limited number of nodes c. Configure the master directories which needs to be backed up for 3x redundancy THAT'S IT - any more work is getting into fiddling. Mehma === On Sun, Feb 7, 2010 at 1:25 AM, Wolfgang Denk w...@denx.de mailto:w...@denx.de wrote: Dear mehma sarja, In message ec5d34681002061636p308947abre20a0b3887454...@mail.gmail.com mailto:ec5d34681002061636p308947abre20a0b3887454...@mail.gmail.com you wrote: PROBLEM I am trying to solve off-site, on-disk backup problem in the Bay Area which is prone to earthquakes. QUESTION Is anyone out there using bittorrent as a backup tool? Hm... have you ever looked into DRBD instead? Bittorrent has not been designed for what you want to use it. DRBD has. I am aware that you can use many tools for things they have not specifically designed for, but I also believe that in most cases the tools that have been designed for a specific purpose are most efficient. Best regards, Wolfgang Denk -- DENX Software Engineering GmbH, MD: Wolfgang Denk Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: w...@denx.de mailto:w...@denx.de There is a time in the tides of men, Which, taken at its flood, leads on to success. On the other hand, don't count on it. - T. K. Lawson -- The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Using Bittorrent as Backup Technology
Dear mehma sarja, In message ec5d34681002061636p308947abre20a0b3887454...@mail.gmail.com you wrote: PROBLEM I am trying to solve off-site, on-disk backup problem in the Bay Area which is prone to earthquakes. QUESTION Is anyone out there using bittorrent as a backup tool? Hm... have you ever looked into DRBD instead? Bittorrent has not been designed for what you want to use it. DRBD has. I am aware that you can use many tools for things they have not specifically designed for, but I also believe that in most cases the tools that have been designed for a specific purpose are most efficient. Best regards, Wolfgang Denk -- DENX Software Engineering GmbH, MD: Wolfgang Denk Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: w...@denx.de There is a time in the tides of men, Which, taken at its flood, leads on to success. On the other hand, don't count on it. - T. K. Lawson -- The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Using Bittorrent as Backup Technology
Thanks to both of you for your thoughts. I am actually looking for anyone who has tried it or someone who has thought about trying it and dropped the idea. I understand 10 instances is very small - but I am not contemplating huge files either. Is the transfer latency something like filesize/number of nodes? Bacula can dump a daily differential and Bittorrent can take it from there. Databases will be more problematic. DRBD assumes, atleast that's what Google says, stable, decent bandwidth. My design takes network cutting out on a daily basis into account. We are talking the US postal service kind of a backup. It should work under adverse situations. I also want to be able to plug another unit in and have it start backing up without any twiddling. I have been there on the rsync front and it is reliable and simple. I just think if a cleaning person bumps the power cord and then plugs it back in (at night) that the machine should right itself. I know it can be done with scripts and rsync. Why invent and maintain something when p2p apps have solved the problem already? For some reason, I still think Bittorrent could be a winner in the set it and forget it backup arena. Technically, this is not really a backup, it is actually backing up a backup. A poor man's disaster planning. To be clear, here is the amount of work I am willing to do with this technology: a. Get the hardware and connection installed b. Configure encrypted transfers among a limited number of nodes c. Configure the master directories which needs to be backed up for 3x redundancy THAT'S IT - any more work is getting into fiddling. Mehma === On Sun, Feb 7, 2010 at 1:25 AM, Wolfgang Denk w...@denx.de wrote: Dear mehma sarja, In message ec5d34681002061636p308947abre20a0b3887454...@mail.gmail.com you wrote: PROBLEM I am trying to solve off-site, on-disk backup problem in the Bay Area which is prone to earthquakes. QUESTION Is anyone out there using bittorrent as a backup tool? Hm... have you ever looked into DRBD instead? Bittorrent has not been designed for what you want to use it. DRBD has. I am aware that you can use many tools for things they have not specifically designed for, but I also believe that in most cases the tools that have been designed for a specific purpose are most efficient. Best regards, Wolfgang Denk -- DENX Software Engineering GmbH, MD: Wolfgang Denk Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: w...@denx.de There is a time in the tides of men, Which, taken at its flood, leads on to success. On the other hand, don't count on it. - T. K. Lawson -- The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Using Bittorrent as Backup Technology
My question may or may not be appropriate for this forum, so sorry beforehand if I am intruding. PROBLEM I am trying to solve off-site, on-disk backup problem in the Bay Area which is prone to earthquakes. QUESTION Is anyone out there using bittorrent as a backup tool? APPLICATION D-Link D323 2-bay NAS boxes (with bittorrent clients) run around USD 150 at the local store w/o drives. Plop in a couple of USD 100-1.5 TB's w/o any RAID configured. That's 3 TB or TiB (I never understood the difference. I have been to sites and read the words, but they flow way over my head). So, if a company has a dozen offices spread over a 100-mile radius territory, and we put, let's say 10 of these boxes - that is 30 TiB of raw space. Now, let's introduce redundancy.. let's say triple, so, 30/3 = 10 TiB for USD 3.5K. We can explore how much risk reduction has taken place with this redundancy...but that's a point for another time. BENEFITS - fast backup, cheap SATA drives, reliable system - data is very safe in case of earthquakes, fires, power outages, theft, communication outages and easy to maintain - one box goes down, plug another one in and point it to the mother ship. Another benefit is by spreading around the backups, you can locate the systems within a high-risk area (of fire, earthquake, flooding, tornadoes, etc) and make it reasonably reliable and easily accessible (easy driving distance to go pick up the hard drive) WHY POST THIS? I am posting this message for two reasons: a) Bacula and other enterprise backup tools do not particularly like unreliable bandwidth connections and a bittorrent-like technology fills the gap. The gap is that Bacula and other tools are making disk backups convenient. As people move away from tape, the disk-based systems are increasingly at risk from natural disasters and wear and tear over time. Thus increasing risk as compared to tape backups. Although they too are susceptible to wear and tear. Do you consider some sort of off-site as a natural cost of doing on-disk backups? b) If someone is doing it or thinking about using bittorrent technologies - I'd like to know how of your experiences (and config files) and what hardware you use. Mehma -- The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Using Bittorrent as Backup Technology
On 02/06/10 19:36, mehma sarja wrote: WHY POST THIS? I am posting this message for two reasons: a) Bacula and other enterprise backup tools do not particularly like unreliable bandwidth connections and a bittorrent-like technology fills the gap. The gap is that Bacula and other tools are making disk backups convenient. As people move away from tape, the disk-based systems are increasingly at risk from natural disasters and wear and tear over time. Thus increasing risk as compared to tape backups. Although they too are susceptible to wear and tear. Do you consider some sort of off-site as a natural cost of doing on-disk backups? b) If someone is doing it or thinking about using bittorrent technologies - I'd like to know how of your experiences (and config files) and what hardware you use. My first thoughts on this: (1) With a comparatively small number of hosts and clients, all with reasonably fast business-grade network connections, there are no advantages to using BitTorrent. BitTorrent is fast because it distributes the server load across a large number of peers to serve a large number of clients, which themselves re-seed to pick up the server load. You do not have that situation. BitTorrent is a good technology for software distribution, but it's not really applicable to the problem of network backup. (2) The problem of unreliable network connections can be trivially solved using rsync. Server A rsyncs changes in its data set every hour, say, to mirrors at sites B and C, which sites B and C then back up every night. Likewise, site B rsyncs *its* server data pool to mirrors at sites A and C, which they then back up. And this is a worst case assuming sires A, B and C have different data sets. If the three sites are hosting the same data, then they simply maintain nightly local backups of their own data, and rsync the changes between themselves as needed to keep themselves in sync. For that matter, if all three sites A, B and C have the same data, you need not even necessarily bother backing them up at all; you could have a master repository at a hardened facility somewhere geographically remote from all three, which has a master copy of the data and simply pushes all changes out to the slave sites. In short, your question isn't really a backup-software question so much as it is a distributed site architecture and data consistency question. -- Phil Stracchino, CDK#2 DoD#299792458 ICBM: 43.5607, -71.355 ala...@caerllewys.net ala...@metrocast.net p...@co.ordinate.org Renaissance Man, Unix ronin, Perl hacker, Free Stater It's not the years, it's the mileage. -- The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users