Re: [BackupPC-users] Backuppc mirroring with rdiff-backup or not?
On Tue, 18 Nov 2008, Adam Goryachev wrote: Christian Völker wrote: My way to backup the backup ;-) is LVM The pool is on a LVM as a LV. To backup the pool while backuppc is running I can take a snapshot of the pool's LV and I rsync this one. So there are no filesystem issues and backuppc can stay running while the rsync is running. Works like a champ! Except your filesystem is not in a stable state. You should stop backuppc, umount your filesystem, then take your snapshot, and then remount filesystem, start backuppc, and finally copy the snapshot remotely as needed With your existing method, you should fsck your filesystem after it is copied to the remote location BTW, 99% of the time it won't matter, but if it screws up that one time when you need it, then oops :( Actually, lvm hooks into most filesystems including ext3 to make sure they are self consistent at the point of snapshot. See http://www.google.com/[EMAIL PROTECTED] Note that a 'fsck -p' is needed on the snapshot, but it will only encounter orphaned inodes if all has gone well (and if all hasn't gone well, then there are bigger problems with the real filesystem, and it should be rebooted and force fscked as soon as possible, as per the thread above). -- Tim Connors - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Backuppc mirroring with rdiff-backup or not?
On Mon, 17 Nov 2008, [EMAIL PROTECTED] wrote: > My biggest worry regarding these outside-of-BackupPC hacks is that when I > need them, I'm going to find that they're not going to work because it was > running, say, simultaneous to an actual backup. > > Don't get me wrong: I'll take the hacks. It's better than nothing. I, > like I think *most* of us, would kill (or even pay for!) a method of > replicating a pool in a guaranteed-correct way, especially at the host or > even backup level. But I still worry about using these hacks for > production. What if you did an lvm snapshot of the pool before while you knew no backups/nightlies etc were running, then rsynced/dded the lvm snapshot? Speaking of which - I'm still, several months later, writing a script that backuppc can use to sync to lvm snapshots. If only I wasn't so busy screwing around doing other things. Has anyone done similar? -- Tim Connors - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Backuppc mirroring with rdiff-backup or not?
Hello, the commentary from Rob Steele bellow the article in http://jeremy.zawodny.com/blog/archives/010037.html seems quite relevant to me. I have tried rsync and cp in the past but had the already mentioned memory and time issues so I'm planning to give his suggestion (BackupPC_tarPCCopy to make the first copy) a try. Cheers, -- Fernando Laudares Camargos Révolution Linux http://www.revolutionlinux.com --- * Tout opinion et prise de position exprimée dans ce message est celle de son auteur et pas nécessairement celle de Révolution Linux. ** Any views and opinion presented in this e-mail are solely those of the author and do not necessarily represent those of Révolution Linux. Ermanno Novali wrote: > Hi everyone, > i'm a backuppc user and i use it on several different linux servers, > with backuppc backupping in some cases to internal hdd, raid or external hdds. > > I'd like to mirror the backuppc pool - I searched through ml archives > and found that mirroring the backuppc pool (wherever it is) with rsync > on an external hard drive isn't efficient and doesn't scale good - > i've tried myself and is cpu and time consuming and very very long for > big pools - not very reliable. > > So i've tried to mirror the pool with rdiff-backup, and it seems a > little better, but not the optimal solution. > > In this ml the best solutions for this task are two hdd with pool on > them (two external, or two in raid maybe) or dd form pool to external > mirror disk - but NOT mirroring the backup with rsync or something > like that - right? can you confirm that? > > And dd is time consuming like rsync but more reliable for backuppc pool? > > Thank you so much, > have a nice day > > Ermanno > > - > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > ___ > BackupPC-users mailing list > BackupPC-users@lists.sourceforge.net > List:https://lists.sourceforge.net/lists/listinfo/backuppc-users > Wiki:http://backuppc.wiki.sourceforge.net > Project: http://backuppc.sourceforge.net/ - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Backuppc mirroring with rdiff-backup or not?
Christian Völker <[EMAIL PROTECTED]> wrote on 11/18/2008 03:34:45 AM: > > I think that you can use rsync v3 (on both sides) to sync pools without > > issue. > > I'm assuming that you are using linux here also. With *solaris you have > > the zfs option as well. > > I did some research a while back to use a cluster filesystem for the > > storage pool but all cluster filesystems have much lower I/O performance > > than an on-disk filesystem. > > My way to backup the backup ;-) is LVM > > The pool is on a LVM as a LV. To backup the pool while backuppc is > running I can take a snapshot of the pool's LV and I rsync this one. So > there are no filesystem issues and backuppc can stay running while the > rsync is running. I've been looking into this for a while now, both for BackupPC and for VMWare. How do you syncronize the snapshot with BackupPC to ensure that you are getting a consistent snapshot. For VMWare Server, that means shutting down the guests, taking the snapshot, and starting the guests, which is disruptive to say the least. (ESX Server has tools for doing this internal to the VM system.) Do you do the same for BackupPC? Or do you just hope that you're getting a consistent snapshot? Tim Massey - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Backuppc mirroring with rdiff-backup or not?
dan wrote: > Also, I did try to do some software raid mirroring over iscsi but did > not do much more that basic testing. The problem here is that the raid > mirroring is syncronous so the slow iscsi connection will effect backup > performance quite a bit. I couldnt find any info on making the linux > software raid work in async mode with the local drive being the priority > drive. > Would the --write-mostly option do this? (man mdadm) -Rob The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the addressee, any disclosure, reproduction, copying, distribution, or other dissemination or use of this transmission in error please notify the sender immediately and then delete this e-mail. E-mail transmission cannot be guaranteed to be secure or error free as information could be intercepted, corrupted lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. If verification is required please request a hard copy version. - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Backuppc mirroring with rdiff-backup or not?
Nils Breunese (Lemonbit) wrote: > [EMAIL PROTECTED] wrote: > >> Don't get me wrong: I'll take the hacks. It's better than >> nothing. I, like I think *most* of us, would kill (or even pay >> for!) a method of replicating a pool in a guaranteed-correct way, >> especially at the host or even backup level. But I still worry >> about using these hacks for production. > > You could also set up two BackupPC servers in two different locations. > No hacks needed. > That's what I do, and it works well. My "offsite" BackupPC server is actually an "onsite" BackupPC server at another office. And my office holds their offsite backups, so it works well. -Rob The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the addressee, any disclosure, reproduction, copying, distribution, or other dissemination or use of this transmission in error please notify the sender immediately and then delete this e-mail. E-mail transmission cannot be guaranteed to be secure or error free as information could be intercepted, corrupted lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. If verification is required please request a hard copy version. - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Backuppc mirroring with rdiff-backup or not?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Yohoo! >> My way to backup the backup ;-) is LVM >> The pool is on a LVM as a LV. To backup the pool while backuppc is >> running I can take a snapshot of the pool's LV and I rsync this one. So >> there are no filesystem issues and backuppc can stay running while the >> rsync is running. >> Works like a champ! > Except your filesystem is not in a stable state. You're partially right. AFAIK with journaling filesystem (like the ext3) this shouldn't be an issue. I'm not dumping the partition with dd, istead I mount the snapshot read-only. I'm using rsync- so it reads the filesystem. Files not been written totally should not be seen in the snapshot. Greetings Christian -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFJIpCQ0XNIYlAXmzsRAmYGAJwLpoxwu/xsl3OLo+vAG2MtklPEQwCfXeZj zCTUTPFhB7dQ+a+0Muv0xAU= =mWCf -END PGP SIGNATURE- - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Backuppc mirroring with rdiff-backup or not?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Christian Völker wrote: > My way to backup the backup ;-) is LVM > > The pool is on a LVM as a LV. To backup the pool while backuppc is > running I can take a snapshot of the pool's LV and I rsync this one. So > there are no filesystem issues and backuppc can stay running while the > rsync is running. > Works like a champ! Except your filesystem is not in a stable state. You should stop backuppc, umount your filesystem, then take your snapshot, and then remount filesystem, start backuppc, and finally copy the snapshot remotely as needed With your existing method, you should fsck your filesystem after it is copied to the remote location BTW, 99% of the time it won't matter, but if it screws up that one time when you need it, then oops :( Regards, Adam -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkkijhMACgkQGyoxogrTyiXQNQCfW7EIyuYErCfUnWYXwzIouq5k wg0AoLej/lIAth/L+lFNK8Zo7Led2iJR =SqM4 -END PGP SIGNATURE- - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Backuppc mirroring with rdiff-backup or not?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Yohoo! > I think that you can use rsync v3 (on both sides) to sync pools without > issue. > I'm assuming that you are using linux here also. With *solaris you have > the zfs option as well. > I did some research a while back to use a cluster filesystem for the > storage pool but all cluster filesystems have much lower I/O performance > than an on-disk filesystem. My way to backup the backup ;-) is LVM The pool is on a LVM as a LV. To backup the pool while backuppc is running I can take a snapshot of the pool's LV and I rsync this one. So there are no filesystem issues and backuppc can stay running while the rsync is running. As the pool is connected to the backuppc through iSCSI the backup runs on a different host and does not influence any running backuppc tasks. Works like a champ! Greetings Christian -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFJIn4l0XNIYlAXmzsRAmC1AKCbXoPjMYoM+w8TEQ7N8aGb/+t5rACgxx57 BF2UumTRS3pD7mqcxyf8wDk= =YU7g -END PGP SIGNATURE- - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Backuppc mirroring with rdiff-backup or not?
[EMAIL PROTECTED] wrote: > Don't get me wrong: I'll take the hacks. It's better than > nothing. I, like I think *most* of us, would kill (or even pay > for!) a method of replicating a pool in a guaranteed-correct way, > especially at the host or even backup level. But I still worry > about using these hacks for production. You could also set up two BackupPC servers in two different locations. No hacks needed. Nils Breunese. - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Backuppc mirroring with rdiff-backup or not?
dan <[EMAIL PROTECTED]> wrote on 11/17/2008 09:29:19 PM: > I use rsync v3 on a pool of about 280GB and about a few million > files. With rsync v3, the writes start within a few seconds of > starting the sync and it traverses the entire pool in 10-20 minutes. > I only transfer about 3-4GB of files each night with rsync reducing > that to about 1GB over a T1 at 196KB/s in about 1.5-2 hours. > Considering the transfer itself is going to take 90 minutes just by > bandwidth restrictions, this is not bad. How does this work while running this simultaneously with, say, a backup (or link or nightly or whatever)? Do you worry about that, or do you just try to make sure the two don't run simultaneously? My biggest worry regarding these outside-of-BackupPC hacks is that when I need them, I'm going to find that they're not going to work because it was running, say, simultaneous to an actual backup. Don't get me wrong: I'll take the hacks. It's better than nothing. I, like I think *most* of us, would kill (or even pay for!) a method of replicating a pool in a guaranteed-correct way, especially at the host or even backup level. But I still worry about using these hacks for production. Tim Massey - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Backuppc mirroring with rdiff-backup or not?
I use rsync v3 on a pool of about 280GB and about a few million files. With rsync v3, the writes start within a few seconds of starting the sync and it traverses the entire pool in 10-20 minutes. I only transfer about 3-4GB of files each night with rsync reducing that to about 1GB over a T1 at 196KB/s in about 1.5-2 hours. Considering the transfer itself is going to take 90 minutes just by bandwidth restrictions, this is not bad. I used to run this on rsync 2.x but it took at least 3 hours to complete as the initial file list would take ages and ages. I also began push memory off to swap to make room for rsync which ground system performance to a halt. I think that you can use rsync v3 (on both sides) to sync pools without issue. I'm assuming that you are using linux here also. With *solaris you have the zfs option as well. I did some research a while back to use a cluster filesystem for the storage pool but all cluster filesystems have much lower I/O performance than an on-disk filesystem. Also, I did try to do some software raid mirroring over iscsi but did not do much more that basic testing. The problem here is that the raid mirroring is syncronous so the slow iscsi connection will effect backup performance quite a bit. I couldnt find any info on making the linux software raid work in async mode with the local drive being the priority drive. Unfortunately ZFS only does 2 redundant device raid workalike and you would want more redundancy to make this work. If you could do raidz* with any number of redundant drives you could also put local disk cache and log drives in place and let zfs handle the slow link on iscsi. local | remote disk1 disk2 disk3 |disk4 disk5 disk6 disk7=log,disk8=cache | On Mon, Nov 17, 2008 at 4:26 PM, Ermanno Novali <[EMAIL PROTECTED]>wrote: > > Yes use dd (or even better dd-rescue that is restartable and gives > progress > > indication) for big pools. For smaller pools you might use "cp -a" or > "rsync > > -aH" (restartable). You have to find out the practical upper limit for > the > > latter methods depending on your requirements. > > Thanks for dd-rescue suggestion, i'll take a look at it > > > > Another alternative is to use at least three disks in a rotating scheme > and > > RAID1. (Those of you who have been reading the list for more than a few > days are > > getting tired of hearing this by now, I imagine...!) Say you have three > disks > > labeled 1, 2 and 3. Then you would rotate them according to the schedule > below, > > which guarantees that: > > - there is always at least one disk in the BackupPC server. > > - there is always at least one disk in the off-site storage. > > - all disks are never at the same location. > > > > 1 2 3 (a = attached, o = off-site) > > a o o > > a a o -> RAID sync > > o a o > > o a a -> RAID sync > > o o a > > a o a -> RAID sync > > . . . > > > > i'll try this too, where i have a raid or i need it > thanks! > > Ermanno > > - > This SF.Net email is sponsored by the Moblin Your Move Developer's > challenge > Build the coolest Linux based applications with Moblin SDK & win great > prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > ___ > BackupPC-users mailing list > BackupPC-users@lists.sourceforge.net > List:https://lists.sourceforge.net/lists/listinfo/backuppc-users > Wiki:http://backuppc.wiki.sourceforge.net > Project: http://backuppc.sourceforge.net/ > - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Backuppc mirroring with rdiff-backup or not?
> Yes use dd (or even better dd-rescue that is restartable and gives progress > indication) for big pools. For smaller pools you might use "cp -a" or "rsync > -aH" (restartable). You have to find out the practical upper limit for the > latter methods depending on your requirements. Thanks for dd-rescue suggestion, i'll take a look at it > Another alternative is to use at least three disks in a rotating scheme and > RAID1. (Those of you who have been reading the list for more than a few days > are > getting tired of hearing this by now, I imagine...!) Say you have three disks > labeled 1, 2 and 3. Then you would rotate them according to the schedule > below, > which guarantees that: > - there is always at least one disk in the BackupPC server. > - there is always at least one disk in the off-site storage. > - all disks are never at the same location. > > 1 2 3 (a = attached, o = off-site) > a o o > a a o -> RAID sync > o a o > o a a -> RAID sync > o o a > a o a -> RAID sync > . . . > i'll try this too, where i have a raid or i need it thanks! Ermanno - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Backuppc mirroring with rdiff-backup or not?
> Also, if your pool is only 10G of data, and your filesystem is 2TB, then > rsync or cp will work better. The above discussion mostly applies to > large pools. Although the definition of "large pools" is somewhat > murky, and it differs depending on your backuppc hardware, I would guess > something around 500G would be large... I've tested rsync and rdiff-backup (http://www.nongnu.org/rdiff-backup/) and rdiff-backup seems to be more responsive, and seems to begin to write before rsync... anyone here with experience with rdiff-backup? I think that (except being -maybe- less time consuming) it's like rsync (good & bads included) And thanks for your reply, Ermanno - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Backuppc mirroring with rdiff-backup or not?
Ermanno Novali wrote: > [...] > I'd like to mirror the backuppc pool - I searched through ml archives > and found that mirroring the backuppc pool (wherever it is) with rsync > on an external hard drive isn't efficient and doesn't scale good - > i've tried myself and is cpu and time consuming and very very long for > big pools - not very reliable. > > So i've tried to mirror the pool with rdiff-backup, and it seems a > little better, but not the optimal solution. > > In this ml the best solutions for this task are two hdd with pool on > them (two external, or two in raid maybe) or dd form pool to external > mirror disk - but NOT mirroring the backup with rsync or something > like that - right? can you confirm that? > > And dd is time consuming like rsync but more reliable for backuppc pool? > > Thank you so much, > have a nice day > > Ermanno Hi, Yes use dd (or even better dd-rescue that is restartable and gives progress indication) for big pools. For smaller pools you might use "cp -a" or "rsync -aH" (restartable). You have to find out the practical upper limit for the latter methods depending on your requirements. Another alternative is to use at least three disks in a rotating scheme and RAID1. (Those of you who have been reading the list for more than a few days are getting tired of hearing this by now, I imagine...!) Say you have three disks labeled 1, 2 and 3. Then you would rotate them according to the schedule below, which guarantees that: - there is always at least one disk in the BackupPC server. - there is always at least one disk in the off-site storage. - all disks are never at the same location. 1 2 3 (a = attached, o = off-site) a o o a a o -> RAID sync o a o o a a -> RAID sync o o a a o a -> RAID sync . . . On top of the RAID1 I recommend that you use LVM even though it might not be strictly necessary right now if your backups fit on one disk. The reason for doing that is when your disk are too small you can expand by using an second set of disks in a similar setup and add that second RAID1 to the volume. Good luck! /Martin - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Backuppc mirroring with rdiff-backup or not?
Ermanno Novali wrote: > Hi everyone, > > I'd like to mirror the backuppc pool - I searched through ml archives > and found that mirroring the backuppc pool (wherever it is) with rsync > on an external hard drive isn't efficient and doesn't scale good - > i've tried myself and is cpu and time consuming and very very long for > big pools - not very reliable. > In this ml the best solutions for this task are two hdd with pool on > them (two external, or two in raid maybe) or dd form pool to external > mirror disk - but NOT mirroring the backup with rsync or something > like that - right? can you confirm that? > Yes. RAID1 and dd are equally "good" except that to use dd you must stop backuppc and unmount the filesystem dor the duration of the dd (with RAID1 you only need to unmount the FS just before you break the RAID to remove the external drive). > And dd is time consuming like rsync but more reliable for backuppc pool? > Generally dd and RAID1 will copy the entire pool in the time it takes to read the entire disk and write the entire disk. The problem with cp and rsync etc is that they need to read the filesystem structure, and make a huge number of small reads and small writes. Especially rsync needs to read the entire list of filenames and store them in memory before even starting to copy the data to the destination. This is what makes rsync a poor choice. Of course, I've not discussed if or how the changes in rsync v3 modify the above discussion, AFAIK, it is meant to solve or improve the situation by starting to copy the content before reading the entire file list, and also being less memory intensive while copying the data. Also, if your pool is only 10G of data, and your filesystem is 2TB, then rsync or cp will work better. The above discussion mostly applies to large pools. Although the definition of "large pools" is somewhat murky, and it differs depending on your backuppc hardware, I would guess something around 500G would be large... Regards, Adam - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] Backuppc mirroring with rdiff-backup or not?
Hi everyone, i'm a backuppc user and i use it on several different linux servers, with backuppc backupping in some cases to internal hdd, raid or external hdds. I'd like to mirror the backuppc pool - I searched through ml archives and found that mirroring the backuppc pool (wherever it is) with rsync on an external hard drive isn't efficient and doesn't scale good - i've tried myself and is cpu and time consuming and very very long for big pools - not very reliable. So i've tried to mirror the pool with rdiff-backup, and it seems a little better, but not the optimal solution. In this ml the best solutions for this task are two hdd with pool on them (two external, or two in raid maybe) or dd form pool to external mirror disk - but NOT mirroring the backup with rsync or something like that - right? can you confirm that? And dd is time consuming like rsync but more reliable for backuppc pool? Thank you so much, have a nice day Ermanno - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
