i have tested the send and receive functionality of zfs on openbsd. the
problem is that it sends the entire fileset, block by block. this is not
going to work for remote replication within a reasonable timeframe.
rsync is about the only option. i know it does not scale well with file
count but it still does the best job of syncing up filesystems remotely.
the best solution here is to get with the rsync team and see what can be
done about the memory usage. maybe an option to write the file list to a
temp file or maybe compress the file list in memory.
I do remote rsyncs for a large fileset with millions of files. It does
work, reliably even, though i have 4GB of ram available on both computers.
I find that raid5 is quite evil for backuppc and/or rsync. though this is
already known on this list, maybe the degree of evil is not known. raid5
will more than double the write time of an rsync transfer. one might think
that this would be hidden behing the low bandwidth of a WAN connection but
it is not because these are not parrallel transactions, they are serial so
those I/O delays cause a waiting period before the next chunk is downloaded.
so, dont use raid5. do use raid1+0.
I have also tried some cluster filesystems recently and lets just put all of
those on the 'dont use' list.
I have done massive amounts of testing and trial and error and have found
that this is the best setup for my needs, and purhaps many people's needs
a raid set with raid1+0 running reiserfs on linux -or- zfs on bsd or solaris
with hardware raid. zfs is dramatically faster because it is much much
better at cacheing I/O than any filesystem I have tested. also, I find that
exporting a zfs snapshot via iscsi and mounting that on the remote machine
and doing the rsync *local* from the iscsi mount to the local backuppc share
is MUCH(maybe 20-30%) faster that just regular rsync as iscsi is much better
at handling I/O than rsync. use many spindles in raid0! pair the drives in
raid1 and get as many spindles as possible. raid0 will do a good job of
spreading I/O(raid5 does NOT). if using zfs, do NOT use compression, and do
not do a raidz or raidz2 as you will have a parity penalty though it is much
smaller than a regular raid5. if using bsd or solaris, use a hardware raid
card(a good one) and run zfs on top of that, again dont use zfs' raid
functionalty or you will get a parity penalty.
On Wed, Jul 23, 2008 at 12:25 PM, Les Mikesell <[EMAIL PROTECTED]>
wrote:
> Holger Parplies wrote:
> >
> > Chances are, you
> > should be aware of what will happen. But still: extensive, potentially
> > destructive operations happening automatically without a chance of user
> > interaction is something that belongs in the world of Windoze. Linux is
> > becoming more like that fast enough already, so I won't let this chance
> pass
> > to argument against it :-).
>
> But this is the backuppc list where you might not be running Linux... I
> still think zfs with it's incremental send/receive operation might be
> the way to go to duplicate changes in the archive filesystem, but
> haven't been able to do any testing myself yet.
>
> > Of course there's also the chance of you simply plugging the drive in at
> an
> > inappropriate moment, when you wouldn't want the sync to happen. You
> might
> > notice it when logging in to call the script to start the sync, but you
> won't
> > log in just to check, if you went to the trouble of automating the sync
> in the
> > first place, will you?
> >
> > Just because you *can* automate something, that doesn't necessarily mean
> it's
> > a good idea to do it. Then again, under some circumstances it may well
> be.
>
> Another alternative that seems like it should work is to plug the
> external drive into some other machine on a fast lan, export the
> partition via iscsi, then add the iscsi device to the raid set. Really
> the same scenario, but you wouldn't have to walk over to the server room
> or keep the server at your desk.
>
> --
> Les Mikesell
> [EMAIL PROTECTED]
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's
> challenge
> Build the coolest Linux based applications with Moblin SDK & win great
> prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki: http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
