rsync wont use swap, it will only use RAM. This can be found on google or
that mailing lists. rsync developers say that it will take nothing short of
a rewrite to get rsync to use a on-disk log rather ram-log of the transfer
files so a qemu with a bunch of swap is not going to do you any good. you
need real RAM. 35M files x ~100Bytes per file for rsync is 3.26GB of RAM
needed. the initial scan from rsync will take ages and a restart will take
ages if the network link huckups. you really need to cut down on the amount
being transfered at any one time, so you basically need to transfer each of
the subdirectories of pc/ seperately. also, you cannot transfer the
cpool/pool directories effectively unless you can rsync the whole $TopDIR.
the hardlinks wont resolve out if you do the transfers seperately so you
will end up transfering the file, not just the hardlink. also, the files
in the pc/ directory wont be a hardlink to the identical copy in pool/cpool
so you will each up at LEAST twice the disk space.
best solution is to transfer each pc subdirectory seperately, then re-link
all the files essentially rebuilding the pool/cpool directories.
On Sat, Mar 29, 2008 at 5:32 AM, Tino Schwarze <[EMAIL PROTECTED]>
wrote:
> Hi James,
>
> On Fri, Mar 28, 2008 at 01:51:43PM -0500, James Harr wrote:
>
> > I need to move the archive for backuppc from one NAS to another. We're
> > dealing with around 800GB of data (35 million files) over a 100 Mbit
> > connection (for the time being, will be 1Gbit in the future), so the
> move
> > will take more than one day.
> >
> > - dump & restore is out of the question since each NAS is basically a
> sealed
> > box.
> > - I'd like to avoid just 'cp'ing it across since it will take a long
> time
> > and I'd like to maintain our backup schedule while we transfer.
> > - rsync -aH pukes because it's running out of address space (the 35
> million
> > files are hard to keep track of)
> > - rsync -a will work, but it doesn't do the hardlinks.
>
> I had a similar situation last year (you may want to look up my posts in
> the archive).
>
> IIRC I managed to copy the whole pool (about 500 GB) via the following
> steps:
> 1. rsync -a only the pool and cpool
> 2. run BackupPC_tarPCCopy for each pc, transferring it's output to the
> destination machine und unTARing there (I did that via netcat).
>
> Altogether this took over a week, but I suppose this is a performance
> issue with the RAID/LVM/FS combination I'm using.
>
> > The last part got me to thinking -- if I could manually run the backuppc
> > linker, I could rsync the backups, and just have the linker take care of
> the
> > rest. Does anyone know how to run the linker like this? If there isn't a
> > convenient way right now, could I bug one of the developers to add a
> --force
> > flag to it so it'd go through and scan all the files?
> >
> > I suppose one other option is to build a 64 bit virtual machine with
> qemu,
> > give it a large amount of swap, and then use that to do the rsync -aH :P
>
> We're talking about tens of GB of rsync memory here. So you'd probably
> wait months for that to complete. :-|
>
> HTH,
>
> Tino.
>
> --
> www.craniosacralzentrum.de
> www.spiritualdesign-chemnitz.de
> www.forteego.de
>
> Tino Schwarze * Lortzingstraße 21 * 09119 Chemnitz
>
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
>
> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
> _______________________________________________
> BackupPC-users mailing list
> [email protected]
> List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki: http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>
-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
_______________________________________________
BackupPC-users mailing list
[email protected]
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
