On Mon, Dec 10, 2001 at 09:18:33PM +0100, [EMAIL PROTECTED] wrote: > Dave Dykstra schrieb am Mon, Dec 10, 2001 at 01:44:26PM -0600: > * > I have to copy a structure of 70 directories where the data of these > * > directories are hardlinked to the data of the 1st directory. Within this > * > "orig data" directory, I have about 30.000 files, so the amount of files > * > to sync is approx. 2.100.000. The overall size is about 9.2GB. > * > * ... [runs out of memory] ... > * > * In case you weren't aware, rsync uses a lot less memory overall if you can > * split up the copies into smaller pieces, because it keeps some memory for > * every file it touches in a run. In your case, it may be especially > * difficult to break up the copy into smaller pieces because of all the > * hardlinks. > > I thought of that possibility and it won't work exactly of the reason you > described (the hardlink issue). > > > * > * Ideas: > * 1. Would it be possible to use symlinks instead of hardlinks? That > * would give you more flexibility to split things up however you > * like. > * 2. Perhaps you could break it up into ~70 copies, where each time you > * give it the first directory that contains the data and another > * one that contains one of the hardlinks. > > Both alternatives will eat up huge amounts of disk space as the numbers > above suggest. I will therefore consider plugging in more mem/swap before > trying them.
Why would symlinks eat up disk space? I only suggested the second alternative because I thought that it would end up with all the destination files hardlinked together as on the original system. I hadn't tested it, but now I did and it works: $ mkdir s s/d1 s/d2 s/d3 t $ touch s/d1/l1 $ ln s/d1/l1 s/d2/l1 $ ln s/d1/l1 s/d3/l1 $ ls -li s/*/l1 286226 -rw-rw-r-- 3 dwd dwd 0 Dec 10 14:20 s/d1/l1 286226 -rw-rw-r-- 3 dwd dwd 0 Dec 10 14:20 s/d2/l1 286226 -rw-rw-r-- 3 dwd dwd 0 Dec 10 14:20 s/d3/l1 $ rsync -aH s/d1 s/d2 t $ rsync -aH s/d1 s/d3 t $ ls -li t/*/l1 622728 -rw-rw-r-- 3 dwd dwd 0 Dec 10 14:20 t/d1/l1 622728 -rw-rw-r-- 3 dwd dwd 0 Dec 10 14:20 t/d2/l1 622728 -rw-rw-r-- 3 dwd dwd 0 Dec 10 14:20 t/d3/l1 > But I thank you for the hint anyway, - Dave Dykstra