raf via rsync wrote: > Georg Limbach via rsync wrote: > > > Hello, > > > > I have a problem while rsyncing a directory with perhaps 1.000.000 > > files. Before this I generate some files locally and add them to temp > > files. So this files should be excluded from syncing. > > > > My rsync call looks like this: > > > > rsync --exclude='tmp/' \ > > --exclude-from=/tmp/tmp.GF7SsFPnS3 \ > > --exclude-from=/tmp/tmp.8SjJNCHyaI \ > > --exclude-from=/tmp/tmp.CxZXEoPjgV \ > > --exclude-from=/tmp/tmp.G3g2iMo4bs \ > > --exclude-from=/tmp/tmp.H9KJYPMfMS \ > > --exclude-from=/tmp/tmp.PNi7cJaREP \ > > --exclude-from=/tmp/tmp.S4N9H4lsU7 \ > > --exclude-from=/tmp/tmp.a5Zlgh6pUK \ > > --exclude-from=/tmp/tmp.eiUlMluAe8 \ > > --exclude-from=/tmp/tmp.ma0S1YSewc \ > > --exclude-from=/tmp/tmp.sLR95oVbVD \ > > --exclude-from=/tmp/tmp.zbfeLpezMX \ > > -ax --info=progress2 \ > > -e 'ssh -x -T -o Compression=no' \ > > '/srv/project/shared/uploads/' \ > > 'project@host:/srv/project/shared/uploads/' > > > > In every temp file are 1000s of lines with files like this: > > > > orders/scan/file/2234480/scan.pdf > > > > When I started rsync it only should transfer perhaps 1000 files and > > delete 20 ones. But it takes 15 minutes to do that. I think the big > > count of excluded files tear down the compare speed of files. > > > > What can I do to speed up this process? > > > > Thanks for your advise! > > > > Georg > > I found a big speed up when I starting providing a list of > candidate files to rsync with the --files-from option. > If you have some way of identifying which files might need > to be rsynced that's quicker than rsync itself checking > everything, it can make a big difference. That's not why > I did it, but it was a nice bonus.
But it might not help if doing --delete at the same time. Can the --delete be done occasionally instead of every time? Unless the files to be deleted can be added to the list of files handed to rsync. I don't know if that'll do what I think but it might. cheers, raf -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html