I create a new thread, because the issue is not really the same, but i copy here the thread that made me jump into the list.
My issue is not really that it waits before starting copying, but a general performance issue, specially when there are multiple rsync running at the same time. Here is my situation : I have multiple clients (around 20) with users and i want to rsync their home dirs with my server to keep a copy of their local files. On the server, files are hosted on a iSCSI volume (on a Thecus RAID) where i never had any performance issue before. When there is only one client, i have no real performance issues. In a few minutes, even with a very large number of files (some users have up to ), the sync is done if there are not too many changed files. But when there are 3 or more rsync at the same time, all rsync become very very slow and can take a few hours to complete. Here are my options : /usr/local/bin/rsync3 --rsync-path=/usr/local/bin/rsync3 -aHXxvE --stats --numeric-ids --delete-excluded --delete-before --human-readable —rsh="ssh -T -c aes128-ctr -o Compression=no -x" -z --skip-compress=gz/bz2/jpg/jpeg/ogg/mp3/mp4/mov/avi/vmdk/vmem --inplace --chmod=u+w --timeout=60 —exclude=‘Caches' —exclude=‘SyncService' —exclude=‘.FileSync' —exclude=‘IMAP*' —exclude=‘.Trash' —exclude='Saved Application State' —exclude='Autosave Information' --exclude-from=/Users/pabittan/.UserSync/exclude-list --max-size=1000M /Users/pabittan/ xserve.local.fftir:./ Here is the version i use (self compiled) : $ /usr/local/bin/rsync3 --version rsync version 3.1.2-jsp protocol version 31 Copyright (C) 1996-2015 by Andrew Tridgell, Wayne Davison, and others. Web site: http://rsync.samba.org/ Capabilities: 64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints, socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace, append, ACLs, xattrs, iconv, symtimes, no prealloc, file-flags I had to put in place a sort of queue to not allow more than 4 simultaneous rsync to be sure they run at least once a day each. Even limiting to 4 rsync makes some wait hours before starting a backup. I’m open to any help to improve perfs. (i have put my whole script calling rsync on github : https://github.com/jpiel/UserSync <https://github.com/jpiel/UserSync> ) PS: I checked, CPU is not under pressure, each rsync instance use between 2 and 5% CPU. The whole CPU usage 30%. I also checked network, and it’s not either an issue. Disk usage doesn’t seem to be at a high load either… (peak at 300 IO/sec) > Le 20 mars 2018 à 13:00, rsync-requ...@lists.samba.org a écrit : > > De: Kevin Korb <k...@sanitarium.net <mailto:k...@sanitarium.net>> > Objet: Rép : Very slow to start sync with millions of directories and files > Date: 19 mars 2018 à 15:33:31 UTC+1 > À: rsync@lists.samba.org <mailto:rsync@lists.samba.org> > > > The performance of rsync with a huge number of files is greatly > determined by every option you are using. So, what is your whole > command line? > > On 03/19/2018 09:05 AM, Bráulio Bhavamitra via rsync wrote: >> Hi all, >> >> I'm using rsync 3 to copy all files from one disk to another. The files >> were writen by Minio, an S3 compatible opensource backend. >> >> The number of files is dozens of millions, almost each of them within >> its own directory. >> >> Rsync takes a long time, when not several hours, to even start syncing >> files. I already see a few reasons: >> - it first create all directories to put files in, that could be done >> along with the sync >> - it needs to generate the list of all files before starting, and cannot >> start syncing and keep the list generation in a different thread. >> >> Cheers, >> bráulio >> >> > > -- > ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., > Kevin Korb Phone: (407) 252-6853 > Systems Administrator Internet: > FutureQuest, Inc. ke...@futurequest.net > <mailto:ke...@futurequest.net> (work) > Orlando, Florida k...@sanitarium.net > <mailto:k...@sanitarium.net> (personal) > Web page: http://www.sanitarium.net/ > <http://www.sanitarium.net/> > PGP public key available on web site. > ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Jayce Piel — jayce.p...@gmail.com -- 0616762431 Responsable Informatique F.F.Tir
-- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
