I've been running rsync as a cygwin task on Windows Server 2008 for about two months now. I'm using the --link-dest option to do a daily 'snapshot' of the contents of a server containing about 10TB of data, about 13 million files, to a Linux based NAS server. Things started out great but I soon noticed that the time take to complete was slowly incrementing. It started at around three hours, but is now around fifteen.
The command is as follows... rsync -rlptDhPR \ --password-file=password \ --Chmod=Du=rwx,Dgo=rx,Fu=rw,Fgo=r \ --Stats \ --delete \ --log-file=logfilename \ --link-dest=linkdestpath \ sourceDirectory \ rsync@192.168.1.2::destinationDirectory I'm not using the full -a option as the differences between the Windows and Linux ownership stuff messed things up. The first log file looked like this... 2018/10/01 23:00:14 [2164] building file list ...transfer file list here 2018/10/02 02:11:30 [2164] Number of files: 13,759,998 (reg: 12,260,176, dir: 1,499,821, link: 1) 2018/10/02 02:11:30 [2164] Number of created files: 302 (reg: 291, dir: 11) 2018/10/02 02:11:30 [2164] Number of regular files transferred: 310 2018/10/02 02:11:30 [2164] Total file size: 10.40T bytes 2018/10/02 02:11:30 [2164] Total transferred file size: 664.31K bytes 2018/10/02 02:11:30 [2164] Literal data: 277.91K bytes 2018/10/02 02:11:30 [2164] Matched data: 386.40K bytes 2018/10/02 02:11:30 [2164] File list size: 10.42M 2018/10/02 02:11:30 [2164] File list generation time: 0.154 seconds 2018/10/02 02:11:30 [2164] File list transfer time: 0.000 seconds 2018/10/02 02:11:30 [2164] Total bytes sent: 235.68M 2018/10/02 02:11:30 [2164] Total bytes received: 7.51M 2018/10/02 02:11:30 [2164] sent 235.68M bytes received 7.51M bytes 21.17K bytes/sec 2018/10/02 02:11:30 [2164] total size is 10.40T speedup is 42,753.79 the most recent looks like this... 2018/11/24 23:00:15 [2924] building file list 2018/11/24 23:00:17 [2924] cd..t...... /cygdrive/ 2018/11/25 13:21:16 [2924] Number of files: 13,776,423 (reg: 12,274,642, dir: 1,501,780, link: 1) 2018/11/25 13:21:16 [2924] Number of created files: 0 2018/11/25 13:21:16 [2924] Number of regular files transferred: 0 2018/11/25 13:21:16 [2924] Total file size: 10.49T bytes 2018/11/25 13:21:16 [2924] Total transferred file size: 0 bytes 2018/11/25 13:21:16 [2924] Literal data: 0 bytes 2018/11/25 13:21:16 [2924] Matched data: 0 bytes 2018/11/25 13:21:16 [2924] File list size: 10.35M 2018/11/25 13:21:16 [2924] File list generation time: 0.316 seconds 2018/11/25 13:21:16 [2924] File list transfer time: 0.000 seconds 2018/11/25 13:21:16 [2924] Total bytes sent: 236.55M 2018/11/25 13:21:16 [2924] Total bytes received: 7.51M 2018/11/25 13:21:16 [2924] sent 236.55M bytes received 7.51M bytes 4.72K bytes/sec 2018/11/25 13:21:16 [2924] total size is 10.49T speedup is 42,996.96 As you can see the start time is 11:00PM (23:00) in both cases. The first log shows that identifying the files to transfer took about three hours (I've omitted the file list - it's quite long), the second log takes fourteen hours to do the same job (in this case this was done at the weekend and I've include the whole log file which correctly identifies that no files have changed) The number of files is as might be expected as is everything else. It's just the time taken "building file list" is significantly larger. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html