Hi, As you are on Cygwin, you should consider the notexec & noacl mount options : https://cygwin.com/cygwin-ug-net/using.html#mount-table <https://cygwin.com/cygwin-ug-net/using.html#mount-table>
They impact stat() performance. Ben > On 8 Jan 2019, at 10:56, John Simpson via rsync <rsync@lists.samba.org> wrote: > > Any ideas anyone? > > I still need at least a weekly backup of all data. > > The current workaround is just for the most active directories. > > Are there any diagnostics I can do which might shed some light on this? > > Thanks > > John > >> On 4 Jan 2019 09:53, John Simpson via rsync <rsync@lists.samba.org> wrote: >> >> Kevin >> >> The link-dest parameter is a single directory (the previous day's >> directory), the destination is today's directory. >> >> I haven't tried deleting a backup, there's no particular need in space >> terms, at the current rate there's enough space for several years of daily >> backups. >> >> I've reverted to daily backups on a small subset of the total; the full >> backup now takes around 30 hours. Clearly not practical. >> >> As the small subset takes only a few minutes to complete I can't yet see if >> this time is incrementing too. >> >> John On 3 Jan 2019 17:06, Kevin Korb via rsync <rsync@lists.samba.org> >> wrote: >>> >>> It does normally take some time to analyze large trees of files. It has >>> to call stat() on each file to get the size and timestamp. >>> >>> However, 15 hours seems a bit excessive even though I have never tried >>> to do this on Windows or a NAS system. Just to be clear, is your >>> --link-dest parameter a single directory or are you trying to tell it to >>> use all of the previous backups? >>> >>> Also, have you deleted a backup yet? In my experience that takes a lot >>> longer than running one so if you need 15 hours to run a backup I would >>> expect deleting one to take about a week. >>> >>> On 1/3/19 4:23 AM, John Simpson via rsync wrote: >>>> >>>> >>>> I've been running rsync as a cygwin task on Windows Server 2008 for about >>>> two months now. I'm using the --link-dest option to do a daily 'snapshot' >>>> of the contents of a server containing about 10TB of data, about 13 >>>> million files, to a Linux based NAS server. Things started out great but I >>>> soon noticed that the time take to complete was slowly incrementing. It >>>> started at around three hours, but is now around fifteen. >>>> >>>> The command is as follows... >>>> >>>> rsync -rlptDhPR \ >>>> --password-file=password \ >>>> --Chmod=Du=rwx,Dgo=rx,Fu=rw,Fgo=r \ >>>> --Stats \ >>>> --delete \ >>>> --log-file=logfilename \ >>>> --link-dest=linkdestpath \ >>>> sourceDirectory \ >>>> rsync@192.168.1.2::destinationDirectory >>>> >>>> I'm not using the full -a option as the differences between the Windows >>>> and Linux ownership stuff messed things up. >>>> >>>> The first log file looked like this... >>>> >>>> 2018/10/01 23:00:14 [2164] building file list >>>> ...transfer file list here >>>> 2018/10/02 02:11:30 [2164] Number of files: 13,759,998 (reg: 12,260,176, >>>> dir: 1,499,821, link: 1) >>>> 2018/10/02 02:11:30 [2164] Number of created files: 302 (reg: 291, dir: >>>> 11) >>>> 2018/10/02 02:11:30 [2164] Number of regular files transferred: 310 >>>> 2018/10/02 02:11:30 [2164] Total file size: 10.40T bytes >>>> 2018/10/02 02:11:30 [2164] Total transferred file size: 664.31K bytes >>>> 2018/10/02 02:11:30 [2164] Literal data: 277.91K bytes >>>> 2018/10/02 02:11:30 [2164] Matched data: 386.40K bytes >>>> 2018/10/02 02:11:30 [2164] File list size: 10.42M >>>> 2018/10/02 02:11:30 [2164] File list generation time: 0.154 seconds >>>> 2018/10/02 02:11:30 [2164] File list transfer time: 0.000 seconds >>>> 2018/10/02 02:11:30 [2164] Total bytes sent: 235.68M >>>> 2018/10/02 02:11:30 [2164] Total bytes received: 7.51M >>>> 2018/10/02 02:11:30 [2164] sent 235.68M bytes received 7.51M bytes >>>> 21.17K bytes/sec >>>> 2018/10/02 02:11:30 [2164] total size is 10.40T speedup is 42,753.79 >>>> >>>> the most recent looks like this... >>>> >>>> 2018/11/24 23:00:15 [2924] building file list >>>> 2018/11/24 23:00:17 [2924] cd..t...... /cygdrive/ >>>> 2018/11/25 13:21:16 [2924] Number of files: 13,776,423 (reg: 12,274,642, >>>> dir: 1,501,780, link: 1) >>>> 2018/11/25 13:21:16 [2924] Number of created files: 0 >>>> 2018/11/25 13:21:16 [2924] Number of regular files transferred: 0 >>>> 2018/11/25 13:21:16 [2924] Total file size: 10.49T bytes >>>> 2018/11/25 13:21:16 [2924] Total transferred file size: 0 bytes >>>> 2018/11/25 13:21:16 [2924] Literal data: 0 bytes >>>> 2018/11/25 13:21:16 [2924] Matched data: 0 bytes >>>> 2018/11/25 13:21:16 [2924] File list size: 10.35M >>>> 2018/11/25 13:21:16 [2924] File list generation time: 0.316 seconds >>>> 2018/11/25 13:21:16 [2924] File list transfer time: 0.000 seconds >>>> 2018/11/25 13:21:16 [2924] Total bytes sent: 236.55M >>>> 2018/11/25 13:21:16 [2924] Total bytes received: 7.51M >>>> 2018/11/25 13:21:16 [2924] sent 236.55M bytes received 7.51M bytes 4.72K >>>> bytes/sec >>>> 2018/11/25 13:21:16 [2924] total size is 10.49T speedup is 42,996.96 >>>> >>>> As you can see the start time is 11:00PM (23:00) in both cases. The first >>>> log shows that identifying the files to transfer took about three hours >>>> (I've omitted the file list - it's quite long), the second log takes >>>> fourteen hours to do the same job (in this case this was done at the >>>> weekend and I've include the whole log file which correctly identifies >>>> that no files have changed) >>>> >>>> The number of files is as might be expected as is everything else. It's >>>> just the time taken "building file list" is significantly larger.
-- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
