Re: --link-dest. Time to 'building file list' incrementing

[John Simpson via rsync]

Don't know how I missed this when I was trying to figure out the cheapest 
solution. I don't need the complexity but it seems to be worth investigating. 
ThanksOn 8 Jan 2019 16:15, Andrew McGlashan via rsync  
wrote:
>
> On 8/1/19 8:56 pm, John Simpson via rsync wrote:
> > Any ideas anyone?
>
> How about using snapshots and doing the rsync off those?
>
> https://www.thewindowsclub.com/vss-volume-shadow-copy-service
>
> https://blogs.technet.microsoft.com/josebda/2007/10/10/the-basics-of-the-volume-shadow-copy-service-vss/
>
> Cheers
> A.
>
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: --link-dest. Time to 'building file list' incrementing

[Andrew McGlashan via rsync]

On 8/1/19 8:56 pm, John Simpson via rsync wrote:
> Any ideas anyone?

How about using snapshots and doing the rsync off those?

https://www.thewindowsclub.com/vss-volume-shadow-copy-service

https://blogs.technet.microsoft.com/josebda/2007/10/10/the-basics-of-the-volume-shadow-copy-service-vss/


Cheers
A.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: --link-dest. Time to 'building file list' incrementing

[John Simpson via rsync]

Thanks. I'll give this a go and report back.
On 8 Jan 2019 10:03, Ben RUBSON  wrote:Hi,As you are on Cygwin, you should consider the notexec & noacl mount options :https://cygwin.com/cygwin-ug-net/using.html#mount-tableThey impact stat() performance.BenOn 8 Jan 2019, at 10:56, John Simpson via rsync  wrote:Any ideas anyone?I still need at least a weekly backup of all data.The current workaround is just for the most active directories.Are there any diagnostics I can do which might shed some light on this?ThanksJohnOn 4 Jan 2019 09:53, John Simpson via rsync  wrote:KevinThe link-dest parameter is a single directory (the previous day's directory), the destination is today's directory.I haven't tried deleting a backup,  there's no particular need in space terms,  at the current rate there's enough space for several years of daily backups.I've reverted to daily backups on a small subset of the total; the full backup now takes around 30 hours.  Clearly not practical.As the small subset takes only a few minutes to complete I can't yet see if this time is incrementing too.John   On 3 Jan 2019 17:06, Kevin Korb via rsync  wrote:It does normally take some time to analyze large trees of files.  It has to call stat() on each file to get the size and timestamp. However, 15 hours seems a bit excessive even though I have never tried to do this on Windows or a NAS system.  Just to be clear, is your --link-dest parameter a single directory or are you trying to tell it to use all of the previous backups? Also, have you deleted a backup yet?  In my experience that takes a lot longer than running one so if you need 15 hours to run a backup I would expect deleting one to take about a week. On 1/3/19 4:23 AM, John Simpson via rsync wrote: I've been running rsync as a cygwin task on Windows Server 2008 for about two months now. I'm using the --link-dest option to do a daily 'snapshot' of the contents of a server containing about 10TB of data, about 13 million files, to a Linux based NAS server. Things started out great but I soon noticed that the time take to complete was slowly incrementing. It started at around three hours, but is now around fifteen. The command is as follows... rsync -rlptDhPR \  --password-file=password \  --Chmod=Du=rwx,Dgo=rx,Fu=rw,Fgo=r \  --Stats \  --delete \  --log-file=logfilename \  --link-dest=linkdestpath \  sourceDirectory \  rsync@192.168.1.2::destinationDirectory I'm not using the full -a option as the differences between the Windows and Linux ownership stuff messed things up. The first log file looked like this... 2018/10/01 23:00:14 [2164] building file list ...transfer file list here 2018/10/02 02:11:30 [2164] Number of files: 13,759,998 (reg: 12,260,176, dir: 1,499,821, link: 1) 2018/10/02 02:11:30 [2164] Number of created files: 302 (reg: 291, dir: 11) 2018/10/02 02:11:30 [2164] Number of regular files transferred: 310 2018/10/02 02:11:30 [2164] Total file size: 10.40T bytes 2018/10/02 02:11:30 [2164] Total transferred file size: 664.31K bytes 2018/10/02 02:11:30 [2164] Literal data: 277.91K bytes 2018/10/02 02:11:30 [2164] Matched data: 386.40K bytes 2018/10/02 02:11:30 [2164] File list size: 10.42M 2018/10/02 02:11:30 [2164] File list generation time: 0.154 seconds 2018/10/02 02:11:30 [2164] File list transfer time: 0.000 seconds 2018/10/02 02:11:30 [2164] Total bytes sent: 235.68M 2018/10/02 02:11:30 [2164] Total bytes received: 7.51M 2018/10/02 02:11:30 [2164] sent 235.68M bytes  received 7.51M bytes  21.17K bytes/sec 2018/10/02 02:11:30 [2164] total size is 10.40T  speedup is 42,753.79 the most recent looks like this... 2018/11/24 23:00:15 [2924] building file list 2018/11/24 23:00:17 [2924] cd..t.. /cygdrive/ 2018/11/25 13:21:16 [2924] Number of files: 13,776,423 (reg: 12,274,642, dir: 1,501,780, link: 1) 2018/11/25 13:21:16 [2924] Number of created files: 0 2018/11/25 13:21:16 [2924] Number of regular files transferred: 0 2018/11/25 13:21:16 [2924] Total file size: 10.49T bytes 2018/11/25 13:21:16 [2924] Total transferred file size: 0 bytes 2018/11/25 13:21:16 [2924] Literal data: 0 bytes 2018/11/25 13:21:16 [2924] Matched data: 0 bytes 2018/11/25 13:21:16 [2924] File list size: 10.35M 2018/11/25 13:21:16 [2924] File list generation time: 0.316 seconds 2018/11/25 13:21:16 [2924] File list transfer time: 0.000 seconds 2018/11/25 13:21:16 [2924] Total bytes sent: 236.55M 2018/11/25 13:21:16 [2924] Total bytes received: 7.51M 2018/11/25 13:21:16 [2924] sent 236.55M bytes  received 7.51M bytes  4.72K bytes/sec 2018/11/25 13:21:16 [2924] total size is 10.49T  speedup is 42,996.96 As you can see the start time is 11:00PM (23:00) in both cases. The first log shows that identifying the files to transfer took about three hours (I've omitted the file list - it's quite long), the second log takes fourteen hours to do the same job (in this case this was done at the weekend and I've include 

Re: --link-dest. Time to 'building file list' incrementing

[Ben RUBSON via rsync]

Hi,

As you are on Cygwin, you should consider the notexec & noacl mount options :
https://cygwin.com/cygwin-ug-net/using.html#mount-table 


They impact stat() performance.

Ben

> On 8 Jan 2019, at 10:56, John Simpson via rsync  wrote:
> 
> Any ideas anyone?
> 
> I still need at least a weekly backup of all data.
> 
> The current workaround is just for the most active directories.
> 
> Are there any diagnostics I can do which might shed some light on this?
> 
> Thanks
> 
> John
> 
>> On 4 Jan 2019 09:53, John Simpson via rsync  wrote:
>> 
>> Kevin
>> 
>> The link-dest parameter is a single directory (the previous day's 
>> directory), the destination is today's directory.
>> 
>> I haven't tried deleting a backup,  there's no particular need in space 
>> terms,  at the current rate there's enough space for several years of daily 
>> backups.
>> 
>> I've reverted to daily backups on a small subset of the total; the full 
>> backup now takes around 30 hours.  Clearly not practical.
>> 
>> As the small subset takes only a few minutes to complete I can't yet see if 
>> this time is incrementing too.
>> 
>> John   On 3 Jan 2019 17:06, Kevin Korb via rsync  
>> wrote:
>>> 
>>> It does normally take some time to analyze large trees of files.  It has 
>>> to call stat() on each file to get the size and timestamp. 
>>> 
>>> However, 15 hours seems a bit excessive even though I have never tried 
>>> to do this on Windows or a NAS system.  Just to be clear, is your 
>>> --link-dest parameter a single directory or are you trying to tell it to 
>>> use all of the previous backups? 
>>> 
>>> Also, have you deleted a backup yet?  In my experience that takes a lot 
>>> longer than running one so if you need 15 hours to run a backup I would 
>>> expect deleting one to take about a week. 
>>> 
>>> On 1/3/19 4:23 AM, John Simpson via rsync wrote: 
 
 
 I've been running rsync as a cygwin task on Windows Server 2008 for about 
 two months now. I'm using the --link-dest option to do a daily 'snapshot' 
 of the contents of a server containing about 10TB of data, about 13 
 million files, to a Linux based NAS server. Things started out great but I 
 soon noticed that the time take to complete was slowly incrementing. It 
 started at around three hours, but is now around fifteen. 
 
 The command is as follows... 
 
 rsync -rlptDhPR \ 
  --password-file=password \ 
  --Chmod=Du=rwx,Dgo=rx,Fu=rw,Fgo=r \ 
  --Stats \ 
  --delete \ 
  --log-file=logfilename \ 
  --link-dest=linkdestpath \ 
  sourceDirectory \ 
  rsync@192.168.1.2::destinationDirectory 
 
 I'm not using the full -a option as the differences between the Windows 
 and Linux ownership stuff messed things up. 
 
 The first log file looked like this... 
 
 2018/10/01 23:00:14 [2164] building file list 
 ...transfer file list here 
 2018/10/02 02:11:30 [2164] Number of files: 13,759,998 (reg: 12,260,176, 
 dir: 1,499,821, link: 1) 
 2018/10/02 02:11:30 [2164] Number of created files: 302 (reg: 291, dir: 
 11) 
 2018/10/02 02:11:30 [2164] Number of regular files transferred: 310 
 2018/10/02 02:11:30 [2164] Total file size: 10.40T bytes 
 2018/10/02 02:11:30 [2164] Total transferred file size: 664.31K bytes 
 2018/10/02 02:11:30 [2164] Literal data: 277.91K bytes 
 2018/10/02 02:11:30 [2164] Matched data: 386.40K bytes 
 2018/10/02 02:11:30 [2164] File list size: 10.42M 
 2018/10/02 02:11:30 [2164] File list generation time: 0.154 seconds 
 2018/10/02 02:11:30 [2164] File list transfer time: 0.000 seconds 
 2018/10/02 02:11:30 [2164] Total bytes sent: 235.68M 
 2018/10/02 02:11:30 [2164] Total bytes received: 7.51M 
 2018/10/02 02:11:30 [2164] sent 235.68M bytes  received 7.51M bytes  
 21.17K bytes/sec 
 2018/10/02 02:11:30 [2164] total size is 10.40T  speedup is 42,753.79 
 
 the most recent looks like this... 
 
 2018/11/24 23:00:15 [2924] building file list 
 2018/11/24 23:00:17 [2924] cd..t.. /cygdrive/ 
 2018/11/25 13:21:16 [2924] Number of files: 13,776,423 (reg: 12,274,642, 
 dir: 1,501,780, link: 1) 
 2018/11/25 13:21:16 [2924] Number of created files: 0 
 2018/11/25 13:21:16 [2924] Number of regular files transferred: 0 
 2018/11/25 13:21:16 [2924] Total file size: 10.49T bytes 
 2018/11/25 13:21:16 [2924] Total transferred file size: 0 bytes 
 2018/11/25 13:21:16 [2924] Literal data: 0 bytes 
 2018/11/25 13:21:16 [2924] Matched data: 0 bytes 
 2018/11/25 13:21:16 [2924] File list size: 10.35M 
 2018/11/25 13:21:16 [2924] File list generation time: 0.316 seconds 
 2018/11/25 13:21:16 [2924] File list transfer time: 0.000 seconds 
 2018/11/25 13:21:16 [2924] Total bytes sent: 236.55M 
 2018/11/25 13:21:16 [2924] Total bytes 

Re: --link-dest. Time to 'building file list' incrementing

[John Simpson via rsync]

Any ideas anyone?

I still need at least a weekly backup of all data.

The current workaround is just for the most active directories.

Are there any diagnostics I can do which might shed some light on this?

Thanks

JohnOn 4 Jan 2019 09:53, John Simpson via rsync  wrote:
>
> Kevin
>
> The link-dest parameter is a single directory (the previous day's directory), 
> the destination is today's directory.
>
> I haven't tried deleting a backup,  there's no particular need in space 
> terms,  at the current rate there's enough space for several years of daily 
> backups.
>
> I've reverted to daily backups on a small subset of the total; the full 
> backup now takes around 30 hours.  Clearly not practical.
>
> As the small subset takes only a few minutes to complete I can't yet see if 
> this time is incrementing too.
>
> John   On 3 Jan 2019 17:06, Kevin Korb via rsync  
> wrote:
> >
> > It does normally take some time to analyze large trees of files.  It has 
> > to call stat() on each file to get the size and timestamp. 
> >
> > However, 15 hours seems a bit excessive even though I have never tried 
> > to do this on Windows or a NAS system.  Just to be clear, is your 
> > --link-dest parameter a single directory or are you trying to tell it to 
> > use all of the previous backups? 
> >
> > Also, have you deleted a backup yet?  In my experience that takes a lot 
> > longer than running one so if you need 15 hours to run a backup I would 
> > expect deleting one to take about a week. 
> >
> > On 1/3/19 4:23 AM, John Simpson via rsync wrote: 
> > > 
> > > 
> > > I've been running rsync as a cygwin task on Windows Server 2008 for about 
> > > two months now. I'm using the --link-dest option to do a daily 'snapshot' 
> > > of the contents of a server containing about 10TB of data, about 13 
> > > million files, to a Linux based NAS server. Things started out great but 
> > > I soon noticed that the time take to complete was slowly incrementing. It 
> > > started at around three hours, but is now around fifteen. 
> > > 
> > > The command is as follows... 
> > > 
> > > rsync -rlptDhPR \ 
> > > --password-file=password \ 
> > > --Chmod=Du=rwx,Dgo=rx,Fu=rw,Fgo=r \ 
> > > --Stats \ 
> > > --delete \ 
> > > --log-file=logfilename \ 
> > > --link-dest=linkdestpath \ 
> > > sourceDirectory \ 
> > > rsync@192.168.1.2::destinationDirectory 
> > > 
> > > I'm not using the full -a option as the differences between the Windows 
> > > and Linux ownership stuff messed things up. 
> > > 
> > > The first log file looked like this... 
> > > 
> > > 2018/10/01 23:00:14 [2164] building file list 
> > > ...transfer file list here 
> > > 2018/10/02 02:11:30 [2164] Number of files: 13,759,998 (reg: 12,260,176, 
> > > dir: 1,499,821, link: 1) 
> > > 2018/10/02 02:11:30 [2164] Number of created files: 302 (reg: 291, dir: 
> > > 11) 
> > > 2018/10/02 02:11:30 [2164] Number of regular files transferred: 310 
> > > 2018/10/02 02:11:30 [2164] Total file size: 10.40T bytes 
> > > 2018/10/02 02:11:30 [2164] Total transferred file size: 664.31K bytes 
> > > 2018/10/02 02:11:30 [2164] Literal data: 277.91K bytes 
> > > 2018/10/02 02:11:30 [2164] Matched data: 386.40K bytes 
> > > 2018/10/02 02:11:30 [2164] File list size: 10.42M 
> > > 2018/10/02 02:11:30 [2164] File list generation time: 0.154 seconds 
> > > 2018/10/02 02:11:30 [2164] File list transfer time: 0.000 seconds 
> > > 2018/10/02 02:11:30 [2164] Total bytes sent: 235.68M 
> > > 2018/10/02 02:11:30 [2164] Total bytes received: 7.51M 
> > > 2018/10/02 02:11:30 [2164] sent 235.68M bytes  received 7.51M bytes  
> > > 21.17K bytes/sec 
> > > 2018/10/02 02:11:30 [2164] total size is 10.40T  speedup is 42,753.79 
> > > 
> > > the most recent looks like this... 
> > > 
> > > 2018/11/24 23:00:15 [2924] building file list 
> > > 2018/11/24 23:00:17 [2924] cd..t.. /cygdrive/ 
> > > 2018/11/25 13:21:16 [2924] Number of files: 13,776,423 (reg: 12,274,642, 
> > > dir: 1,501,780, link: 1) 
> > > 2018/11/25 13:21:16 [2924] Number of created files: 0 
> > > 2018/11/25 13:21:16 [2924] Number of regular files transferred: 0 
> > > 2018/11/25 13:21:16 [2924] Total file size: 10.49T bytes 
> > > 2018/11/25 13:21:16 [2924] Total transferred file size: 0 bytes 
> > > 2018/11/25 13:21:16 [2924] Literal data: 0 bytes 
> > > 2018/11/25 13:21:16 [2924] Matched data: 0 bytes 
> > > 2018/11/25 13:21:16 [2924] File list size: 10.35M 
> > > 2018/11/25 13:21:16 [2924] File list generation time: 0.316 seconds 
> > > 2018/11/25 13:21:16 [2924] File list transfer time: 0.000 seconds 
> > > 2018/11/25 13:21:16 [2924] Total bytes sent: 236.55M 
> > > 2018/11/25 13:21:16 [2924] Total bytes received: 7.51M 
> > > 2018/11/25 13:21:16 [2924] sent 236.55M bytes  received 7.51M bytes  
> > > 4.72K bytes/sec 
> > > 2018/11/25 13:21:16 [2924] total size is 10.49T  speedup is 42,996.96 
> > > 
> > > As you can see the start time is 11:00PM (23:00) in both cases. The first 
> > > log shows that 

Re: --link-dest. Time to 'building file list' incrementing

[John Simpson via rsync]

Kevin

The link-dest parameter is a single directory (the previous day's directory), 
the destination is today's directory.

I haven't tried deleting a backup,  there's no particular need in space terms,  
at the current rate there's enough space for several years of daily backups.

I've reverted to daily backups on a small subset of the total; the full backup 
now takes around 30 hours.  Clearly not practical.

As the small subset takes only a few minutes to complete I can't yet see if 
this time is incrementing too.

John   On 3 Jan 2019 17:06, Kevin Korb via rsync  wrote:
>
> It does normally take some time to analyze large trees of files.  It has 
> to call stat() on each file to get the size and timestamp. 
>
> However, 15 hours seems a bit excessive even though I have never tried 
> to do this on Windows or a NAS system.  Just to be clear, is your 
> --link-dest parameter a single directory or are you trying to tell it to 
> use all of the previous backups? 
>
> Also, have you deleted a backup yet?  In my experience that takes a lot 
> longer than running one so if you need 15 hours to run a backup I would 
> expect deleting one to take about a week. 
>
> On 1/3/19 4:23 AM, John Simpson via rsync wrote: 
> > 
> > 
> > I've been running rsync as a cygwin task on Windows Server 2008 for about 
> > two months now. I'm using the --link-dest option to do a daily 'snapshot' 
> > of the contents of a server containing about 10TB of data, about 13 million 
> > files, to a Linux based NAS server. Things started out great but I soon 
> > noticed that the time take to complete was slowly incrementing. It started 
> > at around three hours, but is now around fifteen. 
> > 
> > The command is as follows... 
> > 
> > rsync -rlptDhPR \ 
> > --password-file=password \ 
> > --Chmod=Du=rwx,Dgo=rx,Fu=rw,Fgo=r \ 
> > --Stats \ 
> > --delete \ 
> > --log-file=logfilename \ 
> > --link-dest=linkdestpath \ 
> > sourceDirectory \ 
> > rsync@192.168.1.2::destinationDirectory 
> > 
> > I'm not using the full -a option as the differences between the Windows and 
> > Linux ownership stuff messed things up. 
> > 
> > The first log file looked like this... 
> > 
> > 2018/10/01 23:00:14 [2164] building file list 
> > ...transfer file list here 
> > 2018/10/02 02:11:30 [2164] Number of files: 13,759,998 (reg: 12,260,176, 
> > dir: 1,499,821, link: 1) 
> > 2018/10/02 02:11:30 [2164] Number of created files: 302 (reg: 291, dir: 11) 
> > 2018/10/02 02:11:30 [2164] Number of regular files transferred: 310 
> > 2018/10/02 02:11:30 [2164] Total file size: 10.40T bytes 
> > 2018/10/02 02:11:30 [2164] Total transferred file size: 664.31K bytes 
> > 2018/10/02 02:11:30 [2164] Literal data: 277.91K bytes 
> > 2018/10/02 02:11:30 [2164] Matched data: 386.40K bytes 
> > 2018/10/02 02:11:30 [2164] File list size: 10.42M 
> > 2018/10/02 02:11:30 [2164] File list generation time: 0.154 seconds 
> > 2018/10/02 02:11:30 [2164] File list transfer time: 0.000 seconds 
> > 2018/10/02 02:11:30 [2164] Total bytes sent: 235.68M 
> > 2018/10/02 02:11:30 [2164] Total bytes received: 7.51M 
> > 2018/10/02 02:11:30 [2164] sent 235.68M bytes  received 7.51M bytes  21.17K 
> > bytes/sec 
> > 2018/10/02 02:11:30 [2164] total size is 10.40T  speedup is 42,753.79 
> > 
> > the most recent looks like this... 
> > 
> > 2018/11/24 23:00:15 [2924] building file list 
> > 2018/11/24 23:00:17 [2924] cd..t.. /cygdrive/ 
> > 2018/11/25 13:21:16 [2924] Number of files: 13,776,423 (reg: 12,274,642, 
> > dir: 1,501,780, link: 1) 
> > 2018/11/25 13:21:16 [2924] Number of created files: 0 
> > 2018/11/25 13:21:16 [2924] Number of regular files transferred: 0 
> > 2018/11/25 13:21:16 [2924] Total file size: 10.49T bytes 
> > 2018/11/25 13:21:16 [2924] Total transferred file size: 0 bytes 
> > 2018/11/25 13:21:16 [2924] Literal data: 0 bytes 
> > 2018/11/25 13:21:16 [2924] Matched data: 0 bytes 
> > 2018/11/25 13:21:16 [2924] File list size: 10.35M 
> > 2018/11/25 13:21:16 [2924] File list generation time: 0.316 seconds 
> > 2018/11/25 13:21:16 [2924] File list transfer time: 0.000 seconds 
> > 2018/11/25 13:21:16 [2924] Total bytes sent: 236.55M 
> > 2018/11/25 13:21:16 [2924] Total bytes received: 7.51M 
> > 2018/11/25 13:21:16 [2924] sent 236.55M bytes  received 7.51M bytes  4.72K 
> > bytes/sec 
> > 2018/11/25 13:21:16 [2924] total size is 10.49T  speedup is 42,996.96 
> > 
> > As you can see the start time is 11:00PM (23:00) in both cases. The first 
> > log shows that identifying the files to transfer took about three hours 
> > (I've omitted the file list - it's quite long), the second log takes 
> > fourteen hours to do the same job (in this case this was done at the 
> > weekend and I've include the whole log file which correctly identifies that 
> > no files have changed) 
> > 
> > The number of files is as might be expected as is everything else. It's 
> > just the time taken "building file list" is significantly larger. 
> > 
> > 

Re: --link-dest. Time to 'building file list' incrementing

[Kevin Korb via rsync]

It does normally take some time to analyze large trees of files.  It has
to call stat() on each file to get the size and timestamp.

However, 15 hours seems a bit excessive even though I have never tried
to do this on Windows or a NAS system.  Just to be clear, is your
--link-dest parameter a single directory or are you trying to tell it to
use all of the previous backups?

Also, have you deleted a backup yet?  In my experience that takes a lot
longer than running one so if you need 15 hours to run a backup I would
expect deleting one to take about a week.

On 1/3/19 4:23 AM, John Simpson via rsync wrote:
> 
> 
> I've been running rsync as a cygwin task on Windows Server 2008 for about two 
> months now. I'm using the --link-dest option to do a daily 'snapshot' of the 
> contents of a server containing about 10TB of data, about 13 million files, 
> to a Linux based NAS server. Things started out great but I soon noticed that 
> the time take to complete was slowly incrementing. It started at around three 
> hours, but is now around fifteen.
> 
> The command is as follows...
> 
> rsync -rlptDhPR \
> --password-file=password \
> --Chmod=Du=rwx,Dgo=rx,Fu=rw,Fgo=r \
> --Stats \
> --delete \
> --log-file=logfilename \
> --link-dest=linkdestpath \
> sourceDirectory \
> rsync@192.168.1.2::destinationDirectory
> 
> I'm not using the full -a option as the differences between the Windows and 
> Linux ownership stuff messed things up.
> 
> The first log file looked like this...
> 
> 2018/10/01 23:00:14 [2164] building file list
> ...transfer file list here
> 2018/10/02 02:11:30 [2164] Number of files: 13,759,998 (reg: 12,260,176, dir: 
> 1,499,821, link: 1)
> 2018/10/02 02:11:30 [2164] Number of created files: 302 (reg: 291, dir: 11)
> 2018/10/02 02:11:30 [2164] Number of regular files transferred: 310
> 2018/10/02 02:11:30 [2164] Total file size: 10.40T bytes
> 2018/10/02 02:11:30 [2164] Total transferred file size: 664.31K bytes
> 2018/10/02 02:11:30 [2164] Literal data: 277.91K bytes
> 2018/10/02 02:11:30 [2164] Matched data: 386.40K bytes
> 2018/10/02 02:11:30 [2164] File list size: 10.42M
> 2018/10/02 02:11:30 [2164] File list generation time: 0.154 seconds
> 2018/10/02 02:11:30 [2164] File list transfer time: 0.000 seconds
> 2018/10/02 02:11:30 [2164] Total bytes sent: 235.68M
> 2018/10/02 02:11:30 [2164] Total bytes received: 7.51M
> 2018/10/02 02:11:30 [2164] sent 235.68M bytes  received 7.51M bytes  21.17K 
> bytes/sec
> 2018/10/02 02:11:30 [2164] total size is 10.40T  speedup is 42,753.79
> 
> the most recent looks like this...
> 
> 2018/11/24 23:00:15 [2924] building file list
> 2018/11/24 23:00:17 [2924] cd..t.. /cygdrive/
> 2018/11/25 13:21:16 [2924] Number of files: 13,776,423 (reg: 12,274,642, dir: 
> 1,501,780, link: 1)
> 2018/11/25 13:21:16 [2924] Number of created files: 0
> 2018/11/25 13:21:16 [2924] Number of regular files transferred: 0
> 2018/11/25 13:21:16 [2924] Total file size: 10.49T bytes
> 2018/11/25 13:21:16 [2924] Total transferred file size: 0 bytes
> 2018/11/25 13:21:16 [2924] Literal data: 0 bytes
> 2018/11/25 13:21:16 [2924] Matched data: 0 bytes
> 2018/11/25 13:21:16 [2924] File list size: 10.35M
> 2018/11/25 13:21:16 [2924] File list generation time: 0.316 seconds
> 2018/11/25 13:21:16 [2924] File list transfer time: 0.000 seconds
> 2018/11/25 13:21:16 [2924] Total bytes sent: 236.55M
> 2018/11/25 13:21:16 [2924] Total bytes received: 7.51M
> 2018/11/25 13:21:16 [2924] sent 236.55M bytes  received 7.51M bytes  4.72K 
> bytes/sec
> 2018/11/25 13:21:16 [2924] total size is 10.49T  speedup is 42,996.96
> 
> As you can see the start time is 11:00PM (23:00) in both cases. The first log 
> shows that identifying the files to transfer took about three hours (I've 
> omitted the file list - it's quite long), the second log takes fourteen hours 
> to do the same job (in this case this was done at the weekend and I've 
> include the whole log file which correctly identifies that no files have 
> changed)
> 
> The number of files is as might be expected as is everything else. It's just 
> the time taken "building file list" is significantly larger.
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

--link-dest. Time to 'building file list' incrementing

[John Simpson via rsync]

I've been running rsync as a cygwin task on Windows Server 2008 for about two 
months now. I'm using the --link-dest option to do a daily 'snapshot' of the 
contents of a server containing about 10TB of data, about 13 million files, to 
a Linux based NAS server. Things started out great but I soon noticed that the 
time take to complete was slowly incrementing. It started at around three 
hours, but is now around fifteen.

The command is as follows...

rsync -rlptDhPR \
--password-file=password \
--Chmod=Du=rwx,Dgo=rx,Fu=rw,Fgo=r \
--Stats \
--delete \
--log-file=logfilename \
--link-dest=linkdestpath \
sourceDirectory \
rsync@192.168.1.2::destinationDirectory

I'm not using the full -a option as the differences between the Windows and 
Linux ownership stuff messed things up.

The first log file looked like this...

2018/10/01 23:00:14 [2164] building file list
...transfer file list here
2018/10/02 02:11:30 [2164] Number of files: 13,759,998 (reg: 12,260,176, dir: 
1,499,821, link: 1)
2018/10/02 02:11:30 [2164] Number of created files: 302 (reg: 291, dir: 11)
2018/10/02 02:11:30 [2164] Number of regular files transferred: 310
2018/10/02 02:11:30 [2164] Total file size: 10.40T bytes
2018/10/02 02:11:30 [2164] Total transferred file size: 664.31K bytes
2018/10/02 02:11:30 [2164] Literal data: 277.91K bytes
2018/10/02 02:11:30 [2164] Matched data: 386.40K bytes
2018/10/02 02:11:30 [2164] File list size: 10.42M
2018/10/02 02:11:30 [2164] File list generation time: 0.154 seconds
2018/10/02 02:11:30 [2164] File list transfer time: 0.000 seconds
2018/10/02 02:11:30 [2164] Total bytes sent: 235.68M
2018/10/02 02:11:30 [2164] Total bytes received: 7.51M
2018/10/02 02:11:30 [2164] sent 235.68M bytes  received 7.51M bytes  21.17K 
bytes/sec
2018/10/02 02:11:30 [2164] total size is 10.40T  speedup is 42,753.79

the most recent looks like this...

2018/11/24 23:00:15 [2924] building file list
2018/11/24 23:00:17 [2924] cd..t.. /cygdrive/
2018/11/25 13:21:16 [2924] Number of files: 13,776,423 (reg: 12,274,642, dir: 
1,501,780, link: 1)
2018/11/25 13:21:16 [2924] Number of created files: 0
2018/11/25 13:21:16 [2924] Number of regular files transferred: 0
2018/11/25 13:21:16 [2924] Total file size: 10.49T bytes
2018/11/25 13:21:16 [2924] Total transferred file size: 0 bytes
2018/11/25 13:21:16 [2924] Literal data: 0 bytes
2018/11/25 13:21:16 [2924] Matched data: 0 bytes
2018/11/25 13:21:16 [2924] File list size: 10.35M
2018/11/25 13:21:16 [2924] File list generation time: 0.316 seconds
2018/11/25 13:21:16 [2924] File list transfer time: 0.000 seconds
2018/11/25 13:21:16 [2924] Total bytes sent: 236.55M
2018/11/25 13:21:16 [2924] Total bytes received: 7.51M
2018/11/25 13:21:16 [2924] sent 236.55M bytes  received 7.51M bytes  4.72K 
bytes/sec
2018/11/25 13:21:16 [2924] total size is 10.49T  speedup is 42,996.96

As you can see the start time is 11:00PM (23:00) in both cases. The first log 
shows that identifying the files to transfer took about three hours (I've 
omitted the file list - it's quite long), the second log takes fourteen hours 
to do the same job (in this case this was done at the weekend and I've include 
the whole log file which correctly identifies that no files have changed)

The number of files is as might be expected as is everything else. It's just 
the time taken "building file list" is significantly larger.


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html