Re: Link-dest breaks on uid/gid mismatches

2012-06-12 Thread Chris Davies
On 11/06/12 21:24, Kevin Korb wrote: You might want to look into --fake-super. It can store metadata things (like ownership) in the file attributes instead of setting them on the file. Thank you for the suggestion. Again, that seems to be a work-around (albeit possibly a better one)? Chris

rsync takes long pauses in xfer ?

2012-06-12 Thread Alan McKay
Hey folks, I did some googling on this but did not come up with much. I'm using rsnapshot which uses rsync, and I notice some pretty long pauses in the xfers as you can see on this graph from munin. THe machine in question right at the moment is doing nothing but rsyncing ( rsnapshoting ) some

RE: rsync takes long pauses in xfer ?

2012-06-12 Thread Claude Juif
Hi, What is the state of your rsync process ? I'm pretty you enter the uninterruptible sleep state (D in ps) Regards, -Message d'origine- De : rsync-boun...@lists.samba.org [mailto:rsync-boun...@lists.samba.org] De la part de Alan McKay Envoyé : mardi 12 juin 2012 15:25 À :

Re: rsync takes long pauses in xfer ?

2012-06-12 Thread Alan McKay
On Tue, Jun 12, 2012 at 9:33 AM, Claude Juif claude.j...@mediagong.tv wrote: What is the state of your rsync process ? I'm pretty you enter the uninterruptible sleep state (D in ps) What option do I need to pass ps to find out? Right now it is going along find but I can set a cron job to

Re: rsync takes long pauses in xfer ?

2012-06-12 Thread Alan McKay
On Tue, Jun 12, 2012 at 9:39 AM, ericbamba...@discover.com wrote: Alan,        What is the rsync command used? If it hits a large group of files with no differences (easy to believe across 12TB) but has to compare the checksums anyways (forced by you or failing the size+time compare) you'll

Re: rsync takes long pauses in xfer ?

2012-06-12 Thread Alan McKay
Oh I should also mention that this is the initial rsync to get it all across Sorry for not mentioning that -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read:

Re: rsync takes long pauses in xfer ?

2012-06-12 Thread ericbambach1
Alan, What is the rsync command used? If it hits a large group of files with no differences (easy to believe across 12TB) but has to compare the checksums anyways (forced by you or failing the size+time compare) you'll be I/O bound with almost no corresponding network traffic as it

Re: rsync takes long pauses in xfer ?

2012-06-12 Thread Alan McKay
On Tue, Jun 12, 2012 at 9:43 AM, Alan McKay alan.mckay+rs...@gmail.com wrote: What option do I need to pass ps to find out? OK, I checked the man page and I see that using BSD-style options gives me this so I set up my cron job to collect data If I see another drop off like that I'll post the

Re: rsync takes long pauses in xfer ?

2012-06-12 Thread Marc Rechté
Eric, How would you know (debug) that size+time compare fails in a rsync session ? Thanks Marc Le 12/06/2012 15:39, ericbamba...@discover.com a écrit : Alan, What is the rsync command used? If it hits a large group of files with no differences (easy to believe across 12TB) but has

Re: rsync takes long pauses in xfer ?

2012-06-12 Thread Kevin Korb
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 - --itemize-changes On 06/12/12 10:08, Marc Rechté wrote: Eric, How would you know (debug) that size+time compare fails in a rsync session ? Thanks Marc Le 12/06/2012 15:39, ericbamba...@discover.com a écrit : Alan, What is the rsync

--stats and performance issue

2012-06-12 Thread Marc Rechté
Thanks Eric, I have been struggling for a few days (see my former message --stats and performance issue) with a similar problem to Alan's. They are very long pauses and very few files / byte transferred. For instance one run gave: Mon Jun 11 18:01:34 CEST 2012 Number of files: 1187751

Re: rsync takes long pauses in xfer ?

2012-06-12 Thread ericbambach1
Marc, Well you can remove the timestamp checks by adding --size-only. If that speeds things up or removes the long pauses the timestamps were to blame. If the behavior is the same then I'm wrong. If I'm right then you might want to synchronize your clocks or use the --modify-window

Re: rsync takes long pauses in xfer ?

2012-06-12 Thread Kevin Korb
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 The time stamp and file size are both returned by a single call to stat(). Not looking at the time stamp value doesn't make it any faster. The option is there for cases where you know the time stamps don't match but you don't want to copy the file

Re: --stats and performance issue

2012-06-12 Thread Alan McKay
On Tue, Jun 12, 2012 at 10:42 AM, Marc Rechté ma...@rechte.fr wrote: I have been struggling for a few days (see my former message --stats and performance issue) with a similar problem to Alan's If there is any sort of debug or other info I can collect for you that will help you out, just let me

Re: rsync takes long pauses in xfer ?

2012-06-12 Thread ericbambach1
Kevin, I'm not saying the stat() call is slow or trying to reduce syscalls. If the quick check fails due to differing timestamps the entire transfer will be significantly slower because rsync has to defer to the full checksum algorithm just to find out the files are identical which