Thanks Eric,
I have been struggling for a few days (see my former message --stats and
performance issue) with a similar problem to Alan's. They are very long
pauses and very few files / byte transferred.
For instance one run gave:
Mon Jun 11 18:01:34 CEST 2012
Number of files: 1187751
Number of files transferred: 1494
Total file size: 203.90G bytes
Total transferred file size: 11.72G bytes
Literal data: 139.17M bytes
Matched data: 11.58G bytes
File list size: 32.74M
File list generation time: 0.059 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 5.09M
Total bytes received: 175.26M
sent 5.09M bytes received 175.26M bytes 246.20K bytes/sec
total size is 203.90G speedup is 1130.61
Mon Jun 11 18:13:46 CEST 2012
And another one with half the number of files (by exluding a few big
directories) taking the same amount of time !
Tue Jun 12 16:01:52 CEST 2012
Number of files: 570880
Number of files transferred: 258
Total file size: 163.37G bytes
Total transferred file size: 5.16G bytes
Literal data: 76.84M bytes
Matched data: 5.09G bytes
File list size: 15.25M
File list generation time: 0.014 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 2.35M
Total bytes received: 93.78M
sent 2.35M bytes received 93.78M bytes 129.47K bytes/sec
total size is 163.37G speedup is 1699.43
Tue Jun 12 16:14:14 CEST 2012
I really don't know what test to undertake to farther investigate the problem.
Thanks
Le 12/06/2012 16:21, ericbamba...@discover.com a écrit :
Marc,
Well you can remove the timestamp checks by adding "--size-only".
If that speeds things up or removes the long "pauses" the timestamps were
to blame. If the behavior is the same then I'm wrong. If I'm right then
you might want to synchronize your clocks or use the " --modify-window"
argument to rsync.
Eric Bambach | Discover
Senior Assoc. Programmer, Warehouse Infrastructure and Tools
2500 Lake Cook Road, Riverwoods IL 60015
P: 224.405.2896 ericbamba...@discover.com
From: Marc Rechté<ma...@rechte.fr>
To:<rsync@lists.samba.org>
Date: 06/12/2012 09:12 AM
Subject: Re: rsync takes long pauses in xfer ?
Sent by:<rsync-boun...@lists.samba.org>
Eric,
How would you know (debug) that size+time compare fails in a rsync session
?
Thanks
Marc
Le 12/06/2012 15:39, ericbamba...@discover.com a écrit :
Alan,
What is the rsync command used? If it hits a large group of
files
with no differences (easy to believe across 12TB) but has to compare the
checksums anyways (forced by you or failing the size+time compare)
you'll
be I/O bound with almost no corresponding network traffic as it just has
to send the checksums.
Eric Bambach | Discover
Senior Assoc. Programmer, Warehouse Infrastructure and Tools
2500 Lake Cook Road, Riverwoods IL 60015
P: 224.405.2896 ericbamba...@discover.com
From: Alan McKay<alan.mckay+rs...@gmail.com>
To:<rsync@lists.samba.org>
Date: 06/12/2012 08:30 AM
Subject: rsync takes long pauses in xfer ?
Sent by:<rsync-boun...@lists.samba.org>
Hey folks,
I did some googling on this but did not come up with much. I'm using
rsnapshot which uses rsync, and I notice some pretty long pauses in
the xfers as you can see on this graph from "munin". THe machine in
question right at the moment is doing nothing but rsyncing (
rsnapshoting ) some 12T of NAS storage to local disk, so there is
nothing else going on at all.
https://plus.google.com/u/0/photos/113518077816006248932/albums/5732421463318682369/5753139625064539250
You can see a couple of spots in the graph where it just drops to
nothing ( checking more closely you do see a very small bit of
activity )
Here is another graph from MRTG which records the traffic from the
switch port side. You can see around 2am it drops off pretty low,
then from about 4am til about 7am it drops even lower.
https://plus.google.com/u/0/photos/113518077816006248932/albums/5732421463318682369/5753140504922328322
Is that normal? Or is this something I should ask on the rsnapshot
list (if there is one)?
I did find something in google about depending on the types of files
it can change xfer rate but not much detail.
thanks,
-Alan
--
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
