RE: Performance and simultaneous connections over SSH

2010-11-25 Thread Francois Begin
 Actually I recently noticed other issues where the rsync process dumps core
so I am not even sure if I am getting all the data synced up correctly. The
files I transfer are fairly large (hundreds of thousands of lines). 

 

I have not been able to dedicate myself to addressing this issue further. So
my solution just limps along for now.

François Bégin
Security Consultant
Security Development Design and Implementation
TELUS Communications
Office: (780) 493-3381

Honesty, Integrity and Accountability 

 

 tbs frog and lock (tiny)Be Security savy, visit CSO
http://habitat.tmi.telus.com/collaborate/display/corpsec/Security+Awareness
+and+Risk+Assessment  Security Awareness today!

 

 

From: Ricardo Olguin [mailto:rolg...@bwg.ie] 
Sent: November 22, 2010 11:10 AM
To: Matt McCutchen; Francois Begin
Cc: rsync
Subject: RE: Performance and simultaneous connections over SSH

 

Hi Francois,

 

 Have you got any progress about the performance?

 

 I am asking that because I have rsync running in a Linux RH5 in our
main office making the rsync for 22 sites and I am getting the same issue:

 

 I have tried to schedule it to start at different times and no luck.

 

 The average band size for the sites are 4 mbytes.

 

- the rsync drops for Connection timed out, but that DOES NOT happens
when it run once site at time.

 

 The last parameters that I have added are : --bwlimit=2048
--timeout=900 but that did not make difference.

 

 

Matt/Francois and anyone:

 

 Any reply will be appreciate.

 

Thank you.

 

Kind Regards,

Ricardo Olguin

 

-Original Message-
From: rsync-boun...@lists.samba.org [mailto:rsync-boun...@lists.samba.org]
On Behalf Of Matt McCutchen
Sent: 15 August 2010 20:38
To: Francois Begin
Cc: rsync
Subject: Re: Performance and simultaneous connections over SSH

 

On Sun, 2010-08-15 at 10:50 -0600, Francois Begin wrote:

 I just started to notice some 'sync sputtering': Sometime, all 4 

 server's latest access log will have the same timestamp e.g. 09:30, 

 while at other times I would see something like this: It is 09:35 and 

 I have 2 servers at 09:20 and 2 at 09:30 i.e. they could not all sync 

 themselves during the last round.

 

I would suggest enabling logging with --log-file on either the client or the
server side to get more information about what is happening.

 

 How does rsync handle mutliple simultaneous connections over SSH. I am 

 guessing that it is up to the log repo server to allocate 4 separate 

 SSH sessions, and that within each of these, it will use rsync to sync 

 up the logs. Is that correct?

 

Correct.  The four rsync instances on the log repo server will work
independently.

 

--

Matt

 

--

Please use reply-all for most replies to avoid omitting the mailing list.

To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
https://lists.samba.org/mailman/listinfo/rsync

Before posting, read:  http://www.catb.org/~esr/faqs/smart-questions.html
http://www.catb.org/~esr/faqs/smart-questions.html

*
BWG Foods E-mail Notice
This email and any files transmitted with it are confidential and intended
solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error, please notify the system manager
by email at postmas...@bwg.ie.
Please note that any views or opinions presented in this email are solely
those of the author and do not necessarily represent those of the company.
Finally, this email has been scanned for the presence of viruses. The 
recipient should also check this email and any attachments for the presence
of viruses. The company accepts no liability for any damage caused by any
virus transmitted by this email.
BWG Foods; 
BWG Foods is a private unlimited company.
Registered Office: Greenhills Road, Walkinstown, Dublin 12 
Registered in Ireland - Registered Number: 20469
 
*
image001.jpg

smime.p7s
Description: S/MIME cryptographic signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Performance and simultaneous connections over SSH

2010-08-15 Thread Francois Begin
Hi all,

I have about 15 LDAP servers that will be using Rsync over SSH to sync up their 
access logs to a centralized server for further processing. The estimated log 
volume is around 12 Gigs / days total. On the centralized server, the directory 
where the logs are being synced is an enterprise-grade NAS. So it looks like 
this:

LDAP servers -- rsync over SSH --  NFS mount on log repo server -- NAS device

I have 4 servers currently configured and everything appeared to work fine at 
first. I do not have any visibility to the LDAP servers but I assume their 
clocks are synced. The cron is running at the same times: HH:00, HH:10, 20, 30, 
40 and 50. 

I just started to notice some 'sync sputtering': Sometime, all 4 server's 
latest access log will have the same timestamp e.g. 09:30, while at other times 
I would see something like this: It is 09:35 and I have 2 servers at 09:20 and 
2 at 09:30 i.e. they could not all sync themselves during the last round. 

How does rsync handle mutliple simultaneous connections over SSH. I am guessing 
that it is up to the log repo server to allocate 4 separate SSH sessions, and 
that within each of these, it will use rsync to sync up the logs. Is that 
correct? If so, then the only issue would be the log repo server requesting a 
lot of information from the NAS device at the same time, hence causing the 
operation to possibly fail. The actual rsync sommand looks like this:

/bin/rsync --timeout=300 --rsync-path=/usr/local/bin/rsync -avz -e 
/usr/local/bin/ssh -i key `find $LDAP_LOGS_LOCAL/access* -mtime -5 -type f` 
flmu...@$preparser_server:$LDAP_LOGS_REMOTE

Timeout is set to 300 and logs being checked and synced are at most 100 megs in 
size (they rotate once they hit 100 megs).

Cheers,

François



-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html