Thanks for the long post, the answer is unfortunately very short: I know there is an issue, but I am trying for quite a while no to reproduce it without any aid. Some people have reported the issue, but nobody so far had the time to go through the (probably considerable) hassle of helping me to debug this (that requires multiple iterations of instrumentation and new tries as I do not have a solid idea what the cause is). Many folks who wanted to help also have not been able to reproduce the issue reliably enough (or it took them days).
If you can help finding the cause, I would be more than happy to address it. (but unfortunately I will be out of office most part of tomorrow and Thursday). Rainer > -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Gavin McDonald > Sent: Tuesday, October 26, 2010 6:20 AM > To: [email protected] > Subject: [rsyslog] rsyslog + tls fails under load. > > Greetings, > > I recently tried installing rsyslog on a small server farm to do > centralized > logging over TLS. I am now seeing some unexpected behavior, and was > hoping > the list would have some insight; > > When I start/restart rsyslog on the central server, I get less than a > minute > or two of logs from the web farm, then nothing. the central server just > stops receiving remote messages. (they don't stop being sent though, > which > leads to a compounding issue...) In an effort to debug the problem, I > stopped the service, and ran rsyslogd manually, adding the -d option. > Rsyslog appears to run ok. and I see no explicit errors, but the log > clients themselves will start to seize after about 30 minutes, forcing > at > minimum an apache restart. (which I have since learned they will do, > as > long as there is no server to connect to.) If I run 'rsyslog -r -c4 -d > > > /dev/null' the original problem manifests. In test conditions under > minimal > load, (one log client, dev traffic only,) this doesn't happen at all. > (or at > least, takes far too long to happen.) > > I have found multiple bug-reports relating to the clients' halting > problem, > and understand that it may have even been fixed in the latest 5.x > branch. > Which is why I'm not asking about why the log clients need to be > rebooted > if the logging server goes down. What I am more interested in is how > quickly rsyslog dies when not throttled by writing to the console. - We > run > a modest web farm, of about 60 LAMP servers, doing a sustained 10 > access/s, > with peaks around 30/s. I direct the apache access.log at rsyslog > using the > O'reilly recipe, and the messages appear to all make it through at pace > until rsyslogd falls over on the central server. in -d mode, > estimating > 10-15 output lines per log message, I'd expect to be seeing ~10k debug > lines > per second of debug output given fictionally optimal console > throughput. I > know that my connection is incapable of this, thus I suspect that the > lag > introduced by actually viewing the debug messages through an ssh > session is > enough to delay the leak/race that makes rsyslog fall over so quickly > when > run as a proper daemon. This is further supported by the behaviour when > dumping the debug output to /dev/null. > > As this issue manifested in production, I've rolled the configs back to > simple insecure logging over tcp, and everything has been stable since. > I > ask, Is tls support _supposed_ to work in the 4.2.0-2 release, or am I > just > tilting at windmills? > > as for version, etc; All log clients are ubuntu 8.04 LTS, with rsyslog > 4.2.0-2 installed from alessandro bono's ppa archive. ( > https://launchpad.net/~a.bono/+archive/rsyslog) The log server is also > running 4.2.0-2 but from the official repositories, as it is ubuntu > 9.10. > > I have created the certificates for each machine, and authentication is > (initially) successful. my client rsyslog.conf is as follows: > (comments etc > stripped for brevity, and obviously sanitized) > $DefaultNetstreamDriver gtls > $ModLoad imuxsock > $ActionSendStreamDriverAuthMode x509/name > $DefaultNetstreamDriverCAFile /etc/tls/pkey/ca.pem > $DefaultNetstreamDriverCertFile /etc/tls/cert/client1.pem > $DefaultNetstreamDriverKeyFile /etc/tls/hkey/client1.pem > $ActionSendStreamDriverPermittedPeer logserver.mydomain.com > $ActionSendStreamDriverMode 1 > $RepeatedMsgReduction on > $ActionFileDefaultTemplate RSYSLOG_ForwardFormat > auth,authpriv.* /var/log/auth.log > *.* @@logserver.mydomain.com:10514 > > > The server config is nearly identical, save for the ruleset definitions > in > an include file, and the relevant cert changes. > $ModLoad imuxsock > $ModLoad imklog > $KLogPath /var/run/rsyslog/kmsg > $ModLoad imtcp > $DefaultNetstreamDriver gtls > $DefaultNetstreamDriverCAFile /etc/tls/pkey/ca.pem > $DefaultNetstreamDriverCertFile /etc/tls/cert/server.pem > $DefaultNetstreamDriverKeyFile /etc/tls/hkey/server.pem > $InputTCPServerStreamDriverAuthMode x509/name > $InputTCPServerStreamDriverPermittedPeer *.mydomain.com > $InputTCPServerStreamDriverMode 1 > $InputTCPServerRun 10514 > $RepeatedMsgReduction on > $CreateDirs on > $FileOwner syslog > $FileGroup adm > $FileCreateMode 0640 > $DirCreateMode 0755 > $Umask 0022 > $IncludeConfig /etc/rsyslog.d/*.conf > > > Regards, > > -G > > Gavin McDonald > EVI Logistic Enterprises > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
