Greetings, I recently tried installing rsyslog on a small server farm to do centralized logging over TLS. I am now seeing some unexpected behavior, and was hoping the list would have some insight;
When I start/restart rsyslog on the central server, I get less than a minute or two of logs from the web farm, then nothing. the central server just stops receiving remote messages. (they don't stop being sent though, which leads to a compounding issue...) In an effort to debug the problem, I stopped the service, and ran rsyslogd manually, adding the -d option. Rsyslog appears to run ok. and I see no explicit errors, but the log clients themselves will start to seize after about 30 minutes, forcing at minimum an apache restart. (which I have since learned they will do, as long as there is no server to connect to.) If I run 'rsyslog -r -c4 -d > /dev/null' the original problem manifests. In test conditions under minimal load, (one log client, dev traffic only,) this doesn't happen at all. (or at least, takes far too long to happen.) I have found multiple bug-reports relating to the clients' halting problem, and understand that it may have even been fixed in the latest 5.x branch. Which is why I'm not asking about why the log clients need to be rebooted if the logging server goes down. What I am more interested in is how quickly rsyslog dies when not throttled by writing to the console. - We run a modest web farm, of about 60 LAMP servers, doing a sustained 10 access/s, with peaks around 30/s. I direct the apache access.log at rsyslog using the O'reilly recipe, and the messages appear to all make it through at pace until rsyslogd falls over on the central server. in -d mode, estimating 10-15 output lines per log message, I'd expect to be seeing ~10k debug lines per second of debug output given fictionally optimal console throughput. I know that my connection is incapable of this, thus I suspect that the lag introduced by actually viewing the debug messages through an ssh session is enough to delay the leak/race that makes rsyslog fall over so quickly when run as a proper daemon. This is further supported by the behaviour when dumping the debug output to /dev/null. As this issue manifested in production, I've rolled the configs back to simple insecure logging over tcp, and everything has been stable since. I ask, Is tls support _supposed_ to work in the 4.2.0-2 release, or am I just tilting at windmills? as for version, etc; All log clients are ubuntu 8.04 LTS, with rsyslog 4.2.0-2 installed from alessandro bono's ppa archive. ( https://launchpad.net/~a.bono/+archive/rsyslog) The log server is also running 4.2.0-2 but from the official repositories, as it is ubuntu 9.10. I have created the certificates for each machine, and authentication is (initially) successful. my client rsyslog.conf is as follows: (comments etc stripped for brevity, and obviously sanitized) $DefaultNetstreamDriver gtls $ModLoad imuxsock $ActionSendStreamDriverAuthMode x509/name $DefaultNetstreamDriverCAFile /etc/tls/pkey/ca.pem $DefaultNetstreamDriverCertFile /etc/tls/cert/client1.pem $DefaultNetstreamDriverKeyFile /etc/tls/hkey/client1.pem $ActionSendStreamDriverPermittedPeer logserver.mydomain.com $ActionSendStreamDriverMode 1 $RepeatedMsgReduction on $ActionFileDefaultTemplate RSYSLOG_ForwardFormat auth,authpriv.* /var/log/auth.log *.* @@logserver.mydomain.com:10514 The server config is nearly identical, save for the ruleset definitions in an include file, and the relevant cert changes. $ModLoad imuxsock $ModLoad imklog $KLogPath /var/run/rsyslog/kmsg $ModLoad imtcp $DefaultNetstreamDriver gtls $DefaultNetstreamDriverCAFile /etc/tls/pkey/ca.pem $DefaultNetstreamDriverCertFile /etc/tls/cert/server.pem $DefaultNetstreamDriverKeyFile /etc/tls/hkey/server.pem $InputTCPServerStreamDriverAuthMode x509/name $InputTCPServerStreamDriverPermittedPeer *.mydomain.com $InputTCPServerStreamDriverMode 1 $InputTCPServerRun 10514 $RepeatedMsgReduction on $CreateDirs on $FileOwner syslog $FileGroup adm $FileCreateMode 0640 $DirCreateMode 0755 $Umask 0022 $IncludeConfig /etc/rsyslog.d/*.conf Regards, -G Gavin McDonald EVI Logistic Enterprises _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
