Hello list, recently i had big trouble with rsyslog 5.8.5 64bit on RHEL5 and would like to know why it happened and how to fix it.
the setup: machine A with rsyslog logs locally and forwards everything to machine B. rsyslog configuration of A: # cat /etc/rsyslog.conf $ModLoad imuxsock $ModLoad imklog $ModLoad immark $MarkMessagePeriod 1200 $SystemLogRateLimitInterval 0 $ActionFileDefaultTemplate RSYSLOG_FileFormat *.info;mail.none;authpriv.none;cron.none;local6.none;local0.none /var/log/messages authpriv.* /var/log/secure mail.* -/var/log/maillog cron.* /var/log/cron *.emerg * uucp,news.crit /var/log/spooler local7.* /var/log/boot $IncludeConfig /etc/rsyslog.d/*.conf # cat /etc/rsyslog.d/forwarder.conf $ActionForwardDefaultTemplate RSYSLOG_ForwardFormat $WorkDirectory /var/spool/rsyslog $ActionQueueType LinkedList $ActionQueueMaxDiskSpace 1024m $ActionQueueHighWatermark 100 $ActionQueueLowWatermark 10 $ActionQueueCheckpointInterval 10 $ActionQueueFileName forward $ActionQueueMaxFileSize 10m $ActionResumeRetryCount -1 $ActionQueueSaveOnShutdown on $ActionWriteAllMarkMessages on *.* @@machine_B:514;RSYSLOG_ForwardFormat what happened: machine B kindly stopped with a kernel panic. for unknown reasons all TCP packets were dropped, TCP sessions hung until TCP timed out. machine A stopped local logging (no disk spool file was created). after some time all applicaions which log via syslog became unresponsive (a simple login took ages). after stopping rsyslog everything went back to normal. i was able to reproduce it by simulation the TCP black-hole with iptables: machine A: # while true; do date | logger; sleep 1; done machine B: 2011-10-11T17:38:38.000000+02:00 machine_A logger: Tue Oct 11 17:38:38 CEST 2011 2011-10-11T17:38:39.000000+02:00 machine_A logger: Tue Oct 11 17:38:39 CEST 2011 2011-10-11T17:38:40.000000+02:00 machine_Alogger: Tue Oct 11 17:38:40 CEST 2011 2011-10-11T17:38:41.000000+02:00 machine_Alogger: Tue Oct 11 17:38:41 CEST 2011 2011-10-11T17:38:42.000000+02:00 machine_Alogger: Tue Oct 11 17:38:42 CEST 2011 2011-10-11T17:38:43.000000+02:00 machine_Alogger: Tue Oct 11 17:38:43 CEST 2011 2011-10-11T17:38:44.000000+02:00 machine_Alogger: Tue Oct 11 17:38:44 CEST 2011 2011-10-11T17:38:45.000000+02:00 machine_Alogger: Tue Oct 11 17:38:45 CEST 2011 2011-10-11T17:38:46.000000+02:00 machine_Alogger: Tue Oct 11 17:38:46 CEST 2011 2011-10-11T17:38:47.000000+02:00 machine_Alogger: Tue Oct 11 17:38:47 CEST 2011 <iptables -A OUTPUT -p tcp --dport 514 -j DROP> machine A: 2011-10-11T17:39:20+02:00 machine_Alogger: Tue Oct 11 17:39:20 CEST 2011 2011-10-11T17:39:21+02:00 machine_Alogger: Tue Oct 11 17:39:21 CEST 2011 2011-10-11T17:39:22+02:00 machine_Alogger: Tue Oct 11 17:39:22 CEST 2011 2011-10-11T17:39:23+02:00 machine_Alogger: Tue Oct 11 17:39:23 CEST 2011 2011-10-11T17:39:24+02:00 machine_Alogger: Tue Oct 11 17:39:24 CEST 2011 2011-10-11T17:39:25+02:00 machine_Alogger: Tue Oct 11 17:39:25 CEST 2011 2011-10-11T17:39:26+02:00 machine_Alogger: Tue Oct 11 17:39:26 CEST 2011 2011-10-11T17:39:27+02:00 machine_Alogger: Tue Oct 11 17:39:27 CEST 2011 <logging stops> i expected that rsyslog starts spooling to disk if the remote server is unreachable but this is not the case if the TCP sessions hangs: # ls -l /var/spool/rsyslog/forward* ls: /var/spool/rsyslog/forward*: No such file or directory i ran the same test with iptables .. -j REJECT and everything worked like expected. is the observed behaviour correct? should local logging stop if the remote server is unresponsive? why didn't switch rsyslog to spooling? because the TCP session to the remote server hung? what can i do on the rsyslog part to prevent such a scenario from happen again? i would like to accomplish the following: if remote destination is unreachable/unresponsive switch to disk spooling but continue to log locally. after the connection has been re-established transfer the backlog to the remote destination. is this possible? and if yes, how? regards, -ap _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
