Watermarks are extremely low, this may cause problems. Can you provide debug log?
Rainer Mike Forbes <[email protected]> hat geschrieben:I can confirm this too on ubuntu lucid with rsyslogd 5.8.5, 64bit. On Fri, Oct 14, 2011 at 6:44 AM, Andreas Piesk <[email protected]> wrote: > Hello list, > > recently i had big trouble with rsyslog 5.8.5 64bit on RHEL5 and would like > to know why it happened > and how to fix it. > > the setup: > > machine A with rsyslog logs locally and forwards everything to machine B. > > rsyslog configuration of A: > > # cat /etc/rsyslog.conf > $ModLoad imuxsock > $ModLoad imklog > $ModLoad immark > $MarkMessagePeriod 1200 > $SystemLogRateLimitInterval 0 > $ActionFileDefaultTemplate RSYSLOG_FileFormat > *.info;mail.none;authpriv.none;cron.none;local6.none;local0.none > /var/log/messages > authpriv.* /var/log/secure > mail.* -/var/log/maillog > cron.* /var/log/cron > *.emerg * > uucp,news.crit /var/log/spooler > local7.* /var/log/boot > $IncludeConfig /etc/rsyslog.d/*.conf > > # cat /etc/rsyslog.d/forwarder.conf > $ActionForwardDefaultTemplate RSYSLOG_ForwardFormat > $WorkDirectory /var/spool/rsyslog > $ActionQueueType LinkedList > $ActionQueueMaxDiskSpace 1024m > $ActionQueueHighWatermark 100 > $ActionQueueLowWatermark 10 > $ActionQueueCheckpointInterval 10 > $ActionQueueFileName forward > $ActionQueueMaxFileSize 10m > $ActionResumeRetryCount -1 > $ActionQueueSaveOnShutdown on > $ActionWriteAllMarkMessages on > *.* @@machine_B:514;RSYSLOG_ForwardFormat > > what happened: > > machine B kindly stopped with a kernel panic. > for unknown reasons all TCP packets were dropped, TCP sessions hung until TCP > timed out. > > machine A stopped local logging (no disk spool file was created). after some > time all applicaions > which log via syslog became unresponsive (a simple login took ages). after > stopping rsyslog > everything went back to normal. > > > i was able to reproduce it by simulation the TCP black-hole with iptables: > > machine A: > # while true; do date | logger; sleep 1; done > > machine B: > > 2011-10-11T17:38:38.000000+02:00 machine_A logger: Tue Oct 11 17:38:38 CEST > 2011 > 2011-10-11T17:38:39.000000+02:00 machine_A logger: Tue Oct 11 17:38:39 CEST > 2011 > 2011-10-11T17:38:40.000000+02:00 machine_Alogger: Tue Oct 11 17:38:40 CEST > 2011 > 2011-10-11T17:38:41.000000+02:00 machine_Alogger: Tue Oct 11 17:38:41 CEST > 2011 > 2011-10-11T17:38:42.000000+02:00 machine_Alogger: Tue Oct 11 17:38:42 CEST > 2011 > 2011-10-11T17:38:43.000000+02:00 machine_Alogger: Tue Oct 11 17:38:43 CEST > 2011 > 2011-10-11T17:38:44.000000+02:00 machine_Alogger: Tue Oct 11 17:38:44 CEST > 2011 > 2011-10-11T17:38:45.000000+02:00 machine_Alogger: Tue Oct 11 17:38:45 CEST > 2011 > 2011-10-11T17:38:46.000000+02:00 machine_Alogger: Tue Oct 11 17:38:46 CEST > 2011 > 2011-10-11T17:38:47.000000+02:00 machine_Alogger: Tue Oct 11 17:38:47 CEST > 2011 > <iptables -A OUTPUT -p tcp --dport 514 -j DROP> > > machine A: > > 2011-10-11T17:39:20+02:00 machine_Alogger: Tue Oct 11 17:39:20 CEST 2011 > 2011-10-11T17:39:21+02:00 machine_Alogger: Tue Oct 11 17:39:21 CEST 2011 > 2011-10-11T17:39:22+02:00 machine_Alogger: Tue Oct 11 17:39:22 CEST 2011 > 2011-10-11T17:39:23+02:00 machine_Alogger: Tue Oct 11 17:39:23 CEST 2011 > 2011-10-11T17:39:24+02:00 machine_Alogger: Tue Oct 11 17:39:24 CEST 2011 > 2011-10-11T17:39:25+02:00 machine_Alogger: Tue Oct 11 17:39:25 CEST 2011 > 2011-10-11T17:39:26+02:00 machine_Alogger: Tue Oct 11 17:39:26 CEST 2011 > 2011-10-11T17:39:27+02:00 machine_Alogger: Tue Oct 11 17:39:27 CEST 2011 > <logging stops> > > i expected that rsyslog starts spooling to disk if the remote server is > unreachable but this is not > the case if the TCP sessions hangs: > > # ls -l /var/spool/rsyslog/forward* > ls: /var/spool/rsyslog/forward*: No such file or directory > > i ran the same test with iptables .. -j REJECT and everything worked like > expected. > > > is the observed behaviour correct? should local logging stop if the remote > server is unresponsive? > why didn't switch rsyslog to spooling? because the TCP session to the remote > server hung? > what can i do on the rsyslog part to prevent such a scenario from happen > again? > > i would like to accomplish the following: > > if remote destination is unreachable/unresponsive switch to disk spooling but > continue to log locally. > after the connection has been re-established transfer the backlog to the > remote destination. > > is this possible? and if yes, how? > > regards, > -ap > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- // Mike GPG: BFC7 3F32 2CCF D91F 53E1 DF88 1578 B2E4 1399 6844 _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
