Hi I have a problem with rsyslog, services which send logs via syslog do not respond anymore and are in a blocked state. For example:
"logger test" => no respond, will run forever. Login via ssh not possible anymore since openssh sends logs via syslog, slapd crash, apache crash, nginx crash and so on. I have to force reboot the system since I can't login anymore. This problem occurs randomly on different server since last week. All server do forward their logs to a central syslog server, reckon the problem is related to this. But the central log system is running fine. I can not detect any problems there. So my question, why can (r)syslog kill an entire linux system? Shouldn't it be that rsyslog discards messages which should be sent to a remote system if something is wrong with the connection or whatever? 2nd question how does a correct configuration look like to ignore all syslog messages if something is wrong with the remote syslogger (I'm just guessing that the problem is the remote forwarding)? This was the configuration we always had (No problems until last week): $WorkDirectory /var/spool/rsyslog # where to place spool files $ActionQueueFileName queue # unique name prefix for spool files $ActionQueueMaxDiskSpace 1g # spool space limit (use as much as possible) $ActionQueueSaveOnShutdown on # save messages to disk on shutdown #$ActionQueueType LinkedList # run asynchronously $ActionResumeRetryCount -1 # infinety retries if host is down # Define custom logging templates $template RFC3164fmt,"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag%%msg%" # Use default format $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # Sending logs that match *.* to syslog.example.org via UDP on 5514 using RFC3164fmt format. *.* @@syslog.example.org:5514;RFC3164fmt After those problems occurred, I tried to switch to udp, same problem randomly occurred. Then I tried it with the following two variants, still the same problem: $SystemLogRateLimitInterval 1 $SystemLogRateLimitBurst 50 $WorkDirectory /var/spool/rsyslog # where to place spool files $ActionResumeInterval 10 $ActionQueueSize 100000 $ActionQueueDiscardMark 97500 $ActionQueueHighWaterMark 80000 $ActionQueueType LinkedList $ActionQueueFileName queue $ActionQueueCheckpointInterval 100 $ActionQueueMaxDiskSpace 1g $ActionResumeRetryCount 10 $ActionQueueSaveOnShutdown on $ActionQueueTimeoutEnqueue 10 $ActionQueueDiscardSeverity 0 # Define custom logging templates $template RFC3164fmt,"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag%%msg%" # Use default format $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # Sending logs that match *.* to syslog.example.org via UDP on 5514 using RFC3164fmt format. *.* @syslog.example.org:5514;RFC3164fmt ----------------------- $SystemLogRateLimitInterval 1 $SystemLogRateLimitBurst 50 $WorkDirectory /var/spool/rsyslog # where to place spool files $MainMsgQueyeType LinkedList $MainMsgQueueSize 10000 $MainMsgQueusDiscardMark 8000 $MainMsgQueueDiscardSeverity 1 $MainMsgQueueSaveOnShutdown off $MainMsgQueueTimeoutEnqueue 0 $ActionQueueType LinkedList # in memory queue $ActionQueueFileName fwdRule1 # unique name prefix for spool files $ActionQueueSize 10000 # Only allow 10000 elements in the queue $ActionQueueDiscardMark 8000 # Only allow 8000 elements in the queue before dropping msgs $ActionQueueDiscardSeverity 1 # Discard Alert,Critical,Error,Warning,Notice,Info,Debug, NOT Emergency $ActionQueueSaveOnShutdown off # save messages to disk on shutdown $ActionQueueTimeoutEnqueue 0 $ActionResumeRetryCount -1 # infinite retries if host is down $RepeatedMsgReduction off # Define custom logging templates $template RFC3164fmt,"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag%%msg%" # Use default format $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # Sending logs that match *.* to syslog.example.org via UDP on 5514 using RFC3164fmt format. *.* @syslog.example.org:5514;RFC3164fmt I have to disable remote logging now, since this kills our entire production system. I don't care if I loose logs, can just anybody tell me how such a configuration should look like? It can not be that syslog kills a system if the remote forwarding part does not work somehow... Cheers -- Raffael Sahli pub...@raffaelsahli.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
