Posted earlier on this week, but was sick at the time and thus, didn't post enough info to warrant informed response. Here's some more information along with some specific questions:
Conditions: - Version: rsyslog-3.22.1-3.el5_5.1 - System: Linux ******* 2.6.18-92.1.22.el5 #1 SMP Tue Dec 16 11:57:43 EST 2008 x86_64 x86_64 x86_64 GNU/Linux - Rsyslog Clients w/ Syslog-NG servers - Transport: TCP Problem: Replaced Sysklog with Rsyslog. After several hours of high traffic, Apache processes jumped from 50 to 250 and Apache eventually stopped working. As soon as I shutdown Rsyslog and turned up Sysklog, Apache procs went back to normal and all was resolved. Questions: 1. Running old version of Rsyslog (3.22) because this is latest version available in CentOS repository. Will I get performance/stability improvements upgrading to 5x (e.g. 5.6.2) or even 6x? If so, I'll build RPMs, but assumed latest version in CentOS 5 repository was sufficient if I don't need latest features. Am I wrong? Should I upgrade to latest? 2. Couple design deficiencies (Apache & Log4 logs are double logged), but deployed as-is until Engineering could fix deficiencies because I wanted to emulate existing Sysklog deployment. Before fixing, I just need to understand if there's anything about my Rsyslog configuration that would make duplicate logging an issue when it was not an issue with SysKlog. 3. The one major difference between Sysklog and Rsyslog is the use of TCP. I know this trades performance for efficiency, but I don't know how to determine if this is the problem. I don't see issues on the server side and even if there are problems, I have Rsyslog configured to queue locally if anything happens on the server side. During the problem, there is no local queuing going on. Are there any diagnostics I can pull to determine if remote syslog is the issue? Below is my config file. Is there anything in here that is a glaring issue? If not, My ideas are to: 1. Upgrade, then retry. 2. Remove duplication, then retry. 3. Change from TCP back to UDP, then retry. until I determine the source of the problem. # Configuration File # Provides kernel logging support (previously done by rklogd) $ModLoad imklog # Provides support for local system logging (e.g. via logger command) $ModLoad imuxsock # Max Message Size (default 2k) $MaxMessageSize 8192 # Must listen on localhost for Log4j. Need engineering to change this $ModLoad imudp $UDPServerAddress 127.0.0.1 $UDPServerRun 514 # Use traditional timestamp format $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # ownership/permissions $umask 0000 $FileOwner root $FileGroup wheel $FileCreateMode 0640 # include directory for breaking directives into separate files (future) $IncludeConfig /etc/rsyslog.d/ # forward to remote host, queueing to local disk if host is down and memory fills up # work (spool) files directory $WorkDirectory /var/log/rsyslog # loghost1 # in-memory queue; set for asynchronous processing (?) $ActionQueueType LinkedList # failover queue filename; also enables disk mode $ActionQueueFileName failqueue-loghost1 # infinite retries on insert failure $ActionResumeRetryCount -1 # save in-memory data if rsyslog shuts down $ActionQueueSaveOnShutdown on # remote logging of everything *.* @@loghost1:5140 # loghost2 # in-memory queue; set for asynchronous processing (?) $ActionQueueType LinkedList # failover queue filename; also enables disk mode $ActionQueueFileName failqueue-loghost2 # infinite retries on insert failure $ActionResumeRetryCount -1 # save in-memory data if rsyslog shuts down $ActionQueueSaveOnShutdown on # remote logging of everything *.* @@loghost2:5140 # Log Filtering Rules # Emergency Messages if $syslogseverity <= '0' then * if $syslogseverity <= '0' then /var/log/messages if $syslogseverity <= '0' then ~ # Apache if $programname == 'logger' and ($msg contains 'access_log' or $msg contains 'cookie_log' or $msg contains 'r equest_log') then /var/log/http & ~ if $programname == 'httpd' and ($syslogfacility-text == 'local5' or $syslogfacility-text == 'local6') then /var/log/http_err & ~ # Log4j (App Logs) if $programname == 'com.redacted.infra.syslog.Log4jSystemLogger' then /var/log/log4j & ~ # Kernel & IPTables if $programname == 'kernel' and ($msg contains 'LOGACCEPT' or $msg contains 'LOGDROP') then /var/log/iptables & ~ # Auth Messages if $syslogfacility-text == 'auth' or $syslogfacility-text == 'authpriv' then /var/log/secure & ~ # Mail if $syslogfacility-text == 'mail' then /var/log/maillog & ~ # Catchall for remaining log messages *.* /var/log/messages _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
