On Mon, 7 Mar 2011, Todd Michael Bushnell wrote:
Appreciate the feedback. Sorry last night's message so sparse - wasn't feeling
too great and wanted to crash. Few points followed by my config file:
1. Using TCP, not RELP, because I'm still using syslog-NG as central
loghost with rsyslog on servers.
Ok, this is still a mechanism that will stall if the server stops
accepting messages
2. Already have configured to queue locally in the event of outage.
See config below. I've tested successfully in the past, but yesterday,
when there were problems, I checked the local queue and did not see
local queueing occurring. Perhaps it was just slow enough to slow
things to a crawl, killing Apache, but not quite slow enough to result
in local queueing. Is that possible? Should I look to tune this?
no, if things are slow it will queue locally, and apache will only see
things slow down if the queue fills up (or if you are writing the queue to
disk, if the disk can't keep up)
what are you looking at to decide that rsyslog is not queuing messages?
3. Running "ancient" version of Rsyslog because this is the latest in
CentOS 5 repo. Figured this is because it's stable which is what I
want. No need for some of the newer bells at this point. If I guessed
wrong here and latest will give me better stability and performance I'll
build new RPMs.
the latest will definantly give you better performance, but the other part
of the problem is that since it is so old, getting help here is a bit
harder, simply because it's ahrd to remember back that far.
4. Have a number of admitted design deficiencies with Apache and Tomcat
that could be contributing to performance although this does not impact
sysklog which is why I proceeded as-is until I could get engineering to
fix.
4a. Apache uses logger to send to local syslog socket (where rsyslog
writes locally and sends to 2 remote servers) and also writes to its own
files locally so we're logging twice locally for every message. Not
good when traffic gets high, I presume. Just noticed this yesterday so
need to get fixed. To make matters worse, all logging is happening on
the same volume. Until fixed, maybe I should just have rsyslog write
local Apache logs to /dev/null and forward to remote syslog - nothing
else. Thoughts?
if you want rsyslog to write the queue to disk you will also have
performance issues (the rsyslog disk queue is not very efficient)
One question to ask is how critical it is that no logs get lost? you may
want to configure rsyslog to discard messages if it gets too many queued
rather than stopping apache.
or you may want to have apache write log files and then have rsyslog use
imfile to read the file.
4b. Log4j sending directly to syslog servers, writing to its own local
files and sending to localhost:514 for local logging. Would prefer all
gets handed to rsyslog for local and remote logging. Need to get
engineering to fix that too. Like mentioned before, to reduce IO
contention and avoid duplication, might just configure rsyslog to write
to /dev/null as long as it's configured like this. Only question with
4a/b is that this never posed a problem with sysklog, but is a problem
with rsyslog. This is the reason I did not try to make any major
changes in phase 1.
remember that sysklog didn't do TCP logging, it only did UDP logging, so
it would send the messages out over the network as fast as it could, and
if the receiver can't keep up the message is lost.
you may want to do a test with rsyslog using UDP instead of TCP and see if
the behavior is what you expect. If it is, then you are loosing logs
because your central server can't keep up with UDP, but with TCP you are
stalling and killing apache instead.
# Configuration File
# Provides kernel logging support (previously done by rklogd)
$ModLoad imklog
# Provides support for local system logging (e.g. via logger command)
$ModLoad imuxsock
# Max Message Size (default 2k)
$MaxMessageSize 8192
hmm, you may want to look at enabling jumbo packets on your network so
that each log message can be pushed in a single packet.
# Must listen on localhost for Log4j. Need engineering to change this
$ModLoad imudp
$UDPServerAddress 127.0.0.1
$UDPServerRun 514
# Use traditional timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# ownership/permissions
$umask 0000
$FileOwner root
$FileGroup wheel
$FileCreateMode 0640
# include directory for breaking directives into separate files (future)
$IncludeConfig /etc/rsyslog.d/
# forward to remote host, queueing to local disk if host is down and memory
fills up
# work (spool) files directory
$WorkDirectory /var/log/rsyslog
# loghost1
# in-memory queue; set for asynchronous processing (?)
$ActionQueueType LinkedList
in my testing LinkedList was slower than the default. everything does
asynchronous processing.
but both the default and LinkedList are limited to memory size and the
$MainMsgQueueSize or $ActionMsgQueueSize (which I think default to 10000)
# failover queue filename; also enables disk mode
$ActionQueueFileName failqueue-loghost1
I don't think this enables disk mode, you also need to set the
$ActionQueueType to a disk related type.
David Lang
# infinite retries on insert failure
$ActionResumeRetryCount -1
# save in-memory data if rsyslog shuts down
$ActionQueueSaveOnShutdown on
# remote logging of everything
*.* @@loghost1:5140
# loghost2
# in-memory queue; set for asynchronous processing (?)
$ActionQueueType LinkedList
# failover queue filename; also enables disk mode
$ActionQueueFileName failqueue-loghost2
# infinite retries on insert failure
$ActionResumeRetryCount -1
# save in-memory data if rsyslog shuts down
$ActionQueueSaveOnShutdown on
# remote logging of everything
*.* @@loghost2:5140
# Log Filtering Rules
# Emergency Messages
if $syslogseverity <= '0' then *
if $syslogseverity <= '0' then /var/log/messages
if $syslogseverity <= '0' then ~
# Apache
if $programname == 'logger' and ($msg contains 'access_log' or $msg contains
'cookie_log' or $msg contains 'r
equest_log') then /var/log/http
& ~
if $programname == 'httpd' and ($syslogfacility-text == 'local5' or
$syslogfacility-text == 'local6') then /var/log/http_err
& ~
# Log4j (App Logs)
if $programname == 'com.redacted.infra.syslog.Log4jSystemLogger' then
/var/log/log4j
& ~
# Kernel & IPTables
if $programname == 'kernel' and ($msg contains 'LOGACCEPT' or $msg contains
'LOGDROP') then /var/log/iptables
& ~
# Auth Messages
if $syslogfacility-text == 'auth' or $syslogfacility-text == 'authpriv' then
/var/log/secure
& ~
# Mail
if $syslogfacility-text == 'mail' then /var/log/maillog
& ~
# Catchall for remaining log messages
*.* /var/log/messages
On Mar 6, 2011, at 10:43 PM, Todd Michael Bushnell wrote:
Been planning an rsyslog deployment for about a month. Everything performed as
expected in my limited use dev environment, but when I deployed rsyslog today
to my production environment multiple systems yielded similar disastrous
results:
After a few hours Apache jumped up to 250+ processes (max=256, normal=~50) and
then started hanging. At this time, rsyslog also stopped logging altogether.
As soon as I killed rsyslog and started sysklog, httpd processes dropped to 50
and everything went back to normal.
I'm not sure if this is a case where rsyslog froze and it's state resulted in
Apache's inability to close processes or if there is a problem with Apache and
Rsyslog when a decent volume of traffic is passed through. I'm happy to
provide additional information if someone could give me some clues as to where
to start looking. At this point we're reverting until I can diagnose this
issue and assure my team that I've fixed the problem for good.
Version: rsyslog-3.22.1-3.el5_5.1
System: Linux ******* 2.6.18-92.1.22.el5 #1 SMP Tue Dec 16 11:57:43 EST 2008
x86_64 x86_64 x86_64 GNU/Linux
Todd Michael Bushnell
[email protected]
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
