Appreciate the stellar advice Rainer. I built a 5.6.x (latest stable), but
before upgrading I wanted to do some tests with my existing version/config and
then apply some of the rule syntax changes you recommended to gauge performance
benefit. I used the syslog_caller tool to perform a few tests. Here are those
results:
Running the following command on a test box and simply recording realtime for
comparison:
time ./syslog_caller -m 50000
Initial (Apache killing) Config w/ Remote TCP logging: 5m38.163s
Switch to UDP Remote Logging:
4m22.497s
Disable Remote Logging:
3m23.322s
Initial w/ Amended Rules (see RULES section):
4m54.055s
Amended Rules w/ expression-based rules commented out: 2m36.023s
Amended Rules w/ MainMsg Disk Queue 7m43.498
Sysklog (for comparison):
2m13.986s
Glad to see just changing my rules improved performance by about 13%. My
initial reaction was to send this info and ask a series of questions based on
the data, but instead decided to give it a whirl with the latest stable
version: 5.6.4.
5.6.4 w/ Amended rules:
0m3.773s
Wow - I almost fell off my chair! This is AMAZING! Thank you! Given these
results, I just have a couple final questions:
In compliance heavy environments (which I'm in), I assume the recommendation is
to add disk queuing for the main queue. Is this correct? Something like:
$MainMsgQueueFileName mainqueue
$MainMsgQueueType LinkedList
$MainMsgQueueSaveOnShutdown on
I understand there is a performance tradeoff, but given PCI-DSS, it'll be worth
it, I think.
Last, I also have a disk queuing setup to cover me if our central loghost goes
down. I Configured per doc recommendations. When I ran my latest, blazing
fast test, it queued to disk, I presume because output to the central loghost
over TCP is so much slower. The problem: even after all the logs made it to
the central loghost, a failqueue file still exists:
-rw------- 1 root root 704235 Mar 10 23:31 failqueue-loghost2.00000007
Furthermore, it continues to receive copies of new messages that I confirm have
arrived at the central loghost. Is this a problem or have I misunderstood the
behavior of the failqueue? My assumption is that it would queue up, but then
gradually go to zero once the central loghost caught up.
Note: when I restart ryslog the file goes away and no duplicates are pushed to
the central loghost.
# Disk Queuing Configuration
# loghost1
# in-memory queue; set for asynchronous processing
$ActionQueueType LinkedList
# failover queue filename; also enables disk mode
$ActionQueueFileName failqueue-loghost1
# infinite retries on insert failure
$ActionResumeRetryCount -1
# save in-memory data if rsyslog shuts down
$ActionQueueSaveOnShutdown on
# remote logging of everything
*.* @@loghost1:5140
If this is unexpected, let me know and I'll file a bug with more info.
Thanks again. This is a happy day.
Todd
On Mar 10, 2011, at 1:05 AM, Rainer Gerhards wrote:
> Just some quick points, but I guess useful ones:
>
> The newer version offer much more performance. For details, see here:
> http://www.gerhards.net/download/LinuxKongress2010rsyslog.pdf
>
> I see you use script-based filters where you could use much simpler ones.
> Script based filters are pretty slow. Go for selector lines or property based
> filters (in that order) to gain more speed.
>
> HTH
> Rainer
>
>> -----Original Message-----
>> From: [email protected] [mailto:rsyslog-
>> [email protected]] On Behalf Of Todd Michael Bushnell
>> Sent: Wednesday, March 09, 2011 11:29 PM
>> To: rsyslog-users
>> Subject: [rsyslog] Troubleshooting Rsyslog/Apache Issues
>>
>> Posted earlier on this week, but was sick at the time and thus, didn't
>> post enough info to warrant informed response. Here's some more
>> information along with some specific questions:
>>
>> Conditions:
>>
>> - Version: rsyslog-3.22.1-3.el5_5.1
>> - System: Linux ******* 2.6.18-92.1.22.el5 #1 SMP Tue Dec 16 11:57:43
>> EST 2008 x86_64 x86_64 x86_64 GNU/Linux
>> - Rsyslog Clients w/ Syslog-NG servers
>> - Transport: TCP
>>
>> Problem:
>>
>> Replaced Sysklog with Rsyslog. After several hours of high traffic,
>> Apache processes jumped from 50 to 250 and Apache eventually stopped
>> working. As soon as I shutdown Rsyslog and turned up Sysklog, Apache
>> procs went back to normal and all was resolved.
>>
>> Questions:
>>
>> 1. Running old version of Rsyslog (3.22) because this is latest
>> version available in CentOS repository. Will I get
>> performance/stability improvements upgrading to 5x (e.g. 5.6.2) or even
>> 6x? If so, I'll build RPMs, but assumed latest version in CentOS 5
>> repository was sufficient if I don't need latest features. Am I wrong?
>> Should I upgrade to latest?
>>
>> 2. Couple design deficiencies (Apache & Log4 logs are double logged),
>> but deployed as-is until Engineering could fix deficiencies because I
>> wanted to emulate existing Sysklog deployment. Before fixing, I just
>> need to understand if there's anything about my Rsyslog configuration
>> that would make duplicate logging an issue when it was not an issue
>> with SysKlog.
>>
>> 3. The one major difference between Sysklog and Rsyslog is the use of
>> TCP. I know this trades performance for efficiency, but I don't know
>> how to determine if this is the problem. I don't see issues on the
>> server side and even if there are problems, I have Rsyslog configured
>> to queue locally if anything happens on the server side. During the
>> problem, there is no local queuing going on. Are there any diagnostics
>> I can pull to determine if remote syslog is the issue?
>>
>> Below is my config file. Is there anything in here that is a glaring
>> issue? If not, My ideas are to:
>>
>> 1. Upgrade, then retry.
>> 2. Remove duplication, then retry.
>> 3. Change from TCP back to UDP, then retry.
>> until I determine the source of the problem.
>>
>> # Configuration File
>>
>> # Provides kernel logging support (previously done by rklogd)
>> $ModLoad imklog
>> # Provides support for local system logging (e.g. via logger command)
>> $ModLoad imuxsock
>>
>> # Max Message Size (default 2k)
>> $MaxMessageSize 8192
>>
>> # Must listen on localhost for Log4j. Need engineering to change this
>> $ModLoad imudp
>> $UDPServerAddress 127.0.0.1
>> $UDPServerRun 514
>>
>> # Use traditional timestamp format
>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>
>> # ownership/permissions
>> $umask 0000
>> $FileOwner root
>> $FileGroup wheel
>> $FileCreateMode 0640
>>
>> # include directory for breaking directives into separate files
>> (future)
>> $IncludeConfig /etc/rsyslog.d/
>>
>> # forward to remote host, queueing to local disk if host is down and
>> memory fills up
>> # work (spool) files directory
>> $WorkDirectory /var/log/rsyslog
>>
>> # loghost1
>> # in-memory queue; set for asynchronous processing (?)
>> $ActionQueueType LinkedList
>> # failover queue filename; also enables disk mode
>> $ActionQueueFileName failqueue-loghost1
>> # infinite retries on insert failure
>> $ActionResumeRetryCount -1
>> # save in-memory data if rsyslog shuts down
>> $ActionQueueSaveOnShutdown on
>> # remote logging of everything
>> *.* @@loghost1:5140
>>
>> # loghost2
>> # in-memory queue; set for asynchronous processing (?)
>> $ActionQueueType LinkedList
>> # failover queue filename; also enables disk mode
>> $ActionQueueFileName failqueue-loghost2
>> # infinite retries on insert failure
>> $ActionResumeRetryCount -1
>> # save in-memory data if rsyslog shuts down
>> $ActionQueueSaveOnShutdown on
>> # remote logging of everything
>> *.* @@loghost2:5140
>>
>> # Log Filtering Rules
>>
>> # Emergency Messages
>> if $syslogseverity <= '0' then *
>> if $syslogseverity <= '0' then /var/log/messages
>> if $syslogseverity <= '0' then ~
>>
>> # Apache
>> if $programname == 'logger' and ($msg contains 'access_log' or $msg
>> contains 'cookie_log' or $msg contains 'r
>> equest_log') then /var/log/http
>> & ~
>> if $programname == 'httpd' and ($syslogfacility-text == 'local5' or
>> $syslogfacility-text == 'local6') then /var/log/http_err
>> & ~
>>
>> # Log4j (App Logs)
>> if $programname == 'com.redacted.infra.syslog.Log4jSystemLogger' then
>> /var/log/log4j
>> & ~
>>
>> # Kernel & IPTables
>> if $programname == 'kernel' and ($msg contains 'LOGACCEPT' or $msg
>> contains 'LOGDROP') then /var/log/iptables
>> & ~
>>
>> # Auth Messages
>> if $syslogfacility-text == 'auth' or $syslogfacility-text == 'authpriv'
>> then /var/log/secure
>> & ~
>>
>> # Mail
>> if $syslogfacility-text == 'mail' then /var/log/maillog
>> & ~
>>
>> # Catchall for remaining log messages
>> *.* /var/log/messages
>>
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
