5.8 is ancient (current version is 8.2). Rsyslog has solved the problems that
you are describing, but the solutions are in newer versions.
If you are unwilling to upgrade to newer versions that are supported by the
rsyslog community because you will lose support from RedHat, then you should ask
RedHat for support of this version.
All that being said, it is possible to improve the performance of 5.8 up to
several hundred thousand logs/sec.
While there are a lot of things that can be done, they will all be complex,
require a lot of time to hammer out, and still not get you the results that
you would probably get by just upgrading to a current release with no changes.
Adiscon does offer paid support for older versions, and someone here on the list
may choose to start going into the details with you on the old version, but if
not, see the note about RedHat support above
I don't have a lot of time to go back and forth with things, but you need to
understand where your bottlenecks are (impstats), probably disable DNS lookups,
look at using dynamic filenames (and set your dynafilecachesize appropriately)
creating more rulesets and queues for writing to local files is probably not the
right answer (and _really_ messy to do with v5 anyway)
David Lang
On Tue, 13 May 2014, Justin Haynes wrote:
I'm happy to share more details, but I thought it wise to at least ask the
question first. I will turn on debugging today in dev and aim some traffic
at it with tcpreplay to gather some more information. I expect much of
that information may be more education to me about how rsyslog works rather
than what's "wrong". I expect rsyslog is functioning as it should, and
that I simply need to learn how to configure it better for our purposes.
On Tue, May 13, 2014 at 10:02 AM, Justin Haynes <[email protected]>wrote:
I have input of about 2500 packets per second of UDP traffic. rsyslog
will only process about 1500 and the queue continues to accumulate
messages. My main question is how can I improve performance?
I have more detailed questions to ask below, but that is the main one. I
am open to any input you believe might be helpful. I'm as interested to
understand better how this works as I am to find a workable solution.
This is rsyslog 5.8.10 which is standard for RHEL 6. An upgrade is not an
option due to policies about straying from software RedHat can support.
In one instance we have 500 rules, where each rule is a single hostname.
I have prioritized top talkers to the top of the ruleset, thiinking that
doing so would improve performance as it would with a firewall. Each rules
has a drop as the last action. For example:
:hostname, isequal, "<hostname>"
@?template_to_write_files_to_a_specific_folder
@xxx.xxx.xx.xxx:xxx
&~
Moving busier rules to the top did not make much difference in performance
with regard to pps. maybe 100pps differnce even though the top talker
constituted 80% of the traffic and that rule used to be ~rule 200.
I only the Main Queue (excluding all the queues needed for each and every
action). I only have one ruleset, and configuration is very simple and
shown below.
Should I distribute my rules across rulesets somehow and have a thread
assigned to each one?
Does rule processing begin at the beginning and end at the end?
Do worker threads automatically get assigned unique workload or are they
competing to match their messages to the rules?
Do the action queues matter in this case? I would think not, but FYI, all
destinations are UDP.
$ModLoad imudp
$UDPServerAddress xxx.xxx.xxx.xxx
$UDPServerRun 514
$UDPServerTimeRequery 8
$ModLoad imtcp
$InputTCPServerRun xxxxx
#$ActionQueueType Direct
$MainMsgQueueWorkerThreadMinimumMessages 100
$MainMsgQueueTimeoutEnqueue 0
$MainMsgQueueWorkerThreads 4
$MainMsgQueueWorkerTimeoutThreadShutdown 20000
$MainMsgQueueSize 50000000
$MainMsgQueueDequeueBatchSize 1000
#$MainMsgQueueType FixedArray
$MainMsgQueueType LinkedList
$IncludeConfig rules1.conf
$IncludeConfig rules2.conf
$IncludeConfig rules3.conf
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
