Thanks David. You are preaching to the choir on upgrading. I'd love for us to have a support contract with Adiscon. I would immediately have yum to keep us up to date with the latest against the adiscon rpm repos, I would immediately fund this feature: http://www.rsyslog.com/doc/lookup_tables.html, and I would use Adiscon professional services to verify our whole design.
Unfortunately, this is not an option in our case due to organizational and other policy factors. This is exactly the kind of answer I was looking for, as I can now check out impstats as a starting point and use some of your other tips. I am about halfway through a re-reading of the docs to understand my options and how they work. I do not want to waste the community's time. I will post again either with my resolution and share the results, or I will ask additional questions after I have made the best effort possible with what I learn. Thanks again, Justin On Tue, May 13, 2014 at 1:00 PM, David Lang <[email protected]> wrote: > 5.8 is ancient (current version is 8.2). Rsyslog has solved the problems > that you are describing, but the solutions are in newer versions. > > If you are unwilling to upgrade to newer versions that are supported by > the rsyslog community because you will lose support from RedHat, then you > should ask RedHat for support of this version. > > All that being said, it is possible to improve the performance of 5.8 up > to several hundred thousand logs/sec. > > While there are a lot of things that can be done, they will all be > complex, require a lot of time to hammer out, and still not get you the > results that you would probably get by just upgrading to a current release > with no changes. > > Adiscon does offer paid support for older versions, and someone here on > the list may choose to start going into the details with you on the old > version, but if not, see the note about RedHat support above > > I don't have a lot of time to go back and forth with things, but you need > to understand where your bottlenecks are (impstats), probably disable DNS > lookups, look at using dynamic filenames (and set your dynafilecachesize > appropriately) > > creating more rulesets and queues for writing to local files is probably > not the right answer (and _really_ messy to do with v5 anyway) > > David Lang > > > On Tue, 13 May 2014, Justin Haynes wrote: > > I'm happy to share more details, but I thought it wise to at least ask the >> question first. I will turn on debugging today in dev and aim some >> traffic >> at it with tcpreplay to gather some more information. I expect much of >> that information may be more education to me about how rsyslog works >> rather >> than what's "wrong". I expect rsyslog is functioning as it should, and >> that I simply need to learn how to configure it better for our purposes. >> >> >> On Tue, May 13, 2014 at 10:02 AM, Justin Haynes <[email protected]> >> wrote: >> >> I have input of about 2500 packets per second of UDP traffic. rsyslog >>> will only process about 1500 and the queue continues to accumulate >>> messages. My main question is how can I improve performance? >>> >>> I have more detailed questions to ask below, but that is the main one. I >>> am open to any input you believe might be helpful. I'm as interested to >>> understand better how this works as I am to find a workable solution. >>> >>> This is rsyslog 5.8.10 which is standard for RHEL 6. An upgrade is not >>> an >>> option due to policies about straying from software RedHat can support. >>> >>> In one instance we have 500 rules, where each rule is a single hostname. >>> I have prioritized top talkers to the top of the ruleset, thiinking that >>> doing so would improve performance as it would with a firewall. Each >>> rules >>> has a drop as the last action. For example: >>> >>> :hostname, isequal, "<hostname>" >>> @?template_to_write_files_to_a_specific_folder >>> @xxx.xxx.xx.xxx:xxx >>> &~ >>> >>> Moving busier rules to the top did not make much difference in >>> performance >>> with regard to pps. maybe 100pps differnce even though the top talker >>> constituted 80% of the traffic and that rule used to be ~rule 200. >>> >>> I only the Main Queue (excluding all the queues needed for each and every >>> action). I only have one ruleset, and configuration is very simple and >>> shown below. >>> >>> Should I distribute my rules across rulesets somehow and have a thread >>> assigned to each one? >>> Does rule processing begin at the beginning and end at the end? >>> Do worker threads automatically get assigned unique workload or are they >>> competing to match their messages to the rules? >>> Do the action queues matter in this case? I would think not, but FYI, >>> all >>> destinations are UDP. >>> >>> >>> $ModLoad imudp >>> $UDPServerAddress xxx.xxx.xxx.xxx >>> $UDPServerRun 514 >>> >>> $UDPServerTimeRequery 8 >>> >>> $ModLoad imtcp >>> $InputTCPServerRun xxxxx >>> >>> >>> #$ActionQueueType Direct >>> $MainMsgQueueWorkerThreadMinimumMessages 100 >>> $MainMsgQueueTimeoutEnqueue 0 >>> $MainMsgQueueWorkerThreads 4 >>> $MainMsgQueueWorkerTimeoutThreadShutdown 20000 >>> $MainMsgQueueSize 50000000 >>> $MainMsgQueueDequeueBatchSize 1000 >>> #$MainMsgQueueType FixedArray >>> $MainMsgQueueType LinkedList >>> >>> >>> $IncludeConfig rules1.conf >>> $IncludeConfig rules2.conf >>> $IncludeConfig rules3.conf >>> >>> >>> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >> _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
