I second David's thought to switch to imptcp. It is designed for plain TCP performance.
I would also like to say that I previously thought having rsyslog write to a file for a SIEM’s agent ingestion was a bad idea. Generally I make the SIEM do the TCP listening itself and relay through rsyslog to the SIEM in this manner. However, David makes some excellent points, and sometimes it’s easier to do some of the in-transit work with a SIEM agent than with rsyslog. Throw in the fact that storage virtualization often hides the rust or bits beneath, and I could see it being viable and equally as performant. Things that make you go, “Hmm…” Thanks Gents! > On Apr 26, 2022, at 11:21, Steven D via rsyslog <rsyslog@lists.adiscon.com> > wrote: > > Interesting... thanks for the tips. As long as there's no overt issues, I > won't press my luck by clobbering the config 🙂 > > I think what we're seeing at this point is some system side inability to > handle TCP at the same volume as UDP.... Could be OS side, still plinko-ing > down that rabbithole. > ________________________________ > From: David Lang <da...@lang.hm> > Sent: Tuesday, April 26, 2022 10:23 AM > To: Steven D <pheerl...@hotmail.com> > Cc: David Lang <da...@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com> > Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting > > that looks pretty good to me, if it's still running slow, I would look at the > threads in top to see what their utilization is, and try the imptcp vs imtcp > and > see if there's a difference > > the way you are specifying a ruleset in an input means that you don't have > lots > of rule processing happening. > > one thing you could have is contention for the lock on the main queue. By > default, all messages go to the main queue. If you are tieing an input to a > ruleset, you can give that ruleset a queue and the messages will never touch > the > main queue (in effect, it's like you were running a separate rsyslog instance > for that input and ruleset) > > David Lang > > > On Tue, 26 Apr 2022, Steven D wrote: > >> Date: Tue, 26 Apr 2022 14:12:36 +0000 >> From: Steven D <pheerl...@hotmail.com> >> To: David Lang <da...@lang.hm> >> Cc: rsyslog-users <rsyslog@lists.adiscon.com> >> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting >> >> Good link thanks for that.... Lack of file probably on me, i'm sure I forgot >> to attach. Coffee hadn't kicked in. >> ________________________________ >> From: David Lang <da...@lang.hm> >> Sent: Tuesday, April 26, 2022 9:57 AM >> To: Steven D <pheerl...@hotmail.com> >> Cc: David Lang <da...@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com> >> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting >> >> hmm, looks like it didn't make it through the mail server >> >> https://cromwell-intl.com/open-source/performance-tuning/tcp.html >> >> (also note there are similar UDP buffer tuning parameters, if those buffers >> fill >> up udp logs will be dropped silently) >> >> David Lang >> >> On Tue, 26 Apr 2022, Steven D wrote: >> >>> Date: Tue, 26 Apr 2022 11:00:16 +0000 >>> From: Steven D <pheerl...@hotmail.com> >>> To: David Lang <da...@lang.hm>, rsyslog-users <rsyslog@lists.adiscon.com> >>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting >>> >>> Here's most recent few rotations of pstats data, any additional input would >>> be appreciated. >>> >>> With the keepalives set, TCP connections don't seem to be growing...however >>> it still seems that using TCP (at least for the high throughput firewalls) >>> causes ingest to drop significantly. Are there any OS side (RHEL 8) >>> optimizations that youz could suggestion that make help? >>> >>> Regards, >>> Steven >>> ________________________________ >>> From: rsyslog <rsyslog-boun...@lists.adiscon.com> on behalf of Steven D via >>> rsyslog <rsyslog@lists.adiscon.com> >>> Sent: Monday, April 25, 2022 9:52 AM >>> To: David Lang <da...@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com> >>> Cc: Steven D <pheerl...@hotmail.com> >>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting >>> >>> Kind of depends on SIEM agent... With our set up, having the files written >>> to disk allows us to target/optimize the front-end parsing applied by the >>> SIEM agent a little better. Make is so there's less work to be done >>> elsewhere in the solution, but it could be done other ways. >>> >>> Glad there are other SIEM jockeys on here... haha. >>> ________________________________ >>> From: rsyslog <rsyslog-boun...@lists.adiscon.com> on behalf of Mariusz Kruk >>> via rsyslog <rsyslog@lists.adiscon.com> >>> Sent: Monday, April 25, 2022 9:08 AM >>> To: David Lang <da...@lang.hm> >>> Cc: Mariusz Kruk <k...@epsilon.eu.org>; Mariusz Kruk via rsyslog >>> <rsyslog@lists.adiscon.com> >>> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting >>> >>> >>> On 25.04.2022 15:04, David Lang wrote: >>>>>>> Sure. I work with them, I know ;-) >>>>>>> >>>>>>> It's just that for some, you can do the same but using rsyslog to >>>>>>> process the message (even filter some events out or trim them or do >>>>>>> many other fancy stuff) an send them directly to SIEM (by means of >>>>>>> native SIEM API, not by syslog) >>>>>>> >>>>>>> instead of killing the server with IOPS. That's all. >>>>>> >>>>>> fair point, but I'll point out that since rsyslog is buffering the >>>>>> writes to disk, the IOPS load is not as high as you would think. >>>>>> It's sequential writes and reads, and in practice should be just >>>>>> sequential writes as the reads should be able to be served from ram >>>>>> (the files should be read shortly after they are written, so should >>>>>> still be cached) >>>>>> >>>>>> It would be interesting to look into the performance (CPU and I/O) >>>>>> of writing to disk and having the files read in batches vs posting >>>>>> each message to the SIEM. I think there are enough variables >>>>>> involved that it's not an obvious win either way. >>>>>> >>>>> Sure, there is also another layer of OS-level caching/buffering. But >>>>> in the end it all has to get written to the disk. ;-) I just think >>>>> that passing the events straight through rulesets from input to >>>>> output (and I mean the real destination, not to the files) is simply >>>>> more straightforward and easier to manage (after all you don't have >>>>> to - for example - deal with file rotation and so on). On the other >>>>> hand, you may run into some other problems when your rulesets get too >>>>> complicated and "run away". >>>> >>>> buffered, sequential writes to disk are pretty cheap and perform well >>>> even on slow spinning rust, and file rotation (should) happen during >>>> off-peak periods, and as previously mentioned gives you a handy backup >>>> for those cases where something goes wrong with your SIEM. Yes, you >>>> could have disk backed queues in rsyslog, but the performance of such >>>> queues is horrid and significantly more I/O load than writing to disk. >>>> >>>> Not saying that sending directly is wrong, just listing things I've >>>> run into that make me default the other way. >>>> >>> Yep. I wasn't talking about file rotation in terms of performance (since >>> it should be relatively cheap anyway, especially if you don't want to >>> compress old files). I was referring more to the manageability of the >>> whole solution. >>> >>> Adding another ruleset seems easier than making sure that you have a >>> proper directory prepared for the files, possibly reconfiguring your >>> SIEM agent, making sure the files are included in the rotation scheme... >>> Seems like a headache. >>> >>> _______________________________________________ >>> rsyslog mailing list >>> https://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >>> LIKE THAT. >>> _______________________________________________ >>> rsyslog mailing list >>> https://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >>> LIKE THAT. >>> >> > _______________________________________________ > rsyslog mailing list > https://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
