Hi Yuri Unfortunately mmfields module is not installed for our RedHat machines, and these are air gapped machines which cannot download the package.
Would it be possible to use property replacer technique to parse out the field? Regards, Scorsese P. On Sun, Oct 4, 2020 at 11:12 PM Yuri Bushmelev <[email protected]> wrote: > Hello! > > From what I see you have 5 conditions. 4 of them doing full-scan of $msg > on every incoming message. What I'd suggest is to parse the message first > using a `mmnormalize`/`mmfields`/`mmpstructdata` module. Then you can refer > to fields extracted in your conditions instead to prevent full message scan. > > I can guess you may be using the iptables message format. So you may check > this liblognorm field type: > https://www.liblognorm.com/files/manual/configuration.html#iptables > > > On Sun, 4 Oct 2020 at 21:30, Kype Ahamed via rsyslog < > [email protected]> wrote: > >> Hello, >> >> I am using rsyslog mainly as a syslog relay, to forward messages from 1 >> source device to multiple destination devices. >> >> Right now i am receiving about 50k messages per second, and i noticed CPU >> usage is constantly above 80% >> >> Are there any further tweaks that can be done to below config to reduce >> the >> CPU usage? >> >> ------------------------------------------------- >> rsyslog config file: >> >> module(load="impstats" >> interval="20" >> severity="7" >> log.syslog="off" >> log.file="/var/log/impstats.log") >> >> global(parser.escapecontrolcharactertab="off") >> >> # Load Modules # >> module(load="imudp" TimeRequery="5" BatchSize="64") >> >> # rsyslog Templates # >> template(name="testMachineHeader" type="string" >> string="%TIMESTAMP:::date-rfc3164% testMachine %rawmsg:::drop-last-lf%\n") >> template(name="rawTemplate" type="string" >> string="%rawmsg:::drop-last-lf%\n") >> >> # rsyslog Input Modules # >> input(type="imudp" >> port="10514" >> ruleset="forwardToDestRule" >> device="eth0" >> ) >> >> >> >> # rsyslog RuleSets # >> ruleset(name="forwardToDestRule" >> queue.type="fixedArray" >> queue.size="25000" >> ) { >> if ($msg contains "interface=inbound" and $msg contains "source=10.1.1.1") >> then { >> action(type="omfwd" >> Target="10.1.1.5" >> Port="514" >> Protocol="tcp" >> Device="eth0" >> queue.type="fixedArray" >> queue.size="50000" >> queue.dequeueBatchSize="1024" >> template="testMachineHeader") >> } >> else{ >> action(type="omfwd" >> Target="10.1.1.6" >> Port="514" >> Protocol="udp" >> Device="eth0" >> queue.type="fixedArray" >> queue.size="50000" >> action.resumeRetryCount="-1" >> template="rawTemplate") >> } >> >> if ($msg contains "interface=outbound" and $msg contains >> "source=10.1.1.1") >> then { >> if ($msg contains "proto=17") then { >> action(type="omfwd" >> Target="10.1.1.7" >> Port="514" >> Protocol="udp" >> Device="eth0" >> queue.type="linkedlist" >> queue.size="50000" >> action.resumeRetryCount="-1" >> template="rawTemplate") >> } >> } >> >> } >> >> ------------------------------------------------- >> >> Top -H output: >> >> top - 08:53:26 up 1:55, 1 user, load average: 2.32, 1.50, 1.07 >> Threads: 112 total, 7 running, 105 sleeping, 0 stopped, 0 zombie >> %Cpu(s): 23.4 us, 53.5 sy, 0.0 ni, 3.5 id, 0.4 wa, 0.0 hi, 19.1 si, >> 0.0 st >> KiB Mem : 7972668 total, 7475888 free, 250088 used, 246692 >> buff/cache >> KiB Swap: 4063228 total, 4063228 free, 0 used. 7449764 avail Mem >> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >> COMMAND >> 2567 root 20 0 639540 16836 12416 R 32.4 0.2 1:11.53 >> rs:action 2 que >> 2566 root 20 0 639540 16836 12416 R 30.1 0.2 1:11.32 >> rs:action 1 que >> 2551 root 20 0 639540 16836 12416 R 14.1 0.2 0:30.68 >> in:imudp >> 2565 root 20 0 639540 16836 12416 R 11.1 0.2 0:31.94 >> rs:forwardToDes >> 600 root 20 0 100676 38508 38184 S 6.2 0.5 0:33.62 >> systemd-journal >> >> ------------------------------------------------- >> >> impstats output: >> >> Sun Oct 4 08:46:49 2020: global: origin=dynstats >> Sun Oct 4 08:46:49 2020: imuxsock: origin=imuxsock submitted=0 >> ratelimit.discarded=0 ratelimit.numratelimiters=0 >> Sun Oct 4 08:46:49 2020: action 0: origin=core.action processed=0 >> failed=0 >> suspended=0 suspended.duration=0 resumed=0 >> Sun Oct 4 08:46:49 2020: action 1: origin=core.action processed=1341545 >> failed=0 suspended=0 suspended.duration=0 resumed=0 >> Sun Oct 4 08:46:49 2020: action 2: origin=core.action processed=1341545 >> failed=0 suspended=0 suspended.duration=0 resumed=0 >> Sun Oct 4 08:46:49 2020: action 3: origin=core.action processed=4022 >> failed=0 suspended=0 suspended.duration=0 resumed=0 >> Sun Oct 4 08:46:49 2020: action 4: origin=core.action processed=1 >> failed=0 >> suspended=0 suspended.duration=0 resumed=0 >> Sun Oct 4 08:46:49 2020: action 5: origin=core.action processed=0 >> failed=0 >> suspended=0 suspended.duration=0 resumed=0 >> Sun Oct 4 08:46:49 2020: action 6: origin=core.action processed=0 >> failed=0 >> suspended=0 suspended.duration=0 resumed=0 >> Sun Oct 4 08:46:49 2020: action 7: origin=core.action processed=0 >> failed=0 >> suspended=0 suspended.duration=0 resumed=0 >> Sun Oct 4 08:46:49 2020: action 8: origin=core.action processed=0 >> failed=0 >> suspended=0 suspended.duration=0 resumed=0 >> Sun Oct 4 08:46:49 2020: action 9: origin=core.action processed=0 >> failed=0 >> suspended=0 suspended.duration=0 resumed=0 >> Sun Oct 4 08:46:49 2020: imudp(*:10514): origin=imudp submitted=1341849 >> Sun Oct 4 08:46:49 2020: imudp(*:10514): origin=imudp submitted=0 >> Sun Oct 4 08:46:49 2020: resource-usage: origin=impstats utime=9190927 >> stime=25608171 maxrss=12244 minflt=166970 majflt=0 inblock=0 oublock=4368 >> nvcsw=37503 nivcsw=339 >> Sun Oct 4 08:46:49 2020: action 0 queue: origin=core.queue size=0 >> enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0 >> Sun Oct 4 08:46:49 2020: action 1 queue: origin=core.queue size=0 >> enqueued=1341545 full=0 discarded.full=0 discarded.nf=0 maxqsize=5227 >> Sun Oct 4 08:46:49 2020: action 2 queue: origin=core.queue size=252 >> enqueued=1341545 full=0 discarded.full=0 discarded.nf=0 maxqsize=6051 >> Sun Oct 4 08:46:49 2020: forwardToDestRule: origin=core.queue size=304 >> enqueued=1341849 full=0 discarded.full=0 discarded.nf=0 maxqsize=1003 >> Sun Oct 4 08:46:49 2020: main Q: origin=core.queue size=0 enqueued=4023 >> full=0 discarded.full=0 discarded.nf=0 maxqsize=64 >> Sun Oct 4 08:46:49 2020: imudp(w0): origin=imudp called.recvmmsg=40859 >> called.recvmsg=0 msgs.received=1341849 >> >> ------------------------------------------------- >> >> >> Regards, >> Scorsese P. >> _______________________________________________ >> rsyslog mailing list >> https://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > > > -- > Yury Bushmelev > _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
