Hello! I guess yes, you should be able to use the property replacer. Though it may be less performant than liblognorm-based.
On Mon, 5 Oct 2020 at 10:52, Kype Ahamed <[email protected]> wrote: > Hi Yuri > > Unfortunately mmfields module is not installed for our RedHat machines, > and these are air gapped machines which cannot download the package. > > Would it be possible to use property replacer technique to parse out the > field? > > > Regards, > Scorsese P. > > On Sun, Oct 4, 2020 at 11:12 PM Yuri Bushmelev <[email protected]> wrote: > >> Hello! >> >> From what I see you have 5 conditions. 4 of them doing full-scan of $msg >> on every incoming message. What I'd suggest is to parse the message first >> using a `mmnormalize`/`mmfields`/`mmpstructdata` module. Then you can refer >> to fields extracted in your conditions instead to prevent full message scan. >> >> I can guess you may be using the iptables message format. So you may >> check this liblognorm field type: >> https://www.liblognorm.com/files/manual/configuration.html#iptables >> >> >> On Sun, 4 Oct 2020 at 21:30, Kype Ahamed via rsyslog < >> [email protected]> wrote: >> >>> Hello, >>> >>> I am using rsyslog mainly as a syslog relay, to forward messages from 1 >>> source device to multiple destination devices. >>> >>> Right now i am receiving about 50k messages per second, and i noticed CPU >>> usage is constantly above 80% >>> >>> Are there any further tweaks that can be done to below config to reduce >>> the >>> CPU usage? >>> >>> ------------------------------------------------- >>> rsyslog config file: >>> >>> module(load="impstats" >>> interval="20" >>> severity="7" >>> log.syslog="off" >>> log.file="/var/log/impstats.log") >>> >>> global(parser.escapecontrolcharactertab="off") >>> >>> # Load Modules # >>> module(load="imudp" TimeRequery="5" BatchSize="64") >>> >>> # rsyslog Templates # >>> template(name="testMachineHeader" type="string" >>> string="%TIMESTAMP:::date-rfc3164% testMachine >>> %rawmsg:::drop-last-lf%\n") >>> template(name="rawTemplate" type="string" >>> string="%rawmsg:::drop-last-lf%\n") >>> >>> # rsyslog Input Modules # >>> input(type="imudp" >>> port="10514" >>> ruleset="forwardToDestRule" >>> device="eth0" >>> ) >>> >>> >>> >>> # rsyslog RuleSets # >>> ruleset(name="forwardToDestRule" >>> queue.type="fixedArray" >>> queue.size="25000" >>> ) { >>> if ($msg contains "interface=inbound" and $msg contains >>> "source=10.1.1.1") >>> then { >>> action(type="omfwd" >>> Target="10.1.1.5" >>> Port="514" >>> Protocol="tcp" >>> Device="eth0" >>> queue.type="fixedArray" >>> queue.size="50000" >>> queue.dequeueBatchSize="1024" >>> template="testMachineHeader") >>> } >>> else{ >>> action(type="omfwd" >>> Target="10.1.1.6" >>> Port="514" >>> Protocol="udp" >>> Device="eth0" >>> queue.type="fixedArray" >>> queue.size="50000" >>> action.resumeRetryCount="-1" >>> template="rawTemplate") >>> } >>> >>> if ($msg contains "interface=outbound" and $msg contains >>> "source=10.1.1.1") >>> then { >>> if ($msg contains "proto=17") then { >>> action(type="omfwd" >>> Target="10.1.1.7" >>> Port="514" >>> Protocol="udp" >>> Device="eth0" >>> queue.type="linkedlist" >>> queue.size="50000" >>> action.resumeRetryCount="-1" >>> template="rawTemplate") >>> } >>> } >>> >>> } >>> >>> ------------------------------------------------- >>> >>> Top -H output: >>> >>> top - 08:53:26 up 1:55, 1 user, load average: 2.32, 1.50, 1.07 >>> Threads: 112 total, 7 running, 105 sleeping, 0 stopped, 0 zombie >>> %Cpu(s): 23.4 us, 53.5 sy, 0.0 ni, 3.5 id, 0.4 wa, 0.0 hi, 19.1 si, >>> 0.0 st >>> KiB Mem : 7972668 total, 7475888 free, 250088 used, 246692 >>> buff/cache >>> KiB Swap: 4063228 total, 4063228 free, 0 used. 7449764 avail >>> Mem >>> >>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >>> COMMAND >>> 2567 root 20 0 639540 16836 12416 R 32.4 0.2 1:11.53 >>> rs:action 2 que >>> 2566 root 20 0 639540 16836 12416 R 30.1 0.2 1:11.32 >>> rs:action 1 que >>> 2551 root 20 0 639540 16836 12416 R 14.1 0.2 0:30.68 >>> in:imudp >>> 2565 root 20 0 639540 16836 12416 R 11.1 0.2 0:31.94 >>> rs:forwardToDes >>> 600 root 20 0 100676 38508 38184 S 6.2 0.5 0:33.62 >>> systemd-journal >>> >>> ------------------------------------------------- >>> >>> impstats output: >>> >>> Sun Oct 4 08:46:49 2020: global: origin=dynstats >>> Sun Oct 4 08:46:49 2020: imuxsock: origin=imuxsock submitted=0 >>> ratelimit.discarded=0 ratelimit.numratelimiters=0 >>> Sun Oct 4 08:46:49 2020: action 0: origin=core.action processed=0 >>> failed=0 >>> suspended=0 suspended.duration=0 resumed=0 >>> Sun Oct 4 08:46:49 2020: action 1: origin=core.action processed=1341545 >>> failed=0 suspended=0 suspended.duration=0 resumed=0 >>> Sun Oct 4 08:46:49 2020: action 2: origin=core.action processed=1341545 >>> failed=0 suspended=0 suspended.duration=0 resumed=0 >>> Sun Oct 4 08:46:49 2020: action 3: origin=core.action processed=4022 >>> failed=0 suspended=0 suspended.duration=0 resumed=0 >>> Sun Oct 4 08:46:49 2020: action 4: origin=core.action processed=1 >>> failed=0 >>> suspended=0 suspended.duration=0 resumed=0 >>> Sun Oct 4 08:46:49 2020: action 5: origin=core.action processed=0 >>> failed=0 >>> suspended=0 suspended.duration=0 resumed=0 >>> Sun Oct 4 08:46:49 2020: action 6: origin=core.action processed=0 >>> failed=0 >>> suspended=0 suspended.duration=0 resumed=0 >>> Sun Oct 4 08:46:49 2020: action 7: origin=core.action processed=0 >>> failed=0 >>> suspended=0 suspended.duration=0 resumed=0 >>> Sun Oct 4 08:46:49 2020: action 8: origin=core.action processed=0 >>> failed=0 >>> suspended=0 suspended.duration=0 resumed=0 >>> Sun Oct 4 08:46:49 2020: action 9: origin=core.action processed=0 >>> failed=0 >>> suspended=0 suspended.duration=0 resumed=0 >>> Sun Oct 4 08:46:49 2020: imudp(*:10514): origin=imudp submitted=1341849 >>> Sun Oct 4 08:46:49 2020: imudp(*:10514): origin=imudp submitted=0 >>> Sun Oct 4 08:46:49 2020: resource-usage: origin=impstats utime=9190927 >>> stime=25608171 maxrss=12244 minflt=166970 majflt=0 inblock=0 oublock=4368 >>> nvcsw=37503 nivcsw=339 >>> Sun Oct 4 08:46:49 2020: action 0 queue: origin=core.queue size=0 >>> enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0 >>> Sun Oct 4 08:46:49 2020: action 1 queue: origin=core.queue size=0 >>> enqueued=1341545 full=0 discarded.full=0 discarded.nf=0 maxqsize=5227 >>> Sun Oct 4 08:46:49 2020: action 2 queue: origin=core.queue size=252 >>> enqueued=1341545 full=0 discarded.full=0 discarded.nf=0 maxqsize=6051 >>> Sun Oct 4 08:46:49 2020: forwardToDestRule: origin=core.queue size=304 >>> enqueued=1341849 full=0 discarded.full=0 discarded.nf=0 maxqsize=1003 >>> Sun Oct 4 08:46:49 2020: main Q: origin=core.queue size=0 enqueued=4023 >>> full=0 discarded.full=0 discarded.nf=0 maxqsize=64 >>> Sun Oct 4 08:46:49 2020: imudp(w0): origin=imudp called.recvmmsg=40859 >>> called.recvmsg=0 msgs.received=1341849 >>> >>> ------------------------------------------------- >>> >>> >>> Regards, >>> Scorsese P. >>> _______________________________________________ >>> rsyslog mailing list >>> https://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >> >> >> -- >> Yury Bushmelev >> > -- Yury Bushmelev _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
