Hi all, Thanks both David and John for your replies.
David I used your suggestion and I now have it working. Thank you so much for pointing me in the right direction! Cheers, James On Thu, Oct 22, 2020 at 1:46 PM David Lang via rsyslog < [email protected]> wrote: > you can also set mmnormalize to parse a variable > > if $programname == "CEE:" then > set $.msg="CEE:" & $msg; > else > set $.msg = $msg; > > then parse with variable="$.msg" in the mmnormalize action() statement > > David Lang > > > On Wed, 21 Oct 2020, John Chivian via rsyslog wrote: > > > Date: Wed, 21 Oct 2020 19:21:45 -0500 > > From: John Chivian via rsyslog <[email protected]> > > To: [email protected] > > Cc: John Chivian <[email protected]> > > Subject: Re: [rsyslog] Add tag before normalization > > > > One thing you could do is tell mmnormalize to use rawmsg instead of > > msg. Then just parse the pieces apart yourself. > > > > Regards, > > > > > > On 10/21/20 5:42 PM, James Fitzsimons via rsyslog wrote: > >> Hi all, > >> > >> We are trying to use mmnormalize to process incoming CEF messages from a > >> fortigate firewall. Unfortunately the fortigate doesn't seem to include > a > >> tag and so the 'CEF:' component of the message actually gets parsed as > the > >> tag. This means that mmnormalize is then unable to parse the CEF > message. > >> > >> Here is an example incoming message: > >> > >> Oct 21 03:07:16 CassiniTest > >> CEF:0|Fortinet|Fortigate|v6.4.2|40704|event:system > >> perf-stats|3|deviceExternalId=FGTAWS0-FPMC6A76 FTNTFGTlogid=0100040704 > >> cat=event:system FTNTFGTsubtype=system FTNTFGTlevel=notice > FTNTFGTvd=root > >> FTNTFGTeventtime=1603274837221415347 FTNTFGTtz=-0700 > FTNTFGTlogdesc=System > >> performance statistics act=perf-stats FTNTFGTcpu=0 FTNTFGTmem=43 > >> FTNTFGTtotalsession=12 FTNTFGTdisk=1 FTNTFGTbandwidth=1/1 > >> FTNTFGTsetuprate=0 FTNTFGTdisklograte=0 FTNTFGTfazlograte=0 > >> FTNTFGTfreediskstorage=28514 FTNTFGTsysuptime=10525 FTNTFGTwaninfo=N/A > >> msg=Performance statistics: average CPU: 0, memory: 43, concurrent > >> sessions: 12, setup-rate: 0 > >> > >> Here is one that I have modified by hand that works - I just added the > "-" > >> character to indicate a nil tag value: > >> > >> Oct 21 03:07:16 CassiniTest - > >> CEF:0|Fortinet|Fortigate|v6.4.2|40704|event:system > >> perf-stats|3|deviceExternalId=FGTAWS0-FPMC6A76 FTNTFGTlogid=0100040704 > >> cat=event:system FTNTFGTsubtype=system FTNTFGTlevel=notice > FTNTFGTvd=root > >> FTNTFGTeventtime=1603274837221415347 FTNTFGTtz=-0700 > FTNTFGTlogdesc=System > >> performance statistics act=perf-stats FTNTFGTcpu=0 FTNTFGTmem=43 > >> FTNTFGTtotalsession=12 FTNTFGTdisk=1 FTNTFGTbandwidth=1/1 > >> FTNTFGTsetuprate=0 FTNTFGTdisklograte=0 FTNTFGTfazlograte=0 > >> FTNTFGTfreediskstorage=28514 FTNTFGTsysuptime=10525 FTNTFGTwaninfo=N/A > >> msg=Performance statistics: average CPU: 0, memory: 43, concurrent > >> sessions: 12, setup-rate: 0 > >> > >> This is what my test config looks like: > >> > >> module(load="mmnormalize") > >> > >> template(name="csv" type="list") { > >> > >> # syslog properties > >> property(name="timegenerated") > >> constant(value=",") > >> property(name="hostname") > >> > >> # CSV > >> constant(value=",") > >> property(name="$!cef!DeviceVendor" format="csv") > >> constant(value=",") > >> property(name="$!cef!DeviceProduct" format="csv") > >> constant(value=",") > >> property(name="$!cef!DeviceVersion" format="csv") > >> constant(value=",") > >> property(name="$!cef!SignatureID" format="csv") > >> constant(value=",") > >> property(name="$!cef!Name" format="csv") > >> constant(value=",") > >> property(name="$!cef!Severity" format="csv") > >> constant(value=",") > >> property(name="$!cef!Extensions!act" format="csv") > >> constant(value="\n") > >> } > >> > >> ruleset(name="cef_ruleset") { > >> action(type="mmnormalize" ruleBase="/etc/rsyslog.d/cef.rb" > useRawMsg="off") > >> action(type="omfile" dirCreateMode="0700" FileCreateMode="0644" > >> File="/var/log/fortigate/parsed.log" template="csv") > >> } > >> input(type="imtcp" port="514" ruleset="cef_ruleset") > >> > >> > >> Is there an action I can use to add the missing tag field? I did look at > >> the mmtaghostname module but it's not in the ubuntu or centos builds > (and > >> looks like it won't be added). > >> > >> Many thanks, > >> James > >> _______________________________________________ > >> rsyslog mailing list > >> https://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad of > > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T > > LIKE THAT. > > > > > > _______________________________________________ > > rsyslog mailing list > > https://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of > > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T > > LIKE THAT. > _______________________________________________ > rsyslog mailing list > https://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
