The legacy code for this very old style "message repeated n times" does not consider rfc5424 structured data, so the empty MSG field makes the messages all equal.
As David said, that feature is not useful any longer today. This is why it is turned off by default. HTH Rainer Sent from phone, thus brief. Marki via rsyslog <[email protected]> schrieb am Fr., 11. Okt. 2019, 21:41: > I have rechecked it... as noted before, this "message repeated" string > appears nowhere in the packet capture I took. Also the source is not > repeating any of those messages. It must be rsyslog thinking there are > duplicates and inserting this message. > > What would the empty square brackets trailing the message mean? > > The interesting thing is that it indeed only happens with the messages > from Checkpoint Smartcenter / Log Exporter. What's special about it is > that it's the only source in our setup sending RFC5424-style messages > containing a structured data field. > > The general interesting thing in our setup is that we're using multiple > queues. > > Was there any bug of consequence somehow related to all of this in or > after 8.33.1 (which is the latest one available in SLES)? > > Thanks. > > On 10/11/2019 8:37 PM, David Lang wrote: > > repeated message reduction is not recommended. It is a feature that > > was useful back when it was only people looking at the logs, but when > > you are analyzing the logs with automation, the 'message repeated' > > logs reduce your accuracy, not improve it. > > > > that 'message repeated' is happening on the sender, so look to fix it > > there (in rsyslog we started having the message repeated log include > > the log being repeated so that it was easier to track what's happening) > > > > David Lang > > > > On Fri, 11 Oct 2019, Marki via rsyslog wrote: > > > >> Date: Fri, 11 Oct 2019 19:14:27 +0200 > >> From: Marki via rsyslog <[email protected]> > >> To: [email protected] > >> Cc: Marki <[email protected]> > >> Subject: [rsyslog] Rsyslog seems to think there are duplicates > >> > >> > >> Hey, > >> > >> We are using Checkpoint Log Exporter > >> ( > https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk122323) > > >> I don't think that it matters, except maybe for the fact that they > >> are implementing RFC5424. > >> > >> The logs are being written to file by Rsyslog as follows: > >> > >> --------------------- > >> module(load="imtcp") > >> module(load="imudp") > >> > >> input(type="imtcp" port="514") > >> input(type="imudp" port="514") > >> > >> template(name="FileNameByHost" type="string" > >> string="/syslog/%fromhost-ip%.log") > >> if ($fromhost-ip startswith "192.168.x.y") then { # RFC5424 > >> action(type="omfile" dynaFile="FileNameByHost" > >> template="RSYSLOG_SyslogProtocol23Format") <------------------- > >> } else { > >> action(type="omfile" dynaFile="FileNameByHost" > >> template="RSYSLOG_FileFormat") > >> } > >> > >> call asyncelk > >> ... > >> --------------------- > >> > >> Now Rsyslog thinks some messages are duplicates. And apparently it > >> thinks there are _many_ duplicates. I have sniffed the traffic, and > >> no duplicates are being transmitted AFAICS. Here is an example of > >> what is logged nevertheless: > >> > >> <134>1 2019-10-11T16:27:51Z chkpt-mgt CheckPoint 15439 - > >> [action:"Reject"; ifdir:"inbound"; ifname:"eth2.53"; logid:"0"; > >> loguid:"{0x0,0x0,0x0,0x0}"; origin:"192.168.1.2"; > >> originsicname:"xxxxx"; sequencenum:"32"; time:"1570811271"; > >> version:"5"; dst:"224.0.0.18"; hll_key:"2008789500278145398"; > >> inzone:"Internal"; layer_name:"Unified"; layer_uuid:"a-b-c-d-e"; > >> match_id:"177"; parent_rule:"0"; rule_action:"Reject"; > >> rule_name:"xyz"; rule_uid:"a-b-c-d-e"; product:"VPN-1 & FireWall-1"; > >> proto:"51"; service_id:"AH"; src:"1.2.3.4"; ] message repeated 49 > >> times: [] > >> > >> Other times it says "99 times" or "149 times", ... I've created somes > >> stats in fact :D > >> > >> # tail -n 10000 //syslog/1.2.3.4.log | grep -o -E 'repeated [0-9]+ > >> times' | sort | uniq -c | sort -n | tail -10 > >> 23 repeated 44 times > >> 26 repeated 199 times > >> 35 repeated 159 times > >> 37 repeated 9 times > >> 38 repeated 4 times > >> 101 repeated 149 times > >> 155 repeated 109 times > >> 408 repeated 59 times > >> 417 repeated 99 times > >> 2926 repeated 49 times > >> > >> This is SLES 15-SP1 running rsyslog-8.33.1-3.17.1.x86_64 > >> > >> Any idea? > >> > >> Thanks, > >> Marki > >> > >> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > >> POST if you DON'T LIKE THAT. > >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
