Hi Rainer, It does not look like the central rate-limiting (by adjusting SysSock.RateLimit.Interval & Burst applied to the imuxsock module) also controls the messages generated by rsyslogd/omfwd - is the rate-limiting for this done separately? Whilst the journal does have this: "rsyslogd: rsyslogd[internal_messages]: 573 messages lost due to rate-limiting", I could not find a way of modifying (even disabling) the rate-limiting for rsyslogd error messages.
Also the filtering of duplicated messages (using e.g. "$RepeatedMsgReduction on") does not have an impact here, but that is to be expected as there is a set of 4 messages (rather than just 1 message) that is repeated n times. Before I look into the suspend/resume handling for omfwd (udp), is this something that is configurable or hard-coded in rsyslogd? Thanks Mike On 05/06/18 07:25, Rainer Gerhards wrote: > Unfortunately, I am totally swamped with Adiscon work :-( For this > case it would probably make most sense to actually suspend the action > (as it does not work). Looks like this is done, but then it resume too > quickly. That's the place to search for. > > The old "rate-limiter" needed to go away as it essentially lead to no > messages at all for longer-running instances. We now have the central > rate limiter, so we really need to utilize it. Maybe it helps for this > use case to just enable the "Last message repeated n times" > processing, albeit I still think this is a bad thing to have. > > Sorry I cannot contribute more concrete solution at the moment. > > Rainer > > 2018-06-02 1:38 GMT+02:00 David Lang <da...@lang.hm>: >> I think we will need to wait until Rainer is back next week to comment on >> this. >> >> David Lang >> >> >> On Fri, 1 Jun 2018, Mike Manning wrote: >> >>> Hi David, >>> Thanks for your reply. I upgraded to the latest v8.34 available on Debian >>> Sid and reran the test. There is no backoff/delay as per the journal output >>> enclosed between suspend & resume - the interval is a few us each time. This >>> example shows a burst of 47 'error 101' msgs over an interval of 11ms before >>> the interface comes up at startup. >>> >>> So is there some way of configuring such a backoff or delay between >>> suspend & resume? >>> >>> Or alternatively, is there some way of rate-limiting just the messages >>> originating from rsyslog, not from other components? >>> >>> This was not a problem in rsyslog v8.24 where only 5 such error messages >>> were generated at startup. Would it make sense to reinstate the rough error >>> rate-limiting in omfwd? >>> >>> I appreciate this is not an issue for tcp, but we would prefer to use udp. >>> >>> Thanks >>> Mike >>> >>> On 01/06/18 00:19, David Lang wrote: >>>> The action is suspending itself because it can't hit the network, but >>>> immediately resuming, there should be some delay (at least after the first >>>> retry) to prevent too much of this. Can you check the timestamps to see fi >>>> there is a growing time gap between the 'suspended' and 'resumed' messages >>>> over time? >>>> >>>> Should there be a way to adjust how agressivly rsyslog tries to resume an >>>> action? >>>> >>>> David Lang >>>> >>>> On Thu, 31 May 2018, Mike Manning via rsyslog wrote: >>>> >>>>> Date: Thu, 31 May 2018 17:10:50 +0100 >>>>> From: Mike Manning via rsyslog <rsyslog@lists.adiscon.com> >>>>> To: rsyslog@lists.adiscon.com >>>>> Cc: Mike Manning <mvrmann...@gmail.com> >>>>> Subject: [rsyslog] rsyslog - omfwd error message rate limiting? >>>>> >>>>> Hi, >>>>> When using omfwd with udp, at startup one gets 'omfwd: error 101 sending >>>>> via udp: Network is unreachable' errors until the interface is up. With >>>>> tcp >>>>> clearly one does not get these, but our preference is to use udp on >>>>> networking devices so as to avoid a potentially large number of queued >>>>> messages being sent after loss of connectivity, also udp is more >>>>> performant >>>>> albeit less reliable. >>>>> >>>>> This approach works with rsyslog v8.24.0 (in Debian Stretch), where >>>>> omfwd itself rate-limits these error messages to 5. However, with rsyslog >>>>> v8.33.1 (in Debian Sid) this rough rate-limiting was removed by the commit >>>>> 35ae24c8a689 ("omfwd/udp: improve error reporting, depricate >>>>> maxerrormessages parameter"), so at startup there are now for example 30 >>>>> to >>>>> 120 of these error messages. >>>>> >>>>> Is there a way of rate-limiting these on client-side without >>>>> rate-limiting other error messages? >>>>> >>>>> Or is there a way of consolidating the error messages to state e.g. >>>>> 'repeated x number of times' when logging and forwarding them? I don't >>>>> think >>>>> this is possible though, as for each such error message there there is >>>>> also >>>>> a sendto() error message, i.e.: >>>>> >>>>> Apr 20 08:24:31 vm2 rsyslogd[468]: omfwd/udp: socket 7: sendto() error: >>>>> Network is unreachable [v8.33.1 try http://www.rsyslog.com/e/2354 ] >>>>> Apr 20 08:24:31 vm2 rsyslogd[468]: omfwd: socket 7: error 101 sending >>>>> via udp: Network is unreachable [v8.33.1 try >>>>> http://www.rsyslog.com/e/2354 ] >>>>> Apr 20 08:24:31 vm2 rsyslogd[468]: action 'action 1' suspended (module >>>>> 'builtin:omfwd'), retry 0. There should be messages before this one giving >>>>> the reason for suspension. [v8. >>>>> Apr 20 08:24:31 vm2 rsyslogd[468]: action 'action 1' resumed (module >>>>> 'builtin:omfwd') [v8.33.1 try http://www.rsyslog.com/e/2359 ] >>>>> >>>>> [so this set is repeated x number of times] >>>>> >>>>> Thanks for any advice, >>>>> Mike >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com/professional-services/ >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>> DON'T >>>>> LIKE THAT. >>>>> >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.