Hi Rainer,
It does not look like the central rate-limiting (by adjusting
SysSock.RateLimit.Interval & Burst applied to the imuxsock module) also
controls the messages generated by rsyslogd/omfwd - is the rate-limiting for
this done separately? Whilst the journal does have this: "rsyslogd:
rsyslogd[internal_messages]: 573 messages lost due to rate-limiting", I could
not find a way of modifying (even disabling) the rate-limiting for rsyslogd
error messages.
Also the filtering of duplicated messages (using e.g. "$RepeatedMsgReduction
on") does not have an impact here, but that is to be expected as there is a set
of 4 messages (rather than just 1 message) that is repeated n times.
Before I look into the suspend/resume handling for omfwd (udp), is this
something that is configurable or hard-coded in rsyslogd?
Thanks
Mike
On 05/06/18 07:25, Rainer Gerhards wrote:
> Unfortunately, I am totally swamped with Adiscon work :-( For this
> case it would probably make most sense to actually suspend the action
> (as it does not work). Looks like this is done, but then it resume too
> quickly. That's the place to search for.
>
> The old "rate-limiter" needed to go away as it essentially lead to no
> messages at all for longer-running instances. We now have the central
> rate limiter, so we really need to utilize it. Maybe it helps for this
> use case to just enable the "Last message repeated n times"
> processing, albeit I still think this is a bad thing to have.
>
> Sorry I cannot contribute more concrete solution at the moment.
>
> Rainer
>
> 2018-06-02 1:38 GMT+02:00 David Lang <da...@lang.hm>:
>> I think we will need to wait until Rainer is back next week to comment on
>> this.
>>
>> David Lang
>>
>>
>> On Fri, 1 Jun 2018, Mike Manning wrote:
>>
>>> Hi David,
>>> Thanks for your reply. I upgraded to the latest v8.34 available on Debian
>>> Sid and reran the test. There is no backoff/delay as per the journal output
>>> enclosed between suspend & resume - the interval is a few us each time. This
>>> example shows a burst of 47 'error 101' msgs over an interval of 11ms before
>>> the interface comes up at startup.
>>>
>>> So is there some way of configuring such a backoff or delay between
>>> suspend & resume?
>>>
>>> Or alternatively, is there some way of rate-limiting just the messages
>>> originating from rsyslog, not from other components?
>>>
>>> This was not a problem in rsyslog v8.24 where only 5 such error messages
>>> were generated at startup. Would it make sense to reinstate the rough error
>>> rate-limiting in omfwd?
>>>
>>> I appreciate this is not an issue for tcp, but we would prefer to use udp.
>>>
>>> Thanks
>>> Mike
>>>
>>> On 01/06/18 00:19, David Lang wrote:
>>>> The action is suspending itself because it can't hit the network, but
>>>> immediately resuming, there should be some delay (at least after the first
>>>> retry) to prevent too much of this. Can you check the timestamps to see fi
>>>> there is a growing time gap between the 'suspended' and 'resumed' messages
>>>> over time?
>>>>
>>>> Should there be a way to adjust how agressivly rsyslog tries to resume an
>>>> action?
>>>>
>>>> David Lang
>>>>
>>>> On Thu, 31 May 2018, Mike Manning via rsyslog wrote:
>>>>
>>>>> Date: Thu, 31 May 2018 17:10:50 +0100
>>>>> From: Mike Manning via rsyslog <rsyslog@lists.adiscon.com>
>>>>> To: rsyslog@lists.adiscon.com
>>>>> Cc: Mike Manning <mvrmann...@gmail.com>
>>>>> Subject: [rsyslog] rsyslog - omfwd error message rate limiting?
>>>>>
>>>>> Hi,
>>>>> When using omfwd with udp, at startup one gets 'omfwd: error 101 sending
>>>>> via udp: Network is unreachable' errors until the interface is up. With
>>>>> tcp
>>>>> clearly one does not get these, but our preference is to use udp on
>>>>> networking devices so as to avoid a potentially large number of queued
>>>>> messages being sent after loss of connectivity, also udp is more
>>>>> performant
>>>>> albeit less reliable.
>>>>>
>>>>> This approach works with rsyslog v8.24.0 (in Debian Stretch), where
>>>>> omfwd itself rate-limits these error messages to 5. However, with rsyslog
>>>>> v8.33.1 (in Debian Sid) this rough rate-limiting was removed by the commit
>>>>> 35ae24c8a689 ("omfwd/udp: improve error reporting, depricate
>>>>> maxerrormessages parameter"), so at startup there are now for example 30
>>>>> to
>>>>> 120 of these error messages.
>>>>>
>>>>> Is there a way of rate-limiting these on client-side without
>>>>> rate-limiting other error messages?
>>>>>
>>>>> Or is there a way of consolidating the error messages to state e.g.
>>>>> 'repeated x number of times' when logging and forwarding them? I don't
>>>>> think
>>>>> this is possible though, as for each such error message there there is
>>>>> also
>>>>> a sendto() error message, i.e.:
>>>>>
>>>>> Apr 20 08:24:31 vm2 rsyslogd[468]: omfwd/udp: socket 7: sendto() error:
>>>>> Network is unreachable [v8.33.1 try http://www.rsyslog.com/e/2354 ]
>>>>> Apr 20 08:24:31 vm2 rsyslogd[468]: omfwd: socket 7: error 101 sending
>>>>> via udp: Network is unreachable [v8.33.1 try
>>>>> http://www.rsyslog.com/e/2354 ]
>>>>> Apr 20 08:24:31 vm2 rsyslogd[468]: action 'action 1' suspended (module
>>>>> 'builtin:omfwd'), retry 0. There should be messages before this one giving
>>>>> the reason for suspension. [v8.
>>>>> Apr 20 08:24:31 vm2 rsyslogd[468]: action 'action 1' resumed (module
>>>>> 'builtin:omfwd') [v8.33.1 try http://www.rsyslog.com/e/2359 ]
>>>>>
>>>>> [so this set is repeated x number of times]
>>>>>
>>>>> Thanks for any advice,
>>>>> Mike
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>>> DON'T
>>>>> LIKE THAT.
>>>>>
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>> LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
