On 04/12/2016 11:31 AM, singh.janmejay wrote:
Im curious, why would you want to do that?
https://github.com/openshift/origin-aggregated-logging#throttling-logs-in-fluentd
As a service that takes those logs and does something with it (store
it, index it etc), you want to absorb acceptable bursts as soon as
possible.
Unless you don't.
If it is an unacceptably big burst, you are better off
dropping messages and catching up with the latest set of messages
(because when someone debugs a "current" problem, the latest logs are
the most useful, and in case of a very large burst current messages
end up waiting to ingest way to long).
Ok, but that's not the way the solution works now, that I'm proposing to
use rsyslog for instead.
In our case, we allow bursts upto x% (using pre-planned burst buffer)
and discard messages when that burst-limit is breached. But we do it
at the service's ingestion-frontend(an array of Rsyslog receivers,
slurping logs in over tcp), so it acts as a gate-keeper to the
service. This allows service users to burst at individual nodes level,
as long as their cluster overall stays within planned-capacity +
burst-buffer.
On Tue, Apr 12, 2016 at 10:26 PM, David Lang <[email protected]> wrote:
On Tue, 12 Apr 2016, Rich Megginson wrote:
I see there is rate limiting for certain types of input
http://blog.gerhards.net/2012/10/rate-limiting-in-rsyslog-732.htm
<http://blog.gerhards.net/2012/10/rate-limiting-in-rsyslog-732.html>
Is it possible to do rate limiting on imfile?
I tried this with rsyslog v 8.12:
input(name="imfile" file="/var/log/mybadapp.log" ratelimit.burst="100")
rsyslogd -N 1 says "parameter ratelimit.burst not known"
yep, that parameter doesn't exist for imfile, look at
http://www.rsyslog.com/doc/v8-stable/configuration/modules/imfile.html for
your options.
it does have the batch size, but there is nothing that limit the rate that
it processes batches.
David Lang
I'm trying to do something like I can do with fluentd - in a file input I
can specify read_lines_limit
I suppose I could do something with the input queue, but I want to do rate
limiting before the log data even gets to the queue - I want to read slowly
from the file.
I could also do some extra-process tricks, like putting some sort of named
pipe script intermediary between the file and rsyslog, and have rsyslog read
from the pipe instead, and have the script do the rate limiting, but that's
an "inelegant" solution.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
