Im curious, why would you want to do that? As a service that takes those logs and does something with it (store it, index it etc), you want to absorb acceptable bursts as soon as possible. If it is an unacceptably big burst, you are better off dropping messages and catching up with the latest set of messages (because when someone debugs a "current" problem, the latest logs are the most useful, and in case of a very large burst current messages end up waiting to ingest way to long).
In our case, we allow bursts upto x% (using pre-planned burst buffer) and discard messages when that burst-limit is breached. But we do it at the service's ingestion-frontend(an array of Rsyslog receivers, slurping logs in over tcp), so it acts as a gate-keeper to the service. This allows service users to burst at individual nodes level, as long as their cluster overall stays within planned-capacity + burst-buffer. On Tue, Apr 12, 2016 at 10:26 PM, David Lang <da...@lang.hm> wrote: > On Tue, 12 Apr 2016, Rich Megginson wrote: > >> >> I see there is rate limiting for certain types of input >> http://blog.gerhards.net/2012/10/rate-limiting-in-rsyslog-732.htm >> <http://blog.gerhards.net/2012/10/rate-limiting-in-rsyslog-732.html> >> Is it possible to do rate limiting on imfile? >> I tried this with rsyslog v 8.12: >> >> input(name="imfile" file="/var/log/mybadapp.log" ratelimit.burst="100") >> rsyslogd -N 1 says "parameter ratelimit.burst not known" > > > yep, that parameter doesn't exist for imfile, look at > http://www.rsyslog.com/doc/v8-stable/configuration/modules/imfile.html for > your options. > > it does have the batch size, but there is nothing that limit the rate that > it processes batches. > > David Lang > > >> I'm trying to do something like I can do with fluentd - in a file input I >> can specify read_lines_limit >> >> I suppose I could do something with the input queue, but I want to do rate >> limiting before the log data even gets to the queue - I want to read slowly >> from the file. >> >> I could also do some extra-process tricks, like putting some sort of named >> pipe script intermediary between the file and rsyslog, and have rsyslog read >> from the pipe instead, and have the script do the rate limiting, but that's >> an "inelegant" solution. >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. -- Regards, Janmejay http://codehunk.wordpress.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
