If you really want to do it, it's possible to use omuxsock and imuxsock pair to pass the logs thru before dispatching it over. If it is replacing fluentd, with this loop it will still be a huge performance win. On Apr 12, 2016 11:24 PM, "Rich Megginson" <[email protected]> wrote:
> On 04/12/2016 11:31 AM, singh.janmejay wrote: > >> Im curious, why would you want to do that? >> > > > https://github.com/openshift/origin-aggregated-logging#throttling-logs-in-fluentd > > >> As a service that takes those logs and does something with it (store >> it, index it etc), you want to absorb acceptable bursts as soon as >> possible. >> > > Unless you don't. > > If it is an unacceptably big burst, you are better off >> dropping messages and catching up with the latest set of messages >> (because when someone debugs a "current" problem, the latest logs are >> the most useful, and in case of a very large burst current messages >> end up waiting to ingest way to long). >> > > Ok, but that's not the way the solution works now, that I'm proposing to > use rsyslog for instead. > > >> In our case, we allow bursts upto x% (using pre-planned burst buffer) >> and discard messages when that burst-limit is breached. But we do it >> at the service's ingestion-frontend(an array of Rsyslog receivers, >> slurping logs in over tcp), so it acts as a gate-keeper to the >> service. This allows service users to burst at individual nodes level, >> as long as their cluster overall stays within planned-capacity + >> burst-buffer. >> >> On Tue, Apr 12, 2016 at 10:26 PM, David Lang <[email protected]> wrote: >> >>> On Tue, 12 Apr 2016, Rich Megginson wrote: >>> >>> I see there is rate limiting for certain types of input >>>> http://blog.gerhards.net/2012/10/rate-limiting-in-rsyslog-732.htm >>>> <http://blog.gerhards.net/2012/10/rate-limiting-in-rsyslog-732.html> >>>> Is it possible to do rate limiting on imfile? >>>> I tried this with rsyslog v 8.12: >>>> >>>> input(name="imfile" file="/var/log/mybadapp.log" >>>> ratelimit.burst="100") >>>> rsyslogd -N 1 says "parameter ratelimit.burst not known" >>>> >>> >>> yep, that parameter doesn't exist for imfile, look at >>> http://www.rsyslog.com/doc/v8-stable/configuration/modules/imfile.html >>> for >>> your options. >>> >>> it does have the batch size, but there is nothing that limit the rate >>> that >>> it processes batches. >>> >>> David Lang >>> >>> >>> I'm trying to do something like I can do with fluentd - in a file input I >>>> can specify read_lines_limit >>>> >>>> I suppose I could do something with the input queue, but I want to do >>>> rate >>>> limiting before the log data even gets to the queue - I want to read >>>> slowly >>>> from the file. >>>> >>>> I could also do some extra-process tricks, like putting some sort of >>>> named >>>> pipe script intermediary between the file and rsyslog, and have rsyslog >>>> read >>>> from the pipe instead, and have the script do the rate limiting, but >>>> that's >>>> an "inelegant" solution. >>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>> DON'T >>>> LIKE THAT. >>>> >>>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of >>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >>> LIKE THAT. >>> >> >> >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
