2016-11-23 13:19 GMT+01:00 [email protected] <[email protected]>: > +1 > > > Our current scenario (dockerized!): > > > imfile_forwarder-->imrelp-->rsyslog-->redis-->logstash(grok+geoip)-->elastic > > We are using redis as memory buffer and to split into multiple > channels/lists (using dynakey ATM). We see kafka on the horizon. > > We are also using several logstash containers to balance load, prevent > single point of failure, etc. > > What we're thinking after past days messages: > > imfile_forwarder-->imrelp-->rsyslog-->elastic > > Having multiple rsyslog instances with simpler configs (instead of 5k lines > with thousand of rulesets, templates and so), being able to geoip, reliable > queues... > > I wont dare to say it's time to review/refactor rsyslog, but > maybe...https://www.youtube.com/watch?v=0O5h4enjrHw >
refactoring per se is not a problem, we just need to keep it in managable pieces. We had big refactoring almost every year :-) Rainer > > El 23/11/16 a las 12:52, Bob Gregory escribió: > >> There've been a few discussions over the last few days that are all >> pointing in the same direction: >> >> * Is it better to use Rsyslog's omelasticsearch rather than pushing to >> logstash? >> * Should we have a minimal log shipper component as distinct from >> rsyslog's >> processing capabilities? >> * Ought we to have an imhiredis module? >> >> Really what we're talking about is replacing Logstash (and the various >> beats) with rsyslog. I'm perfectly happy with that, Logstash is a >> resource-expensive and fickle beast that spoils my otherwise pristine log >> pipeline, but I do think the community ought to think about whether this >> is >> the direction they want to take. >> >> For my part, I'm quite happy to help build an imhiredis (and imkafka?) >> module but only if I can actually dogfood it, which means replacing >> Logstash in our own environment. >> >> For that, I'd like to see better support for GeoIP tagging, a Riemann >> output plugin, some better guidance on "failed message queues", etc. etc. >> etc. >> >> Are we jointly interested in building the REK stack and, if so, can we >> start to work out the feature set we're missing, and the documentation >> we'd >> need for this to work? I'm a little concerned that if we tackle the >> usecase >> piece-meal, we'll end up with lots of disjointed parts that don't really >> solve the problem: logstash is not an adequate logstash. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
