ERK +1, I have published my experiement at http://www.slideshare.net/chenryn/elk-stack-at-weibocom rsyslog-imsock -> rsyslog-omfwd -> rsyslog-imptcp -> rsyslog-mmnormalize/rsyslog-mmgrok/rsyslog-mmdblookup/rsyslog-mmfields/rainerscripts... -> rsyslog-omkafka -> kafka -> hangout -> es cluster -> rsyslog-omprog -> python scripts -> zabbix
I had open my rsyslog-mmdblookup for geoip2 lookup, then david lang tell me this can be done with lookup_table function. I think there should be a good article about this great function and geoip lookup practice. 2016-11-23 21:20 GMT+08:00 Rainer Gerhards <[email protected]>: > 2016-11-23 13:19 GMT+01:00 [email protected] <[email protected]>: > > +1 > > > > > > Our current scenario (dockerized!): > > > > > > imfile_forwarder-->imrelp-->rsyslog-->redis-->logstash( > grok+geoip)-->elastic > > > > We are using redis as memory buffer and to split into multiple > > channels/lists (using dynakey ATM). We see kafka on the horizon. > > > > We are also using several logstash containers to balance load, prevent > > single point of failure, etc. > > > > What we're thinking after past days messages: > > > > imfile_forwarder-->imrelp-->rsyslog-->elastic > > > > Having multiple rsyslog instances with simpler configs (instead of 5k > lines > > with thousand of rulesets, templates and so), being able to geoip, > reliable > > queues... > > > > I wont dare to say it's time to review/refactor rsyslog, but > > maybe...https://www.youtube.com/watch?v=0O5h4enjrHw > > > > refactoring per se is not a problem, we just need to keep it in > managable pieces. We had big refactoring almost every year :-) > > Rainer > > > > > El 23/11/16 a las 12:52, Bob Gregory escribió: > > > >> There've been a few discussions over the last few days that are all > >> pointing in the same direction: > >> > >> * Is it better to use Rsyslog's omelasticsearch rather than pushing to > >> logstash? > >> * Should we have a minimal log shipper component as distinct from > >> rsyslog's > >> processing capabilities? > >> * Ought we to have an imhiredis module? > >> > >> Really what we're talking about is replacing Logstash (and the various > >> beats) with rsyslog. I'm perfectly happy with that, Logstash is a > >> resource-expensive and fickle beast that spoils my otherwise pristine > log > >> pipeline, but I do think the community ought to think about whether this > >> is > >> the direction they want to take. > >> > >> For my part, I'm quite happy to help build an imhiredis (and imkafka?) > >> module but only if I can actually dogfood it, which means replacing > >> Logstash in our own environment. > >> > >> For that, I'd like to see better support for GeoIP tagging, a Riemann > >> output plugin, some better guidance on "failed message queues", etc. > etc. > >> etc. > >> > >> Are we jointly interested in building the REK stack and, if so, can we > >> start to work out the feature set we're missing, and the documentation > >> we'd > >> need for this to work? I'm a little concerned that if we tackle the > >> usecase > >> piece-meal, we'll end up with lots of disjointed parts that don't really > >> solve the problem: logstash is not an adequate logstash. > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T > >> LIKE THAT. > > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of > > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > > LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
