+1
Our current scenario (dockerized!):
imfile_forwarder-->imrelp-->rsyslog-->redis-->logstash(grok+geoip)-->elastic
We are using redis as memory buffer and to split into multiple
channels/lists (using dynakey ATM). We see kafka on the horizon.
We are also using several logstash containers to balance load, prevent
single point of failure, etc.
What we're thinking after past days messages:
imfile_forwarder-->imrelp-->rsyslog-->elastic
Having multiple rsyslog instances with simpler configs (instead of 5k
lines with thousand of rulesets, templates and so), being able to geoip,
reliable queues...
I wont dare to say it's time to review/refactor rsyslog, but
maybe...https://www.youtube.com/watch?v=0O5h4enjrHw
El 23/11/16 a las 12:52, Bob Gregory escribió:
There've been a few discussions over the last few days that are all
pointing in the same direction:
* Is it better to use Rsyslog's omelasticsearch rather than pushing to
logstash?
* Should we have a minimal log shipper component as distinct from rsyslog's
processing capabilities?
* Ought we to have an imhiredis module?
Really what we're talking about is replacing Logstash (and the various
beats) with rsyslog. I'm perfectly happy with that, Logstash is a
resource-expensive and fickle beast that spoils my otherwise pristine log
pipeline, but I do think the community ought to think about whether this is
the direction they want to take.
For my part, I'm quite happy to help build an imhiredis (and imkafka?)
module but only if I can actually dogfood it, which means replacing
Logstash in our own environment.
For that, I'd like to see better support for GeoIP tagging, a Riemann
output plugin, some better guidance on "failed message queues", etc. etc.
etc.
Are we jointly interested in building the REK stack and, if so, can we
start to work out the feature set we're missing, and the documentation we'd
need for this to work? I'm a little concerned that if we tackle the usecase
piece-meal, we'll end up with lots of disjointed parts that don't really
solve the problem: logstash is not an adequate logstash.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
