On Wed, 23 Nov 2016, [email protected] wrote:
Our current scenario (dockerized!):
imfile_forwarder-->imrelp-->rsyslog-->redis-->logstash(grok+geoip)-->elastic
We are using redis as memory buffer and to split into multiple channels/lists
(using dynakey ATM). We see kafka on the horizon.
We are also using several logstash containers to balance load, prevent single
point of failure, etc.
What we're thinking after past days messages:
imfile_forwarder-->imrelp-->rsyslog-->elastic
Having multiple rsyslog instances with simpler configs (instead of 5k lines
with thousand of rulesets, templates and so), being able to geoip, reliable
queues...
I wont dare to say it's time to review/refactor rsyslog, but
maybe...https://www.youtube.com/watch?v=0O5h4enjrHw
there are probably ways to simplify the configs, 5K lines of configs seems
excessive :-) how much of this is rulebase config vs rsyslog config?
Rsyslog is designed to be fast and supports a lot of threading options for speed
(most defined implicitly by the creation of queues), so you should not need to
have lots of different instances.
I've had single instances of rsyslog processing 100K messages/sec in real-world
use, and people have benchmarked rsyslog with simple configs at over 1M
messages/sec in a VM
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
