Logstash needs something like redis because it can't do any queueing
itself. Rsyslog is built around queues, and has the ability to
create multiple queues and piplines internally, you don't need to
run multiple instances.
I want multiples instances in order to:
* Being able to process pipelines on different containers/hosts
much less needed on rsyslog due to the higher effiency. I've had
rsyslog handling over a hundred thousand logs/sec on a single host.
This is our current scenario (each element deployed within a docker
container):
logs-->RELP-->rsyslog-->redis-->logstash_app_1/N...
This allow us to have multiple simpler configurations for logstash,
splitting traffic between multiple workers/containers on different
hosts, high availability, load balancing...
* Isolate pipelines to prevent problems on one affecting others
rulesets with queues on each ruleset solvs this for you.
One segfault while processing one ruleset/action (actually, it happened
a lot with 8.22) crash the whole process.
All processing from that point on will take place in different
threads working on different queues for each category.
Will I be able to "reload" rsyslog configuration to add/delete new
rulesets/pipelines?
you can stop/start rsyslog, but there is not a way to change the
config on the fly.
:(
However, if you really want to go this way, one thing you can do is to
make use of the multicast mac feature in ethernet to distribute the
same logs to multiple systems/containers and have each container throw
away all logs except what it's configured to handle.
This lets you add/remove log processing at any time and even have
multiple systems processing the same logs in different ways
https://www.usenix.org/conference/lisa12/technical-sessions/presentation/lang_david
Network traffic x2
Actually, we are using a similar environment for other things, but I
don't think that's the way to go.
KISS, start simple and only add complexity when you find it's actually
needed. Have plans for how to scale out when you hit limits, but you
usually find that you hit limits far later than expected. Yes, you may
have to eventually do the same work, but by having a solid system now
with less work, you can spend the time saved now to improve other things.
KISS is great, but we are looking to build a dynamic pipeline, and we
found rsyslog is close to be the proper tool, with a couple of changes!
Somehow related with Rainer's new file reader proposal, I think a
rsyslog code review/refactor will help with this.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
