On Tue, 22 Nov 2016, [email protected] wrote:
What sort of log volume are you talking about here? (logs/sec type of
thing)
From 0 to thousand-thousands/sec
Logstash needs something like redis because it can't do any queueing
itself. Rsyslog is built around queues, and has the ability to create
multiple queues and piplines internally, you don't need to run multiple
instances.
I want multiples instances in order to:
* Being able to process pipelines on different containers/hosts
much less needed on rsyslog due to the higher effiency. I've had rsyslog
handling over a hundred thousand logs/sec on a single host.
* Isolate pipelines to prevent problems on one affecting others
rulesets with queues on each ruleset solvs this for you.
* (others)
that's hard to answer :-)
What you would do is create a ruleset for each application (pipeline) and
give that ruleset it's own queue.
I know it can be done, but not what I'm looking for. Moreover, I would love
to be a "dynamic" configuration
As new logs arrive, you then sort them by application, and for each
application (or application category), you call the appropriate ruleset.
And, if there are a lot of evt/sec, you may have a bottleneck. I'll probably
have a rsyslog cluster based on docker swarm mode
This is unlikly to be a bottleneck. The overhead of recieving a log message,
parsing it, and looking up what ruleset to call is very cheap. At anything under
several hundred thousand logs/sec it's unlikly to max out a single core.
All processing from that point on will take place in different threads
working on different queues for each category.
Will I be able to "reload" rsyslog configuration to add/delete new
rulesets/pipelines?
you can stop/start rsyslog, but there is not a way to change the config on the
fly.
However, if you really want to go this way, one thing you can do is to make use
of the multicast mac feature in ethernet to distribute the same logs to multiple
systems/containers and have each container throw away all logs except what it's
configured to handle.
This lets you add/remove log processing at any time and even have multiple
systems processing the same logs in different ways
https://www.usenix.org/conference/lisa12/technical-sessions/presentation/lang_david
Give it a try, I'll bet that you find the result much simpler and faster.
I expecting your reply ;)
KISS, start simple and only add complexity when you find it's actually needed.
Have plans for how to scale out when you hit limits, but you usually find that
you hit limits far later than expected. Yes, you may have to eventually do the
same work, but by having a solid system now with less work, you can spend the
time saved now to improve other things.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
