On Tue, 22 Nov 2016, [email protected] wrote:
We've been playing with logstash, rsyslog and redis for a while in order to
*index into elasticsearch a bunch of application logs*. Briefly:
app1-file1.log, app1-file2.log...appN-fileX.log -> pipeline -> elasticsearch.
So far, we are using *redis queues and _each application_ processing was made
by one logstash instance* (docker container). Of course, this works with 5-10
applications, but it doesn't when you plan to deploy 100 apps cause each
logstash instance requires ~512MB of RAM.
We've been thinking about rsyslog since the beginning, because it takes fewer
RAM, but just noticed it doesn't have a *redis input module (aka: imhiredis)*
We still plan to have independent instances (one rsyslog for each
application), but we're wondering if you'll consider it makes sense to
implement this module.
What sort of log volume are you talking about here? (logs/sec type of thing)
Logstash needs something like redis because it can't do any queueing itself.
Rsyslog is built around queues, and has the ability to create multiple queues
and piplines internally, you don't need to run multiple instances.
What you would do is create a ruleset for each application (pipeline) and give
that ruleset it's own queue.
As new logs arrive, you then sort them by application, and for each application
(or application category), you call the appropriate ruleset. All processing from
that point on will take place in different threads working on different queues
for each category.
Give it a try, I'll bet that you find the result much simpler and faster.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
