On Fri, 6 Dec 2013, Otis Gospodnetic wrote:
Hi Rainer,
Good questions. Kafka is immensely popular. It's used a lot for log
messaging. It's super scalable and provides a very nice produce/consume
mechanism that can act as both a queue or a topic. It's persistent and has
configurable TTL for data. As such, it's very often used as a buffer from
which consumers pull based on how fast they can pull data. There are a ton
of projects built around Kafka that can take messages from Kafka and push
them into systems like Hadoop/HDFS, S3, Elasticsearch, and so on.
I think you'd want to have omkafka in Rsyslog for a similar reason why you
want to have omelasticsearch.
Ah, I didn't see your earlier message about Kafka - just subscribed to the
ML yesterday. :)
I'm guessing the implementation on the Rsyslog side would be VERY VERY
similar to omelasticsearch one, except instead of pushing a log to ES you'd
push it to Kafka -- using something like
https://github.com/edenhill/librdkafka .
omelasticsearch was written by people who use elasticsearch and contributed to
rsyslog, it would be great if Kafka users were to do the same thing. but to
expect people who have to ask what Kafka is to do so is the wrong way to
approach it.
If Kafka is such a popular tool to handle log messages, why doesn't it implement
any of the standard protocols for delivering log messages? If it does, then
rsyslog should be able to deliver messages to it using those protocols. If there
is some advantage of rsyslog having a omkafka module to do something specific
instead of just using the standard protocols, someone will need to tell us about
it.
David Lang
P.S. using the justification that it's good to send logs to hadoop and
elasticsearch when rsyslog can do that already is a bit odd ;-) Sending logs to
S3 requires that the logs be bundles up and uploaded
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
