On Fri, 6 Dec 2013, Mike Hoskins (michoski) wrote:
On Fri, Dec 6, 2013 at 3:50 PM, Otis Gospodnetic
<[email protected]
wrote:
Hi,
What is the best way to send logs from Rsyslog to Kafka?
I just came across https://github.com/edenhill/librdkafka and thought it
would be cool if Rsyslog could use that ..... omkafka would be cool, no?
Could you elaborate what Kafka is and why it would be cool if rsyslog
connects to it (use cases!)?
Thx,
Rainer
PS: pls also see me other msg earlier this week on implementing it.
i'm just getting forcibly exposed to this myself (drinking from a
firehose). it's just another message broker, so in theory you could have
rsyslog agents filling a durable message store that could be consumed by
geographically dispersed (or not) clients. that could be of interest in
many use cases, in our case a good example would be security devices
generating logs and analytics consumers. this is actually interesting
enough i have been saving these emails and thinking of ways to contribute
myself, but time is my biggest weakness as i am sure it is for many. :-)
again i'm just getting exposed to this in a live devops project, so i
might be oversimplifying. googling for "rabbitmq vs kafka" gives a lot of
good reading. generically i see this just as useful as supporting
amqp/stomp output. sorry i am still pretty new here, but just based on
some past archives from this list i wonder if logstash as a middleman
would be a quick answer for the OP?
If you are using logstash, why bother using rsyslog? you don't need the rsyslog
performance if you can then have rsyslog send the logs to logstash.
It's a good thing to get added modules to rsyslog to let it deliver to things in
their native protocols (it's better if log related things can talk the standard
logging protocols, but some tools that are useful for logs were not designed for
them)
The question is just how should they be written.
OpenSource software is developed by people working on (or paying others to work
on) things that they are interested in.
Sometimes you can get someone else interested in something that you want and
they will then do the development for you for free.
but you can't just say "it would be good to have omkafka", you need to explain
to people who have never heard of kafka before why they should be interested in
it. I don't think anyone has even posted a link to kafka to point us at the
right thing to look at yet.
Or you can create a simple omkafka module (or sponsor the development of one)
and contribute it. Even if the first version isn't the fastest possible, once
you get something that works at all, if others find it useful they will help
improve it (sometimes including re-writitng it)
When the initial post of "what's the best way people have found to get logs from
rsyslog to kafka" was posted, I almost responded with a "what's kafka, what
protocols does it understand" type response, but I decided to be quiet and see
if someone else was familar with it.
In several days we haven't had anyone else indicate that they knew much about it
before the second round "wouldn't it be good to have a omkafka modules".
This indicates that there are at least a couple people interested in using
kafka, but not much else :-)
If you want help, provide info, don't ask others to do a bunch of research
before they can answer your questions.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
