Hi Rainer, Good questions. Kafka is immensely popular. It's used a lot for log messaging. It's super scalable and provides a very nice produce/consume mechanism that can act as both a queue or a topic. It's persistent and has configurable TTL for data. As such, it's very often used as a buffer from which consumers pull based on how fast they can pull data. There are a ton of projects built around Kafka that can take messages from Kafka and push them into systems like Hadoop/HDFS, S3, Elasticsearch, and so on.
I think you'd want to have omkafka in Rsyslog for a similar reason why you want to have omelasticsearch. Ah, I didn't see your earlier message about Kafka - just subscribed to the ML yesterday. :) I'm guessing the implementation on the Rsyslog side would be VERY VERY similar to omelasticsearch one, except instead of pushing a log to ES you'd push it to Kafka -- using something like https://github.com/edenhill/librdkafka . Tschuss, Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Fri, Dec 6, 2013 at 10:08 AM, Rainer Gerhards <[email protected]>wrote: > On Fri, Dec 6, 2013 at 3:50 PM, Otis Gospodnetic < > [email protected] > > wrote: > > > Hi, > > > > What is the best way to send logs from Rsyslog to Kafka? > > > > I just came across https://github.com/edenhill/librdkafka and thought it > > would be cool if Rsyslog could use that ..... omkafka would be cool, no? > > > > > Could you elaborate what Kafka is and why it would be cool if rsyslog > connects to it (use cases!)? > > Thx, > Rainer > > PS: pls also see me other msg earlier this week on implementing it. > > > > I did find https://github.com/rngadam/sendkafka , which you can feed > with > > omprog, but because this reads from stdin I wonder if this would be > > suboptimal? > > > > Thanks, > > Otis > > -- > > Performance Monitoring * Log Analytics * Search Analytics > > Solr & Elasticsearch Support * http://sematext.com/ > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
