Hi, Thanks for the info, David. Hadoop, ES, and S3 were just examples, not the full list. I didn't know about omhdfs, thanks for the pointer. It looks like omhdfs and omelasticsearch were all pushed into Rsyslog through sponsoring, so maybe that's the only way! :)
While one or two sparrows don't make a spring, Kafka (and SAMZA) may be something Rainer may want to keep an eye on. Re standards, I'm all for them! But evolution of everything keeps breaking them over and over... :) Thanks, Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Fri, Dec 6, 2013 at 10:43 AM, David Lang <[email protected]> wrote: > On Fri, 6 Dec 2013, Otis Gospodnetic wrote: > > Hi Rainer, >> >> Good questions. Kafka is immensely popular. It's used a lot for log >> messaging. It's super scalable and provides a very nice produce/consume >> mechanism that can act as both a queue or a topic. It's persistent and >> has >> configurable TTL for data. As such, it's very often used as a buffer from >> which consumers pull based on how fast they can pull data. There are a >> ton >> of projects built around Kafka that can take messages from Kafka and push >> them into systems like Hadoop/HDFS, S3, Elasticsearch, and so on. >> >> I think you'd want to have omkafka in Rsyslog for a similar reason why you >> want to have omelasticsearch. >> >> Ah, I didn't see your earlier message about Kafka - just subscribed to the >> ML yesterday. :) >> >> I'm guessing the implementation on the Rsyslog side would be VERY VERY >> similar to omelasticsearch one, except instead of pushing a log to ES >> you'd >> push it to Kafka -- using something like >> https://github.com/edenhill/librdkafka . >> > > omelasticsearch was written by people who use elasticsearch and > contributed to rsyslog, it would be great if Kafka users were to do the > same thing. but to expect people who have to ask what Kafka is to do so is > the wrong way to approach it. > > If Kafka is such a popular tool to handle log messages, why doesn't it > implement any of the standard protocols for delivering log messages? If it > does, then rsyslog should be able to deliver messages to it using those > protocols. If there is some advantage of rsyslog having a omkafka module to > do something specific instead of just using the standard protocols, someone > will need to tell us about it. > > David Lang > > P.S. using the justification that it's good to send logs to hadoop and > elasticsearch when rsyslog can do that already is a bit odd ;-) Sending > logs to S3 requires that the logs be bundles up and uploaded > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
