So one of the things that the already existing Kafka channel can do is to run without a source. Does this outperform that as well? I have already seen people use it this way.
Thanks, Hari On Thu, Aug 27, 2015 at 4:11 PM, Roshan Naik <ros...@hortonworks.com> wrote: > Wanted to give a heads-up on this idea I have been working on … > > Using Flume as a Kafka producer or consumer has been gaining popularity > thanks to the Flafka components that were recently introduced. > > For the use case of Flume as a Kafka consumer, it appears we can sidestep > the compromise between Mem channel (which is fast but can lose data) and > File channel (which is slow but won't lose data) and get the best of both > worlds. > > I have a prototype of this idea for a "Kafka Consumer" channel. It is > designed to enable the use of Flume as a really light weight and very fast > Kafka consumer without the data loss potential of mem channel. My > measurements indicate it easily outperforms memory channel. > > Additional info here … > https://github.com/roshannaik/kafka-consumer-channel > > I think the same idea could be applied for "Kafka producer channel". > > -roshan >
