My understanding is that the Kafka channel expects "Flume Event" objects to be stored in the Kafka topic. Isn't that right ? -roshan
On 8/27/15 5:47 PM, "Hari Shreedharan" <[email protected]> wrote: >So one of the things that the already existing Kafka channel can do is to >run without a source. Does this outperform that as well? I have already >seen people use it this way. > > >Thanks, >Hari > >On Thu, Aug 27, 2015 at 4:11 PM, Roshan Naik <[email protected]> >wrote: > >> Wanted to give a heads-up on this idea I have been working on Š >> >> Using Flume as a Kafka producer or consumer has been gaining popularity >> thanks to the Flafka components that were recently introduced. >> >> For the use case of Flume as a Kafka consumer, it appears we can >>sidestep >> the compromise between Mem channel (which is fast but can lose data) and >> File channel (which is slow but won't lose data) and get the best of >>both >> worlds. >> >> I have a prototype of this idea for a "Kafka Consumer" channel. It is >> designed to enable the use of Flume as a really light weight and very >>fast >> Kafka consumer without the data loss potential of mem channel. My >> measurements indicate it easily outperforms memory channel. >> >> Additional info here Š >> https://github.com/roshannaik/kafka-consumer-channel >> >> I think the same idea could be applied for "Kafka producer channel". >> >> -roshan >>
