OK that's really good to know. We won't need an additional component if it can function that way. Also in that case I would expect it to be quite fast.
Will try to get some numbers next week. Glad I only spent a couple evenings on that prototype. -roshan On 8/27/15 5:56 PM, "Hari Shreedharan" <[email protected]> wrote: >Nope. You can put anything you want, just set parseAsFlumeEvent to false >and the channel won't attempt to convert it into a Flume event. It just >stashes the whole thing into the body of the returned event. > > >Thanks, >Hari > >On Thu, Aug 27, 2015 at 5:53 PM, Roshan Naik <[email protected]> >wrote: > >> My understanding is that the Kafka channel expects "Flume Event" objects >> to be stored in the Kafka topic. >> Isn't that right ? >> -roshan >> >> >> On 8/27/15 5:47 PM, "Hari Shreedharan" <[email protected]> >>wrote: >> >> >So one of the things that the already existing Kafka channel can do is >>to >> >run without a source. Does this outperform that as well? I have already >> >seen people use it this way. >> > >> > >> >Thanks, >> >Hari >> > >> >On Thu, Aug 27, 2015 at 4:11 PM, Roshan Naik <[email protected]> >> >wrote: >> > >> >> Wanted to give a heads-up on this idea I have been working on Š >> >> >> >> Using Flume as a Kafka producer or consumer has been gaining >>popularity >> >> thanks to the Flafka components that were recently introduced. >> >> >> >> For the use case of Flume as a Kafka consumer, it appears we can >> >>sidestep >> >> the compromise between Mem channel (which is fast but can lose data) >>and >> >> File channel (which is slow but won't lose data) and get the best of >> >>both >> >> worlds. >> >> >> >> I have a prototype of this idea for a "Kafka Consumer" channel. It >>is >> >> designed to enable the use of Flume as a really light weight and very >> >>fast >> >> Kafka consumer without the data loss potential of mem channel. My >> >> measurements indicate it easily outperforms memory channel. >> >> >> >> Additional info here Š >> >> https://github.com/roshannaik/kafka-consumer-channel >> >> >> >> I think the same idea could be applied for "Kafka producer channel". >> >> >> >> -roshan >> >> >> >>
