Wanted to give a heads-up on this idea I have been working on ... Using Flume as a Kafka producer or consumer has been gaining popularity thanks to the Flafka components that were recently introduced.
For the use case of Flume as a Kafka consumer, it appears we can sidestep the compromise between Mem channel (which is fast but can lose data) and File channel (which is slow but won't lose data) and get the best of both worlds. I have a prototype of this idea for a "Kafka Consumer" channel. It is designed to enable the use of Flume as a really light weight and very fast Kafka consumer without the data loss potential of mem channel. My measurements indicate it easily outperforms memory channel. Additional info here ... https://github.com/roshannaik/kafka-consumer-channel I think the same idea could be applied for "Kafka producer channel". -roshan
