We implemented the KafkaChannel with the "sync" producer, because we want to guarantee the transaction success and handle errors appropriately. We think it is worth the performance loss. Even with the sync producer, we found that the Kafka Channel can handle 20K events per second on my (pretty simple) setup. By tuning the batch size you can have some control over the latency and throughput.
I also started testing the new Kafka producer (0.8.2.0 and up) - the way we use the producer in the channel (batches), there was almost no performance improvement. Gwen On Mon, Apr 13, 2015 at 9:41 AM, Tao Li <[email protected]> wrote: > Hi all: > > JIRA: https://issues.apache.org/jira/browse/FLUME-2500 > > I have a question about this issue. > > Flume support transaction between source and channel. So for kafka channel: > 1. If we use "sync" kafka producer, it surely can guarantee the transaction, > but sync send has low performance. > 2. If we use "async" kafka producer, the performance will be better, but it > can't guarantee the transaction. > > How do you think on it?
