Varun, Were there broker failures during the test? In 0.7, since producer doesn't receive acks, some produced data could get lost when there is broker failure. This issue will be addressed in 0.8.
If there were no broker failures, we don't expect to have message loss. Are there any errors in the producer, broker or consumer log? Thanks, Jun On Sat, Oct 27, 2012 at 3:15 PM, Varun Vijayaraghavan <varun....@gmail.com>wrote: > Hi! > > I am testing out Kafka as a message queue replacement for ZMQ, for supposed > reliability and easy scaling. Our traffic peaks at around 6000 messages per > second. However, on comparing the aggregated output of the Kafka consumers > with our existing ZMQ setup, it looks like about 5% - 10% of the messages > are getting dropped (or possibly even corrupted?) somewhere between the > sending on the producers side to the polling on the consumers side. > > Some info - > > We have: > > - 3 Kafka broker machines with 4 GB of RAM each. The disk latency is in the > lower 10s of milliseconds, and about 3.5 GB of memory is used by the cache. > > - 6 producers. We are not yet doing batched writes, but as mentioned in my > previous point, the kafka server machines seem to be able to handle it > quite easily. > > - 120 partitions with about 120 workers, with load unevenly spread among > the workers. Some workers may not get any messages, some workers may > receive as much as 1000 messages per second. > > - The maximum fetch size specified for each consumer is 2MB. > > > > Does anyone have or has anyone had similar issues with pings getting > dropped? Or is there some conventional wisdom which could point to > incorrect / sub-optimal configuration from my side? > > Thanks much! > > > -- > - varun :) >
