Re: Kafka unable to process large number of messages

2013-03-29 Thread Neha Narkhede
Hi Anand, Can you describe your exact test setup ? This bug has been quite elusive so far, it will be great to have a reproducible test case. Also, are you using Kafka 0.7 or 0.8 ? I wonder if you can reproduce this with Kafka 0.8 as well ? Thanks Neha On Fri, Mar 29, 2013 at 8:15 AM, anand naly

Re: Kafka unable to process large number of messages

2013-03-29 Thread anand nalya
Hi Jun, I'm using async java producer. It works fine till the messages are in 100s of thousands but starts failing for anything above a million. Each message is around 2kb. I've tried both with single producer and multiple producers. Rate of this error is much less in single producer then in case

Re: Kafka unable to process large number of messages

2013-03-29 Thread Jun Rao
This indicates that the messages sent to the broker are somehow corrupted. Are you using a java producer? How many instances of producers do you have? Thanks, Jun On Fri, Mar 29, 2013 at 2:46 AM, anand nalya wrote: > Hi, > > I'm running kafka in distributed mode with 2 nodes. It works fine wit

Kafka unable to process large number of messages

2013-03-29 Thread anand nalya
Hi, I'm running kafka in distributed mode with 2 nodes. It works fine with slow ingestion rates but when I increase the ingestion rate, both the nodes starts giving the following error: [2013-03-29 14:51:45,379] ERROR Closing socket for /192.168.145.183 because of error (kafka.network.Processor)