Jun, I wrote a test producer to test if num.retries working or not. But I found that it's not working. No matter how many retries I set, whenever a message send fails, it always never gets to the broker. I am using Kafka 0.7.0
Is this a known problem? Do I need to file a JIRA issue? Because we are using Async producer we have no way to catch the exception ourselves and act on it. Is that right? Any ideas how we can ensure that every single message is sent with retries? Regards, Vaibhav On Tue, Jun 26, 2012 at 4:52 PM, Jun Rao <jun...@gmail.com> wrote: > Set num.retries in producer config property file. It defaults to 0. > > Thanks, > > Jun > > On Tue, Jun 26, 2012 at 4:46 PM, Vaibhav Puranik <vpura...@gmail.com> > wrote: > > > I reduced the batch size and reduced the pooled connections. Number of > > errors have gone down significantly. But they are not eliminated yet. > > > > We definitely don't want to loose any events. > > > > Jun, how do I configure the client resend you mentioned below? I couldn't > > find any configuration. > > > > Regards, > > Vaibhav > > > > On Tue, Jun 26, 2012 at 9:27 AM, Vaibhav Puranik <vpura...@gmail.com> > > wrote: > > > > > These are great pointers. > > > I found some more discussion here: > > > https://forums.aws.amazon.com/thread.jspa?threadID=33427 > > > > > > I can do the following to keep using the elastic load balancer: > > > > > > 1) Reduce the producer pool size to 1 or 2 because looks like > connections > > > are sitting idle. My volume does not desire that big pool. > > > 2) Reduce the batch size so that the webapp frequently dumps the data > to > > > brokers. It's better for us anyways. > > > > > > I will try both of these options and report back. > > > > > > Thank you very much Jun and Niek. > > > > > > Regards, > > > Vaibhav > > > > > > > > > On Tue, Jun 26, 2012 at 8:52 AM, Niek Sanders <niek.sand...@gmail.com > > >wrote: > > > > > >> ELBs will close connections that have no data going across them over a > > >> 60 sec period. A reference to this behavior can be found at the > > >> bottom of this page: > > >> > > >> http://aws.amazon.com/articles/1636185810492479 > > >> > > >> There is currently no way for customers to increase this timeout. If > > >> this timeout is in fact the problem, then the alternative is to use HA > > >> proxy for load balancing instead. > > >> > > >> - Niek > > >> > > >> > > >> > > >> > > >> On Tue, Jun 26, 2012 at 7:55 AM, Jun Rao <jun...@gmail.com> wrote: > > >> > Vaibhav, > > >> > > > >> > Does elastic load balancer have any timeouts or quotas that kill > > >> existing > > >> > socket connections? Does client resend succeed (you can configure > > >> resend in > > >> > DefaultEventHandler)? > > >> > > > >> > Thanks, > > >> > > > >> > Jun > > >> > > > >> > On Mon, Jun 25, 2012 at 6:01 PM, Vaibhav Puranik < > vpura...@gmail.com> > > >> wrote: > > >> > > > >> >> Hi all, > > >> >> > > >> >> We are sending our ad impressions to Kafka 0.7.0. I am using async > > >> >> prouducers in our web app. > > >> >> I am pooling kafak producers with commons pool. Pool size - 10. > > >> batch.size > > >> >> is 100. > > >> >> > > >> >> We have 3 c1.xlarge instances with Kafka brokers installed behind a > > >> elastic > > >> >> load balancer in AWS. > > >> >> Every minute we loose some events because of the following > exception > > >> >> > > >> >> - Disconnecting from > > >> dualstack.kafka-xyz.us-east-1.elb.amazonaws.com:9092 > > >> >> - Error in handling batch of 64 events > > >> >> java.io.IOException: Connection timed out > > >> >> at sun.nio.ch.FileDispatcher.write0(Native Method) > > >> >> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) > > >> >> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104) > > >> >> at sun.nio.ch.IOUtil.write(IOUtil.java:75) > > >> >> at > sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) > > >> >> at > > >> >> > > >> > > > kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:51) > > >> >> at > kafka.network.Send$class.writeCompletely(Transmission.scala:76) > > >> >> at > > >> >> > > >> >> > > >> > > > kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25) > > >> >> at > > kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:88) > > >> >> at kafka.producer.SyncProducer.send(SyncProducer.scala:87) > > >> >> at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:128) > > >> >> at > > >> >> > > >> > > > kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:52) > > >> >> at > > >> >> > > >> >> > > >> > > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:46) > > >> >> at > > >> >> > > >> >> > > >> > > > kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:119) > > >> >> at > > >> >> > > >> >> > > >> > > > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:98) > > >> >> at > > >> >> > > >> >> > > >> > > > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:74) > > >> >> at scala.collection.immutable.Stream.foreach(Stream.scala:254) > > >> >> at > > >> >> > > >> >> > > >> > > > kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:73) > > >> >> at > > >> >> > > >> > kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:43) > > >> >> - Connected to > dualstack.kafka-xyz.us-east-1.elb.amazonaws.com:9092 > > for > > >> >> producing > > >> >> > > >> >> Has anybody faced this kind of timeouts before? Do they indicate > any > > >> >> resource misconfiguration? The CPU usage on broker is pretty small. > > >> >> Also, in spite of setting batch size to 100, the failing batch > > usually > > >> only > > >> >> have 50 to 60 events. Is there any other limit I am hitting? > > >> >> > > >> >> Any help is appreciated. > > >> >> > > >> >> > > >> >> Regards, > > >> >> Vaibhav > > >> >> GumGum > > >> >> > > >> > > > > > > > > >
