log.flush.interval is set to 10000. Should I increase or decrease? thanks Jamie
-----Original Message----- From: Jun Rao [mailto:[email protected]] Sent: Wednesday, November 28, 2012 10:00 PM To: [email protected] Subject: Re: producer queue size Is the server busy on I/O? What's log.flush.interval on the broker? For better performance, you need to set it to a few hundreds or more. Thanks, Jun On Wed, Nov 28, 2012 at 7:12 PM, Jamie Wang <[email protected]> wrote: > Hi, > > Thanks for all the replies. It helped me understand the system. I > appreciate it. > > I tried change the producer properties to async and also changed > queue.enqueueTimeout.ms=-1. But I still get the exception. I then changed > the producer queue size to 20K and 30K. My hope is by making the queue size > bigger, it will buy the producer more time to digest the total amount of > messages. But that didn't help in either 20K or 30K instances. I then > increased max queue time to 5000ms in hopes that would hold some of the > data. But in all these case, I still get the exception. > > I looked the console producer code and it uses synchronous producer class. > In this case, does it make any difference if I change the producer's > properties (e.g., to async)? I guess if the answer is no, it would explain > what saw the above. > > Also is there any definition or explanation on the Java API? As I read the > doc page on the site, there is none... > > Thanks > Jamie > > > > -----Original Message----- > From: Neha Narkhede [mailto:[email protected]] > Sent: Monday, November 26, 2012 4:40 PM > To: [email protected] > Subject: Re: producer queue size > > This will happen if you push data to the producer at a higher rate > than it is able to send to the server. > queue.size allows you to configure the size of the producer queue > (defaults to 10K). Also, > queue.enqueueTimeout.ms, if set to -1 will lead to blocking behaviour > instead of the producer throwing > QueueFullExceptions. > > Thanks, > Neha > > On Mon, Nov 26, 2012 at 4:06 PM, Joel Koshy <[email protected]> wrote: > > To use async, set producer.type to async ; The default queue size is > 10000; > > and the default batch size is 200. > > > > > > On Mon, Nov 26, 2012 at 2:28 PM, Jamie Wang <[email protected]> > wrote: > > > >> Hi, I am running the console demo comes with 0.7.2 in getting started > >> guide. All is working fine. Then I use stdio redirect a file of 30K > lines > >> of messages into the producer and I received an error "ERROR Event > queue > >> is full of unsent messages, could not send event:..." and exception > stack > >> shows: > >> Exception in thread "main" kafka.producer.async.QueueFullException: > Event > >> queue is full of unsent messages, could not send event:... > >> > >> Is there a way in the produce.properties to configure the size of > producer > >> queue? I know the producer is running in synchronous mode. How do I > solve > >> this problem or my usage is totally wrong? > >> > >> Thanks > >> Jamie > >> >
