producer queue size

2015-03-18 Thread sunil kalva
Hi How do i get the size of the inmemory queue which are holding messages and ready to send in async producer, i am using new KafkaProducer class in 0.8.2. Basically instead of dropping the messages silently, i want to avoid sending messages if the queue is already full. I am using async

Re: producer queue size

2015-03-18 Thread sunil kalva
essentially i want to use this property queue.buffering.max.messages with new KafkaProducer class, and also want to access the current value of the queue SunilKalva On Wed, Mar 18, 2015 at 9:51 PM, sunil kalva sambarc...@gmail.com wrote: Hi How do i get the size of the inmemory queue which

Async Producer Callback

2015-03-18 Thread sunil kalva
Hi How do i access the actual message which is failed to send to cluster using Callback interface and onCompletion method. Basically if the sender is failed to send, i want to add it to a temp queue and retry them later. --Sunil

Re: producer queue size

2015-03-18 Thread Ewen Cheslack-Postava
The setting you want is buffer.memory, but I don't think there's a way to get the amount of remaining space. The setting block.on.buffer.full controls the behavior when you run out of space. Neither setting silently drops messages. It will either block until there is space to add the message or

Re: [ANN] sqlstream: Simple MySQL binlog to Kafka stream

2015-03-18 Thread Xiao
Linkedin Gabblin compaction tool is using Hive to perform the compaction. Does it mean Lumos is replaced? Confused… On Mar 17, 2015, at 10:00 PM, Xiao lixiao1...@gmail.com wrote: Hi, all, Do you know whether Linkedin plans to open source Lumos in the near future? I found the answer