Re: Replication bandwidth double what is expected

Hi Guozhang, We're using the default on all of those, except num.replica.fetchers which is set to 4. T# On Mon, Sep 1, 2014 at 9:41 PM, Guozhang Wang wangg...@gmail.com wrote: Hello Theo, What are the values for your replica.fetch.max.bytes, replica.fetch.min.bytes,

Intellij setup for Kafka 0.7.x

Hi, I am trying to setup Kafka 0.7.2 project in Intellij Idea 13 CE. The wiki instructions for developers seems to be pointing to trunk. Since the place I'm working for is using the older version, I was thinking of setting that version up in Intellij. Are there instructions on how to do that ?

Re: Intellij setup for Kafka 0.7.x

To give some more details: * I used the src tarball of Kafka 0.7.2. * Ran sbt idea. This generated a bunch of *.iml files under various folders of Kafka. * I tried to open the project by pointing to the Kafka project directory. This opened an Import Project from SBT project dialog, and when I

Re: Replication bandwidth double what is expected

Hi Theo, You can try to set replica.fetch.min.bytes to some large number (default to 1) and increase replica.fetch.wait.max.ms (default to 500) and see if that helps. In general, with 4 fetchers and min.bytes to 1 the replicas would effectively exchange many small packets over the wire. Guozhang

High Level Consumer and Commit

Hi Kafka Group, I have to pull the data from the Topic and index into Elastic Search with Bulk API and wanted to commit only batch that has been committed and still continue to read from topic further on same topic. I have auto commit to be off. ListMessage batch . while

Re: High Level Consumer and Commit

Either use the SimpleConsumer which gives you much finer-grained control, or (this worked with 0.7) spin up a ConsumerConnection (this is a HighLevel consumer concept) per partition, turn off auto-commit. Philip - http://www.philipotoole.com On

Re: High Level Consumer and Commit

Hi Philip, Thanks for the update. With Simple Consumer I will not get failover and rebalance that is provided out of box. what is other option not to block reading and keep processing and commit only when batch is done. Thanks, Bhavesh On Tue, Sep 2, 2014 at 4:43 PM, Philip O'Toole

Re: High Level Consumer and Commit

No, you'll need to write your own failover. I'm not sure I follow your second question, but the high-level Consumer should be able to do what you want if you disable auto-commit. I'm not sure what else you're asking. Philip -

Re: High Level Consumer and Commit

Hi Philip, Yes, We have disabled auto commit but, we need to be able to read from particular offset if we manage the offset ourself in some storage(DB). High Level consumer does not allow per partition management plug-ability. I like to have the High Level consumers Failover and auto

Re: High Level Consumer and Commit

Yeah, from reading that I suspect you need the SimpleConsumer. Try it out and see. Philip - http://www.philipotoole.com On Tuesday, September 2, 2014 5:43 PM, Bhavesh Mistry mistry.p.bhav...@gmail.com wrote: Hi Philip, Yes, We have disabled

Re: High Level Consumer and Commit

I believe a simpler solution would be to create multiple ConsumerConnector, each with 1 thread (single ConsumerStream) and use commitOffset API to commit all partitions managed by each ConsumerConnector after the thread finished processing the messages. Does that solve the problem, Bhavesh? Gwen