Re: Updated Kafka Roadmap?

Hey Jonathan, I just sent an email on the dev list to discuss this. Not to be double effort but for posterity and good thread communication if you can voice your opinion there it would be great (please) I would volunteer to release 0.8.1.2, not a problem. My concern with 0.8.2 is with any new re

Re: Updated Kafka Roadmap?

Hi, I was wondering whether 0.8.2 is on track for being released this month, as I haven’t read much about betas or release candidates in any of the kafka groups (although I haven’t been following the IRC)? Although many of the new features targeted for 0.8.2, including topic deletion, the new

Re: High Level Consumer and Commit

This is not how I'd expect this to work. Offsets are per-partition and each thread reads its own partition (Assuming you use Balaji's solution). So: Thread 1 reads messages 1..50 from partition 1, processes, indexes, whatever and commits. Thread 2 reads messages 1..55 from partition 2, processes,

Re: message size limit

are you referring to socket.request.max.bytes ? it looks like it could indeed limit the size of a batch accepted by a broker. So, you're right, batch.num.messages * message.max.bytes must be smaller than socket.request.max.bytes. It looks like this case has been addressed in the new producer. See

Re: High Level Consumer and Commit

>>Only problem is no of connections to Kafka is increased. *Why* is it a problem? Philip

Re: High Level Consumer and Commit

Thanks got the idea !! But it will create a fragments for example Main Thread reads 0-50 messages give to Thread 1 for bulk index and commit 0 to 50 offset... Main Thread reads 51-100 message give to Thread 2 for bulk index and commit 51 100 offset... So Zookeeper might have offset that will o

RE: High Level Consumer and Commit

Multiple consumers with single thread each will also work. Only problem is no of connections to Kafka is increased. -Original Message- From: Gwen Shapira [mailto:gshap...@cloudera.com] Sent: Wednesday, September 03, 2014 4:20 PM To: users@kafka.apache.org Cc: Philip O'Toole Subject: Re:

Re: message size limit

I am referring to wiki http://kafka.apache.org/08/configuration.html and following parameter control max batch message bytes as far as I know. Kafka Community, please correct me if I am wrong. I do not want to create confusion for Kafka User Community here. Also, if you increase this limit than

Re: High Level Consumer and Commit

Thanks, Balaji! It looks like your approach depends on specific implementation details, such as the directory structure in ZK. In this case it doesn't matter much since the APIs are not stable yet, but in general, wouldn't you prefer to use public APIs, even if it means multiple consumers without

Re: message size limit

Hi Bhavesh can you explain what limit you're referring to? I'm asking because `message.max.bytes` is applied per message not per batch. is there another limit I should be aware of? thanks On Wed, Sep 3, 2014 at 2:07 PM, Bhavesh Mistry wrote: > Hi Jun, > > We have similar problem. We have var

RE: High Level Consumer and Commit

We can still do with single ConsumerConnector with multiple threads. Each thread updates its own data in zookeeper.The below one is our own implementation of commitOffset. public void commitOffset(DESMetadata metaData) { log.debug("Update offsets only for ->"+ metaData.toString()

Re: message size limit

Hi Jun, We have similar problem. We have variable length of messages. So when we have fixed size of Batch sometime the batch exceed the limit set on the brokers (2MB). So can Producer have some extra logic to determine the optimal batch size by looking at configured message.max.bytes value. D

Error in acceptor (kafka.network.Acceptor)

Hi, I am trying to load a cluster with over than 10K connections, and bumped into the error in the subject. Is there any limitation on Kafka's side? if so it configurable? how? on first look, it looks like the selector accepting the connection is overflowing... Thanks. -- Shlomi

Manual Leader Assignment

Hiya, During leader changes, we see short periods of message loss on some of our higher volume producers. I suspect that this is because it takes a couple of seconds for Zookeeper to notice and notify the producers of the metadata change. During this time, producer buffers can fill up and end

Re: Subscription

Sorry I picked the wrong address 2014-09-03 18:45 GMT+01:00 François Langelier : > You should follow those instructions > > http://kafka.apache.org/contact.html > > > François Langelier > Étudiant en génie Logiciel - École de Technologie Supérieure > > Capitaine Club Capr

Re: Subscription

You should follow those instructions http://kafka.apache.org/contact.html François Langelier Étudiant en génie Logiciel - École de Technologie Supérieure Capitaine Club Capra VP-Communication - CS Games 2014 Jeux de Génie

Subscription

Subscription -- Massimiliano Tomassi web: http://about.me/maxtomassi e-mail: max.toma...@gmail.com mobile: +447751193667

Re: High Level Consumer and Commit

Sorry, I guess I missed that. The followup discussion was around the simple consumer :) I'm not sure why the OP didn't find this solution acceptable. On Wed, Sep 3, 2014 at 8:29 AM, Philip O'Toole wrote: > That's what I said in my first reply. :-) > > - >

Re: High Level Consumer and Commit

That's what I said in my first reply. :-) - http://www.philipotoole.com On Tuesday, September 2, 2014 10:37 PM, Gwen Shapira wrote: I believe a simpler solution would be to create multiple ConsumerConnector, each with 1 thread (single ConsumerStre

Re: message size limit

Thanks Jun. I'll create a jira and try to provide a patch. I think this is pretty serious. On Friday, August 29, 2014, Jun Rao wrote: > The goal of batching is mostly to reduce the # RPC calls to the broker. If > compression is enabled, a larger batch typically implies better compression > rati

Re: Replication bandwidth double what is expected

that makes sense. if the size of each fetch is small then compression won't do much, and that could very well explain the increase in bandwidth. we will try to change these settings and see what happens. thanks a lot for your help. T# On Tue, Sep 2, 2014 at 10:44 PM, Guozhang Wang wrote: > H

Re: Migrating data from old brokers to new borkers question

Hi Marcin, A few weeks ago, I did an upgrade to 0.8.1.1 and then augmented the cluster from 3 to 9 brokers. All went smoothly. In a dev environment, we found out that the biggest pain point is to have to deal with the json file and the error-prone command line interface. So to make our life easier