Faster segment recovery after reboot - 0.8.1

Hi, Just got a lovely email a bunch of our EC2 instances will be rebooted in a few days. Some of them run our Kafka 0.8.1 brokers with a few hundred GBs of data in them. Last time Kafka brokers didn't shut down cleanly it took them mny hours to recover. I just found

LeaderNotAvailableException, although leader elected

Hi, What does the exception mean in the following context? I've got two Kafka brokers out which the second one will be elected as leader: kafka2_1 | [2014-09-26 12:35:07,289] INFO [Kafka Server 2], started (kafka.server.KafkaServer) kafka2_1 | [2014-09-26 12:35:07,394] INFO

RE: Log.scala

Also, would this check be redundant because of https://github.com/apache/kafka/blob/7847e9c703f3a0b70519666cdb8a6e4c8e37c3a7/core/src/main/scala/kafka/log/Log.scala#L243 From: Manjunath Shivakumar [manjunath.shivaku...@betfair.com] Sent: 26 September 2014

Automatic preferred replica election

Hello, A while back, I saw mentioning about making preferred replica election automatic. It would be really nice to have cluster balance restoring automatically. Is this work on the roadmap? Any chance it will be included into 0.8.2? 0.9? Thanks, Yury

Re: LeaderNotAvailableException, although leader elected

And the solution was: Increase retry.backoff.ms from the default 100 ms to 1000 ms, so the output is: 11891 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from broker id:0,host:192.168.59.103,port:9092 with correlation id 0 for 1 topic(s) Set(inputTopic) 11893 [main] INFO

Re: Log.scala

Hi Manjunath, This check !appendInfo.offsetsMonotonic makes sure that there will be no gaps. And the other part appendInfo.firstOffset nextOffset.get I agree that it seems redundant. Guozhang On Fri, Sep 26, 2014 at 5:52 AM, Manjunath Shivakumar manjunath.shivaku...@betfair.com wrote:

Re: Read a specific number of messages using kafka

You can use any offset you like to fetch (provided it is present on the broker). Since the client-side FetchRequest and the server-side Log API don't support reading a specific _number_ of messages you will need to specify the bytes (size) to read. You can then extract as many messages as you like

Re: BytesOutPerSec is more than BytesInPerSec.

Can you confirm that you are using a replication factor of two? As Steven said, the replicas also consume from the leader. So it's your consumer, plus the replica. On Thu, Sep 25, 2014 at 10:04:29PM -0700, ravi singh wrote: Thanks Steven. That answers the difference in Bytes in and bytes Out

Re: Faster segment recovery after reboot - 0.8.1

Unfortunately, I think the only option (going forward) until you pick up KAFKA-1414 is to use a smaller segment size. On Fri, Sep 26, 2014 at 02:04:33AM -0400, Otis Gospodnetic wrote: Hi, Just got a lovely email a bunch of our EC2 instances will be rebooted in a few days. Some of them run

Re: Automatic preferred replica election

I think there are some issues still open with it that are currently slated for 0.9 - e.g., KAFKA-1305. Would be great if someone can pick that up as it is a useful feature to include in 0.8.2 On Fri, Sep 26, 2014 at 06:06:14PM +0400, Yury Ruchin wrote: Hello, A while back, I saw mentioning

Re: LeaderNotAvailableException, although leader elected

kafka2_1 | [2014-09-26 12:35:07,289] INFO [Kafka Server 2], started (kafka.server.KafkaServer) kafka2_1 | [2014-09-26 12:35:07,394] INFO New leader is 2 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener) The above logs are for controller election. Not leader

high-level consumer design

Hi, Is this implemented? I am looking for high level consumer implementation in golang. [Kafka-users] high-level consumer design [http://grokbase.com/ico/sbj/rss/24]http://grokbase.com/g/kafka/users/feed [http://grokbase.com/ico/sbj/rss/24]

Re: BytesOutPerSec is more than BytesInPerSec.

Thanks Steven, I am using replication factor of 3 . As you mentioned, the replicas also consume from the leader that's the reason for deviation in BytesInPerSec and BytesOutPersec Regards, Ravi On Fri, Sep 26, 2014 at 10:27 AM, Joel Koshy jjkosh...@gmail.com wrote: Can you confirm that you

Re: Automatic preferred replica election

+1 on having someone pick up the JIRA. I'm happy to mentor a contributor through that JIRA towards a checkin. On Fri, Sep 26, 2014 at 10:45 AM, Joel Koshy jjkosh...@gmail.com wrote: I think there are some issues still open with it that are currently slated for 0.9 - e.g., KAFKA-1305. Would be

Re: Automatic preferred replica election

Neha, I am interested in picking it up. Assigining JIRA to myself. Thanks, Harsha On Fri, Sep 26, 2014, at 02:05 PM, Neha Narkhede wrote: +1 on having someone pick up the JIRA. I'm happy to mentor a contributor through that JIRA towards a checkin. On Fri, Sep 26, 2014 at 10:45 AM, Joel

Re: Handling errors in the new (0.8.2) Java Client's Producer

Guozhang, Thank you very much for your reply. It is good to know that I have not overlooked something simple in the interface. How to I open a jira. The major change that I would like to see is to have a KafkaProducer.abort() method that closes the producer immediately, aborting attempts to

Re: Repost: Busy CPU while negotiating contentBuffer size at BoundedByteBufferReceive.scala:54

Hi Jun, Thanks for the info, we'll keep you posted on the above topic. Surprisingly in broker controller logs we also see somewhat similar exceptions at the same line BoundedByteBufferReceive.scala:54. (Not sure if there is any correlation here.) 8--