Consumer cannot find leader after leadership changes on broker side

After executing PreferredReplicaLeaderElectionCommand on broker instance, we observed one of the consumers cannot find the leadership and stopped consuming. The following exception is all over the log file and it appears that the consumer cannot recover from it: 2014-10-29 00:53:30,492 WARN

Re: Consumer cannot find leader after leadership changes on broker side

-beta a try on your consumer and retry the test? On Wed, Oct 29, 2014 at 10:34 AM, Allen Wang allenxw...@gmail.com wrote: After executing PreferredReplicaLeaderElectionCommand on broker instance, we observed one of the consumers cannot find the leadership and stopped consuming. The following

Re: High CPU usage of Crc32 on Kafka broker

or something? Thanks, Neha On Thu, Nov 6, 2014 at 12:02 PM, Allen Wang aw...@netflix.com.invalid wrote: After digging more into the stack trace got from flight recorder (which is attached), it seems that Kafka (0.8.1.1) can optimize the usage of Crc32. The stack trace shows that Crc32

Add partitions with replica assignment in same command

I am trying to figure out how to add partitions and assign replicas using one admin command. I tried kafka.admin.TopicCommand to increase the partition number from 9 to 12 with the following options: /apps/kafka/bin/kafka-run-class.sh kafka.admin.TopicCommand --zookeeper ${ZOOKEEPER} --alter

Re: Add partitions with replica assignment in same command

need to fix the tool to support that. It's probably more intuitive to have TopicCommand just take the replica-assignment (for the new partitions) when altering a topic. Could you file a jira? Thanks, Jun On Fri, Nov 7, 2014 at 4:17 PM, Allen Wang aw...@netflix.com.invalid wrote: I am trying

BytesInPerSec greater than BytesOutPerSec

We observed that for a topic, BytesIn is greater than BytesOut. We are under the impression that BytesOut should include replication. The topic has two replicas for each partitions and all replicas are in sync. Then BytesOut should be at least same as BytesIn since it always needs to replicate to

Re: BytesInPerSec greater than BytesOutPerSec

consumer fetcher's requests count in BytesOutPerSec. Guozhang On Fri, Nov 21, 2014 at 11:13 AM, Allen Wang aw...@netflix.com.invalid wrote: We observed that for a topic, BytesIn is greater than BytesOut. We are under the impression that BytesOut should include replication. The topic has two

Re: under replicated topics

Brokers may have temporary problems catching up with the leaders. So I would not worry about it if it happens only once a while and goes away. Occasionally we have seen under replicated topics for long time, which might be caused by ZooKeeper session problem as indicated by such log messages:

Re: Leadership rebalance causing drop of incoming messages

and discover the new leader. Are you using the Java producer? Do you see any errors in the producer logs? On Wed, Jan 14, 2015 at 06:36:27PM -0800, Allen Wang wrote: Hello, We did a manual leadership rebalance (using PreferredReplicaLeaderElectionCommand) under heavy load and found

Re: Leadership rebalance causing drop of incoming messages

consumers appears to have resolved your issues. What config overrides did you use for your consumers? E.g., did you override the max wait time? How many consumers/producers are we talking about here? Thanks, Joel On Thu, Jan 15, 2015 at 11:55:35AM -0800, Allen Wang wrote: We are using

Re: Leadership rebalance causing drop of incoming messages

, Allen Wang wrote: After a closer look to other metrics and broker logs, we found that the instability of the broker cluster might have been caused by a snappy un-compression error. In our case, the consumer and producer happens to be the same application so restarting the client made

Leadership rebalance causing drop of incoming messages

Hello, We did a manual leadership rebalance (using PreferredReplicaLeaderElectionCommand) under heavy load and found that there is a significant drop of incoming messages to the broker cluster for more than an hour. Looking at broker log, we found a lot of errors like this: 2015-01-15

Errors from ReassignPartitionsCommand

Hello, We tried the ReassignPartitionsCommand to move partitions to new brokers. The execution initially showed message Successfully started reassignment of partitions But when I tried to verify using --verify option, it reported some reassignments have failed: ERROR: Assigned replicas

Re: Idle/dead producer connections on broker

try out the 0.8.2 broker version and see if this is still easily re-producible, i.e. starting a bunch of producers to send data for a while, and terminate them? Guozhang On Tue, Mar 10, 2015 at 1:00 PM, Allen Wang aw...@netflix.com.invalid wrote: Hello, We are using Kafka 0.8.1.1

Re: Anyone interested in speaking at Bay Area Kafka meetup @ LinkedIn on March 24?

We (Steven Wu and Allen Wang) can talk about Kafka use cases and operations in Netflix. Specifically, we can talk about how we scale and operate Kafka clusters in AWS and how we migrate our data pipeline to Kafka. Thanks, Allen On Mon, Feb 23, 2015 at 12:15 PM, Ed Yakabosky eyakabo

Re: Errors from ReassignPartitionsCommand

before? Guozhang On Fri, Jan 23, 2015 at 3:56 PM, Allen Wang aw...@netflix.com.invalid wrote: Hello, We tried the ReassignPartitionsCommand to move partitions to new brokers. The execution initially showed message Successfully started reassignment of partitions But when I tried

Kafka High Level Consumer Message Loss?

We have two applications that consume all messages from one Kafka cluster. We found that the MessagesPerSec metric started to diverge after some time. One of them matches the MessagesInPerSec metric from the Kafka broker, while the other is lower than the broker metric and appears to have some

Does quota requires 0.9.X clients?

>From looking at the design document, it seems quota is implemented purely at server side. So it should work with 0.8.X clients. But I would like to get confirmation. Thanks, Allen

Re: Kafka 0.9.0.1 plan

Hi Jun, What about https://issues.apache.org/jira/browse/KAFKA-3100? Thanks, Allen On Fri, Feb 5, 2016 at 1:19 PM, Ismael Juma wrote: > Hi Becket, > > On Fri, Feb 5, 2016 at 9:15 PM, Becket Qin wrote: > > > I am taking KAFKA-3177 off the list because