Re: Kafka 0.9.0.1 partitions shrink and expand frequently after restart the broker

someone can help to analysis it？ > 在 2017年11月10日，上午11:08，Json Tu <kafka...@126.com> 写道： > > I‘m so sorry for my poor english. > > what I really means is my broker machine is configured as 8 core 16G. but my > jvm configure is as below. > java -Xmx1G

Re: Kafka 0.9.0.1 partitions shrink and expand frequently after restart the broker

t;huxi...@hotmail.com> 写道： > > Seems broker `4759750` was always removed for partition [Yelp, 5] every round > of ISR shrinking. Did you check if everything works alright for this broker? > > > 发件人: Json Tu <kafka...@126.com> > 发送时间: 2017年11月10日 11:08 >

Re: Kafka 0.9.0.1 partitions shrink and expand frequently after restart the broker

suggestions. > 在 2017年11月9日，下午9:59，John Yost <hokiege...@gmail.com> 写道： > > I've seen this before and it was due to long GC pauses due in large part to > a memory heap > 8 GB. > > --John > > On Thu, Nov 9, 2017 at 8:17 AM, Json Tu <kafka...@126.com> wrote:

Kafka 0.9.0.1 partitions shrink and expand frequently after restart the broker

Hi, we have a kafka cluster which is made of 6 brokers, with 8 cpu and 16G memory on each broker’s machine, and we have about 1600 topics in the cluster，about 1700 partitions’ leader and 1600 partitions' replica on each broker. when we restart a normal broke, we find that there are

Get Broker metrics timeout

Hi all， we have a cluster with 10 brokers, and our kafka version is 0.9.0.1，we repeatedly get our metric data such as offlinePartition metric from each broker with 2 minutes gap to achieve the goal of cluster’s monitor. but accidental timeout occurs when we get data from some of brokers.

Question about KIP-73 pr merge

Hi all， We are now using kafka 0.9.0 in our product enviroment, and we add one broker to the cluster，and execute reassign partitions between all brokers，we find our network card and disk io is very high. and I know KIP-73 has resolved this problem, but I wonder can I merge it to my

[jira] [Resolved] (KAFKA-4447) Controller resigned but it also acts as a controller for a long time

[ https://issues.apache.org/jira/browse/KAFKA-4447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Json Tu resolved KAFKA-4447. Resolution: Fixed > Controller resigned but it also acts as a controller for a long t

Re: log.flush.interval.messages setting of Kafka 0.9.0.0

Would be grateful to hear opinions from experts out there. Thanks in advance > 在 2016年12月16日，下午6:17，Json Tu <kafka...@126.com> 写道： > > Hi all, > we have a cluster of 0.9.0.0 with 3 nodes, we have a topic with 3 > replicas, and send it with ack -1, our sending

log.flush.interval.messages setting of Kafka 0.9.0.0

Hi all, we have a cluster of 0.9.0.0 with 3 nodes, we have a topic with 3 replicas, and send it with ack -1, our sending latency is avg 7ms. I prepare to optimize performance of cluster through adjusting some params. we find our brokers has set config item as below,

Re: A strange controller log in Kafka 0.9.0.1

Hi, Can someone else help to review the pr in jira: https://issues.apache.org/jira/browse/KAFKA-4447 <https://issues.apache.org/jira/browse/KAFKA-4447>. > 在 2016年11月23日，下午11:28，Json Tu <kafka...@126.com> 写道： > > Hi, > We have a cluster of kafka 0.9.0.1 wi

[jira] [Comment Edited] (KAFKA-4447) Controller resigned but it also acts as a controller for a long time

[ https://issues.apache.org/jira/browse/KAFKA-4447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702384#comment-15702384 ] Json Tu edited comment on KAFKA-4447 at 12/1/16 6:24 AM: - [~hachikuji] thanks

[jira] [Commented] (KAFKA-4447) Controller resigned but it also acts as a controller for a long time

[ https://issues.apache.org/jira/browse/KAFKA-4447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711052#comment-15711052 ] Json Tu commented on KAFKA-4447: [~hachikuji] could you help review it，thanks. > Controller resig

[jira] [Issue Comment Deleted] (KAFKA-4447) Controller resigned but it also acts as a controller for a long time

[ https://issues.apache.org/jira/browse/KAFKA-4447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Json Tu updated KAFKA-4447: --- Comment: was deleted (was: [~wangg...@gmail.com] thank you for such a detailed analysis . as you

[jira] [Commented] (KAFKA-4447) Controller resigned but it also acts as a controller for a long time

[ https://issues.apache.org/jira/browse/KAFKA-4447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703688#comment-15703688 ] Json Tu commented on KAFKA-4447: [~wangg...@gmail.com] thank you for such a detailed analysis . as you

[jira] [Commented] (KAFKA-4447) Controller resigned but it also acts as a controller for a long time

[ https://issues.apache.org/jira/browse/KAFKA-4447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703687#comment-15703687 ] Json Tu commented on KAFKA-4447: [~wangg...@gmail.com] thank you for such a detailed analysis . as you

Re: A strange controller log in Kafka 0.9.0.1

thanks to Jason Gustafson, hope more contributor can take part in this discussion. https://issues.apache.org/jira/browse/KAFKA-4447 <https://issues.apache.org/jira/browse/KAFKA-4447> > 在 2016年11月27日，下午9:20，Json Tu <kafka...@126.com> 写道： > > AnyBody？This is very disconc

[jira] [Commented] (KAFKA-4447) Controller resigned but it also acts as a controller for a long time

[ https://issues.apache.org/jira/browse/KAFKA-4447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702465#comment-15702465 ] Json Tu commented on KAFKA-4447: may be the phenomenon of these listener is the same as one zk's callback

[jira] [Comment Edited] (KAFKA-4447) Controller resigned but it also acts as a controller for a long time

[ https://issues.apache.org/jira/browse/KAFKA-4447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702384#comment-15702384 ] Json Tu edited comment on KAFKA-4447 at 11/28/16 4:45 PM: -- [~skarface] thanks

[jira] [Comment Edited] (KAFKA-4447) Controller resigned but it also acts as a controller for a long time

[ https://issues.apache.org/jira/browse/KAFKA-4447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702384#comment-15702384 ] Json Tu edited comment on KAFKA-4447 at 11/28/16 4:44 PM: -- [~skarface] thanks

[jira] [Issue Comment Deleted] (KAFKA-4447) Controller resigned but it also acts as a controller for a long time

[ https://issues.apache.org/jira/browse/KAFKA-4447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Json Tu updated KAFKA-4447: --- Comment: was deleted (was: [~skarface] thanks for your reply. the latest release version 0.10.1.0

[jira] [Comment Edited] (KAFKA-4447) Controller resigned but it also acts as a controller for a long time

[ https://issues.apache.org/jira/browse/KAFKA-4447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702384#comment-15702384 ] Json Tu edited comment on KAFKA-4447 at 11/28/16 4:26 PM: -- [~skarface] thanks

[jira] [Commented] (KAFKA-4447) Controller resigned but it also acts as a controller for a long time

[ https://issues.apache.org/jira/browse/KAFKA-4447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702383#comment-15702383 ] Json Tu commented on KAFKA-4447: [~skarface] thanks for your reply. the latest release version 0.10.1.0

[jira] [Commented] (KAFKA-4447) Controller resigned but it also acts as a controller for a long time

[ https://issues.apache.org/jira/browse/KAFKA-4447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702384#comment-15702384 ] Json Tu commented on KAFKA-4447: [~skarface] thanks for your reply. the latest release version 0.10.1.0

Re: A strange controller log in Kafka 0.9.0.1

AnyBody？This is very disconcerting! If convenient, Can somebody help to confirm this strange question. > 在 2016年11月26日，上午1:35，Json Tu <kafka...@126.com> 写道： > > thanks guozhang, > if it's convenient，can we disscuss it in the jira > https://issues.apache.org/j

[jira] [Comment Edited] (KAFKA-4447) Controller resigned but it also acts as a controller for a long time

[ https://issues.apache.org/jira/browse/KAFKA-4447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696353#comment-15696353 ] Json Tu edited comment on KAFKA-4447 at 11/26/16 1:17 AM: -- after review the code

[jira] [Comment Edited] (KAFKA-4447) Controller resigned but it also acts as a controller for a long time

[ https://issues.apache.org/jira/browse/KAFKA-4447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696340#comment-15696340 ] Json Tu edited comment on KAFKA-4447 at 11/26/16 1:16 AM: -- after check

[jira] [Comment Edited] (KAFKA-4447) Controller resigned but it also acts as a controller for a long time

[ https://issues.apache.org/jira/browse/KAFKA-4447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696353#comment-15696353 ] Json Tu edited comment on KAFKA-4447 at 11/25/16 5:40 PM: -- after review the code

[jira] [Comment Edited] (KAFKA-4447) Controller resigned but it also acts as a controller for a long time

[ https://issues.apache.org/jira/browse/KAFKA-4447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696353#comment-15696353 ] Json Tu edited comment on KAFKA-4447 at 11/25/16 5:39 PM: -- after review the code

[jira] [Comment Edited] (KAFKA-4447) Controller resigned but it also acts as a controller for a long time

[ https://issues.apache.org/jira/browse/KAFKA-4447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696340#comment-15696340 ] Json Tu edited comment on KAFKA-4447 at 11/25/16 5:38 PM: -- after check

Re: A strange controller log in Kafka 0.9.0.1

ized. But if you continuously see broker > 100's listener fires and it acts like a controller then there may be an > issue with 0.9.0.1 version. > > Guozhang > > On Wed, Nov 23, 2016 at 7:28 AM, Json Tu <kafka...@126.com> wrote: > >> Hi, >>We have a

[jira] [Updated] (KAFKA-4447) Controller resigned but it also acts as a controller for a long time

[ https://issues.apache.org/jira/browse/KAFKA-4447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Json Tu updated KAFKA-4447: --- Reviewer: Guozhang Wang > Controller resigned but it also acts as a controller for a long t

[jira] [Comment Edited] (KAFKA-4447) Controller resigned but it also acts as a controller for a long time

[ https://issues.apache.org/jira/browse/KAFKA-4447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696340#comment-15696340 ] Json Tu edited comment on KAFKA-4447 at 11/25/16 5:27 PM: -- after check

[jira] [Commented] (KAFKA-4447) Controller resigned but it also acts as a controller for a long time

[ https://issues.apache.org/jira/browse/KAFKA-4447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696353#comment-15696353 ] Json Tu commented on KAFKA-4447: after review the code of kafka 0.9.0.0 and some new versions, can we add

[jira] [Commented] (KAFKA-4447) Controller resigned but it also acts as a controller for a long time

[ https://issues.apache.org/jira/browse/KAFKA-4447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696340#comment-15696340 ] Json Tu commented on KAFKA-4447: after check the email's response in the dev's mail list，I review

[jira] [Updated] (KAFKA-4447) Controller resigned but it also acts as a controller for a long time

[ https://issues.apache.org/jira/browse/KAFKA-4447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Json Tu updated KAFKA-4447: --- Attachment: log.tar.gz I upload the log's tar, we can unzip it using tar tools > Controller resig

[jira] [Created] (KAFKA-4447) Controller resigned but it also acts as a controller for a long time

Json Tu created KAFKA-4447: -- Summary: Controller resigned but it also acts as a controller for a long time Key: KAFKA-4447 URL: https://issues.apache.org/jira/browse/KAFKA-4447 Project: Kafka

A strange controller log in Kafka 0.9.0.1

Hi, We have a cluster of kafka 0.9.0.1 with 3 nodes, and we found a strange controller log as below. [2016-11-07 03:14:48,575] INFO [SessionExpirationListener on 100], ZK expired; shut down all controller components and try to re-elect

replica fetch error and shuabing

Hi, when I move __consumer_offsets from old broker to new broker, we encounter error as follow and it always shuabing. server.log.2016-11-07-19:[2016-11-07 19:17:15,392] ERROR Found invalid messages during fetch for partition [__consumer_offsets,10] offset 13973569 error Message found with

[jira] [Commented] (KAFKA-4360) Controller may deadLock when autoLeaderRebalance encounter zk expired

[ https://issues.apache.org/jira/browse/KAFKA-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15631474#comment-15631474 ] Json Tu commented on KAFKA-4360: I'm sorry, I rollback one comment commit, and https://github.com/apache

[jira] [Updated] (KAFKA-4360) Controller may deadLock when autoLeaderRebalance encounter zk expired

[ https://issues.apache.org/jira/browse/KAFKA-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Json Tu updated KAFKA-4360: --- Reviewer: Jiangjie Qin (was: Json Tu) > Controller may deadLock when autoLeaderRebalance encounter

[jira] [Comment Edited] (KAFKA-4360) Controller may deadLock when autoLeaderRebalance encounter zk expired

[ https://issues.apache.org/jira/browse/KAFKA-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15625317#comment-15625317 ] Json Tu edited comment on KAFKA-4360 at 11/1/16 12:38 PM: -- I put a pull

[jira] [Commented] (KAFKA-4360) Controller may deadLock when autoLeaderRebalance encounter zk expired

[ https://issues.apache.org/jira/browse/KAFKA-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15625317#comment-15625317 ] Json Tu commented on KAFKA-4360: I put a pull request：https://github.com/apache/kafka/pull/2085，can

[jira] [Comment Edited] (KAFKA-4360) Controller may deadLock when autoLeaderRebalance encounter zk expired

[ https://issues.apache.org/jira/browse/KAFKA-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15624461#comment-15624461 ] Json Tu edited comment on KAFKA-4360 at 11/1/16 6:22 AM: - it is wonderful，I search

[jira] [Updated] (KAFKA-4360) Controller may deadLock when autoLeaderRebalance encounter zk expired

[ https://issues.apache.org/jira/browse/KAFKA-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Json Tu updated KAFKA-4360: --- Reviewer: Json Tu > Controller may deadLock when autoLeaderRebalance encounter zk expi

[jira] [Comment Edited] (KAFKA-4360) Controller may deadLock when autoLeaderRebalance encounter zk expired

[ https://issues.apache.org/jira/browse/KAFKA-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15624461#comment-15624461 ] Json Tu edited comment on KAFKA-4360 at 11/1/16 5:57 AM: - it is wonderful，I search

[jira] [Comment Edited] (KAFKA-4360) Controller may deadLock when autoLeaderRebalance encounter zk expired

[ https://issues.apache.org/jira/browse/KAFKA-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15624461#comment-15624461 ] Json Tu edited comment on KAFKA-4360 at 11/1/16 5:50 AM: - it is wonderful，I search

[jira] [Comment Edited] (KAFKA-4360) Controller may deadLock when autoLeaderRebalance encounter zk expired

[ https://issues.apache.org/jira/browse/KAFKA-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15624461#comment-15624461 ] Json Tu edited comment on KAFKA-4360 at 11/1/16 5:49 AM: - it is wonderful，I search

[jira] [Commented] (KAFKA-4360) Controller may deadLock when autoLeaderRebalance encounter zk expired

[ https://issues.apache.org/jira/browse/KAFKA-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15624461#comment-15624461 ] Json Tu commented on KAFKA-4360: it is wonderful，I search onControllerResignation() in kafka codes. just

KAFKA-4360 issue

wse/KAFKA-4360 >>Project: Kafka >> Issue Type: Bug >> Components: controller >> Affects Versions: 0.9.0.0, 0.9.0.1, 0.10.0.0, 0.10.0.1 >> Reporter: Json Tu >> Labels: bugfix >>

KAFKA-4360 issue

> URL: https://issues.apache.org/jira/browse/KAFKA-4360 >> Project: Kafka >> Issue Type: Bug >>Components: controller >> Affects Versions: 0.9.0.0, 0.9.0.1, 0.10.0.0, 0.10.0.1 >> Reporter: Json Tu >>

[jira] [Issue Comment Deleted] (KAFKA-4360) Controller may deadLock when autoLeaderRebalance encounter zk expired

[ https://issues.apache.org/jira/browse/KAFKA-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Json Tu updated KAFKA-4360: --- Comment: was deleted (was: add a deadlock_patch to kafka.) > Controller may deadLock w

[jira] [Updated] (KAFKA-4360) Controller may deadLock when autoLeaderRebalance encounter zk expired

[ https://issues.apache.org/jira/browse/KAFKA-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Json Tu updated KAFKA-4360: --- Attachment: deadlock_patch add a deadlock_patch to kafka. > Controller may deadLock when autoLeaderRebala

[jira] [Comment Edited] (KAFKA-4360) Controller may deadLock when autoLeaderRebalance encounter zk expired

[ https://issues.apache.org/jira/browse/KAFKA-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15624174#comment-15624174 ] Json Tu edited comment on KAFKA-4360 at 11/1/16 3:04 AM: - yes，it is this deadlock

[jira] [Comment Edited] (KAFKA-4360) Controller may deadLock when autoLeaderRebalance encounter zk expired

[ https://issues.apache.org/jira/browse/KAFKA-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15624174#comment-15624174 ] Json Tu edited comment on KAFKA-4360 at 11/1/16 3:04 AM: - yes，I think we should

[jira] [Commented] (KAFKA-4360) Controller may deadLock when autoLeaderRebalance encounter zk expired

[ https://issues.apache.org/jira/browse/KAFKA-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15624174#comment-15624174 ] Json Tu commented on KAFKA-4360: yes，I think we should add a AtomicBoolean variable may be called

[jira] [Updated] (KAFKA-4360) Controller may deadLock when autoLeaderRebalance encounter zk expired

[ https://issues.apache.org/jira/browse/KAFKA-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Json Tu updated KAFKA-4360: --- Description: when controller has checkAndTriggerPartitionRebalance task in autoRebalanceScheduler

[jira] [Updated] (KAFKA-4360) Controller may deadLock when autoLeaderRebalance encounter zk expired

[ https://issues.apache.org/jira/browse/KAFKA-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Json Tu updated KAFKA-4360: --- Attachment: yf-mafka2-common02_jstack.txt the jstack file created when kill kafka > Controller may deadL

[jira] [Commented] (KAFKA-4360) Controller may deadLock when autoLeaderRebalance encounter zk expired

[ https://issues.apache.org/jira/browse/KAFKA-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15622377#comment-15622377 ] Json Tu commented on KAFKA-4360: [~guozhang] [~junrao] do you think this is may be a bug, if so, I

[jira] [Updated] (KAFKA-4360) Controller may deadLock when autoLeaderRebalance encounter zk expired

[ https://issues.apache.org/jira/browse/KAFKA-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Json Tu updated KAFKA-4360: --- Description: when controller has checkAndTriggerPartitionRebalance task in autoRebalanceScheduler

[jira] [Created] (KAFKA-4360) Controller may deadLock when autoLeaderRebalance encounter zk expired

Json Tu created KAFKA-4360: -- Summary: Controller may deadLock when autoLeaderRebalance encounter zk expired Key: KAFKA-4360 URL: https://issues.apache.org/jira/browse/KAFKA-4360 Project: Kafka

Kafka cannot shutdown

Hi all, We have a kafka cluster with 11 nodes, and we found there are some partition’s replica num is not equal to isr’s num，because our data traffic is small，we think it should isr’s num should equal to replica’s num at last， but it can not recovery to normal，so we try to shutdown a

Re: handleFetchRequest throw exception

spam > of repeated error log entries since it should at most print one entry (and > should be DEBUG not ERROR) for each delayed request whose partition leaders > have migrated out. > > > > Guozhang > > > > On Wed, Oct 26, 2016 at 7:46 AM, Json Tu <kafka...@126.

[jira] [Commented] (KAFKA-3536) ReplicaFetcherThread should not log errors when leadership changes

[ https://issues.apache.org/jira/browse/KAFKA-3536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15608708#comment-15608708 ] Json Tu commented on KAFKA-3536: Thank you，But I think I can’t ignore it，because it make the cluster can

Re: handleFetchRequest throw exception

er in this request may not be completely be satisfied and return to the fetch broker， which leads some producer and consumer fail for a longtime，I don’t know is it correct > 在 2016年10月25日，下午8:32，Json Tu <kafka...@126.com> 写道： > > Hi all, > I use Kafka 0.9.0.0, and we have a cluster

handleFetchRequest throw exception

Hi all, I use Kafka 0.9.0.0, and we have a cluster with 6 nodes, when I restart a broker，we find there are many logs as below， [2016-10-24 15:29:00,914] ERROR [KafkaApi-2141642] error when handling request Name: FetchRequest; Version: 1; CorrelationId: 4928; ClientId:

Re: client use high cpu which caused by delayedFetch operation immediately return

Thanks. I patch it, and everything goes ok. > 在 2016年10月9日，下午12:39，Becket Qin 写道： > > Can you check if you have KAFKA-3003 when you run the code? > > On Sat, Oct 8, 2016 at 12:52 AM, Kafka wrote: > >> Hi all, >>we found our consumer have high

Lots of warn log in Kafka broker

Hi all, I have a kafka 0.9.0.0 cluster with 11 nodes. First,I found server logs as below， server.log.2016-10-17-22:[2016-10-17 22:22:13,885] WARN [ReplicaFetcherThread-0-4], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@367c9f98. Possible cause: