someone can help to analysis it?

> 在 2017年11月10日,上午11:08,Json Tu <kafka...@126.com> 写道:
> 
> I‘m so sorry for my poor english.
> 
> what I really means is my broker machine is configured as 8 core 16G. but my 
> jvm configure is as below.
> java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 
> -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC 
> -Djava.awt.headless=true -Xloggc:/xx/yy/kafkaServer-gc.log -verbose:gc 
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps 
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=10M 
> -XX:+HeapDumpOnOutOfMemoryError.
> 
> we have 30+ clusters with this jvm configure, and are deployed on the machine 
> which configured as 8 core 16G. compare to other clusters, the current 
> cluster have more than 5 times partitions than other clusters.
> when we restart other clusters,  there is no such phenomenon.
> 
> may be some metrics or logs can leads to find root cause of this phenomenon.
> Looking forward to more suggestions.
> 
> 
>> 在 2017年11月9日,下午9:59,John Yost <hokiege...@gmail.com> 写道:
>> 
>> I've seen this before and it was due to long GC pauses due in large part to
>> a memory heap > 8 GB.
>> 
>> --John
>> 
>> On Thu, Nov 9, 2017 at 8:17 AM, Json Tu <kafka...@126.com> wrote:
>> 
>>> Hi,
>>>   we have a kafka cluster which is made of 6 brokers,  with 8 cpu and
>>> 16G memory on each broker’s machine, and we have about 1600 topics in the
>>> cluster,about 1700 partitions’ leader and 1600 partitions' replica on each
>>> broker.
>>>   when we restart a normal broke,  we find that there are 500+
>>> partitions shrink and expand frequently when restart the broker,
>>> there are many logs as below.
>>> 
>>>  [2017-11-09 17:05:51,173] INFO Partition [Yelp,5] on broker 4759726:
>>> Expanding ISR for partition [Yelp,5] from 4759726 to 4759726,4759750
>>> (kafka.cluster.Partition)
>>> [2017-11-09 17:06:22,047] INFO Partition [Yelp,5] on broker 4759726:
>>> Shrinking ISR for partition [Yelp,5] from 4759726,4759750 to 4759726
>>> (kafka.cluster.Partition)
>>> [2017-11-09 17:06:28,634] INFO Partition [Yelp,5] on broker 4759726:
>>> Expanding ISR for partition [Yelp,5] from 4759726 to 4759726,4759750
>>> (kafka.cluster.Partition)
>>> [2017-11-09 17:06:44,658] INFO Partition [Yelp,5] on broker 4759726:
>>> Shrinking ISR for partition [Yelp,5] from 4759726,4759750 to 4759726
>>> (kafka.cluster.Partition)
>>> [2017-11-09 17:06:47,611] INFO Partition [Yelp,5] on broker 4759726:
>>> Expanding ISR for partition [Yelp,5] from 4759726 to 4759726,4759750
>>> (kafka.cluster.Partition)
>>> [2017-11-09 17:07:19,703] INFO Partition [Yelp,5] on broker 4759726:
>>> Shrinking ISR for partition [Yelp,5] from 4759726,4759750 to 4759726
>>> (kafka.cluster.Partition)
>>> [2017-11-09 17:07:26,811] INFO Partition [Yelp,5] on broker 4759726:
>>> Expanding ISR for partition [Yelp,5] from 4759726 to 4759726,4759750
>>> (kafka.cluster.Partition)
>>> …
>>> 
>>> 
>>>   and repeat shrink and expand after 30 minutes which is the default
>>> value of leader.imbalance.check.interval.seconds, and at that time
>>> we can find the log of controller’s auto rebalance,which can leads some
>>> partition’s leader change to this restarted broker.
>>>   we have no shrink and expand when our cluster is running except when
>>> we restart it,so replica.fetch.thread.num is 1,and it seems enough.
>>> 
>>>   we can reproduce it at each restart,can someone give some suggestions.
>>> thanks before.
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
> 

Reply via email to