[
https://issues.apache.org/jira/browse/KAFKA-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin McCabe resolved KAFKA-4712.
---------------------------------
Resolution: Cannot Reproduce
This seems to be related to overall load on the cluster as a result of the
reassignment. Let's close unless there is more to investigate here.
> kafka-reassign-partitions causes ISR to shrink
> ----------------------------------------------
>
> Key: KAFKA-4712
> URL: https://issues.apache.org/jira/browse/KAFKA-4712
> Project: Kafka
> Issue Type: Bug
> Components: admin
> Affects Versions: 0.10.1.1
> Reporter: Andrew Holford
> Priority: Major
>
> Hi,
> We have been doing some work on our kafka clusters recently which required us
> to shrink the number of brokers and then increase the number again once the
> work on the hosts was complete.
> During the reassign partitions work to move data away from and back to the
> servers we have worked on, the ISR has been seen to drop below our
> min.insync.replicas value causing an outage to our data flow until the ISR
> increases again.
> From the documentation our understanding was that during the reassignment the
> number of replicas should increase to include the new ones and then decrease
> back to the replication factor, is this not the case?
> The process we have followed is below:
> 1. generate a json which contains all our topics..
> for example something like...
> {"version":1,"topics":[{"topic":"topic1"},{"topic":"topic2"},{"topic":"topic3"}]}
> 2. use the topic list json with the kafka-partition-assignment tool with the
> generate option, eg:
> bin/kafka-reassign-partitions --zookeeper
> zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181 --topics-to-move-json-file
> /tmp/topic-list.json --broker-list "1,2,3,4,5,6,7" --generate
> 3. we then take the proposed configuration, place in another file, and
> execute as below:
> bin/kafka-reassign-partitions --zookeeper
> zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181 --reassignment-json-file
> /tmp/topic-to-reassign.json --execute
> During the execution, we see lots of times different topics and different
> partitions reduce to 1 ISR, one example below:
> [2017-01-30 12:22:38,748] INFO Partition
> [com_ig_trade_v1_position_snapshot--live,9] on broker 5: Expanding ISR for
> partition [com_ig_trade_v1_position_snapshot--live,9] from 5,1 to 5,1,7
> (kafka.cluster.Partition)
> [2017-01-30 12:22:38,712] INFO Partition
> [com_ig_trade_v1_position_snapshot--live,9] on broker 5: Expanding ISR for
> partition [com_ig_trade_v1_position_snapshot--live,9] from 5 to 5,1
> (kafka.cluster.Partition)
> [2017-01-30 12:22:38,582] INFO Partition
> [com_ig_trade_v1_position_snapshot--live,9] on broker 5: Shrinking ISR for
> partition [com_ig_trade_v1_position_snapshot--live,9] from 5,1,7 to 5
> (kafka.cluster.Partition)
> [2017-01-30 12:22:38,326] INFO Partition
> [com_ig_trade_v1_position_snapshot--live,9] on broker 2: Expanding ISR for
> partition [com_ig_trade_v1_position_snapshot--live,9] from 2,6,1,5 to
> 5,1,6,2,7 (kafka.cluster.Partition)
> [2017-01-30 12:14:49,906] INFO Partition
> [com_ig_trade_v1_position_snapshot--live,9] on broker 2: Expanding ISR for
> partition [com_ig_trade_v1_position_snapshot--live,9] from 2,6,1 to 2,6,1,5
> (kafka.cluster.Partition)
> Is this what is expected?
> Do you have any recommendations / procedure / best practice which we could
> use when decreasing / increasing the cluster size, without the ISRs shrinking?
> Thanks,
> Andy.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
