[ https://issues.apache.org/jira/browse/KAFKA-1555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14144378#comment-14144378 ]
Joe Stein commented on KAFKA-1555: ---------------------------------- [~gwenshap] patch applied, tests passed however; I ran with 3 brokers all new code and didn't get the expected results {code} Topic:testDefault PartitionCount:4 ReplicationFactor:3 Configs: Topic: testDefault Partition: 0 Leader: 1 Replicas: 1,2,3 Isr: 1 Topic: testDefault Partition: 1 Leader: 1 Replicas: 2,3,1 Isr: 1 Topic: testDefault Partition: 2 Leader: 1 Replicas: 3,1,2 Isr: 1 Topic: testDefault Partition: 3 Leader: 1 Replicas: 1,3,2 Isr: 1 Topic:testNew PartitionCount:4 ReplicationFactor:3 Configs:min.insync.replicas=2 Topic: testNew Partition: 0 Leader: 1 Replicas: 2,1,3 Isr: 1 Topic: testNew Partition: 1 Leader: 1 Replicas: 3,2,1 Isr: 1 Topic: testNew Partition: 2 Leader: 1 Replicas: 1,3,2 Isr: 1 Topic: testNew Partition: 3 Leader: 1 Replicas: 2,3,1 Isr: 1 {code} I am still able to produce to topic testNew (though I shouldn't be able too since 2 brokers are down and only 1 is up with min.isr=2) I got proper exceptions trying to create invalid values for the config {code} root@precise64:/opt/apache/kafka# bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic testNewA --partitions 4 --replication-factor 3 --config min.insync.replicas=-1 Error while executing topic command Wrong value -1 of min.insync.replicas in Topic configuration; Valid values are at least 1 kafka.common.InvalidConfigException: Wrong value -1 of min.insync.replicas in Topic configuration; Valid values are at least 1 at kafka.log.LogConfig$.validateMinInSyncReplicas(LogConfig.scala:191) at kafka.log.LogConfig$.validate(LogConfig.scala:179) at kafka.admin.TopicCommand$.parseTopicConfigsToBeAdded(TopicCommand.scala:204) at kafka.admin.TopicCommand$.createTopic(TopicCommand.scala:84) at kafka.admin.TopicCommand$.main(TopicCommand.scala:54) at kafka.admin.TopicCommand.main(TopicCommand.scala) root@precise64:/opt/apache/kafka# bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic testNewA --partitions 4 --replication-factor 3 --config min.insync.replicas=4 Error while executing topic command replication factor: 3 larger than available brokers: 1 kafka.admin.AdminOperationException: replication factor: 3 larger than available brokers: 1 at kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:70) at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:171) at kafka.admin.TopicCommand$.createTopic(TopicCommand.scala:92) at kafka.admin.TopicCommand$.main(TopicCommand.scala:54) at kafka.admin.TopicCommand.main(TopicCommand.scala) {code} > provide strong consistency with reasonable availability > ------------------------------------------------------- > > Key: KAFKA-1555 > URL: https://issues.apache.org/jira/browse/KAFKA-1555 > Project: Kafka > Issue Type: Improvement > Components: controller > Affects Versions: 0.8.1.1 > Reporter: Jiang Wu > Assignee: Gwen Shapira > Fix For: 0.8.2 > > Attachments: KAFKA-1555.0.patch, KAFKA-1555.1.patch, > KAFKA-1555.2.patch, KAFKA-1555.3.patch > > > In a mission critical application, we expect a kafka cluster with 3 brokers > can satisfy two requirements: > 1. When 1 broker is down, no message loss or service blocking happens. > 2. In worse cases such as two brokers are down, service can be blocked, but > no message loss happens. > We found that current kafka versoin (0.8.1.1) cannot achieve the requirements > due to its three behaviors: > 1. when choosing a new leader from 2 followers in ISR, the one with less > messages may be chosen as the leader. > 2. even when replica.lag.max.messages=0, a follower can stay in ISR when it > has less messages than the leader. > 3. ISR can contains only 1 broker, therefore acknowledged messages may be > stored in only 1 broker. > The following is an analytical proof. > We consider a cluster with 3 brokers and a topic with 3 replicas, and assume > that at the beginning, all 3 replicas, leader A, followers B and C, are in > sync, i.e., they have the same messages and are all in ISR. > According to the value of request.required.acks (acks for short), there are > the following cases. > 1. acks=0, 1, 3. Obviously these settings do not satisfy the requirement. > 2. acks=2. Producer sends a message m. It's acknowledged by A and B. At this > time, although C hasn't received m, C is still in ISR. If A is killed, C can > be elected as the new leader, and consumers will miss m. > 3. acks=-1. B and C restart and are removed from ISR. Producer sends a > message m to A, and receives an acknowledgement. Disk failure happens in A > before B and C replicate m. Message m is lost. > In summary, any existing configuration cannot satisfy the requirements. -- This message was sent by Atlassian JIRA (v6.3.4#6332)