[
https://issues.apache.org/jira/browse/KAFKA-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311617#comment-14311617
]
Jiangjie Qin commented on KAFKA-1908:
-------------------------------------
[~aozeritsky] It looks the scenario should not happen by design. Ideally what
should happen is as below:
1. Initially every broker works well. Leader is on broker 1.
2. After port 9092 is disabled, no further connection could be established to
the broker 1, but existing connections are not affected, so broker
3(controller) and broker 1 are still connected.
3. run a preferred leader election successfully.
4. Given what mentioned in 2), broker 3 should be able to send
UpdateMetadataRequest to broker 1, so broker 1 should become follower in that
case.
One possibility I can think of which might cause the result you saw is that
somehow broker 3 to broker 1 connection were lost after you disabled port 9092
on broker 1. In that case, broker 3 cannot connect to broker 1 again so broker
1 will miss UpdateMetadataRequest thus remain to think itself as leader. I
think this is what mentioned by Gwen as "multi-lan" or network partition.
Could you verify from controller log to see if broker 3 to broker 1 connection
has ever broken?
> Split brain
> -----------
>
> Key: KAFKA-1908
> URL: https://issues.apache.org/jira/browse/KAFKA-1908
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8.2
> Reporter: Alexey Ozeritskiy
>
> In some cases, there may be two leaders for one partition.
> Steps to reproduce:
> # We have 3 brokers, 1 partition with 3 replicas:
> {code}
> TopicAndPartition: [partition,0] Leader: 1 Replicas: [2,1,3]
> ISR: [1,2,3]
> {code}
> # controller works on broker 3
> # let the kafka port be 9092. We execute on broker 1:
> {code}
> iptables -A INPUT -p tcp --dport 9092 -j REJECT
> {code}
> # Initiate replica election
> # As a result:
> Broker 1:
> {code}
> TopicAndPartition: [partition,0] Leader: 1 Replicas: [2,1,3]
> ISR: [1,2,3]
> {code}
> Broker 2:
> {code}
> TopicAndPartition: [partition,0] Leader: 2 Replicas: [2,1,3]
> ISR: [1,2,3]
> {code}
> # Flush the iptables rules on broker 1
> Now we can produce messages to {code}[partition,0]{code}. Replica-1 will not
> receive new data. A consumer can read data from replica-1 or replica-2. When
> it reads from replica-1 it resets the offsets and than can read duplicates
> from replica-2.
> We saw this situation in our production cluster when it had network problems.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
