[
https://issues.apache.org/jira/browse/KAFKA-5758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16186792#comment-16186792
]
Jun Rao commented on KAFKA-5758:
--------------------------------
[~dwvangeest], the problem you described could happen transiently when the
controller changes the assigned replica list since the leader may receive the
changed list before the follower. However, this shouldn't last long. Did you
see NotAssignedReplicaException continuously?
> Reassigning a topic's partitions can adversely impact other topics
> ------------------------------------------------------------------
>
> Key: KAFKA-5758
> URL: https://issues.apache.org/jira/browse/KAFKA-5758
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.10.1.1
> Reporter: David van Geest
> Assignee: Ismael Juma
> Labels: reliability
> Fix For: 1.0.0
>
>
> We've noticed that reassigning a topic's partitions seems to adversely impact
> other topics. Specifically, followers for other topics fall out of the ISR.
> While I'm not 100% sure about why this happens, the scenario seems to be as
> follows:
> 1. Reassignment is manually triggered on topic-partition X-Y, and broker A
> (which used to be a follower for X-Y) is no longer a follower.
> 2. Broker A makes `FetchRequest` including topic-partition X-Y to broker B,
> just after the reassignment.
> 3. Broker B can fulfill the `FetchRequest`, but while trying to do so it
> tries to record the position of "follower" A. This fails, because broker A is
> no longer a follower for X-Y (see exception below).
> 4. The entire `FetchRequest` request fails, and broker A's other followed
> topics start falling behind.
> 5. Depending on the length of the reassignment, this sequence repeats.
> In step 3, we see exceptions like:
> {noformat}
> Error when handling request Name: FetchRequest; Version: 3; CorrelationId:
> 46781859; ClientId: ReplicaFetcherThread-0-1001; ReplicaId: 1006; MaxWait:
> 500 ms; MinBytes: 1 bytes; MaxBytes:10485760 bytes; RequestInfo:
> <LOTS OF PARTITIONS>
> kafka.common.NotAssignedReplicaException: Leader 1001 failed to record
> follower 1006's position -1 since the replica is not recognized to be one of
> the assigned replicas 1001,1004,1005 for partition [topic_being_reassigned,5].
> at kafka.cluster.Partition.updateReplicaLogReadResult(Partition.scala:249)
> at
> kafka.server.ReplicaManager$$anonfun$updateFollowerLogReadResults$2.apply(ReplicaManager.scala:923)
> at
> kafka.server.ReplicaManager$$anonfun$updateFollowerLogReadResults$2.apply(ReplicaManager.scala:920)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at
> kafka.server.ReplicaManager.updateFollowerLogReadResults(ReplicaManager.scala:920)
> at kafka.server.ReplicaManager.fetchMessages(ReplicaManager.scala:481)
> at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:534)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:79)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> Does my assessment make sense? If so, this behaviour seems problematic. A few
> changes that might improve matters (assuming I'm on the right track):
> 1. `FetchRequest` should be able to return partial results
> 2. The broker fulfilling the `FetchRequest` could ignore the
> `NotAssignedReplicaException`, and return results without recording the
> not-any-longer-follower position.
> This behaviour was experienced with 0.10.1.1, although looking at the
> changelogs and the code in question, I don't see any reason why it would have
> changed in later versions.
> Am very interested to have some discussion on this. Thanks!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
