[
https://issues.apache.org/jira/browse/KAFKA-5116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15981382#comment-15981382
]
Ismael Juma commented on KAFKA-5116:
------------------------------------
cc [~onurkaraman] [~junrao]
> Controller updates to ISR holds the controller lock for a very long time
> ------------------------------------------------------------------------
>
> Key: KAFKA-5116
> URL: https://issues.apache.org/jira/browse/KAFKA-5116
> Project: Kafka
> Issue Type: Bug
> Components: controller
> Affects Versions: 0.10.1.0, 0.10.2.0
> Reporter: Justin Downing
> Fix For: 0.11.0.0
>
>
> Hello!
> Lately, we have noticed slow (or no) results when monitoring the broker's ISR
> using JMX. Many of these requests appear to be 'hung' for a very long time
> (eg: >2m). We've dug a bunch, and found that in our case, sometimes the
> controllerLock can be held for multiple minutes in the IsrChangeNotifier
> callback.
> Inside the lock, we are reading from Zookeeper for *each* partition in the
> changeset. With a large changeset (eg: >500 partitions), [this
> operation|https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/controller/KafkaController.scala#L1347]
> can take a long time to complete.
> In KAFKA-2406, throttling was introduced to prevent overwhelming the
> controller with many changesets at once. However, this does not take into
> consideration _large_ changesets.
> We have identified two potential remediations we'd like to discuss further:
> * Move the Zookeeper request outside of the lock. This would then only lock
> for the controller update and processing of the changeset.
> * Send limited changesets to Zookeeper when calling the
> maybePropagateIsrChanges. When dealing with lots of partitions (eg: >1000) it
> may be useful to batch the changesets in groups of 100 rather the send the
> [entire
> list|https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/ReplicaManager.scala#L204]
> to Zookeeper at once.
> We're happy working on patches for either or both of these, but we are unsure
> of the safety around these two proposals. Specifically, moving the Zookeeper
> request out of the lock may be unsafe.
> Holding these locks for long periods of time seems problematic - it means
> that broker failure won't be detected and acted upon quickly.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
