[
https://issues.apache.org/jira/browse/KAFKA-5642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16148549#comment-16148549
]
ASF GitHub Bot commented on KAFKA-5642:
---------------------------------------
GitHub user onurkaraman opened a pull request:
https://github.com/apache/kafka/pull/3765
KAFKA-5642 [WIP]: Use async ZookeeperClient in Controller
Synchronous zookeeper writes means that we wait an entire round trip before
doing the next write. These synchronous writes are happening at a per-partition
granularity in several places, so partition-heavy clusters suffer from the
controller doing many sequential round trips to zookeeper:
- PartitionStateMachine.electLeaderForPartition updates leaderAndIsr in
zookeeper on transition to OnlinePartition. This gets triggered per-partition
sequentially with synchronous writes during controlled shutdown of the shutting
down broker's replicas for which it is the leader.
- ReplicaStateMachine updates leaderAndIsr in zookeeper on transition to
OfflineReplica when calling KafkaController.removeReplicaFromIsr. This gets
triggered per-partition sequentially with synchronous writes for failed or
controlled shutdown brokers.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/onurkaraman/kafka KAFKA-5642
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/3765.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3765
----
commit 4b3e4e8b1d403bf817aeaa9196fb69a15b1a13a0
Author: Onur Karaman <[email protected]>
Date: 2017-07-27T18:29:35Z
KAFKA-5642: Use async ZookeeperClient in Controller
Synchronous zookeeper writes means that we wait an entire round trip before
doing the next write. These synchronous writes are happening at a per-partition
granularity in several places, so partition-heavy clusters suffer from the
controller doing many sequential round trips to zookeeper:
- PartitionStateMachine.electLeaderForPartition updates leaderAndIsr in
zookeeper on transition to OnlinePartition. This gets triggered per-partition
sequentially with synchronous writes during controlled shutdown of the shutting
down broker's replicas for which it is the leader.
- ReplicaStateMachine updates leaderAndIsr in zookeeper on transition to
OfflineReplica when calling KafkaController.removeReplicaFromIsr. This gets
triggered per-partition sequentially with synchronous writes for failed or
controlled shutdown brokers.
----
> Use async ZookeeperClient in Controller
> ---------------------------------------
>
> Key: KAFKA-5642
> URL: https://issues.apache.org/jira/browse/KAFKA-5642
> Project: Kafka
> Issue Type: Sub-task
> Reporter: Onur Karaman
> Assignee: Onur Karaman
>
> Synchronous zookeeper writes means that we wait an entire round trip before
> doing the next write. These synchronous writes are happening at a
> per-partition granularity in several places, so partition-heavy clusters
> suffer from the controller doing many sequential round trips to zookeeper.
> * PartitionStateMachine.electLeaderForPartition updates leaderAndIsr in
> zookeeper on transition to OnlinePartition. This gets triggered per-partition
> sequentially with synchronous writes during controlled shutdown of the
> shutting down broker's replicas for which it is the leader.
> * ReplicaStateMachine updates leaderAndIsr in zookeeper on transition to
> OfflineReplica when calling KafkaController.removeReplicaFromIsr. This gets
> triggered per-partition sequentially with synchronous writes for failed or
> controlled shutdown brokers.
> KAFKA-5501 introduced an async ZookeeperClient that encourages pipelined
> requests to zookeeper. We should replace ZkClient's usage with this client.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
