thetumbled commented on code in PR #4258: URL: https://github.com/apache/bookkeeper/pull/4258#discussion_r1582076373
########## site3/website/src/pages/bps/BP-66-support-throttling-for-zookeeper-read-of-rereplication.md: ########## @@ -0,0 +1,26 @@ +# BP-66: support throttling for zookeeper read of rereplication + +### Motivation + +Each time the cluster trigger the re-replication, all replicators will read data from zookeeper. This can cause a great pressure on Zookeeper. We need to support throttling for zookeeper read of re-replication. + +For example, in a Pulsar cluster, we enable auto-recovery for every bookie. There are 400 bookies in a cluster, which means there are 400 replicators in the cluster. +And there are about 3000 ledgers in each bookie, 1/3 of them are small ledgers, whose size is less than 0.1MB, that is 1000 small ledgers in each bookie. +If we decommission one bookie, the read latency of zookeeper will increase to minutes. + + +### Configuration +add the following configuration: Review Comment: We enable `replicationRateByBytes` to only 3MB as we have 100+ replicators, but the latency of zk read is still very high to minutes level. Each time we decommission a bookie in production cluster, the read latency soar to minute level.  Limitting the byte rate of replication can't relieve the pressure of zk, but avoid the replication throughput of replication is too high that impact the normal client throughput. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@bookkeeper.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
