wuYin opened a new issue #9297:
URL: https://github.com/apache/pulsar/issues/9297
**Describe the bug**
- Certain issue
After broker restarted with 30s zk session expired, client take 51s, even
1min40s to reconnect successfully, recovery time is a bit long.
- Flaky issue
Client reconnect always failed handshake with proxy, stucked at 1~5
**To Reproduce**
1. use pulsar-helm-chart to deploy a cluster
- 1\*zk, 3\*brokers, 3\*bookies, 3\*proxy, all with 2c 2g resource limit
- 4k QPS, 4MBps In/out, for cpu and memory, load < 35%, see
[3_brokers_loadreport.json.log](https://github.com/apache/pulsar/files/5862180/3_brokers_loadreport.json.log)
2. Manually delete one broker and restart, it's owning topics(bundles) will
be unloaded, clients disconnect with proxy and reconnect, 0.1s, 0.2s, 0.4s,
..., 25.6s at least needed, 51.2s is common.
In flaky case, client reconnect always failed, it has been seen in
production env, but it's not easy to reproduce.
**Expected behavior**
After broker restart, client need reconnect as soon as possible, like within
40s(30s zk session expiration + 10s reconnect interval)
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Other**
How to reduce recovery time?
- Reduce zk session expiration time, default 30s, reduce to 10s, but what is
cost?
- Client backoff policy default exponential, may need provide a option to
limit the backoff limit, such as 5s, backoff should be 0.1s, 0.2s, 0.4s,
...,3.2s, 5s, 5s, ...
- Reduce broker znode expiration waiting time, default 10s, reduce to 2s,
configured in
[broker-statefulset.yaml#L197](https://github.com/apache/pulsar-helm-chart/blob/master/charts/pulsar/templates/broker-statefulset.yaml#L197)
- Flaky issue need be fixed...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
