codelipenghui opened a new issue #4959: Pulsar admin always timeout until restart the broker URL: https://github.com/apache/pulsar/issues/4959 **Describe the bug** pulsar-admin be blocked until restart the broker. Check the broker log, some logs looks abnormal: ``` 11:18:28.793 [pulsar-ordered-OrderedExecutor-1-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x30034ce16030052 local:/172.30 .91.40:50486 remoteserver:172.30.91.42/172.30.91.42:2181 lastZxid:4298756812 xid:616893 sent:616893 recv:673431 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: Wa tchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/172.30.91.44:8080 11:18:33.345 [pulsar-ordered-OrderedExecutor-1-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x30034ce16030052 local:/172.30 .91.40:50486 remoteserver:172.30.91.42/172.30.91.42:2181 lastZxid:4298756821 xid:616896 sent:616896 recv:673435 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: Wa tchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/172.30.91.42:8080 11:18:40.207 [pulsar-ordered-OrderedExecutor-1-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x30034ce16030052 local:/172.30 .91.40:50486 remoteserver:172.30.91.42/172.30.91.42:2181 lastZxid:4298756833 xid:616901 sent:616901 recv:673441 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: Wa tchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/172.30.91.41:8080 11:18:53.793 [pulsar-ordered-OrderedExecutor-1-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x30034ce16030052 local:/172.30 .91.40:50486 remoteserver:172.30.91.42/172.30.91.42:2181 lastZxid:4298756842 xid:616908 sent:616908 recv:673449 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: Wa tchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/172.30.91.44:8080 11:18:58.345 [pulsar-ordered-OrderedExecutor-1-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x30034ce16030052 local:/172.30 .91.40:50486 remoteserver:172.30.91.42/172.30.91.42:2181 lastZxid:4298756851 xid:616912 sent:616912 recv:673454 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: Wa tchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/172.30.91.42:8080 11:19:05.207 [pulsar-ordered-OrderedExecutor-1-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x30034ce16030052 local:/172.30 .91.40:50486 remoteserver:172.30.91.42/172.30.91.42:2181 lastZxid:4298756860 xid:616916 sent:616916 recv:673459 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: Wa tchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/172.30.91.41:8080 11:19:18.793 [pulsar-ordered-OrderedExecutor-1-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x30034ce16030052 local:/172.30 .91.40:50486 remoteserver:172.30.91.42/172.30.91.42:2181 lastZxid:4298756885 xid:616925 sent:616925 recv:673469 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: Wa tchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/172.30.91.44:8080 11:19:23.345 [pulsar-ordered-OrderedExecutor-1-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x30034ce16030052 local:/172.30 .91.40:50486 remoteserver:172.30.91.42/172.30.91.42:2181 lastZxid:4298756886 xid:616928 sent:616928 recv:673473 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: Wa tchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/172.30.91.42:8080 11:19:28.793 [pulsar-ordered-OrderedExecutor-1-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x30034ce16030052 local:/172.30 .91.40:50486 remoteserver:172.30.91.42/172.30.91.42:2181 lastZxid:4298756887 xid:616932 sent:616932 recv:673478 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: Wa tchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/172.30.91.44:8080 11:19:30.207 [pulsar-ordered-OrderedExecutor-1-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x30034ce16030052 local:/172.30 .91.40:50486 remoteserver:172.30.91.42/172.30.91.42:2181 lastZxid:4298756896 xid:616934 sent:616934 recv:673481 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: Wa tchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/172.30.91.41:8080 11:19:48.345 [pulsar-ordered-OrderedExecutor-1-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x30034ce16030052 local:/172.30 .91.40:50486 remoteserver:172.30.91.42/172.30.91.42:2181 lastZxid:4298756905 xid:616944 sent:616944 recv:673492 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: Wa tchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/172.30.91.42:8080 11:19:50.207 [pulsar-ordered-OrderedExecutor-1-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x30034ce16030052 local:/172.30 .91.40:50486 remoteserver:172.30.91.42/172.30.91.42:2181 lastZxid:4298756914 xid:616947 sent:616947 recv:673496 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: Wa tchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/172.30.91.41:8080 11:19:53.793 [pulsar-ordered-OrderedExecutor-1-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x30034ce16030052 local:/172.30 .91.40:50486 remoteserver:172.30.91.42/172.30.91.42:2181 lastZxid:4298756915 xid:616949 sent:616949 recv:673499 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: Wa tchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/172.30.91.44:8080 11:20:13.345 [pulsar-ordered-OrderedExecutor-1-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x30034ce16030052 local:/172.30 .91.40:50486 remoteserver:172.30.91.42/172.30.91.42:2181 lastZxid:4298756924 xid:616960 sent:616960 recv:673511 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: Wa tchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/172.30.91.42:8080 11:20:18.793 [pulsar-ordered-OrderedExecutor-1-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x30034ce16030052 local:/172.30 .91.40:50486 remoteserver:172.30.91.42/172.30.91.42:2181 lastZxid:4298756925 xid:616964 sent:616964 recv:673516 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: Wa tchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/172.30.91.44:8080 11:20:20.207 [pulsar-ordered-OrderedExecutor-1-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x30034ce16030052 local:/172.30 .91.40:50486 remoteserver:172.30.91.42/172.30.91.42:2181 lastZxid:4298756926 xid:616966 sent:616966 recv:673519 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: Wa tchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/172.30.91.41:8080 11:20:28.345 [pulsar-ordered-OrderedExecutor-1-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x30034ce16030052 local:/172.30 .91.40:50486 remoteserver:172.30.91.42/172.30.91.42:2181 lastZxid:4298756927 xid:616971 sent:616971 recv:673525 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: Wa tchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/172.30.91.42:8080 11:20:43.793 [pulsar-ordered-OrderedExecutor-1-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x30034ce16030052 local:/172.30 .91.40:50486 remoteserver:172.30.91.42/172.30.91.42:2181 lastZxid:4298756936 xid:616979 sent:616979 recv:673534 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: Wa tchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/172.30.91.44:8080 11:20:45.207 [pulsar-ordered-OrderedExecutor-1-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x30034ce16030052 local:/172.30 .91.40:50486 remoteserver:172.30.91.42/172.30.91.42:2181 lastZxid:4298756937 xid:616981 sent:616981 recv:673537 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: Wa tchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/172.30.91.41:8080 11:20:53.345 [pulsar-ordered-OrderedExecutor-1-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x30034ce16030052 local:/172.30 .91.40:50486 remoteserver:172.30.91.42/172.30.91.42:2181 lastZxid:4298756946 xid:616986 sent:616986 recv:673543 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: Wa tchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/172.30.91.42:8080 11:21:08.793 [pulsar-ordered-OrderedExecutor-1-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x30034ce16030052 local:/172.30 .91.40:50486 remoteserver:172.30.91.42/172.30.91.42:2181 lastZxid:4298756963 xid:616995 sent:616995 recv:673553 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: Wa tchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/172.30.91.44:8080 11:21:10.207 [pulsar-ordered-OrderedExecutor-1-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x30034ce16030052 local:/172.30 .91.40:50486 remoteserver:172.30.91.42/172.30.91.42:2181 lastZxid:4298756964 xid:616997 sent:616997 recv:673556 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: Wa tchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/172.30.91.41:8080 11:21:18.345 [pulsar-ordered-OrderedExecutor-1-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x30034ce16030052 local:/172.30 .91.40:50486 remoteserver:172.30.91.42/172.30.91.42:2181 lastZxid:4298756965 xid:617002 sent:617002 recv:673562 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: Wa tchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/172.30.91.42:8080 11:21:18.793 [pulsar-ordered-OrderedExecutor-1-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x30034ce16030052 local:/172.30 .91.40:50486 remoteserver:172.30.91.42/172.30.91.42:2181 lastZxid:4298756966 xid:617003 sent:617003 recv:673564 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: Wa tchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/172.30.91.44:8080 11:21:40.207 [pulsar-ordered-OrderedExecutor-1-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x30034ce16030052 local:/172.30 .91.40:50486 remoteserver:172.30.91.42/172.30.91.42:2181 lastZxid:4298757073 xid:617015 sent:617015 recv:673577 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: Wa tchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/172.30.91.41:8080 11:21:43.345 [pulsar-ordered-OrderedExecutor-1-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x30034ce16030052 local:/172.30 .91.40:50486 remoteserver:172.30.91.42/172.30.91.42:2181 lastZxid:4298757074 xid:617017 sent:617017 recv:673580 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: Wa tchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/172.30.91.42:8080 11:21:43.793 [pulsar-ordered-OrderedExecutor-1-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x30034ce16030052 local:/172.30 .91.40:50486 remoteserver:172.30.91.42/172.30.91.42:2181 lastZxid:4298757075 xid:617018 sent:617018 recv:673582 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: Wa tchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/172.30.91.44:8080 11:21:47.965 [pulsar-load-manager-4-1] INFO org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - Writing local data to ZooKeeper because maximum change 18.73050332069397% exc eeded threshold 10%; time since last report written is 420.0 seconds ``` And then make the heap dump of the broker: [172.30.91.40.dump.tar.zip](https://github.com/apache/pulsar/files/3504896/meta_block_172.30.91.40.tar.zip)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
