wenjun8888 opened a new issue #7876: URL: https://github.com/apache/pulsar/issues/7876
hello sir. I setup an geo-replication pulsar cluster which called alpha and beta. We use alpha in our daily product env, now i find beta cluster always shut down by itselt, when i start it again, after maybe 30 or 40 minutes, i still shut down, I find some logs in bookie: ``` Aug 22 16:41:15 dc3-pulsar-ec-beta-pro01 pulsar[29710]: 16:41:15.340 [ReplicationWorker] ERROR org.apache.bookkeeper.replication.ReplicationWorker - ReplicationWorker failed to replicate Ledger : 120281 for 16 number of times, so deferring the ledger lock release by 300000 msecs Aug 22 16:41:15 dc3-pulsar-ec-beta-pro01 pulsar[29710]: 16:41:15.563 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:10.30.12.11:3181>, <Bookie:10.30.12.10:3181>, <Bookie:10.30.12.12:3181>], allBookies [<Bookie:10.30.12.11:3181>, <Bookie:10.30.12.12:3181>, <Bookie:10.30.12.10:3181>]. Aug 22 16:41:15 dc3-pulsar-ec-beta-pro01 pulsar[29710]: 16:41:15.564 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie: excluded [<Bookie:10.30.12.11:3181>, <Bookie:10.30.12.10:3181>, <Bookie:10.30.12.12:3181>], fallback to choose bookie randomly from the cluster. Aug 22 16:41:15 dc3-pulsar-ec-beta-pro01 pulsar[29710]: 16:41:15.564 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:10.30.12.11:3181>, <Bookie:10.30.12.10:3181>, <Bookie:10.30.12.12:3181>], allBookies [<Bookie:10.30.12.12:3181>, <Bookie:10.30.12.11:3181>, <Bookie:10.30.12.10:3181>]. Aug 22 16:41:15 dc3-pulsar-ec-beta-pro01 pulsar[29710]: 16:41:15.564 [ReplicationWorker] WARN org.apache.bookkeeper.replication.ReplicationWorker - BKNotEnoughBookiesException while replicating the fragment Aug 22 16:41:15 dc3-pulsar-ec-beta-pro01 pulsar[29710]: org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available Aug 22 16:41:15 dc3-pulsar-ec-beta-pro01 pulsar[29710]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandomInternal(RackawareEnsemblePlacementPolicyImpl.java:772) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] Aug 22 16:41:15 dc3-pulsar-ec-beta-pro01 pulsar[29710]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandom(RackawareEnsemblePlacementPolicyImpl.java:690) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] Aug 22 16:41:15 dc3-pulsar-ec-beta-pro01 pulsar[29710]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:579) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] Aug 22 16:41:15 dc3-pulsar-ec-beta-pro01 pulsar[29710]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:203) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] Aug 22 16:41:15 dc3-pulsar-ec-beta-pro01 pulsar[29710]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:539) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] Aug 22 16:41:15 dc3-pulsar-ec-beta-pro01 pulsar[29710]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:224) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] Aug 22 16:41:15 dc3-pulsar-ec-beta-pro01 pulsar[29710]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.replaceBookie(RackawareEnsemblePlacementPolicyImpl.java:467) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] Aug 22 16:41:15 dc3-pulsar-ec-beta-pro01 pulsar[29710]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.replaceBookie(RackawareEnsemblePlacementPolicy.java:117) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] Aug 22 16:41:15 dc3-pulsar-ec-beta-pro01 pulsar[29710]: at org.apache.bookkeeper.client.BookKeeperAdmin.getReplacementBookiesByIndexes(BookKeeperAdmin.java:1012) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] Aug 22 16:41:15 dc3-pulsar-ec-beta-pro01 pulsar[29710]: at org.apache.bookkeeper.client.BookKeeperAdmin.replicateLedgerFragment(BookKeeperAdmin.java:1061) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] Aug 22 16:41:15 dc3-pulsar-ec-beta-pro01 pulsar[29710]: at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:384) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] Aug 22 16:41:15 dc3-pulsar-ec-beta-pro01 pulsar[29710]: at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:277) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] Aug 22 16:41:15 dc3-pulsar-ec-beta-pro01 pulsar[29710]: at org.apache.bookkeeper.replication.ReplicationWorker.run(ReplicationWorker.java:238) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] Aug 22 16:41:15 dc3-pulsar-ec-beta-pro01 pulsar[29710]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final] Aug 22 16:41:15 dc3-pulsar-ec-beta-pro01 pulsar[29710]: at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112] ``` and in broker, it is: ``` Aug 22 15:59:37 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 15:59:37.390 [bookkeeper-ml-workers-OrderedExecutor-1-0] ERROR org.apache.pulsar.broker.service.persistent.PersistentReplicator - [persistent://dominos_pro/user/send_coupon_log-partition-1][dominos-pulsar-beta -> dominos-pulsar-alpha] Unexpected exception: null Aug 22 15:59:37 dc3-pulsar-ec-beta-pro02 pulsar[13707]: java.lang.NullPointerException: null Aug 22 15:59:37 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.service.persistent.PersistentSubscription.acknowledgeMessage(PersistentSubscription.java:386) ~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 15:59:37 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.service.persistent.ReplicatedSubscriptionsController.receiveSubscriptionUpdated(ReplicatedSubscriptionsController.java:177) ~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 15:59:37 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.service.persistent.ReplicatedSubscriptionsController.receivedReplicatedSubscriptionMarker(ReplicatedSubscriptionsController.java:92) ~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 15:59:37 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.service.persistent.PersistentTopic.receivedReplicatedSubscriptionMarker(PersistentTopic.java:2155) ~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 15:59:37 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.service.persistent.PersistentReplicator.checkReplicatedSubscriptionMarker(PersistentReplicator.java:745) ~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 15:59:37 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.service.persistent.PersistentReplicator.readEntriesComplete(PersistentReplicator.java:351) ~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 15:59:37 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.bookkeeper.mledger.impl.OpReadEntry.lambda$checkReadCompletion$2(OpReadEntry.java:152) ~[org.apache.pulsar-managed-ledger-2.6.0.jar:2.6.0] Aug 22 16:00:08 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:08.268 [pulsar-load-manager-3-1] WARN org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - Error when attempting to update local broker data Aug 22 16:00:08 dc3-pulsar-ec-beta-pro02 pulsar[13707]: java.lang.NullPointerException: null Aug 22 16:00:08 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.getBundleStats(ModularLoadManagerImpl.java:416) ~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:08 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.updateLocalBrokerData(ModularLoadManagerImpl.java:939) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:08 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.writeBrokerDataOnZooKeeper(ModularLoadManagerImpl.java:975) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:08 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerWrapper.writeLoadReportOnZookeeper(ModularLoadManagerWrapper.java:115) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:08 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.LoadReportUpdaterTask.run(LoadReportUpdaterTask.java:41) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:08 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_112] Aug 22 16:00:08 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_112] Aug 22 16:00:08 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_112] Aug 22 16:00:08 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_112] Aug 22 16:00:08 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_112] Aug 22 16:00:08 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_112] Aug 22 16:00:08 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final] Aug 22 16:00:08 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112] Aug 22 16:00:13 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:13.268 [pulsar-load-manager-3-1] WARN org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - Error when attempting to update local broker data Aug 22 16:00:13 dc3-pulsar-ec-beta-pro02 pulsar[13707]: java.lang.NullPointerException: null Aug 22 16:00:13 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.getBundleStats(ModularLoadManagerImpl.java:416) ~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:13 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.updateLocalBrokerData(ModularLoadManagerImpl.java:939) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:13 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.writeBrokerDataOnZooKeeper(ModularLoadManagerImpl.java:975) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:13 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerWrapper.writeLoadReportOnZookeeper(ModularLoadManagerWrapper.java:115) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:13 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.LoadReportUpdaterTask.run(LoadReportUpdaterTask.java:41) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:13 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_112] Aug 22 16:00:13 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_112] Aug 22 16:00:13 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_112] Aug 22 16:00:13 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_112] Aug 22 16:00:13 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_112] Aug 22 16:00:13 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_112] Aug 22 16:00:13 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final] Aug 22 16:00:13 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112] Aug 22 16:00:18 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:18.268 [pulsar-load-manager-3-1] WARN org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - Error when attempting to update local broker data Aug 22 16:00:18 dc3-pulsar-ec-beta-pro02 pulsar[13707]: java.lang.NullPointerException: null Aug 22 16:00:18 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.getBundleStats(ModularLoadManagerImpl.java:416) ~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:18 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.updateLocalBrokerData(ModularLoadManagerImpl.java:939) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:18 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.writeBrokerDataOnZooKeeper(ModularLoadManagerImpl.java:975) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:18 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerWrapper.writeLoadReportOnZookeeper(ModularLoadManagerWrapper.java:115) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:18 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.LoadReportUpdaterTask.run(LoadReportUpdaterTask.java:41) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:18 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_112] Aug 22 16:00:18 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_112] Aug 22 16:00:18 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_112] Aug 22 16:00:18 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_112] Aug 22 16:00:18 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_112] Aug 22 16:00:18 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_112] Aug 22 16:00:18 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final] Aug 22 16:00:18 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112] Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.274 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44863 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.275 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Timed-out 33 operations to channel [id: 0x314748d8, L:/10.30.12.11:60426 - R:10.30.12.11/10.30.12.11:3181] for 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.275 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Timed-out 5 operations to channel [id: 0x1b104795, L:/10.30.12.11:60444 - R:10.30.12.11/10.30.12.11:3181] for 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.275 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Timed-out 67 operations to channel [id: 0xcaacd0ba, L:/10.30.12.11:42884 - R:10.30.12.10/10.30.12.10:3181] for 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.276 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44803 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.276 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44860 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.276 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44818 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.276 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44785 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.276 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44806 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.276 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44845 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.276 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44800 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.276 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44842 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.276 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44854 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.276 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44782 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.276 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44848 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.276 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44773 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.276 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44812 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.276 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44827 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.276 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44857 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.276 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44788 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.276 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44851 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.276 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44794 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.276 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44824 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.276 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44821 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.276 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44797 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.276 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44839 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.276 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44776 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.276 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44830 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.276 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44833 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.276 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44815 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.276 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44767 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.276 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44779 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.276 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44836 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.276 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44809 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.276 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44791 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.276 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44770 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.276 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L144878 E44951 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L144878 E44960 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L144878 E44954 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L144878 E44963 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L144878 E44957 from bookie: 10.30.12.11:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44786 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44852 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44853 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44826 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44823 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44838 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44783 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44856 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44795 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44819 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44829 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44772 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44780 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44813 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44796 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44775 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44834 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44801 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44849 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44807 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44793 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44805 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44814 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44774 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44799 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44850 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44777 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44769 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44858 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44784 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44808 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44768 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44847 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44835 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44765 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44832 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44804 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44778 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44855 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44810 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44825 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44798 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44861 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44844 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44816 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44792 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44843 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44828 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44837 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44790 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44787 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44831 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44864 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44817 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.277 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44841 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44859 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44846 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44766 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44781 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44802 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44789 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44840 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44771 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44862 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44822 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44820 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L145030 E44811 from bookie: 10.30.12.10:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: BookKeeper client is closed while reading L145030 E44863 from bookie: 10.30.12.12:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: BookKeeper client is closed while reading L145030 E44803 from bookie: 10.30.12.12:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: BookKeeper client is closed while reading L145030 E44860 from bookie: 10.30.12.12:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: BookKeeper client is closed while reading L145030 E44818 from bookie: 10.30.12.12:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: BookKeeper client is closed while reading L145030 E44785 from bookie: 10.30.12.12:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: BookKeeper client is closed while reading L145030 E44806 from bookie: 10.30.12.12:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: BookKeeper client is closed while reading L145030 E44845 from bookie: 10.30.12.12:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: BookKeeper client is closed while reading L145030 E44800 from bookie: 10.30.12.12:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: BookKeeper client is closed while reading L145030 E44842 from bookie: 10.30.12.12:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: BookKeeper client is closed while reading L145030 E44854 from bookie: 10.30.12.12:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: BookKeeper client is closed while reading L145030 E44782 from bookie: 10.30.12.12:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: BookKeeper client is closed while reading L145030 E44848 from bookie: 10.30.12.12:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: BookKeeper client is closed while reading L145030 E44773 from bookie: 10.30.12.12:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: BookKeeper client is closed while reading L145030 E44812 from bookie: 10.30.12.12:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: BookKeeper client is closed while reading L145030 E44827 from bookie: 10.30.12.12:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: BookKeeper client is closed while reading L145030 E44857 from bookie: 10.30.12.12:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: BookKeeper client is closed while reading L145030 E44788 from bookie: 10.30.12.12:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: BookKeeper client is closed while reading L145030 E44851 from bookie: 10.30.12.12:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: BookKeeper client is closed while reading L145030 E44794 from bookie: 10.30.12.12:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: BookKeeper client is closed while reading L145030 E44824 from bookie: 10.30.12.12:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: BookKeeper client is closed while reading L145030 E44821 from bookie: 10.30.12.12:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: BookKeeper client is closed while reading L145030 E44797 from bookie: 10.30.12.12:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: BookKeeper client is closed while reading L145030 E44839 from bookie: 10.30.12.12:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: BookKeeper client is closed while reading L145030 E44776 from bookie: 10.30.12.12:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: BookKeeper client is closed while reading L145030 E44830 from bookie: 10.30.12.12:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: BookKeeper client is closed while reading L145030 E44833 from bookie: 10.30.12.12:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: BookKeeper client is closed while reading L145030 E44815 from bookie: 10.30.12.12:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: BookKeeper client is closed while reading L145030 E44767 from bookie: 10.30.12.12:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: BookKeeper client is closed while reading L145030 E44779 from bookie: 10.30.12.12:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: BookKeeper client is closed while reading L145030 E44836 from bookie: 10.30.12.12:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: BookKeeper client is closed while reading L145030 E44809 from bookie: 10.30.12.12:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: BookKeeper client is closed while reading L145030 E44791 from bookie: 10.30.12.12:3181 Aug 22 16:00:22 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:22.278 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: BookKeeper client is closed while reading L145030 E44770 from bookie: 10.30.12.12:3181 Aug 22 16:00:23 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:23.268 [pulsar-load-manager-3-1] WARN org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - Error when attempting to update local broker data Aug 22 16:00:23 dc3-pulsar-ec-beta-pro02 pulsar[13707]: java.lang.NullPointerException: null Aug 22 16:00:23 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.getBundleStats(ModularLoadManagerImpl.java:416) ~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:23 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.updateLocalBrokerData(ModularLoadManagerImpl.java:939) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:23 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.writeBrokerDataOnZooKeeper(ModularLoadManagerImpl.java:975) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:23 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerWrapper.writeLoadReportOnZookeeper(ModularLoadManagerWrapper.java:115) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:23 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.LoadReportUpdaterTask.run(LoadReportUpdaterTask.java:41) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:23 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_112] Aug 22 16:00:23 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_112] Aug 22 16:00:23 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_112] Aug 22 16:00:23 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_112] Aug 22 16:00:23 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_112] Aug 22 16:00:23 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_112] Aug 22 16:00:23 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final] Aug 22 16:00:23 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112] Aug 22 16:00:28 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:28.268 [pulsar-load-manager-3-1] WARN org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - Error when attempting to update local broker data Aug 22 16:00:28 dc3-pulsar-ec-beta-pro02 pulsar[13707]: java.lang.NullPointerException: null Aug 22 16:00:28 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.getBundleStats(ModularLoadManagerImpl.java:416) ~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:28 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.updateLocalBrokerData(ModularLoadManagerImpl.java:939) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:28 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.writeBrokerDataOnZooKeeper(ModularLoadManagerImpl.java:975) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:28 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerWrapper.writeLoadReportOnZookeeper(ModularLoadManagerWrapper.java:115) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:28 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.LoadReportUpdaterTask.run(LoadReportUpdaterTask.java:41) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:28 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_112] Aug 22 16:00:28 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_112] Aug 22 16:00:28 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_112] Aug 22 16:00:28 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_112] Aug 22 16:00:28 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_112] Aug 22 16:00:28 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_112] Aug 22 16:00:28 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final] Aug 22 16:00:28 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112] Aug 22 16:00:33 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:33.268 [pulsar-load-manager-3-1] WARN org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - Error when attempting to update local broker data Aug 22 16:00:33 dc3-pulsar-ec-beta-pro02 pulsar[13707]: java.lang.NullPointerException: null Aug 22 16:00:33 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.getBundleStats(ModularLoadManagerImpl.java:416) ~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:33 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.updateLocalBrokerData(ModularLoadManagerImpl.java:939) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:33 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.writeBrokerDataOnZooKeeper(ModularLoadManagerImpl.java:975) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:33 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerWrapper.writeLoadReportOnZookeeper(ModularLoadManagerWrapper.java:115) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:33 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.LoadReportUpdaterTask.run(LoadReportUpdaterTask.java:41) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:33 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_112] Aug 22 16:00:33 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_112] Aug 22 16:00:33 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_112] Aug 22 16:00:33 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_112] Aug 22 16:00:33 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_112] Aug 22 16:00:33 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_112] Aug 22 16:00:33 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final] Aug 22 16:00:33 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112] Aug 22 16:00:36 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:36.721 [pulsar-ordered-OrderedExecutor-0-0-EventThread] WARN org.apache.pulsar.broker.loadbalance.LeaderElectionService - Type of the event is [NodeDeleted] and path is [/loadbalance/leader] Aug 22 16:00:36 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:36.722 [pulsar-ordered-OrderedExecutor-0-0-EventThread] WARN org.apache.pulsar.broker.loadbalance.LeaderElectionService - Election node /loadbalance/leader is deleted, attempting re-election... Aug 22 16:00:36 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:36.722 [pulsar-ordered-OrderedExecutor-0-0-EventThread] INFO org.apache.pulsar.broker.loadbalance.LeaderElectionService - This should call elect again... Aug 22 16:00:36 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:36.723 [pulsar-1-4] INFO org.apache.pulsar.broker.loadbalance.LeaderElectionService - Broker [http://10.30.12.11:8080] is calling re-election from the thread Aug 22 16:00:36 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:36.726 [pulsar-1-4] INFO org.apache.pulsar.broker.loadbalance.LeaderElectionService - Broker [http://10.30.12.11:8080] is the leader now, notifying the listener... Aug 22 16:00:38 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:38.268 [pulsar-load-manager-3-1] WARN org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - Error when attempting to update local broker data Aug 22 16:00:38 dc3-pulsar-ec-beta-pro02 pulsar[13707]: java.lang.NullPointerException: null Aug 22 16:00:38 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.getBundleStats(ModularLoadManagerImpl.java:416) ~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:38 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.updateLocalBrokerData(ModularLoadManagerImpl.java:939) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:38 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.writeBrokerDataOnZooKeeper(ModularLoadManagerImpl.java:975) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:38 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerWrapper.writeLoadReportOnZookeeper(ModularLoadManagerWrapper.java:115) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:38 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.LoadReportUpdaterTask.run(LoadReportUpdaterTask.java:41) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:38 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_112] Aug 22 16:00:38 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_112] Aug 22 16:00:38 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_112] Aug 22 16:00:38 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_112] Aug 22 16:00:38 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_112] Aug 22 16:00:38 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_112] Aug 22 16:00:38 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final] Aug 22 16:00:38 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112] Aug 22 16:00:43 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:43.268 [pulsar-load-manager-3-1] WARN org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - Error when attempting to update local broker data Aug 22 16:00:43 dc3-pulsar-ec-beta-pro02 pulsar[13707]: java.lang.NullPointerException: null Aug 22 16:00:43 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.getBundleStats(ModularLoadManagerImpl.java:416) ~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:43 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.updateLocalBrokerData(ModularLoadManagerImpl.java:939) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:43 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.writeBrokerDataOnZooKeeper(ModularLoadManagerImpl.java:975) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:43 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerWrapper.writeLoadReportOnZookeeper(ModularLoadManagerWrapper.java:115) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:43 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.LoadReportUpdaterTask.run(LoadReportUpdaterTask.java:41) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:43 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_112] Aug 22 16:00:43 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_112] Aug 22 16:00:43 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_112] Aug 22 16:00:43 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_112] Aug 22 16:00:43 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_112] Aug 22 16:00:43 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_112] Aug 22 16:00:43 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final] Aug 22 16:00:43 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112] Aug 22 16:00:48 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:48.268 [pulsar-load-manager-3-1] WARN org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - Error when attempting to update local broker data Aug 22 16:00:48 dc3-pulsar-ec-beta-pro02 pulsar[13707]: java.lang.NullPointerException: null Aug 22 16:00:48 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.getBundleStats(ModularLoadManagerImpl.java:416) ~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:48 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.updateLocalBrokerData(ModularLoadManagerImpl.java:939) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:48 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.writeBrokerDataOnZooKeeper(ModularLoadManagerImpl.java:975) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:48 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerWrapper.writeLoadReportOnZookeeper(ModularLoadManagerWrapper.java:115) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:48 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at org.apache.pulsar.broker.loadbalance.LoadReportUpdaterTask.run(LoadReportUpdaterTask.java:41) [org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0] Aug 22 16:00:48 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_112] Aug 22 16:00:48 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_112] Aug 22 16:00:48 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_112] Aug 22 16:00:48 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_112] Aug 22 16:00:48 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_112] Aug 22 16:00:48 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_112] Aug 22 16:00:48 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final] Aug 22 16:00:48 dc3-pulsar-ec-beta-pro02 pulsar[13707]: at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112] Aug 22 16:00:51 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:51.945 [bookkeeper-io-12-7] WARN org.apache.pulsar.broker.MessagingServiceShutdownHook - Graceful shutdown timeout expired. Closing now Aug 22 16:00:51 dc3-pulsar-ec-beta-pro02 pulsar[13707]: 16:00:51.946 [bookkeeper-io-12-7] INFO org.apache.pulsar.PulsarBrokerStarter - Halting broker process with code 0 ``` I have no idea how to fix it, please help me ~ thanks ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
