wangrenyi commented on issue #23599: URL: https://github.com/apache/pulsar/issues/23599#issuecomment-2493195637
when bookeeper is started, the log is as follows: there was a problem with this place during init  Nov 22 16:02:33 DFPulsar-172-2-10-119 pulsar[23535]: 2024-11-22T16:02:33,617+0800 [main-EventThread] INFO org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager - Latch countdown due to ZK event: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/ledgers/underreplication/locks zxid: -1 Nov 22 16:02:33 DFPulsar-172-2-10-119 pulsar[23535]: 2024-11-22T16:02:33,653+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie node from network location /default-rack, the bookies in the network location are [<Bookie:172.2.10.118:3181>, <Bookie:172.2.10.121:3181>], excluded bookies [<Bookie:172.2.10.118:3181>, <Bookie:172.2.10.121:3181>], current ensemble org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy$EnsembleForReplacementWithNoConstraints@4fd5850a, fallback to choose bookie randomly from the cluster. Nov 22 16:02:33 DFPulsar-172-2-10-119 pulsar[23535]: 2024-11-22T16:02:33,654+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:172.2.10.118:3181>, <Bookie:172.2.10.121:3181>], allBookies [<Bookie:172.2.10.121:3181>, <Bookie:172.2.10.118:3181>]. Nov 22 16:02:33 DFPulsar-172-2-10-119 pulsar[23535]: 2024-11-22T16:02:33,654+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie: excluded [<Bookie:172.2.10.118:3181>, <Bookie:172.2.10.121:3181>], fallback to choose bookie randomly from the cluster. Nov 22 16:02:33 DFPulsar-172-2-10-119 pulsar[23535]: 2024-11-22T16:02:33,654+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:172.2.10.118:3181>, <Bookie:172.2.10.121:3181>], allBookies [<Bookie:172.2.10.118:3181>, <Bookie:172.2.10.121:3181>]. Nov 22 16:02:33 DFPulsar-172-2-10-119 pulsar[23535]: 2024-11-22T16:02:33,654+0800 [ReplicationWorker] WARN org.apache.bookkeeper.replication.ReplicationWorker - BKNotEnoughBookiesException while replicating the fragment Nov 22 16:02:33 DFPulsar-172-2-10-119 pulsar[23535]: org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available Nov 22 16:02:33 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandomInternal(RackawareEnsemblePlacementPolicyImpl.java:796) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:33 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandom(RackawareEnsemblePlacementPolicyImpl.java:716) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:33 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:605) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:33 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:205) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:33 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:565) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:33 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:226) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:33 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.replaceBookie(RackawareEnsemblePlacementPolicyImpl.java:488) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:33 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.replaceBookie(RackawareEnsemblePlacementPolicy.java:119) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:33 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.client.BookKeeperAdmin.getReplacementBookiesByIndexes(BookKeeperAdmin.java:1092) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:33 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.client.BookKeeperAdmin.replicateLedgerFragment(BookKeeperAdmin.java:1138) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:33 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:473) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:33 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:301) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:33 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.replication.ReplicationWorker.run(ReplicationWorker.java:249) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:33 DFPulsar-172-2-10-119 pulsar[23535]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.113.Final.jar:4.1.113.Final] Nov 22 16:02:33 DFPulsar-172-2-10-119 pulsar[23535]: at java.lang.Thread.run(Thread.java:840) ~[?:?] Nov 22 16:02:33 DFPulsar-172-2-10-119 pulsar[23535]: 2024-11-22T16:02:33,726+0800 [ReplicationWorker] ERROR org.apache.bookkeeper.replication.ReplicationWorker - ReplicationWorker failed to replicate Ledger : 8991607 for 0 number of times, so deferring the ledger lock release by 9375 msecs Nov 22 16:02:33 DFPulsar-172-2-10-119 pulsar[23535]: 2024-11-22T16:02:33,726+0800 [ReplicationWorker] WARN org.apache.bookkeeper.replication.ReplicationWorker - failed while replicating fragments Nov 22 16:02:37 DFPulsar-172-2-10-119 pulsar[23535]: 2024-11-22T16:02:37,785+0800 [bookie-io-8-11] INFO org.apache.bookkeeper.proto.AuthHandler - Authentication success on server side Nov 22 16:02:37 DFPulsar-172-2-10-119 pulsar[23535]: 2024-11-22T16:02:37,786+0800 [bookie-io-8-11] INFO org.apache.bookkeeper.proto.BookieRequestHandler - Channel connected [id: 0xc275f19b, L:/10.2.20.113:3181 - R:/10.2.20.113:48026] Nov 22 16:02:37 DFPulsar-172-2-10-119 pulsar[23535]: 2024-11-22T16:02:37,998+0800 [LedgerDirsMonitorThread] WARN org.apache.bookkeeper.bookie.LedgerDirsMonitor - LedgerDirsMonitor check process: All ledger directories are non writable Nov 22 16:02:38 DFPulsar-172-2-10-119 pulsar[23535]: 2024-11-22T16:02:37,999+0800 [LedgerDirsMonitorThread] ERROR org.apache.bookkeeper.util.DiskChecker - Space left on device /data/bookkeeper/ledgers/current : 103930888192, Used space fraction: 0.94800913 > threshold 0.9. Nov 22 16:02:38 DFPulsar-172-2-10-119 pulsar[23535]: 2024-11-22T16:02:38,740+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie node from network location /default-rack, the bookies in the network location are [ <Bookie:172.2.10.118:3181>, <Bookie:172.2.10.121:3181>], excluded bookies [<Bookie:172.2.10.118:3181>, <Bookie:172.2.10.121:3181>], current ensemble org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy$EnsembleForReplacementWithNoConstraints@4fd5850a, fallback to choose bookie randomly from the cluster. Nov 22 16:02:38 DFPulsar-172-2-10-119 pulsar[23535]: 2024-11-22T16:02:38,740+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:172.2.10.118:3181>, <Bookie:172.2.10.121:3181>], allBookies [<Bookie:172.2.10.121:3181>, <Bookie:172.2.10.118:3181>]. Nov 22 16:02:38 DFPulsar-172-2-10-119 pulsar[23535]: 2024-11-22T16:02:38,741+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie: excluded [<Bookie:172.2.10.118:3181>, <Bookie:172.2.10.121:3181>], fallback to choose bookie randomly from the cluster. Nov 22 16:02:38 DFPulsar-172-2-10-119 pulsar[23535]: 2024-11-22T16:02:38,741+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:172.2.10.118:3181>, <Bookie:172.2.10.121:3181>], allBookies [<Bookie:172.2.10.118:3181>, <Bookie:172.2.10.121:3181>]. Nov 22 16:02:38 DFPulsar-172-2-10-119 pulsar[23535]: 2024-11-22T16:02:38,741+0800 [ReplicationWorker] WARN org.apache.bookkeeper.replication.ReplicationWorker - BKNotEnoughBookiesException while replicating the fragment Nov 22 16:02:38 DFPulsar-172-2-10-119 pulsar[23535]: org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available Nov 22 16:02:38 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandomInternal(RackawareEnsemblePlacementPolicyImpl.java:796) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:38 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandom(RackawareEnsemblePlacementPolicyImpl.java:716) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:38 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:605) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:38 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:205) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:38 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:565) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:38 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:226) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:38 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.replaceBookie(RackawareEnsemblePlacementPolicyImpl.java:488) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:38 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.replaceBookie(RackawareEnsemblePlacementPolicy.java:119) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:38 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.client.BookKeeperAdmin.getReplacementBookiesByIndexes(BookKeeperAdmin.java:1092) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:38 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.client.BookKeeperAdmin.replicateLedgerFragment(BookKeeperAdmin.java:1138) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:38 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:473) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:38 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:301) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:38 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.replication.ReplicationWorker.run(ReplicationWorker.java:249) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:38 DFPulsar-172-2-10-119 pulsar[23535]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.113.Final.jar:4.1.113.Final] Nov 22 16:02:38 DFPulsar-172-2-10-119 pulsar[23535]: at java.lang.Thread.run(Thread.java:840) ~[?:?] Nov 22 16:02:38 DFPulsar-172-2-10-119 pulsar[23535]: 2024-11-22T16:02:38,746+0800 [ReplicationWorker] ERROR org.apache.bookkeeper.replication.ReplicationWorker - ReplicationWorker failed to replicate Ledger : 8904239 for 0 number of times, so deferring the ledger lock release by 9375 msecs Nov 22 16:02:38 DFPulsar-172-2-10-119 pulsar[23535]: 2024-11-22T16:02:38,747+0800 [ReplicationWorker] WARN org.apache.bookkeeper.replication.ReplicationWorker - failed while replicating fragments Nov 22 16:02:43 DFPulsar-172-2-10-119 pulsar[23535]: 2024-11-22T16:02:43,755+0800 [main-EventThread] INFO org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager - Latch countdown due to ZK event: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/ledgers/underreplication/locks zxid: -1 Nov 22 16:02:43 DFPulsar-172-2-10-119 pulsar[23535]: 2024-11-22T16:02:43,763+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie node from network location /default-rack, the bookies in the network location are [<Bookie:172.2.10.118:3181>, <Bookie:172.2.10.121:3181>], excluded bookies [<Bookie:172.2.10.118:3181>, <Bookie:172.2.10.121:3181>], current ensemble org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy$EnsembleForReplacementWithNoConstraints@4fd5850a, fallback to choose bookie randomly from the cluster. Nov 22 16:02:43 DFPulsar-172-2-10-119 pulsar[23535]: 2024-11-22T16:02:43,763+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:172.2.10.118:3181>, <Bookie:172.2.10.121:3181>], allBookies [<Bookie:172.2.10.121:3181>, <Bookie:172.2.10.118:3181>]. Nov 22 16:02:43 DFPulsar-172-2-10-119 pulsar[23535]: 2024-11-22T16:02:43,763+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie: excluded [<Bookie:172.2.10.118:3181>, <Bookie:172.2.10.121:3181>], fallback to choose bookie randomly from the cluster. Nov 22 16:02:43 DFPulsar-172-2-10-119 pulsar[23535]: 2024-11-22T16:02:43,764+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:<Bookie:172.2.10.118:3181>, <Bookie:172.2.10.121:3181>], allBookies [<Bookie:172.2.10.118:3181>,<Bookie:172.2.10.121:3181>]. Nov 22 16:02:43 DFPulsar-172-2-10-119 pulsar[23535]: 2024-11-22T16:02:43,764+0800 [ReplicationWorker] WARN org.apache.bookkeeper.replication.ReplicationWorker - BKNotEnoughBookiesException while replicating the fragment Nov 22 16:02:43 DFPulsar-172-2-10-119 pulsar[23535]: org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available Nov 22 16:02:43 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandomInternal(RackawareEnsemblePlacementPolicyImpl.java:796) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:43 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandom(RackawareEnsemblePlacementPolicyImpl.java:716) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:43 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:605) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:43 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:205) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:43 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:565) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:43 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:226) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:43 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.replaceBookie(RackawareEnsemblePlacementPolicyImpl.java:488) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:43 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.replaceBookie(RackawareEnsemblePlacementPolicy.java:119) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:43 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.client.BookKeeperAdmin.getReplacementBookiesByIndexes(BookKeeperAdmin.java:1092) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:43 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.client.BookKeeperAdmin.replicateLedgerFragment(BookKeeperAdmin.java:1138) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:43 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:473) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 16:02:43 DFPulsar-172-2-10-119 pulsar[23535]: at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:301) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
