wangrenyi commented on issue #23599:
URL: https://github.com/apache/pulsar/issues/23599#issuecomment-2478276294
bin/bookkeeper shell bookiesanity ,the results are as follows:
2024-11-15T16:55:44,433+0800 [main] INFO
org.apache.bookkeeper.client.BookKeeper - Weighted ledger placement is not
enabled
2024-11-15T16:55:44,620+0800 [main-EventThread] INFO
org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache
(writable bookie) 172.2.10.118:3181 -> BookieServiceInfo{properties={},
endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.118,
protocol=bookie-rpc, auth=[], extensions=[]}]}
2024-11-15T16:55:44,620+0800 [main-EventThread] INFO
org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache
(writable bookie) 10.2.20.72:3181 -> BookieServiceInfo{properties={},
endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.121,
protocol=bookie-rpc, auth=[], extensions=[]}]}
2024-11-15T16:55:44,623+0800 [main-EventThread] INFO
org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache
(readonly bookie) 172.2.10.119:3181 -> BookieServiceInfo{properties={},
endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.119,
protocol=bookie-rpc, auth=[], extensions=[]}]}
2024-11-15T16:55:44,623+0800 [main-EventThread] INFO
org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache
(readonly bookie) 172.2.10.120:3181 -> BookieServiceInfo{properties={},
endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.120,
protocol=bookie-rpc, auth=[], extensions=[]}]}
2024-11-15T16:55:44,718+0800 [main-EventThread] INFO
org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [172.2.10.119:3181] for
ledger: 9117303
2024-11-15T16:55:44,719+0800 [main] INFO
org.apache.bookkeeper.tools.cli.commands.bookie.SanityTestCommand - Create
ledger 9117303
2024-11-15T16:55:44,857+0800 [bookkeeper-io-3-1] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to
bookie: 172.2.10.119:3181 [id: 0xd15f532d, L:/172.2.10.119:38528 -
R:172.2.10.119/172.2.10.119:3181]
2024-11-15T16:55:44,868+0800 [bookkeeper-io-3-1] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id:
0xd15f532d, L:/172.2.10.119:38528 - R:172.2.10.119/172.2.10.119:3181]
authenticated as BookKeeperPrincipal{ANONYMOUS}
2024-11-15T16:55:44,889+0800 [BookKeeperClientWorker-OrderedExecutor-7-0]
WARN org.apache.bookkeeper.client.PendingAddOp - Failed to write entry
(9117303, 0) to bookie (0, 172.2.10.119:3181): Attempting to write on ReadOnly
bookie
2024-11-15T16:55:44,902+0800 [BookKeeperClientWorker-OrderedExecutor-7-0]
ERROR org.apache.bookkeeper.client.MetadataUpdateLoop -
UpdateLoop(ledgerId=9117303,loopId=7a0bfa43) Exception updating
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not
enough non-faulty bookies available
at
org.apache.bookkeeper.bookie.LocalBookieEnsemblePlacementPolicy.replaceBookie(LocalBookieEnsemblePlacementPolicy.java:89)
~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at
org.apache.bookkeeper.client.BookieWatcherImpl.replaceBookie(BookieWatcherImpl.java:334)
~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at
org.apache.bookkeeper.client.EnsembleUtils.replaceBookiesInEnsemble(EnsembleUtils.java:71)
~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at
org.apache.bookkeeper.client.LedgerHandle.lambda$ensembleChangeLoop$2(LedgerHandle.java:1947)
~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at
org.apache.bookkeeper.client.MetadataUpdateLoop.writeLoop(MetadataUpdateLoop.java:134)
~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at
org.apache.bookkeeper.client.MetadataUpdateLoop.run(MetadataUpdateLoop.java:123)
~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at
org.apache.bookkeeper.client.LedgerHandle.ensembleChangeLoop(LedgerHandle.java:1966)
~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at
org.apache.bookkeeper.client.LedgerHandle.handleBookieFailure(LedgerHandle.java:1915)
~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:378)
~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at
org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.writeComplete(PerChannelBookieClient.java:2168)
~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at
org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleResponse(PerChannelBookieClient.java:2225)
~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at
org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleV2Response(PerChannelBookieClient.java:2204)
~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at
org.apache.bookkeeper.proto.PerChannelBookieClient$ReadV2ResponseCallback.safeRun(PerChannelBookieClient.java:1382)
~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at
org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36)
~[org.apache.bookkeeper-bookkeeper-common-4.14.8.jar:4.14.8]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
2024-11-15T16:55:44,918+0800 [BookKeeperClientWorker-OrderedExecutor-7-0]
WARN org.apache.bookkeeper.client.LedgerHandle -
[EnsembleChange(ledger:9117303, change-id:0000000001)][attempt:1] Exception
changing ensemble
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not
enough non-faulty bookies available
at
org.apache.bookkeeper.bookie.LocalBookieEnsemblePlacementPolicy.replaceBookie(LocalBookieEnsemblePlacementPolicy.java:89)
~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at
org.apache.bookkeeper.client.BookieWatcherImpl.replaceBookie(BookieWatcherImpl.java:334)
~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at
org.apache.bookkeeper.client.EnsembleUtils.replaceBookiesInEnsemble(EnsembleUtils.java:71)
~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at
org.apache.bookkeeper.client.LedgerHandle.lambda$ensembleChangeLoop$2(LedgerHandle.java:1947)
~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at
org.apache.bookkeeper.client.MetadataUpdateLoop.writeLoop(MetadataUpdateLoop.java:134)
~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at
org.apache.bookkeeper.client.MetadataUpdateLoop.run(MetadataUpdateLoop.java:123)
~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at
org.apache.bookkeeper.client.LedgerHandle.ensembleChangeLoop(LedgerHandle.java:1966)
~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at
org.apache.bookkeeper.client.LedgerHandle.handleBookieFailure(LedgerHandle.java:1915)
~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:378)
~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at
org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.writeComplete(PerChannelBookieClient.java:2168)
~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at
org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleResponse(PerChannelBookieClient.java:2225)
~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at
org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleV2Response(PerChannelBookieClient.java:2204)
~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at
org.apache.bookkeeper.proto.PerChannelBookieClient$ReadV2ResponseCallback.safeRun(PerChannelBookieClient.java:1382)
~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at
org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36)
~[org.apache.bookkeeper-bookkeeper-common-4.14.8.jar:4.14.8]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
2024-11-15T16:55:44,918+0800 [BookKeeperClientWorker-OrderedExecutor-7-0]
ERROR org.apache.bookkeeper.client.LedgerHandle - Closing ledger 9117303 due to
NotEnoughBookiesException: Not enough non-faulty bookies available
2024-11-15T16:55:44,921+0800 [BookKeeperClientWorker-OrderedExecutor-7-0]
ERROR org.apache.bookkeeper.client.PendingAddOp - Write of ledger entry to
quorum failed: L9117303 E0
2024-11-15T16:55:44,921+0800 [main] WARN
org.apache.bookkeeper.tools.cli.commands.bookie.SanityTestCommand - Error in
bookie sanity test
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not
enough non-faulty bookies available
at
org.apache.bookkeeper.client.SyncCallbackUtils.finish(SyncCallbackUtils.java:83)
~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at
org.apache.bookkeeper.client.SyncCallbackUtils$SyncAddCallback.addComplete(SyncCallbackUtils.java:251)
~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at
org.apache.bookkeeper.client.AsyncCallback$AddCallback.addCompleteWithLatency(AsyncCallback.java:92)
~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at
org.apache.bookkeeper.client.PendingAddOp.submitCallback(PendingAddOp.java:431)
~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at
org.apache.bookkeeper.client.LedgerHandle.errorOutPendingAdds(LedgerHandle.java:1811)
~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at
org.apache.bookkeeper.client.LedgerHandle$5.safeRun(LedgerHandle.java:575)
~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at
org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36)
~[org.apache.bookkeeper-bookkeeper-common-4.14.8.jar:4.14.8]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
2024-11-15T16:55:44,930+0800 [main-EventThread] WARN
org.apache.bookkeeper.meta.AbstractZkLedgerManager - Ledger node does not exist
in ZooKeeper: ledgerId=9117303
2024-11-15T16:55:44,930+0800 [main-EventThread] ERROR
org.apache.bookkeeper.client.MetadataUpdateLoop -
UpdateLoop(ledgerId=9117303,loopId=6a2ca3d5) Error writing metadata to store
org.apache.bookkeeper.client.BKException$BKNoSuchLedgerExistsOnMetadataServerException:
No such ledger exists on Metadata Server
at
org.apache.bookkeeper.meta.AbstractZkLedgerManager$5.processResult(AbstractZkLedgerManager.java:547)
~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at
org.apache.bookkeeper.zookeeper.ZooKeeperClient$22$1.processResult(ZooKeeperClient.java:1094)
~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:613)
~[org.apache.zookeeper-zookeeper-3.9.1.jar:3.9.1]
at
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:538)
~[org.apache.zookeeper-zookeeper-3.9.1.jar:3.9.1]
2024-11-15T16:55:44,932+0800 [main-EventThread] WARN
org.apache.bookkeeper.client.LedgerHandle - Close failed:
NoSuchLedgerExistsOnMetadataServerException: No such ledger exists on Metadata
Server
2024-11-15T16:55:44,932+0800 [main-EventThread] ERROR
org.apache.bookkeeper.meta.CleanupLedgerManager - Failed on operating ledger
metadata: -25
2024-11-15T16:55:44,932+0800 [main] INFO
org.apache.bookkeeper.tools.cli.commands.bookie.SanityTestCommand - Deleted
ledger 9117303
2024-11-15T16:55:44,932+0800 [main] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel
bookie client for 172.2.10.119:3181
2024-11-15T16:55:44,936+0800 [main] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel
bookie client for 172.2.10.119:3181
2024-11-15T16:55:44,936+0800 [bookkeeper-io-3-1] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie
channel [id: 0xd15f532d, L:/172.2.10.119:38528 !
R:172.2.10.119/172.2.10.119:3181]
2024-11-15T16:55:45,060+0800 [main] INFO org.apache.zookeeper.ZooKeeper -
Session: 0x1004929423a01c5 closed
2024-11-15T16:55:45,060+0800 [main-EventThread] INFO
org.apache.zookeeper.ClientCnxn - EventThread shut down for session:
0x1004929423a01c5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
