mrtworo opened a new issue, #23807: URL: https://github.com/apache/pulsar/issues/23807
### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Read release policy - [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker. ### Version Apache pulsar-3.8.0 chart with pulsar 4.0.1 deployed in EKS. ### Minimal reproduce step Pulsar consisting of: 4 bookies 3 brokers 3 zookeepers two persistent topics configured with E=3, Qw=3, Qa=2 Two bookies and zookeeper restart and are unavailable for minutes. Both topics owned by the same brooker. ### What did you expect to see? 1. Both topics affected by 2/4 bookies unavailable. 2. Broker not being able to form new ensembles to handle writes. 3. Both recovered after bookies are up. ### What did you see instead? Both were rightly complaining on "org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available", however when bookies were back, only first topic a7u_transactions was able to recover, second topic a7u_transactions_sanitized was stuck in endless loop of un-fencing and failed writes until broker was restarted an it opened normally on the new owner. Before the restart in following morning, we've double checked the bookies status: ``` pulsar-toolset-0:/pulsar$ bin/bookkeeper shell listbookies -rw -h ... 2025-01-02T09:28:46,406+0000 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - ReadWrite Bookies : 2025-01-02T09:28:46,422+0000 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - BookieID:pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local:3181, IP:10.11.219.28, Port:3181, Hostname:pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local 2025-01-02T09:28:46,423+0000 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - BookieID:pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local:3181, IP:10.11.219.36, Port:3181, Hostname:pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local 2025-01-02T09:28:46,424+0000 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - BookieID:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181, IP:10.11.225.176, Port:3181, Hostname:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local 2025-01-02T09:28:46,426+0000 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - BookieID:pulsar-bookie-1.pulsar-bookie.pulsar.svc.cluster.local:3181, IP:10.11.232.43, Port:3181, Hostname:pulsar-bookie-1.pulsar-bookie.pulsar.svc.cluster.local ``` all were up, there were no other topics were reporting any issues with bookies. We don't understand why a7u_transactions recovered while a7u_transactions_sanitized remained impacted for hours while all bookies were up. It seemed like it's an edge case where there was no attempt to form new ensemble for the affected topic. Detailed logs attached. ### Anything else? bookie-0 returns at: ``` 2025-01-01T20:53:09,319+0000 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 ... 2025-01-01T20:53:09,624+0000 [main] INFO org.apache.bookkeeper.common.component.ComponentStarter - Started component bookie-server. ``` bookie-1 returns at: ``` 2025-01-01T20:53:34,979+0000 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/pulsar-bookie-1.pulsar-bookie.pulsar.svc.cluster.local:3181 ... 2025-01-01T20:53:35,305+0000 [main] INFO org.apache.bookkeeper.common.component.ComponentStarter - Started component bookie-server. ``` broker-1 (owner) logs when topics seemed to start behaving differently: ``` 2025-01-01T20:53:09,314+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Cannot connect to pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 as endpoint resolution failed (probably bookie is down) err org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181, bookie does not exist or it is not running 2025-01-01T20:53:09,314+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] WARN org.apache.bookkeeper.client.PendingAddOp - Failed to write entry (535058, 1) to bookie (0, pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181): Bookie handle is not available 2025-01-01T20:53:09,315+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.client.DefaultBookieAddressResolver - Cannot resolve pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181, bookie is unknown org.apache.bookkeeper.client.BKException$BKBookieHandleNotAvailableException: Bookie handle is not available 2025-01-01T20:53:09,315+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Cannot connect to pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 as endpoint resolution failed (probably bookie is down) err org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181, bookie does not exist or it is not running 2025-01-01T20:53:09,315+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] WARN org.apache.bookkeeper.client.PendingAddOp - Failed to write entry (535058, 2) to bookie (0, pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181): Bookie handle is not available 2025-01-01T20:53:09,315+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.client.DefaultBookieAddressResolver - Cannot resolve pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181, bookie is unknown org.apache.bookkeeper.client.BKException$BKBookieHandleNotAvailableException: Bookie handle is not available 2025-01-01T20:53:09,315+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Cannot connect to pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 as endpoint resolution failed (probably bookie is down) err org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181, bookie does not exist or it is not running 2025-01-01T20:53:09,315+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] WARN org.apache.bookkeeper.client.PendingAddOp - Failed to write entry (535058, 3) to bookie (0, pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181): Bookie handle is not available 2025-01-01T20:53:09,314+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] WARN org.apache.bookkeeper.client.LedgerHandle - [EnsembleChange(ledger:535058, change-id:0000000001)][attempt:1] Exception changing ensemble org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandomInternal(RackawareEnsemblePlacementPolicyImpl.java:850) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1] at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandom(RackawareEnsemblePlacementPolicyImpl.java:770) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1] at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:659) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1] at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:225) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1] at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:619) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1] at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:246) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1] at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.replaceBookie(RackawareEnsemblePlacementPolicyImpl.java:542) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1] at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.replaceBookie(RackawareEnsemblePlacementPolicy.java:139) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1] at org.apache.bookkeeper.client.BookieWatcherImpl.replaceBookie(BookieWatcherImpl.java:333) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1] at org.apache.bookkeeper.client.EnsembleUtils.replaceBookiesInEnsemble(EnsembleUtils.java:69) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1] at org.apache.bookkeeper.client.LedgerHandle.lambda$ensembleChangeLoop$11(LedgerHandle.java:2222) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1] at org.apache.bookkeeper.client.MetadataUpdateLoop.writeLoop(MetadataUpdateLoop.java:133) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1] at org.apache.bookkeeper.client.MetadataUpdateLoop.run(MetadataUpdateLoop.java:120) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1] at org.apache.bookkeeper.client.LedgerHandle.ensembleChangeLoop(LedgerHandle.java:2241) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1] at org.apache.bookkeeper.client.LedgerHandle.handleBookieFailure(LedgerHandle.java:2189) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1] at org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1] at org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:287) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1] at org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:79) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1] at org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:414) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1] at org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:373) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1] at org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2687) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1] at org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2592) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1] at org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:336) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1] at org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:545) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1] at org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:679) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1] at org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1] at org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1] at org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:312) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1] at org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1] at org.apache.bookkeeper.client.PendingAddOp.initiate(PendingAddOp.java:258) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1] at org.apache.bookkeeper.client.LedgerHandle.doAsyncAddEntry(LedgerHandle.java:1640) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1] at org.apache.bookkeeper.client.LedgerHandle.asyncAddEntry(LedgerHandle.java:1338) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1] at org.apache.bookkeeper.mledger.impl.OpAddEntry.initiate(OpAddEntry.java:153) ~[org.apache.pulsar-managed-ledger-4.0.1.jar:4.0.1] at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.updateLedgersIdsComplete(ManagedLedgerImpl.java:1736) ~[org.apache.pulsar-managed-ledger-4.0.1.jar:4.0.1] at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl$14.operationComplete(ManagedLedgerImpl.java:1606) ~[org.apache.pulsar-managed-ledger-4.0.1.jar:4.0.1] at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl$14.operationComplete(ManagedLedgerImpl.java:1592) ~[org.apache.pulsar-managed-ledger-4.0.1.jar:4.0.1] at org.apache.bookkeeper.mledger.impl.MetaStoreImpl.lambda$asyncUpdateLedgerIds$7(MetaStoreImpl.java:187) ~[org.apache.pulsar-managed-ledger-4.0.1.jar:4.0.1] at java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(Unknown Source) ~[?:?] at java.base/java.util.concurrent.CompletableFuture$Completion.run(Unknown Source) ~[?:?] at org.apache.bookkeeper.common.util.OrderedExecutor$TimedRunnable.run(OrderedExecutor.java:203) ~[org.apache.bookkeeper-bookkeeper-common-4.17.1.jar:4.17.1] at org.apache.bookkeeper.common.util.SingleThreadSafeScheduledExecutorService$SafeRunnable.run(SingleThreadSafeScheduledExecutorService.java:46) ~[org.apache.bookkeeper-bookkeeper-common-4.17.1.jar:4.17.1] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?] at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?] at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final] at java.base/java.lang.Thread.run(Unknown Source) [?:?] 2025-01-01T20:53:09,315+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] ERROR org.apache.bookkeeper.client.LedgerHandle - Closing ledger 535058 due to NotEnoughBookiesException: Not enough non-faulty bookies available 2025-01-01T20:53:09,315+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] ERROR org.apache.bookkeeper.client.PendingAddOp - Write of ledger entry to quorum failed: L535058 E0 2025-01-01T20:53:09,315+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] ERROR org.apache.bookkeeper.client.PendingAddOp - Write of ledger entry to quorum failed: L535058 E1 2025-01-01T20:53:09,315+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] ERROR org.apache.bookkeeper.client.PendingAddOp - Write of ledger entry to quorum failed: L535058 E2 2025-01-01T20:53:09,315+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] ERROR org.apache.bookkeeper.client.PendingAddOp - Write of ledger entry to quorum failed: L535058 E3 2025-01-01T20:53:09,315+0000 [metadata-store-12-1] INFO org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Bookie pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 created. path: /ledgers/available/pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 2025-01-01T20:53:09,315+0000 [metadata-store-12-1] INFO org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Bookie pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 created. path: /ledgers/available/pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 2025-01-01T20:53:09,315+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] ERROR org.apache.pulsar.broker.transaction.buffer.impl.TopicTransactionBuffer - Failed to append buffer to txn (11,2578) org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,317+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.pulsar.broker.service.Producer - Disconnecting producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions_sanitized}, client=[id: 0xd93fe9b7, L:/10.11.217.164:6650 - R:/10.11.219.27:58438] [SR:10.11.219.27, state:Connected], producerName=DynamoDBStreamProcessor-a7u_transactions-a7u_transactions_sanitized - 0d7bfcbc-724e-4674-aa90-8a48ec8d861a, producerId=0}, assignedBrokerLookupData: Optional.empty 2025-01-01T20:53:09,318+0000 [metadata-store-12-1] INFO org.apache.pulsar.metadata.impl.AbstractMetadataStore - Deleted path: /ledgers/00/0053/L5053 (v. Optional.empty) 2025-01-01T20:53:09,318+0000 [metadata-store-12-1] INFO org.apache.pulsar.metadata.impl.AbstractMetadataStore - Deleted path: /ledgers/00/0053/L5052 (v. Optional.empty) 2025-01-01T20:53:09,319+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,319+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,319+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,319+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,319+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,321+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Un-fencing topic... 2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [a7u/ddbs/persistent/a7u_transactions_sanitized] Managed ledger is now ready to accept writes again 2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [a7u/ddbs/persistent/a7u_transactions] Creating a new ledger after closed 535058 2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [a7u/ddbs/persistent/a7u_transactions_sanitized] Delete complete for empty ledger 535053. rc=0 2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [a7u/ddbs/persistent/a7u_transactions] Delete complete for empty ledger 535052. rc=0 2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local:3181>, <Bookie:pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local:3181>], allBookies [<Bookie:pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local:3181>, <Bookie:pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local:3181>]. 2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local:3181>, <Bookie:pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local:3181>], allBookies [<Bookie:pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local:3181>, <Bookie:pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local:3181>]. 2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.client.LedgerCreateOp - Not enough bookies to create ledger with ensembleSize=3, writeQuorumSize=3 and ackQuorumSize=2 2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [a7u/ddbs/persistent/a7u_transactions] Error creating ledger rc=-6 Not enough non-faulty bookies available 2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.pulsar.broker.service.Producer - Disconnecting producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions}, client=[id: 0xe3c4b8be, L:/10.11.217.164:6650 - R:/10.11.233.64:37306] [SR:10.11.233.64, state:Connected], producerName=pulsar-313-191, producerId=0}, assignedBrokerLookupData: Optional.empty 2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.pulsar.broker.service.Producer - Disconnecting producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions}, client=[id: 0x320c7da6, L:/10.11.217.164:6650 - R:/10.11.219.27:57008] [SR:10.11.219.27, state:Connected], producerName=pulsar-313-193, producerId=0}, assignedBrokerLookupData: Optional.empty 2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.pulsar.broker.service.Producer - Disconnecting producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions}, client=[id: 0x96ac0dc3, L:/10.11.217.164:6650 - R:/10.11.233.64:37318] [SR:10.11.233.64, state:Connected], producerName=pulsar-313-192, producerId=0}, assignedBrokerLookupData: Optional.empty 2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.pulsar.broker.service.Producer - Disconnecting producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions}, client=[id: 0x89f930ab, L:/10.11.217.164:6650 - R:/10.11.219.27:57010] [SR:10.11.219.27, state:Connected], producerName=pulsar-313-194, producerId=0}, assignedBrokerLookupData: Optional.empty 2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,323+0000 [pulsar-io-5-2] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions] Un-fencing topic... 2025-01-01T20:53:09,323+0000 [pulsar-io-5-2] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [a7u/ddbs/persistent/a7u_transactions] Managed ledger is now ready to accept writes again 2025-01-01T20:53:09,324+0000 [pulsar-io-5-1] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.11.233.64:37318 2025-01-01T20:53:09,325+0000 [pulsar-io-5-2] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.11.219.27:57008 2025-01-01T20:53:09,325+0000 [pulsar-io-5-3] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.11.219.27:57010 2025-01-01T20:53:09,325+0000 [metadata-store-12-1] INFO org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=httpserver, port=8000, host=0.0.0.0, protocol=http, auth=[], extensions=[]}, EndpointInfo{id=bookie, port=3181, host=pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local, protocol=bookie-rpc, auth=[], extensions=[]}]} 2025-01-01T20:53:09,326+0000 [metadata-store-12-1] INFO org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=httpserver, port=8000, host=0.0.0.0, protocol=http, auth=[], extensions=[]}, EndpointInfo{id=bookie, port=3181, host=pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local, protocol=bookie-rpc, auth=[], extensions=[]}]} 2025-01-01T20:53:09,326+0000 [pulsar-registration-client-38-1] WARN org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy - Failed to resolve network location for pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local, using default rack for it : /default-rack. 2025-01-01T20:53:09,326+0000 [pulsar-registration-client-19-1] WARN org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy - Failed to resolve network location for pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local, using default rack for it : /default-rack. 2025-01-01T20:53:09,326+0000 [pulsar-registration-client-38-1] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 2025-01-01T20:53:09,326+0000 [pulsar-registration-client-19-1] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 2025-01-01T20:53:09,328+0000 [pulsar-io-5-2] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.11.219.27:58438 2025-01-01T20:53:09,463+0000 [pulsar-io-5-3] INFO org.apache.pulsar.broker.service.ServerCnx - [/10.11.219.27:57018] connected role=system:serviceaccount:pulsar:pulsar-proxy and originalAuthRole=system:serviceaccount:a7u:ddbss-records-processor using authMethod=token, clientVersion=Pulsar-Java-v3.0.0, clientProtocolVersion=21, proxyVersion=4.0.1 2025-01-01T20:53:09,466+0000 [broker-topic-workers-OrderedExecutor-0-0] INFO org.apache.pulsar.broker.service.ServerCnx - [/10.11.219.27:57018] Created new producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions_sanitized}, client=[id: 0x3133e9f2, L:/10.11.217.164:6650 - R:/10.11.219.27:57018] [SR:10.11.219.27, state:Connected], producerName=DynamoDBStreamProcessor-a7u_transactions-a7u_transactions_sanitized - 0d7bfcbc-724e-4674-aa90-8a48ec8d861a, producerId=0}, role: system:serviceaccount:a7u:ddbss-records-processor 2025-01-01T20:53:09,469+0000 [pulsar-io-5-3] INFO org.apache.pulsar.broker.service.Producer - Disconnecting producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions_sanitized}, client=[id: 0x3133e9f2, L:/10.11.217.164:6650 - R:/10.11.219.27:57018] [SR:10.11.219.27, state:Connected], producerName=DynamoDBStreamProcessor-a7u_transactions-a7u_transactions_sanitized - 0d7bfcbc-724e-4674-aa90-8a48ec8d861a, producerId=0}, assignedBrokerLookupData: Optional.empty 2025-01-01T20:53:09,469+0000 [pulsar-io-5-3] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,469+0000 [pulsar-io-5-3] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Un-fencing topic... 2025-01-01T20:53:09,474+0000 [pulsar-io-5-3] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.11.219.27:57018 2025-01-01T20:53:09,482+0000 [pulsar-io-5-4] INFO org.apache.pulsar.broker.service.ServerCnx - [/10.11.230.148:60912] connected role=system:serviceaccount:pulsar:pulsar-proxy and originalAuthRole=system:serviceaccount:a7u:ddbss-records-processor using authMethod=token, clientVersion=Pulsar-Java-v3.0.0, clientProtocolVersion=21, proxyVersion=4.0.1 2025-01-01T20:53:09,550+0000 [pulsar-io-5-3] INFO org.apache.pulsar.broker.service.ServerCnx - [/10.11.219.27:57036] connected role=system:serviceaccount:pulsar:pulsar-proxy and originalAuthRole=a7u-ddbss using authMethod=token, clientVersion=Pulsar-CPP-v3.5.1, clientProtocolVersion=20, proxyVersion=4.0.1 2025-01-01T20:53:09,553+0000 [broker-topic-workers-OrderedExecutor-0-0] INFO org.apache.pulsar.broker.service.ServerCnx - [/10.11.219.27:57036] Created new producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions}, client=[id: 0x90f74dd6, L:/10.11.217.164:6650 - R:/10.11.219.27:57036] [SR:10.11.219.27, state:Connected], producerName=pulsar-313-193, producerId=0}, role: a7u-ddbss 2025-01-01T20:53:09,554+0000 [pulsar-io-5-1] INFO org.apache.pulsar.broker.service.ServerCnx - [/10.11.219.27:57030] connected role=system:serviceaccount:pulsar:pulsar-proxy and originalAuthRole=a7u-ddbss using authMethod=token, clientVersion=Pulsar-CPP-v3.5.1, clientProtocolVersion=20, proxyVersion=4.0.1 2025-01-01T20:53:09,555+0000 [pulsar-io-5-2] INFO org.apache.pulsar.broker.service.ServerCnx - [/10.11.233.64:37332] connected role=system:serviceaccount:pulsar:pulsar-proxy and originalAuthRole=a7u-ddbss using authMethod=token, clientVersion=Pulsar-CPP-v3.5.1, clientProtocolVersion=20, proxyVersion=4.0.1 2025-01-01T20:53:09,555+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [a7u/ddbs/persistent/a7u_transactions] Creating a new ledger 2025-01-01T20:53:09,555+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.bookkeeper.client.BookieWatcherImpl - New ensemble: [pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local:3181, pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local:3181, pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181] is not adhering to Placement Policy. quarantinedBookies: [] 2025-01-01T20:53:09,557+0000 [broker-topic-workers-OrderedExecutor-0-0] INFO org.apache.pulsar.broker.service.ServerCnx - [/10.11.233.64:37332] Created new producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions}, client=[id: 0x563b128e, L:/10.11.217.164:6650 - R:/10.11.233.64:37332] [SR:10.11.233.64, state:Connected], producerName=pulsar-313-192, producerId=0}, role: a7u-ddbss 2025-01-01T20:53:09,566+0000 [broker-topic-workers-OrderedExecutor-0-0] INFO org.apache.pulsar.broker.service.ServerCnx - [/10.11.219.27:57030] Created new producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions}, client=[id: 0x7c18fd74, L:/10.11.217.164:6650 - R:/10.11.219.27:57030] [SR:10.11.219.27, state:Connected], producerName=pulsar-313-194, producerId=0}, role: a7u-ddbss 2025-01-01T20:53:09,571+0000 [metadata-store-12-1] INFO org.apache.pulsar.metadata.impl.AbstractMetadataStore - Deleting path: /ledgers/idgen/ID-0000535064 (v. Optional.empty) 2025-01-01T20:53:09,575+0000 [broker-topic-workers-OrderedExecutor-0-0] INFO org.apache.pulsar.broker.service.ServerCnx - [/10.11.230.148:60912] Created new producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions_sanitized}, client=[id: 0x6e4d4bd8, L:/10.11.217.164:6650 - R:/10.11.230.148:60912] [SR:10.11.230.148, state:Connected], producerName=DynamoDBStreamProcessor-a7u_transactions-a7u_transactions_sanitized - 0d7bfcbc-724e-4674-aa90-8a48ec8d861a, producerId=0}, role: system:serviceaccount:a7u:ddbss-records-processor 2025-01-01T20:53:09,578+0000 [metadata-store-12-1] INFO org.apache.pulsar.metadata.impl.AbstractMetadataStore - Deleted path: /ledgers/idgen/ID-0000535064 (v. Optional.empty) 2025-01-01T20:53:09,578+0000 [metadata-store-12-1] INFO org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local:3181, pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local:3181, pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181] for ledger: 535064 2025-01-01T20:53:09,578+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [a7u/ddbs/persistent/a7u_transactions] Created new ledger 535064 2025-01-01T20:53:09,580+0000 [pulsar-io-5-4] INFO org.apache.pulsar.broker.service.Producer - Disconnecting producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions_sanitized}, client=[id: 0x6e4d4bd8, L:/10.11.217.164:6650 - R:/10.11.230.148:60912] [SR:10.11.230.148, state:Connected], producerName=DynamoDBStreamProcessor-a7u_transactions-a7u_transactions_sanitized - 0d7bfcbc-724e-4674-aa90-8a48ec8d861a, producerId=0}, assignedBrokerLookupData: Optional.empty 2025-01-01T20:53:09,580+0000 [pulsar-io-5-4] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6 2025-01-01T20:53:09,580+0000 [pulsar-io-5-4] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Un-fencing topic... 2025-01-01T20:53:09,585+0000 [pulsar-io-5-4] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.11.230.148:60912 2025-01-01T20:53:09,589+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO org.apache.pulsar.metadata.impl.AbstractMetadataStore - Deleting path: /ledgers/00/0053/L5058 (v. Optional.empty) 2025-01-01T20:53:09,602+0000 [pulsar-io-5-4] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0x3c1af6f7, L:/10.11.217.164:49698 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.11.225.176:3181] 2025-01-01T20:53:09,603+0000 [pulsar-io-5-4] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0x3c1af6f7, L:/10.11.217.164:49698 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.11.225.176:3181] authenticated as BookKeeperPrincipal{ANONYMOUS} 2025-01-01T20:53:09,609+0000 [metadata-store-12-1] INFO org.apache.pulsar.metadata.impl.AbstractMetadataStore - Deleted path: /ledgers/00/0053/L5058 (v. Optional.empty) 2025-01-01T20:53:09,609+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [a7u/ddbs/persistent/a7u_transactions] Delete complete for empty ledger 535058. rc=0 ``` ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
