GitHub user sindhushreem25 edited a discussion: Bookie went down on the power
outage on the cluster
Bookies went down when 2 out of 3 worker nodes went down due to some power
outage .Post the machines and Kubernetes cluster is up bookies are running but
one of the topics stats is showing the below exception and data flow is not
happening to that topic. Any ways to recover this topic ?
./pulsar-admin topics stats persistent://public/default/profile1_9-partition-1
--- An unexpected error occurred in the server ---
Message: No such entry
Stacktrace:
org.apache.bookkeeper.mledger.ManagedLedgerException$NonRecoverableLedgerException:
No such entry
Reason:
--- An unexpected error occurred in the server ---
Message: No such entry
Stacktrace:
org.apache.bookkeeper.mledger.ManagedLedgerException$NonRecoverableLedgerException:
No such entry
On further debugging in the broker logs we could see that read entry was
failing .
Then we ran bookie ledger metadata and readledger below are the ouput .
metadata :
2024-05-16T04:48:22,297 [main] INFO
org.apache.bookkeeper.tools.cli.commands.client.LedgerMetaDataCommand -
LedgerMetadata{formatVersion=3, ensembleSize=2, writeQuorumSize=2,
ackQuorumSize=2, state=CLOSED, length=8336855, lastEntryId=474,
digestType=CRC32C, password=base64:,
ensembles={0=[itomdipulsar-bookkeeper-2.itomdipulsar-bookkeeper.chotest:3181,
itomdipulsar-bookkeeper-1.itomdipulsar-bookkeeper.chotest:3181],
475=[itomdipulsar-bookkeeper-2.itomdipulsar-bookkeeper.chotest:3181,
itomdipulsar-bookkeeper-0.itomdipulsar-bookkeeper.chotest:3181]},
customMetadata={component=base64:bWFuYWdlZC1sZWRnZXI=,
pulsar/managed-ledger=base64:cHVibGljL2RlZmF1bHQvcGVyc2lzdGVudC9wcm9maWxlMV85LXBhcnRpdGlvbi0x,
application=base64:cHVsc2Fy}}
read ledger :
ERROR org.apache.bookkeeper.client.PendingReadOp - Read of ledger entry failed:
L315020253 E474-E474, Sent to
[itomdipulsar-bookkeeper-1.itomdipulsar-bookkeeper.chotest:3181,
itomdipulsar-bookkeeper-2.itomdipulsar-bookkeeper.chotest:3181], Heard from []
: bitset = {}, Error = 'No such entry'. First unread entry is (-1, rc = null)
GitHub link: https://github.com/apache/pulsar/discussions/22706
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
