golden-yang opened a new issue, #3519:
URL: https://github.com/apache/bookkeeper/issues/3519
**QUESTION**
BookKeeper 4.10.0 recovery issue.
A Bookie is broken beyond repair. When I try to restore a Ledger, I have
this problem:
host3 is the broken host. When I try to recover it, it throws a
ConnectTimeoutException with host3.
Am I using the recover command incorrectly?
`bin/bookkeeper shell recover host3:3181 -l 28190099`
This is the metadata of the ledger.
**ledgerID: 28190099
LedgerMetadata{formatVersion=3, ensembleSize=2, writeQuorumSize=2,
ackQuorumSize=2, state=OPEN, digestType=CRC32C, password=base64:,
ensembles={0=[ {{ host2 }} :3181, {{ host3 }}:3181]},
customMetadata={component=XX, pulsar/managed-ledger=XX,
application=base64:XX}}**
`
11:30:14.419 [bookkeeper-io-3-1] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id:
0x4c920fe9, L:/{{ host1 }}:33605 - R:/{{ host2 }}:3181] authenticated as
BookKeeperPrincipal{ANONYMOUS}
11:30:24.396 [bookkeeper-io-3-2] ERROR
org.apache.bookkeeper.proto.PerChannelBookieClient - Could not connect to
bookie: [id: 0x224fd5e3]/{{ host3 }}:3181, current state CONNECTING :
io.netty.channel.ConnectTimeoutException: connection timed out: /{{ host3
}}:3181
at
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:261)
[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:150)
[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:510)
[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:518)
[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050)
[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_302]
11:30:24.403 [BookKeeperClientWorker-OrderedExecutor-19-0] ERROR
org.apache.bookkeeper.client.ReadLastConfirmedOp - While readLastConfirmed
ledger: 28190099 did not hear success responses from all quorums
11:30:24.411 [BookKeeperClientWorker-OrderedExecutor-19-0] ERROR
org.apache.bookkeeper.client.BookKeeperAdmin - BK error opening ledger: 28190099
org.apache.bookkeeper.client.BKException$BKReadException: Error while
reading ledger
at
org.apache.bookkeeper.client.BKException.create(BKException.java:62)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.client.BookKeeperAdmin$6.openComplete(BookKeeperAdmin.java:751)
[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.client.LedgerOpenOp.openComplete(LedgerOpenOp.java:232)
[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.client.LedgerOpenOp$2.readLastConfirmedComplete(LedgerOpenOp.java:215)
[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.client.LedgerHandle$10.readLastConfirmedDataComplete(LedgerHandle.java:1405)
[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.client.ReadLastConfirmedOp.readEntryComplete(ReadLastConfirmedOp.java:142)
[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.proto.BookieClientImpl$2.safeRun(BookieClientImpl.java:369)
[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36)
[org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_302]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_302]
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_302]
11:30:24.412 [BookKeeperClientWorker-OrderedExecutor-19-0] INFO
org.apache.bookkeeper.client.BookKeeperAdmin - Recover bookie for 28190099
completed with rc : ReadException: Error while reading ledger
11:30:24.412 [main] INFO org.apache.bookkeeper.proto.PerChannelBookieClient
- Closing the per channel bookie client for {{ host2 }}:3181
11:30:24.416 [main] INFO org.apache.bookkeeper.proto.PerChannelBookieClient
- Closing the per channel bookie client for {{ host3 }}:3181
11:30:24.417 [bookkeeper-io-3-1] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie
channel [id: 0x4c920fe9, L:/{{ host1 }}:33605 ! R:/{{ host2 }}:3181]
11:30:24.530 [main] INFO org.apache.zookeeper.ZooKeeper - Session:
0x5826bcd8e8f081c closed
11:30:24.530 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn -
EventThread shut down for session: 0x5826bcd8e8f081c
Exception in thread "main"
com.google.common.util.concurrent.UncheckedExecutionException: Error while
reading ledger
at
org.apache.bookkeeper.tools.cli.commands.bookies.RecoverCommand.apply(RecoverCommand.java:110)
at
org.apache.bookkeeper.bookie.BookieShell$RecoverCmd.runCmd(BookieShell.java:509)
at
org.apache.bookkeeper.bookie.BookieShell$MyCommand.runCmd(BookieShell.java:223)
at
org.apache.bookkeeper.bookie.BookieShell.run(BookieShell.java:1976)
at
org.apache.bookkeeper.bookie.BookieShell.main(BookieShell.java:2067)
Caused by: org.apache.bookkeeper.client.BKException$BKReadException: Error
while reading ledger
at
org.apache.bookkeeper.client.BKException.create(BKException.java:62)
at
org.apache.bookkeeper.client.BookKeeperAdmin.recoverBookieData(BookKeeperAdmin.java:613)
at
org.apache.bookkeeper.tools.cli.commands.bookies.RecoverCommand.bkRecoveryLedger(RecoverCommand.java:213)
at
org.apache.bookkeeper.tools.cli.commands.bookies.RecoverCommand.recover(RecoverCommand.java:155)
at
org.apache.bookkeeper.tools.cli.commands.bookies.RecoverCommand.apply(RecoverCommand.java:108)
... 4 more
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
