dragonls commented on issue #2945:
URL: https://github.com/apache/bookkeeper/issues/2945#issuecomment-995493274
The log shows that lots of `No ledger found while trying to read last entry:
xxx` first:
```
22:25:19.742 [db-storage-14-1] INFO org.apache.bookkeeper.bookie.Journal -
garbage collected journal 17ce633929b.txn
22:25:20.292 [bookie-io-1-9] INFO org.apache.bookkeeper.proto.AuthHandler -
Authentication success on server side
22:25:20.292 [bookie-io-1-9] INFO
org.apache.bookkeeper.proto.BookieRequestHandler - Channel connected [id:
0x793e94c8, L:/9.146.225.131:3181 - R:/9.146.225.179:40152]
22:25:20.440 [BookieReadThreadPool-OrderedExecutor-0-0] ERROR
org.apache.bookkeeper.proto.ReadLacProcessorV3 - No ledger found while
performing readLac from ledger: 930007
org.apache.bookkeeper.bookie.Bookie$NoLedgerException: Ledger 930007 not
found
at
org.apache.bookkeeper.bookie.LedgerDescriptor.createReadOnly(LedgerDescriptor.java:52)
~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at
org.apache.bookkeeper.bookie.HandleFactoryImpl.getReadOnlyHandle(HandleFactoryImpl.java:61)
~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at
org.apache.bookkeeper.bookie.Bookie.getExplicitLac(Bookie.java:1364)
~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at
org.apache.bookkeeper.proto.ReadLacProcessorV3.getReadLacResponse(ReadLacProcessorV3.java:71)
[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at
org.apache.bookkeeper.proto.ReadLacProcessorV3.safeRun(ReadLacProcessorV3.java:118)
[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at
org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36)
[org.apache.bookkeeper-bookkeeper-common-4.12.0.jar:4.12.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_302]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_302]
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[io.netty-netty-common-4.1.60.Final.jar:4.1.60.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_302]
22:25:20.441 [BookieReadThreadPool-OrderedExecutor-7-0] INFO
org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while
reading entry: 0 from ledger: 930007
22:25:20.441 [BookieReadThreadPool-OrderedExecutor-0-0] ERROR
org.apache.bookkeeper.proto.ReadLacProcessorV3 - No ledger found while trying
to read last entry: 930007
org.apache.bookkeeper.bookie.Bookie$NoLedgerException: Ledger 930007 not
found
at
org.apache.bookkeeper.bookie.LedgerDescriptor.createReadOnly(LedgerDescriptor.java:52)
~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at
org.apache.bookkeeper.bookie.HandleFactoryImpl.getReadOnlyHandle(HandleFactoryImpl.java:61)
~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.bookie.Bookie.readEntry(Bookie.java:1441)
~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at
org.apache.bookkeeper.proto.ReadLacProcessorV3.getReadLacResponse(ReadLacProcessorV3.java:86)
[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at
org.apache.bookkeeper.proto.ReadLacProcessorV3.safeRun(ReadLacProcessorV3.java:118)
[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at
org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36)
[org.apache.bookkeeper-bookkeeper-common-4.12.0.jar:4.12.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_302]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_302]
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[io.netty-netty-common-4.1.60.Final.jar:4.1.60.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_302]
22:25:20.441 [BookieReadThreadPool-OrderedExecutor-0-0] ERROR
org.apache.bookkeeper.proto.ReadLacProcessorV3 - No ledger found while
performing readLac from ledger: 930018
org.apache.bookkeeper.bookie.Bookie$NoLedgerException: Ledger 930018 not
found
at
org.apache.bookkeeper.bookie.LedgerDescriptor.createReadOnly(LedgerDescriptor.java:52)
~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at
org.apache.bookkeeper.bookie.HandleFactoryImpl.getReadOnlyHandle(HandleFactoryImpl.java:61)
~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at
org.apache.bookkeeper.bookie.Bookie.getExplicitLac(Bookie.java:1364)
~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at
org.apache.bookkeeper.proto.ReadLacProcessorV3.getReadLacResponse(ReadLacProcessorV3.java:71)
[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at
org.apache.bookkeeper.proto.ReadLacProcessorV3.safeRun(ReadLacProcessorV3.java:118)
[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at
org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36)
[org.apache.bookkeeper-bookkeeper-common-4.12.0.jar:4.12.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_302]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_302]
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[io.netty-netty-common-4.1.60.Final.jar:4.1.60.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_302]
```
And then lots of `Bookie operation timeout`:
```
22:25:32.828 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Timed-out 2 operations to
channel [id: 0xbc5f70b3, L:/9.146.225.131:33454 -
R:9.146.225.169/9.146.225.169:3181] for 9.146.225.169:3181
22:25:32.829 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Timed-out 2 operations to
channel [id: 0x0191e854, L:/9.146.225.131:55480 -
R:9.146.225.110/9.146.225.110:3181] for 9.146.225.110:3181
22:25:32.829 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Timed-out 2 operations to
channel [id: 0x2feb6cb8, L:/9.146.225.131:40600 -
R:9.146.225.131/9.146.225.131:3181] for 9.146.225.131:3181
22:25:32.829 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Timed-out 2 operations to
channel [id: 0x9ce6a3c1, L:/9.146.225.131:36128 -
R:9.146.225.128/9.146.225.128:3181] for 9.146.225.128:3181
22:25:32.829 [ReplicationWorker] INFO
org.apache.bookkeeper.client.LedgerFragmentReplicator - Replicating fragment
Fragment(LedgerID: 1212480, FirstEntryID: 0[0], LastKnownEntryID: 49999[49999],
Host: [9.146.225.169:3181, 9.146.225.110:3181, 9.146.225.131:3181,
9.146.225.128:3181], Closed: true) in 50 sub fragments.
22:25:40.828 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout
while reading L1212480 E36190 from bookie: 9.146.225.131:3181
22:25:40.828 [BookKeeperClientWorker-OrderedExecutor-0-0] ERROR
org.apache.bookkeeper.client.PendingReadOp - Received error from a host which
is not in the ensemble 9.146.225.131:3181 [9.146.225.110:3181,
9.146.225.169:3181, 9.146.225.131:3181, 9.146.225.128:3181].
22:25:40.828 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout
while reading L1212480 E36298 from bookie: 9.146.225.131:3181
22:25:40.828 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout
while reading L1212480 E36050 from bookie: 9.146.225.131:3181
22:25:40.828 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout
while reading L1212480 E36006 from bookie: 9.146.225.131:3181
22:25:40.828 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout
while reading L1212480 E36022 from bookie: 9.146.225.131:3181
22:25:40.828 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout
while reading L1212480 E36326 from bookie: 9.146.225.131:3181
22:25:40.828 [BookKeeperClientWorker-OrderedExecutor-0-0] ERROR
org.apache.bookkeeper.client.PendingReadOp - Received error from a host which
is not in the ensemble 9.146.225.131:3181 [9.146.225.110:3181,
9.146.225.169:3181, 9.146.225.131:3181, 9.146.225.128:3181].
22:25:40.828 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout
while reading L1212480 E36762 from bookie: 9.146.225.131:3181
```
Also some `LedgerFencedException`:
```
22:26:17.715 [BookieHighPriorityThread-3181-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.proto.ReadEntryProcessorV3 - Ledger fence request
received for ledger: 1212888 from address: /9.146.225.170:56668
22:26:17.719 [bookie-io-1-7] ERROR
org.apache.bookkeeper.proto.WriteEntryProcessor - Attempt to write to fenced
ledger
org.apache.bookkeeper.bookie.BookieException$LedgerFencedException: null
at
org.apache.bookkeeper.bookie.BookieException.create(BookieException.java:57)
~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.bookie.Bookie.addEntry(Bookie.java:1399)
~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at
org.apache.bookkeeper.proto.WriteEntryProcessor.processPacket(WriteEntryProcessor.java:80)
[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at
org.apache.bookkeeper.proto.PacketProcessorBase.safeRun(PacketProcessorBase.java:85)
[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at
org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36)
[org.apache.bookkeeper-bookkeeper-common-4.12.0.jar:4.12.0]
at
org.apache.bookkeeper.proto.BookieRequestProcessor.processAddRequest(BookieRequestProcessor.java:625)
[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at
org.apache.bookkeeper.proto.BookieRequestProcessor.processRequest(BookieRequestProcessor.java:365)
[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at
org.apache.bookkeeper.proto.BookieRequestHandler.channelRead(BookieRequestHandler.java:80)
[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[io.netty-netty-transport-4.1.60.Final.jar:4.1.60.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[io.netty-netty-transport-4.1.60.Final.jar:4.1.60.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[io.netty-netty-transport-4.1.60.Final.jar:4.1.60.Final]
at
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
[io.netty-netty-transport-4.1.60.Final.jar:4.1.60.Final]
at
org.apache.bookkeeper.proto.AuthHandler$ServerSideHandler.channelRead(AuthHandler.java:95)
[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[io.netty-netty-transport-4.1.60.Final.jar:4.1.60.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[io.netty-netty-transport-4.1.60.Final.jar:4.1.60.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[io.netty-netty-transport-4.1.60.Final.jar:4.1.60.Final]
at
org.apache.bookkeeper.proto.BookieProtoEncoding$RequestDecoder.channelRead(BookieProtoEncoding.java:450)
[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[io.netty-netty-transport-4.1.60.Final.jar:4.1.60.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[io.netty-netty-transport-4.1.60.Final.jar:4.1.60.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[io.netty-netty-transport-4.1.60.Final.jar:4.1.60.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
[io.netty-netty-codec-4.1.60.Final.jar:4.1.60.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
[io.netty-netty-codec-4.1.60.Final.jar:4.1.60.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[io.netty-netty-transport-4.1.60.Final.jar:4.1.60.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[io.netty-netty-transport-4.1.60.Final.jar:4.1.60.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[io.netty-netty-transport-4.1.60.Final.jar:4.1.60.Final]
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
[io.netty-netty-transport-4.1.60.Final.jar:4.1.60.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[io.netty-netty-transport-4.1.60.Final.jar:4.1.60.Final]
```
And `Too many requests to the same Bookie`:
```
22:26:23.356 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.client.PendingReadOp - Error: Too many requests to the
same Bookie while reading L1212480 E562 from bookie: 9.146.225.131:3181
22:26:23.356 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.client.PendingReadOp - Error: Too many requests to the
same Bookie while reading L1212480 E566 from bookie: 9.146.225.131:3181
22:26:23.356 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.client.PendingReadOp - Error: Too many requests to the
same Bookie while reading L1212480 E570 from bookie: 9.146.225.131:3181
22:26:23.356 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.client.PendingReadOp - Error: Too many requests to the
same Bookie while reading L1212480 E574 from bookie: 9.146.225.131:3181
22:26:23.356 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.client.PendingReadOp - Error: Too many requests to the
same Bookie while reading L1212480 E578 from bookie: 9.146.225.131:3181
```
Also `Queue at limit of 5000 items`:
```
22:26:31.915 [bookie-io-1-11] ERROR
org.apache.bookkeeper.proto.BookieRequestHandler - Unhandled exception occurred
in I/O thread or handler on [id: 0xffe37d0c, L:/9.146.225.131:3181 -
R:/9.146.225.131:40600]
java.util.concurrent.RejectedExecutionException: Queue at limit of 5000 items
at
org.apache.bookkeeper.common.util.BoundedExecutorService.checkQueue(BoundedExecutorService.java:56)
~[org.apache.bookkeeper-bookkeeper-common-4.12.0.jar:4.12.0]
at
org.apache.bookkeeper.common.util.BoundedExecutorService.execute(BoundedExecutorService.java:88)
~[org.apache.bookkeeper-bookkeeper-common-4.12.0.jar:4.12.0]
at
com.google.common.util.concurrent.ForwardingExecutorService.execute(ForwardingExecutorService.java:99)
~[com.google.guava-guava-30.1-jre.jar:?]
at
org.apache.bookkeeper.common.util.OrderedExecutor$1.execute(OrderedExecutor.java:341)
~[org.apache.bookkeeper-bookkeeper-common-4.12.0.jar:4.12.0]
at
org.apache.bookkeeper.common.util.OrderedExecutor.executeOrdered(OrderedExecutor.java:493)
~[org.apache.bookkeeper-bookkeeper-common-4.12.0.jar:4.12.0]
at
org.apache.bookkeeper.proto.BookieRequestProcessor.processReadLacRequestV3(BookieRequestProcessor.java:416)
~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at
org.apache.bookkeeper.proto.BookieRequestProcessor.processRequest(BookieRequestProcessor.java:334)
~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at
org.apache.bookkeeper.proto.BookieRequestHandler.channelRead(BookieRequestHandler.java:80)
~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[io.netty-netty-transport-4.1.60.Final.jar:4.1.60.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[io.netty-netty-transport-4.1.60.Final.jar:4.1.60.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[io.netty-netty-transport-4.1.60.Final.jar:4.1.60.Final]
at
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
[io.netty-netty-transport-4.1.60.Final.jar:4.1.60.Final]
at
org.apache.bookkeeper.proto.AuthHandler$ServerSideHandler.channelRead(AuthHandler.java:95)
[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[io.netty-netty-transport-4.1.60.Final.jar:4.1.60.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[io.netty-netty-transport-4.1.60.Final.jar:4.1.60.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[io.netty-netty-transport-4.1.60.Final.jar:4.1.60.Final]
at
org.apache.bookkeeper.proto.BookieProtoEncoding$RequestDecoder.channelRead(BookieProtoEncoding.java:450)
[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[io.netty-netty-transport-4.1.60.Final.jar:4.1.60.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[io.netty-netty-transport-4.1.60.Final.jar:4.1.60.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[io.netty-netty-transport-4.1.60.Final.jar:4.1.60.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
[io.netty-netty-codec-4.1.60.Final.jar:4.1.60.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
[io.netty-netty-codec-4.1.60.Final.jar:4.1.60.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[io.netty-netty-transport-4.1.60.Final.jar:4.1.60.Final]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
