kaykillyou opened a new issue #6751: org.apache.bookkeeper.proto.WriteEntryProcessor - Attempt to write to fenced ledger URL: https://github.com/apache/pulsar/issues/6751 **Describe the bug** there a exception after restart bookie, and all machines in the cluster keep reporting errors as below: 2020-04-17 12:52:12.009 INFO [GarbageCollectorThread-11-1] org.apache.bookkeeper.bookie.GarbageCollectorThread - Enter minor compaction, suspendMinor false 2020-04-17 12:52:12.009 INFO [GarbageCollectorThread-11-1] org.apache.bookkeeper.bookie.GarbageCollectorThread - Do compaction to compact those files lower than 0.2 2020-04-17 12:52:12.040 INFO [db-storage-cleanup-10-1] org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndex - Deleting indexes for ledgers: [28960, 28725, 28965, 28726, 28727, 28967, 28280] 2020-04-17 12:52:12.172 INFO [db-storage-cleanup-10-1] org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndex - Deleted indexes for 100016 entries from 7 ledgers in 0.131 seconds 2020-04-17 12:54:19.134 INFO [GarbageCollectorThread-11-1] org.apache.bookkeeper.bookie.EntryLogCompactor - Removing entry log 214 after compaction 2020-04-17 12:54:19.200 INFO [GarbageCollectorThread-11-1] org.apache.bookkeeper.bookie.GarbageCollectorThread - Removing entry log metadata for 214 2020-04-17 12:54:19.200 INFO [GarbageCollectorThread-11-1] org.apache.bookkeeper.bookie.GarbageCollectorThread - Compaction: entry log usage buckets[10% 20% 30% 40% 50% 60% 70% 80% 90% 100%] = [0, 1, 0, 0, 0, 0, 0, 0, 0, 0] 2020-04-17 13:01:50.892 INFO [BookieJournal-3181] org.apache.bookkeeper.bookie.JournalChannel - Opening journal /data1/bookkeeper/journal/current/16f3c9ea3e4.txn 2020-04-17 13:02:12.041 INFO [SyncThread-7-1] org.apache.bookkeeper.bookie.Journal - garbage collected journal 16f3c9ea3de.txn 2020-04-17 13:07:12.036 INFO [db-storage-cleanup-10-1] org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndex - Deleting indexes for ledgers: [28976, 28978, 28981, 28982, 28983, 28973] 2020-04-17 13:07:12.374 INFO [db-storage-cleanup-10-1] org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndex - Deleted indexes for 250008 entries from 6 ledgers in 0.337 seconds 2020-04-17 13:22:12.036 INFO [db-storage-cleanup-10-1] org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndex - Deleting indexes for ledgers: [28980, 28984] 2020-04-17 13:22:12.380 INFO [db-storage-cleanup-10-1] org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndex - Deleted indexes for 100000 entries from 2 ledgers in 0.344 seconds 2020-04-17 13:26:48.176 INFO [bookie-io-1-12] org.apache.bookkeeper.proto.AuthHandler - Authentication success on server side 2020-04-17 13:26:48.176 INFO [bookie-io-1-12] org.apache.bookkeeper.proto.BookieRequestHandler - Channel connected [id: 0x410eafe2, L:/10.52.10.114:3181 - R:/10.52.10.115:47648] 2020-04-17 13:37:12.036 INFO [db-storage-cleanup-10-1] org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndex - Deleting indexes for ledgers: [28992, 28963, 28986] 2020-04-17 13:37:12.127 INFO [db-storage-cleanup-10-1] org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndex - Deleted indexes for 100984 entries from 3 ledgers in 0.09 seconds 2020-04-17 13:41:47.029 INFO [bookie-io-1-13] org.apache.bookkeeper.proto.AuthHandler - Authentication success on server side 2020-04-17 13:41:47.029 INFO [bookie-io-1-13] org.apache.bookkeeper.proto.BookieRequestHandler - Channel connected [id: 0x490c94fc, L:/10.52.10.114:3181 - R:/10.52.10.114:49494] 2020-04-17 13:41:47.385 INFO [bookie-io-1-14] org.apache.bookkeeper.proto.AuthHandler - Authentication success on server side 2020-04-17 13:41:47.385 INFO [bookie-io-1-14] org.apache.bookkeeper.proto.BookieRequestHandler - Channel connected [id: 0xceb71e0d, L:/10.52.10.114:3181 - R:/10.52.10.116:56866] 2020-04-17 13:41:47.386 INFO [bookie-io-1-15] org.apache.bookkeeper.proto.AuthHandler - Authentication success on server side 2020-04-17 13:41:47.386 INFO [bookie-io-1-15] org.apache.bookkeeper.proto.BookieRequestHandler - Channel connected [id: 0x2c0138fa, L:/10.52.10.114:3181 - R:/10.52.10.115:47792] 2020-04-17 13:41:47.388 INFO [bookie-io-1-16] org.apache.bookkeeper.proto.AuthHandler - Authentication success on server side 2020-04-17 13:41:47.388 INFO [bookie-io-1-16] org.apache.bookkeeper.proto.BookieRequestHandler - Channel connected [id: 0xcad36398, L:/10.52.10.114:3181 - R:/10.52.10.114:49502] 2020-04-17 13:42:11.346 ERROR [bookie-io-1-11] org.apache.bookkeeper.proto.WriteEntryProcessor - Attempt to write to fenced ledger org.apache.bookkeeper.bookie.BookieException$LedgerFencedException: null at org.apache.bookkeeper.bookie.BookieException.create(BookieException.java:57) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.bookie.Bookie.addEntry(Bookie.java:1374) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.proto.WriteEntryProcessor.processPacket(WriteEntryProcessor.java:79) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.proto.PacketProcessorBase.safeRun(PacketProcessorBase.java:85) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0] at org.apache.bookkeeper.proto.BookieRequestProcessor.processAddRequest(BookieRequestProcessor.java:616) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.proto.BookieRequestProcessor.processRequest(BookieRequestProcessor.java:365) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at org.apache.bookkeeper.proto.BookieRequestHandler.channelRead(BookieRequestHandler.java:80) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final] at org.apache.bookkeeper.proto.AuthHandler$ServerSideHandler.channelRead(AuthHandler.java:89) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final] at org.apache.bookkeeper.proto.BookieProtoEncoding$RequestDecoder.channelRead(BookieProtoEncoding.java:441) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:326) [io.netty-netty-codec-4.1.43.Final.jar:4.1.43.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:300) [io.netty-netty-codec-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144] **To Reproduce** 1. restart the bookie service incorrectly with root user 2. stop the bookie service and start service with normal user 3. an error will occur after the service is started 4. See error **Expected behavior** expected to solve this issue **Screenshots** **Desktop (please complete the following information):** **Additional context**
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
