hrsakai edited a comment on issue #2103: Ledger directory is 100% used URL: https://github.com/apache/bookkeeper/issues/2103#issuecomment-501535512 Similar issue has happened in our environment. Compaction suddenly stoped with following exception. ``` 09:23:42.310 [GarbageCollectorThread-11-1] INFO o.a.b.bookie.GarbageCollectorThread - Enter minor compaction, suspendMinor false 09:23:42.310 [GarbageCollectorThread-11-1] INFO o.a.b.bookie.GarbageCollectorThread - Do compaction to compact those files lower than 0.2 . . 09:25:17.681 [GarbageCollectorThread-11-1] INFO o.a.b.bookie.EntryLogCompactor - Removing entry log 27135 after compaction 09:25:17.810 [GarbageCollectorThread-11-1] INFO o.a.b.bookie.GarbageCollectorThread - Removing entry log metadata for 27135 09:25:17.810 [GarbageCollectorThread-11-1] DEBUG o.a.b.bookie.GarbageCollectorThread - Compacting entry log 6743 below threshold 0.2 09:25:17.984 [GarbageCollectorThread-11-1] ERROR o.a.b.common.util.SafeRunnable - Unexpected throwable caught java.lang.IllegalArgumentException: Keys and values must be >= 0 at org.apache.bookkeeper.util.collections.ConcurrentLongLongHashMap.checkBiggerEqualZero(ConcurrentLongLongHashMap.java:731) at org.apache.bookkeeper.util.collections.ConcurrentLongLongHashMap.get(ConcurrentLongLongHashMap.java:141) at org.apache.bookkeeper.util.collections.ConcurrentLongLongHashMap.containsKey(ConcurrentLongLongHashMap.java:147) at org.apache.bookkeeper.bookie.EntryLogMetadata.containsLedger(EntryLogMetadata.java:51) at org.apache.bookkeeper.bookie.EntryLogCompactor$CompactionScannerFactory$1.accept(EntryLogCompactor.java:85) at org.apache.bookkeeper.bookie.EntryLogger.scanEntryLog(EntryLogger.java:1211) at org.apache.bookkeeper.bookie.EntryLogCompactor.compact(EntryLogCompactor.java:56) at org.apache.bookkeeper.bookie.GarbageCollectorThread.compactEntryLog(GarbageCollectorThread.java:536) at org.apache.bookkeeper.bookie.GarbageCollectorThread.doCompactEntryLogs(GarbageCollectorThread.java:475) at org.apache.bookkeeper.bookie.GarbageCollectorThread.runWithFlags(GarbageCollectorThread.java:381) at org.apache.bookkeeper.bookie.GarbageCollectorThread.safeRun(GarbageCollectorThread.java:333) at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 09:25:38.374 [bookie-io-1-53] INFO o.a.bookkeeper.proto.AuthHandler - Authentication success on server side 09:25:38.374 [bookie-io-1-53] INFO o.a.b.proto.BookieRequestHandler - Channel connected [id: 0x614a946d, L:/xxxxxx:3181 - R:/xxxxxx:39748] ``` At this time, it seems that the `compacting` flag does not be changed to` false` because following line is not executed. https://github.com/apache/bookkeeper/blob/release-4.7.3/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java#L538 Therefore, after this compaction, entry log files are not deleted by any other compaction until restarting the bookie. https://github.com/apache/bookkeeper/blob/release-4.7.3/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java#L530-L534 The specific entry log id(`6743`) raises above exception on every first compaction after restarting. There are log usage buckets logs outputted by compactions after this compaction. (This compaction has not been reported) ``` 10:58:48.317 [GarbageCollectorThread-11-1] INFO o.a.b.bookie.GarbageCollectorThread - Compaction: entry log usage buckets[10% 20% 30% 40% 50% 60% 70% 80% 90% 100%] = [0, 13425, 721, 320, 195, 99, 46, 18, 10, 4] 11:58:48.392 [GarbageCollectorThread-11-1] INFO o.a.b.bookie.GarbageCollectorThread - Compaction: entry log usage buckets[10% 20% 30% 40% 50% 60% 70% 80% 90% 100%] = [0, 13439, 716, 318, 197, 101, 45, 18, 8, 4] ``` **Bookkeeper version** 4.7.3 **settings** ``` minorCompactionThreshold=0.2 minorCompactionInterval=3600 majorCompactionThreshold=0.5 majorCompactionInterval=86400 ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
