This is an automated email from the ASF dual-hosted git repository.
chenhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/bookkeeper.git
The following commit(s) were added to refs/heads/master by this push:
new 8309b1895c Fix garbage collection blocked by runtime exception (#3901)
8309b1895c is described below
commit 8309b1895c240a7fba8719cacf551b2cc4abda42
Author: Hang Chen <[email protected]>
AuthorDate: Mon May 15 21:56:35 2023 +0800
Fix garbage collection blocked by runtime exception (#3901)
### Motivation
When one ledger file is broken, `getEntryLogMetadata` will throw an
IllegalArgumentException, which is a RuntimeException and won't be caught in
the whole garbage collector execution path. The exception will be caught by the
thread SafeRunnable and interrupt the garbage collector, leading to those
deleted ledgers can't be recycled and the ledger disk usage up.
```
2023-04-01T00:55:29,497+0000 [GarbageCollectorThread-11-1] INFO
org.apache.bookkeeper.bookie.GarbageCollectorThread - Garbage collector thread
forced to perform GC before expiry of wait time.
2023-04-01T00:55:29,498+0000 [GarbageCollectorThread-11-1] INFO
org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta
from entryLogId: 17
2023-04-01T00:55:29,498+0000 [GarbageCollectorThread-11-1] INFO
org.apache.bookkeeper.bookie.EntryLogger - Failed to get ledgers map index
from: 17.log : Cannot deserialize ledgers map from ledger 9062744587808030975
2023-04-01T00:55:29,553+0000 [GarbageCollectorThread-11-1] ERROR
org.apache.bookkeeper.common.util.SafeRunnable - Unexpected throwable caught
java.lang.IllegalArgumentException: Negative position
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:785) ~[?:?]
at
org.apache.bookkeeper.bookie.BufferedReadChannel.read(BufferedReadChannel.java:93)
~[org.apache.bookkeeper-bookkeeper-server-4.14.5.jar:4.14.5]
at
org.apache.bookkeeper.bookie.BufferedReadChannel.read(BufferedReadChannel.java:65)
~[org.apache.bookkeeper-bookkeeper-server-4.14.5.jar:4.14.5]
at
org.apache.bookkeeper.bookie.EntryLogger.readFromLogChannel(EntryLogger.java:418)
~[org.apache.bookkeeper-bookkeeper-server-4.14.5.jar:4.14.5]
at
org.apache.bookkeeper.bookie.EntryLogger.scanEntryLog(EntryLogger.java:996)
~[org.apache.bookkeeper-bookkeeper-server-4.14.5.jar:4.14.5]
at
org.apache.bookkeeper.bookie.EntryLogger.extractEntryLogMetadataByScanning(EntryLogger.java:1136)
~[org.apache.bookkeeper-bookkeeper-server-4.14.5.jar:4.14.5]
at
org.apache.bookkeeper.bookie.EntryLogger.getEntryLogMetadata(EntryLogger.java:1045)
~[org.apache.bookkeeper-bookkeeper-server-4.14.5.jar:4.14.5]
at
org.apache.bookkeeper.bookie.GarbageCollectorThread.extractMetaFromEntryLogs(GarbageCollectorThread.java:607)
~[org.apache.bookkeeper-bookkeeper-server-4.14.5.jar:4.14.5]
at
org.apache.bookkeeper.bookie.GarbageCollectorThread.runWithFlags(GarbageCollectorThread.java:348)
~[org.apache.bookkeeper-bookkeeper-server-4.14.5.jar:4.14.5]
at
org.apache.bookkeeper.bookie.GarbageCollectorThread.safeRun(GarbageCollectorThread.java:329)
~[org.apache.bookkeeper-bookkeeper-server-4.14.5.jar:4.14.5]
at
org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36)
~[org.apache.bookkeeper-bookkeeper-common-4.14.5.jar:4.14.5]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
~[?:?]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
```
### Changes
Catch the RuntimeException when getting the metadata for each entry log
file.
---
.../main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java
b/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java
index ab66f03e09..552592f624 100644
---
a/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java
+++
b/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java
@@ -760,7 +760,7 @@ public class GarbageCollectorThread implements Runnable {
} else {
entryLogMetaMap.put(entryLogId, entryLogMeta);
}
- } catch (IOException e) {
+ } catch (IOException | RuntimeException e) {
LOG.warn("Premature exception when processing " + entryLogId
+ " recovery will take care of the problem", e);
}
