This is an automated email from the ASF dual-hosted git repository.
yong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/bookkeeper.git
The following commit(s) were added to refs/heads/master by this push:
new 48a87a8 Support multi ledger directories for rocksdb backend
entryMetadataMap (#2965)
48a87a8 is described below
commit 48a87a88f7d98fb890ed2aef58487f19f32a8296
Author: Hang Chen <[email protected]>
AuthorDate: Fri Feb 11 11:12:27 2022 +0800
Support multi ledger directories for rocksdb backend entryMetadataMap
(#2965)
### Motivation
When we use RocksDB backend entryMetadataMap for multi ledger directories
configured, the bookie start up failed, and throw the following exception.
```
12:24:28.530 [main] ERROR org.apache.pulsar.PulsarStandaloneStarter -
Failed to start pulsar service.
java.io.IOException: Error open RocksDB database
at
org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:202)
~[org.apache.bookkeeper-bookkeeper-server-4.15.0-SNAPSHOT.jar:4.15.0-SNAPSHOT]
at
org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:89)
~[org.apache.bookkeeper-bookkeeper-server-4.15.0-SNAPSHOT.jar:4.15.0-SNAPSHOT]
at
org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.lambda$static$0(KeyValueStorageRocksDB.java:62)
~[org.apache.bookkeeper-bookkeeper-server-4.15.0-SNAPSHOT.jar:4.15.0-SNAPSHOT]
at
org.apache.bookkeeper.bookie.storage.ldb.PersistentEntryLogMetadataMap.<init>(PersistentEntryLogMetadataMap.java:87)
~[org.apache.bookkeeper-bookkeeper-server-4.15.0-SNAPSHOT.jar:4.15.0-SNAPSHOT]
at
org.apache.bookkeeper.bookie.GarbageCollectorThread.createEntryLogMetadataMap(GarbageCollectorThread.java:265)
~[org.apache.bookkeeper-bookkeeper-server-4.15.0-SNAPSHOT.jar:4.15.0-SNAPSHOT]
at
org.apache.bookkeeper.bookie.GarbageCollectorThread.<init>(GarbageCollectorThread.java:154)
~[org.apache.bookkeeper-bookkeeper-server-4.15.0-SNAPSHOT.jar:4.15.0-SNAPSHOT]
at
org.apache.bookkeeper.bookie.GarbageCollectorThread.<init>(GarbageCollectorThread.java:133)
~[org.apache.bookkeeper-bookkeeper-server-4.15.0-SNAPSHOT.jar:4.15.0-SNAPSHOT]
at
org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage.<init>(SingleDirectoryDbLedgerStorage.java:182)
~[org.apache.bookkeeper-bookkeeper-server-4.15.0-SNAPSHOT.jar:4.15.0-SNAPSHOT]
at
org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.newSingleDirectoryDbLedgerStorage(DbLedgerStorage.java:190)
~[org.apache.bookkeeper-bookkeeper-server-4.15.0-SNAPSHOT.jar:4.15.0-SNAPSHOT]
at
org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.initialize(DbLedgerStorage.java:150)
~[org.apache.bookkeeper-bookkeeper-server-4.15.0-SNAPSHOT.jar:4.15.0-SNAPSHOT]
at
org.apache.bookkeeper.bookie.BookieResources.createLedgerStorage(BookieResources.java:110)
~[org.apache.bookkeeper-bookkeeper-server-4.15.0-SNAPSHOT.jar:4.15.0-SNAPSHOT]
at
org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.buildBookie(LocalBookkeeperEnsemble.java:328)
~[org.apache.pulsar-pulsar-zookeeper-utils-2.8.1.jar:2.8.1]
at
org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.runBookies(LocalBookkeeperEnsemble.java:391)
~[org.apache.pulsar-pulsar-zookeeper-utils-2.8.1.jar:2.8.1]
at
org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.startStandalone(LocalBookkeeperEnsemble.java:521)
~[org.apache.pulsar-pulsar-zookeeper-utils-2.8.1.jar:2.8.1]
at
org.apache.pulsar.PulsarStandalone.start(PulsarStandalone.java:264)
~[org.apache.pulsar-pulsar-broker-2.8.1.jar:2.8.1]
at
org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:121)
[org.apache.pulsar-pulsar-broker-2.8.1.jar:2.8.1]
Caused by: org.rocksdb.RocksDBException: lock hold by current process,
acquire time 1640492668 acquiring thread 123145515651072:
data/standalone/bookkeeper00/entrylogIndexCache/metadata-cache/LOCK: No locks
available
at org.rocksdb.RocksDB.open(Native Method)
~[org.rocksdb-rocksdbjni-6.10.2.jar:?]
at org.rocksdb.RocksDB.open(RocksDB.java:239)
~[org.rocksdb-rocksdbjni-6.10.2.jar:?]
at
org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:199)
~[org.apache.bookkeeper-bookkeeper-server-4.15.0-SNAPSHOT.jar:4.15.0-SNAPSHOT]
... 15 more
```
The reason is multi garbageCollectionThread will open the same RocksDB and
own the LOCK, and then throw the above exception.
### Modification
1. Change the default GcEntryLogMetadataCachePath from
`getLedgerDirNames()[0] + "/" + ENTRYLOG_INDEX_CACHE` to `null`. If it is
`null`, it will use each ledger's directory.
2. Remove the internal directory `entrylogIndexCache`. The data structure
looks like:
```
└── current
├── lastMark
├── ledgers
│ ├── 000003.log
│ ├── CURRENT
│ ├── IDENTITY
│ ├── LOCK
│ ├── LOG
│ ├── MANIFEST-000001
│ └── OPTIONS-000005
├── locations
│ ├── 000003.log
│ ├── CURRENT
│ ├── IDENTITY
│ ├── LOCK
│ ├── LOG
│ ├── MANIFEST-000001
│ └── OPTIONS-000005
└── metadata-cache
├── 000003.log
├── CURRENT
├── IDENTITY
├── LOCK
├── LOG
├── MANIFEST-000001
└── OPTIONS-000005
```
3. If user configured `GcEntryLogMetadataCachePath` in `bk_server.conf`, it
only support one ledger directory configured for `ledgerDirectories`.
Otherwise, the best practice is to keep it default.
4. The PR is better to release with #1949
---
.../java/org/apache/bookkeeper/bookie/BookieImpl.java | 11 +++++++----
.../bookkeeper/bookie/GarbageCollectorThread.java | 17 +++++++++++++----
.../bookkeeper/bookie/InterleavedLedgerStorage.java | 3 ++-
.../storage/ldb/PersistentEntryLogMetadataMap.java | 5 +++--
.../storage/ldb/SingleDirectoryDbLedgerStorage.java | 2 +-
.../org/apache/bookkeeper/conf/ServerConfiguration.java | 12 +++++++-----
.../org/apache/bookkeeper/util/BookKeeperConstants.java | 2 +-
7 files changed, 34 insertions(+), 18 deletions(-)
diff --git
a/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/BookieImpl.java
b/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/BookieImpl.java
index f0a002f..94b7525 100644
---
a/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/BookieImpl.java
+++
b/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/BookieImpl.java
@@ -25,6 +25,7 @@ import static
org.apache.bookkeeper.bookie.BookKeeperServerStats.JOURNAL_SCOPE;
import static
org.apache.bookkeeper.bookie.BookKeeperServerStats.LD_INDEX_SCOPE;
import static
org.apache.bookkeeper.bookie.BookKeeperServerStats.LD_LEDGER_SCOPE;
import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Strings;
import com.google.common.collect.Lists;
import io.netty.buffer.ByteBuf;
import io.netty.buffer.ByteBufAllocator;
@@ -1192,10 +1193,12 @@ public class BookieImpl extends BookieCriticalThread
implements Bookie {
// Clean up metadata directories if they are separate from the
// ledger dirs
- File metadataDir = new File(conf.getGcEntryLogMetadataCachePath());
- if (!cleanDir(metadataDir)) {
- LOG.error("Formatting ledger metadata directory {} failed",
metadataDir);
- return false;
+ if (!Strings.isNullOrEmpty(conf.getGcEntryLogMetadataCachePath())) {
+ File metadataDir = new File(conf.getGcEntryLogMetadataCachePath());
+ if (!cleanDir(metadataDir)) {
+ LOG.error("Formatting ledger metadata directory {} failed",
metadataDir);
+ return false;
+ }
}
LOG.info("Bookie format completed successfully");
return true;
diff --git
a/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java
b/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java
index 8e46a63..123ab14 100644
---
a/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java
+++
b/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java
@@ -21,7 +21,9 @@
package org.apache.bookkeeper.bookie;
+import static org.apache.bookkeeper.util.BookKeeperConstants.METADATA_CACHE;
import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Strings;
import io.netty.util.concurrent.DefaultThreadFactory;
import java.io.IOException;
@@ -121,6 +123,7 @@ public class GarbageCollectorThread extends SafeRunnable {
final GarbageCleaner garbageCleaner;
final ServerConfiguration conf;
+ final LedgerDirsManager ledgerDirsManager;
/**
* Create a garbage collector thread.
@@ -130,8 +133,10 @@ public class GarbageCollectorThread extends SafeRunnable {
* @throws IOException
*/
public GarbageCollectorThread(ServerConfiguration conf, LedgerManager
ledgerManager,
- final CompactableLedgerStorage ledgerStorage, StatsLogger
statsLogger) throws IOException {
- this(conf, ledgerManager, ledgerStorage, statsLogger,
+ final LedgerDirsManager ledgerDirsManager,
+ final CompactableLedgerStorage ledgerStorage,
+ StatsLogger statsLogger) throws IOException {
+ this(conf, ledgerManager, ledgerDirsManager, ledgerStorage,
statsLogger,
Executors.newSingleThreadScheduledExecutor(new
DefaultThreadFactory("GarbageCollectorThread")));
}
@@ -144,6 +149,7 @@ public class GarbageCollectorThread extends SafeRunnable {
*/
public GarbageCollectorThread(ServerConfiguration conf,
LedgerManager ledgerManager,
+ final LedgerDirsManager ledgerDirsManager,
final CompactableLedgerStorage ledgerStorage,
StatsLogger statsLogger,
ScheduledExecutorService gcExecutor)
@@ -151,6 +157,7 @@ public class GarbageCollectorThread extends SafeRunnable {
this.gcExecutor = gcExecutor;
this.conf = conf;
+ this.ledgerDirsManager = ledgerDirsManager;
this.entryLogger = ledgerStorage.getEntryLogger();
this.entryLogMetaMap = createEntryLogMetadataMap();
this.ledgerStorage = ledgerStorage;
@@ -261,11 +268,13 @@ public class GarbageCollectorThread extends SafeRunnable {
private EntryLogMetadataMap createEntryLogMetadataMap() throws IOException
{
if (conf.isGcEntryLogMetadataCacheEnabled()) {
- String baseDir = this.conf.getGcEntryLogMetadataCachePath();
+ String baseDir =
Strings.isNullOrEmpty(conf.getGcEntryLogMetadataCachePath())
+ ? this.ledgerDirsManager.getAllLedgerDirs().get(0).getPath() :
conf.getGcEntryLogMetadataCachePath();
try {
return new PersistentEntryLogMetadataMap(baseDir, conf);
} catch (IOException e) {
- LOG.error("Failed to initialize persistent-metadata-map ,
clean up {}", baseDir, e);
+ LOG.error("Failed to initialize persistent-metadata-map ,
clean up {}",
+ baseDir + "/" + METADATA_CACHE, e);
throw e;
}
} else {
diff --git
a/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/InterleavedLedgerStorage.java
b/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/InterleavedLedgerStorage.java
index dcfeed0..0960c66 100644
---
a/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/InterleavedLedgerStorage.java
+++
b/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/InterleavedLedgerStorage.java
@@ -193,7 +193,8 @@ public class InterleavedLedgerStorage implements
CompactableLedgerStorage, Entry
this.entryLogger.addListener(this);
ledgerCache = new LedgerCacheImpl(conf, activeLedgers,
null == indexDirsManager ? ledgerDirsManager :
indexDirsManager, statsLogger);
- gcThread = new GarbageCollectorThread(conf, ledgerManager, this,
statsLogger.scope("gc"));
+ gcThread = new GarbageCollectorThread(conf, ledgerManager,
ledgerDirsManager,
+ this, statsLogger.scope("gc"));
pageSize = conf.getPageSize();
ledgerDirsManager.addLedgerDirsListener(getLedgerDirsListener());
// Expose Stats
diff --git
a/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/PersistentEntryLogMetadataMap.java
b/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/PersistentEntryLogMetadataMap.java
index 812ab84..95b3659 100644
---
a/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/PersistentEntryLogMetadataMap.java
+++
b/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/PersistentEntryLogMetadataMap.java
@@ -20,6 +20,7 @@
*/
package org.apache.bookkeeper.bookie.storage.ldb;
+import static org.apache.bookkeeper.util.BookKeeperConstants.METADATA_CACHE;
import io.netty.util.concurrent.FastThreadLocal;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
@@ -77,14 +78,14 @@ public class PersistentEntryLogMetadataMap implements
EntryLogMetadataMap {
};
public PersistentEntryLogMetadataMap(String metadataPath,
ServerConfiguration conf) throws IOException {
- LOG.info("Loading persistent entrylog metadata-map from {}",
metadataPath);
+ LOG.info("Loading persistent entrylog metadata-map from {}",
metadataPath + "/" + METADATA_CACHE);
File dir = new File(metadataPath);
if (!dir.mkdirs() && !dir.exists()) {
String err = "Unable to create directory " + dir;
LOG.error(err);
throw new IOException(err);
}
- metadataMapDB =
KeyValueStorageRocksDB.factory.newKeyValueStorage(metadataPath,
"metadata-cache",
+ metadataMapDB =
KeyValueStorageRocksDB.factory.newKeyValueStorage(metadataPath, METADATA_CACHE,
DbConfigType.Small, conf);
}
diff --git
a/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/SingleDirectoryDbLedgerStorage.java
b/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/SingleDirectoryDbLedgerStorage.java
index 4d2e4ad..ce8af36 100644
---
a/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/SingleDirectoryDbLedgerStorage.java
+++
b/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/SingleDirectoryDbLedgerStorage.java
@@ -183,7 +183,7 @@ public class SingleDirectoryDbLedgerStorage implements
CompactableLedgerStorage
TransientLedgerInfo.LEDGER_INFO_CACHING_TIME_MINUTES,
TimeUnit.MINUTES);
entryLogger = new EntryLogger(conf, ledgerDirsManager, null,
statsLogger, allocator);
- gcThread = new GarbageCollectorThread(conf, ledgerManager, this,
statsLogger);
+ gcThread = new GarbageCollectorThread(conf, ledgerManager,
ledgerDirsManager, this, statsLogger);
dbLedgerStorageStats = new DbLedgerStorageStats(
ledgerDirStatsLogger,
diff --git
a/bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ServerConfiguration.java
b/bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ServerConfiguration.java
index a587145..3f099eb 100644
---
a/bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ServerConfiguration.java
+++
b/bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ServerConfiguration.java
@@ -17,7 +17,6 @@
*/
package org.apache.bookkeeper.conf;
-import static
org.apache.bookkeeper.util.BookKeeperConstants.ENTRYLOG_INDEX_CACHE;
import static
org.apache.bookkeeper.util.BookKeeperConstants.MAX_LOG_SIZE_LIMIT;
import com.google.common.annotations.Beta;
@@ -515,17 +514,20 @@ public class ServerConfiguration extends
AbstractConfiguration<ServerConfigurati
* gcPersistentEntrylogMetadataMapEnabled is true.
*
* @return entrylog metadata-map persistent store dir path.(default: it
- * creates a sub-directory under a first available base ledger
- * directory with name "entrylogIndexCache").
+ * creates a sub-directory under each ledger
+ * directory with name "metadata-cache". If it set, it only works
for one ledger directory
+ * configured for ledgerDirectories).
*/
public String getGcEntryLogMetadataCachePath() {
- return getString(GC_ENTRYLOG_METADATA_CACHE_PATH,
getLedgerDirNames()[0] + "/" + ENTRYLOG_INDEX_CACHE);
+ return getString(GC_ENTRYLOG_METADATA_CACHE_PATH, null);
}
/**
* Set directory to persist Entrylog metadata if
gcPersistentEntrylogMetadataMapEnabled is true.
+ * If it set, it only works for one ledger directory configured for
ledgerDirectories. For multi ledgerDirectory
+ * configured, keep the default value is the best practice.
*
- * @param gcPersistentEntrylogMetadataMapPath.
+ * @param gcEntrylogMetadataCachePath
* @return server configuration.
*/
public ServerConfiguration setGcEntryLogMetadataCachePath(String
gcEntrylogMetadataCachePath) {
diff --git
a/bookkeeper-server/src/main/java/org/apache/bookkeeper/util/BookKeeperConstants.java
b/bookkeeper-server/src/main/java/org/apache/bookkeeper/util/BookKeeperConstants.java
index 8e10a15..d928aac 100644
---
a/bookkeeper-server/src/main/java/org/apache/bookkeeper/util/BookKeeperConstants.java
+++
b/bookkeeper-server/src/main/java/org/apache/bookkeeper/util/BookKeeperConstants.java
@@ -31,7 +31,7 @@ public class BookKeeperConstants {
public static final String BOOKIE_STATUS_FILENAME = "BOOKIE_STATUS";
public static final String PASSWD = "passwd";
public static final String CURRENT_DIR = "current";
- public static final String ENTRYLOG_INDEX_CACHE = "entrylogIndexCache";
+ public static final String METADATA_CACHE = "metadata-cache";
public static final String READONLY = "readonly";
// //////////////////////////
