This is an automated email from the ASF dual-hosted git repository.
ethanfeng pushed a commit to branch branch-0.4
in repository https://gitbox.apache.org/repos/asf/celeborn.git
The following commit(s) were added to refs/heads/branch-0.4 by this push:
new 3f3e0a78b [CELEBORN-1280] Change default value of
celeborn.worker.graceful.shutdown.recoverDbBackend to ROCKSDB
3f3e0a78b is described below
commit 3f3e0a78b76858aa1e9993f4ce7caf4512ba162b
Author: SteNicholas <[email protected]>
AuthorDate: Fri Feb 23 14:53:24 2024 +0800
[CELEBORN-1280] Change default value of
celeborn.worker.graceful.shutdown.recoverDbBackend to ROCKSDB
### What changes were proposed in this pull request?
Change the default value of
`celeborn.worker.graceful.shutdown.recoverDbBackend` from `LEVELDB` to
`ROCKSDB`.
### Why are the changes needed?
Because the LevelDB support will be removed, the default value of
`celeborn.worker.graceful.shutdown.recoverDbBackend` could be changed to
ROCKSDB instead of LEVELDB for preparation of LevelDB deprecation.
Backport:
[[SPARK-45351][CORE] Change spark.shuffle.service.db.backend default value
to ROCKSDB](https://github.com/apache/spark/pull/43142)
[[SPARK-45413][CORE] Add warning for prepare drop LevelDB
support](https://github.com/apache/spark/pull/43217)
### Does this PR introduce _any_ user-facing change?
The default value of `celeborn.worker.graceful.shutdown.recoverDbBackend`
is changed from `LEVELDB` to `ROCKSDB`.
### How was this patch tested?
No.
Closes #2320 from SteNicholas/CELEBORN-1280.
Lead-authored-by: SteNicholas <[email protected]>
Co-authored-by: Nicholas Jiang <[email protected]>
Signed-off-by: mingji <[email protected]>
(cherry picked from commit b9bdea3c72fcc7ffc45bd24d53e920cea5516c5c)
---
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala | 4 ++--
docs/configuration/worker.md | 2 +-
docs/migration.md | 3 +++
.../apache/celeborn/service/deploy/worker/shuffledb/DBProvider.java | 6 ++++++
4 files changed, 12 insertions(+), 3 deletions(-)
diff --git
a/common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala
b/common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala
index 261e97478..2d9552448 100644
--- a/common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala
+++ b/common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala
@@ -2917,12 +2917,12 @@ object CelebornConf extends Logging {
val WORKER_GRACEFUL_SHUTDOWN_RECOVER_DB_BACKEND: ConfigEntry[String] =
buildConf("celeborn.worker.graceful.shutdown.recoverDbBackend")
.categories("worker")
- .doc("Specifies a disk-based store used in local db. LEVELDB or
ROCKSDB.")
+ .doc("Specifies a disk-based store used in local db. ROCKSDB or LEVELDB
(deprecated).")
.version("0.4.0")
.stringConf
.transform(_.toUpperCase(Locale.ROOT))
.checkValues(Set("LEVELDB", "ROCKSDB"))
- .createWithDefault("LEVELDB")
+ .createWithDefault("ROCKSDB")
val WORKER_PARTITION_SORTER_SHUTDOWN_TIMEOUT: ConfigEntry[Long] =
buildConf("celeborn.worker.graceful.shutdown.partitionSorter.shutdownTimeout")
diff --git a/docs/configuration/worker.md b/docs/configuration/worker.md
index 1f7f48a24..8a62d9936 100644
--- a/docs/configuration/worker.md
+++ b/docs/configuration/worker.md
@@ -63,7 +63,7 @@ license: |
| celeborn.worker.graceful.shutdown.checkSlotsFinished.timeout | 480s | The
wait time of waiting for the released slots to be committed or destroyed during
worker graceful shutdown. | 0.2.0 | |
| celeborn.worker.graceful.shutdown.enabled | false | When true, during worker
shutdown, the worker will wait for all released slots to be committed or
destroyed. | 0.2.0 | |
| celeborn.worker.graceful.shutdown.partitionSorter.shutdownTimeout | 120s |
The wait time of waiting for sorting partition files during worker graceful
shutdown. | 0.2.0 | |
-| celeborn.worker.graceful.shutdown.recoverDbBackend | LEVELDB | Specifies a
disk-based store used in local db. LEVELDB or ROCKSDB. | 0.4.0 | |
+| celeborn.worker.graceful.shutdown.recoverDbBackend | ROCKSDB | Specifies a
disk-based store used in local db. ROCKSDB or LEVELDB (deprecated). | 0.4.0 |
|
| celeborn.worker.graceful.shutdown.recoverPath | <tmp>/recover | The
path to store DB. | 0.2.0 | |
| celeborn.worker.graceful.shutdown.saveCommittedFileInfo.interval | 5s |
Interval for a Celeborn worker to flush committed file infos into Level DB. |
0.3.1 | |
| celeborn.worker.graceful.shutdown.saveCommittedFileInfo.sync | false |
Whether to call sync method to save committed file infos into Level DB to
handle OS crash. | 0.3.1 | |
diff --git a/docs/migration.md b/docs/migration.md
index 6128357e7..fd4fcdc7f 100644
--- a/docs/migration.md
+++ b/docs/migration.md
@@ -26,6 +26,9 @@ license: |
- Since 0.4.1, Celeborn master adds a limit to the estimated partition size
used for computing worker slots.
This size is now constrained within the range specified by
`celeborn.master.estimatedPartitionSize.minSize` and
`celeborn.master.estimatedPartitionSize.maxSize`.
+- Since 0.5.0, Celeborn changed the default value of
`celeborn.worker.graceful.shutdown.recoverDbBackend` from `LEVELDB` to
`ROCKSDB`, which means Celeborn will use RocksDB store for recovery backend.
+ To restore the behavior before Celeborn 0.5, you can set
`celeborn.worker.graceful.shutdown.recoverDbBackend` to `LEVELDB`.
+
## Upgrading from 0.3 to 0.4
- Since 0.4.0, Celeborn won't be compatible with Celeborn client that versions
below 0.3.0.
diff --git
a/worker/src/main/java/org/apache/celeborn/service/deploy/worker/shuffledb/DBProvider.java
b/worker/src/main/java/org/apache/celeborn/service/deploy/worker/shuffledb/DBProvider.java
index 8223e6a77..41df2a4dc 100644
---
a/worker/src/main/java/org/apache/celeborn/service/deploy/worker/shuffledb/DBProvider.java
+++
b/worker/src/main/java/org/apache/celeborn/service/deploy/worker/shuffledb/DBProvider.java
@@ -20,14 +20,20 @@ package org.apache.celeborn.service.deploy.worker.shuffledb;
import java.io.File;
import java.io.IOException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
/** Note: code copied from Apache Spark. */
public class DBProvider {
+ private static final Logger logger =
LoggerFactory.getLogger(DBProvider.class);
+
public static DB initDB(DBBackend dbBackend, File dbFile, StoreVersion
version)
throws IOException {
if (dbFile != null) {
switch (dbBackend) {
case LEVELDB:
org.iq80.leveldb.DB levelDB = LevelDBProvider.initLevelDB(dbFile,
version);
+ logger.warn("The LEVELDB is deprecated. Please use ROCKSDB
instead.");
return levelDB != null ? new LevelDB(levelDB) : null;
case ROCKSDB:
org.rocksdb.RocksDB rocksDB = RocksDBProvider.initRockDB(dbFile,
version);
