This is an automated email from the ASF dual-hosted git repository.
roryqi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-uniffle.git
The following commit(s) were added to refs/heads/master by this push:
new 80cc3d1f8 [MINOR] refactor: Rename MultiStorage to HybridStorage
(#1191)
80cc3d1f8 is described below
commit 80cc3d1f87798cc4f656fa6ccb8effaf26874748
Author: roryqi <[email protected]>
AuthorDate: Mon Sep 11 15:21:39 2023 +0800
[MINOR] refactor: Rename MultiStorage to HybridStorage (#1191)
### What changes were proposed in this pull request?
Rename MultiStorage to HybridStorage.
### Why are the changes needed?
HybridStorage seems more accurate
### Does this PR introduce _any_ user-facing change?
We deprecated two config options.
### How was this patch tested?
CI passed.
---
docs/server_guide.md | 72 +++++++++++-----------
...e.java => HybridStorageFaultToleranceBase.java} | 2 +-
...t.java => HybridStorageHadoopFallbackTest.java} | 2 +-
...ava => HybridStorageLocalFileFallbackTest.java} | 8 +--
.../apache/uniffle/test/ShuffleServerGrpcTest.java | 4 +-
...RepartitionWithHadoopHybridStorageRssTest.java} | 4 +-
...RepartitionWithMemoryHybridStorageRssTest.java} | 2 +-
.../apache/uniffle/server/ShuffleServerConf.java | 16 ++---
...orageManager.java => HybridStorageManager.java} | 12 ++--
.../server/storage/StorageManagerFactory.java | 2 +-
.../DefaultStorageManagerSelector.java | 2 +-
.../FallbackBasedStorageManagerSelector.java | 2 +-
...gePartitionSensitiveStorageManagerSelector.java | 2 +-
.../{multi => hybrid}/StorageManagerSelector.java | 2 +-
.../uniffle/server/ShuffleFlushManagerTest.java | 16 ++---
...agerTest.java => HybridStorageManagerTest.java} | 18 +++---
16 files changed, 84 insertions(+), 82 deletions(-)
diff --git a/docs/server_guide.md b/docs/server_guide.md
index d1865c6e0..96e6f6c25 100644
--- a/docs/server_guide.md
+++ b/docs/server_guide.md
@@ -64,42 +64,42 @@ This document will introduce how to deploy Uniffle shuffle
servers.
```
## Configuration
-| Property Name | Default |
Description
|
-|---------------------------------------------------------|---------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| rss.coordinator.quorum | - |
Coordinator quorum
|
-| rss.rpc.server.port | - | RPC port
for Shuffle server, if set zero, grpc server start on random port.
|
-| rss.jetty.http.port | - | Http
port for Shuffle server
|
-| rss.server.netty.port | -1 | Netty
port for Shuffle server, if set zero, netty server start on random port.
|
-| rss.server.netty.epoll.enable | false | If
enable epoll model with netty server.
|
-| rss.server.netty.accept.thread | 10 | Accept
thread count in netty.
|
-| rss.server.netty.worker.thread | 100 | Worker
thread count in netty.
|
-| rss.server.netty.connect.backlog | 0 | For
netty server, requested maximum length of the queue of incoming connections.
|
-| rss.server.netty.connect.timeout | 5000 | Timeout
for connection in netty.
|
-| rss.server.netty.receive.buf | 0 | Receive
buffer size (SO_RCVBUF). Note: the optimal size for receive buffer and send
buffer should be latency * network_bandwidth. Assuming latency = 1ms,
network_bandwidth = 10Gbps, buffer size should be ~ 1.25MB. Default is 0, the
operating system automatically estimates the receive buffer size based on
default settings. |
-| rss.server.netty.send.buf | 0 | Send
buffer size (SO_SNDBUF).
|
-| rss.server.buffer.capacity | -1 | Max
memory of buffer manager for shuffle server. If negative, JVM heap size *
buffer.ratio is used
|
-| rss.server.buffer.capacity.ratio | 0.8 | when
`rss.server.buffer.capacity`=-1, then the buffer capacity is JVM heap size *
ratio
|
-| rss.server.memory.shuffle.highWaterMark.percentage | 75.0 |
Threshold of spill data to storage, percentage of rss.server.buffer.capacity
|
-| rss.server.memory.shuffle.lowWaterMark.percentage | 25.0 |
Threshold of keep data in memory, percentage of rss.server.buffer.capacity
|
-| rss.server.read.buffer.capacity | -1 | Max size
of buffer for reading data. If negative, JVM heap size * read.buffer.ratio is
used
|
-| rss.server.read.buffer.capacity.ratio | 0.4 | when
`rss.server.read.buffer.capacity`=-1, then read buffer capacity is JVM heap
size * ratio
|
-| rss.server.heartbeat.interval | 10000 |
Heartbeat interval to Coordinator (ms)
|
-| rss.server.flush.localfile.threadPool.size | 10 | Thread
pool for flush data to local file
|
-| rss.server.flush.hadoop.threadPool.size | 60 | Thread
pool for flush data to hadoop storage
|
-| rss.server.commit.timeout | 600000 | Timeout
when commit shuffle data (ms)
|
-| rss.storage.type | - | Supports
MEMORY_LOCALFILE, MEMORY_HDFS, MEMORY_LOCALFILE_HDFS
|
-| rss.server.flush.cold.storage.threshold.size | 64M | The
threshold of data size for LOACALFILE and HADOOP if MEMORY_LOCALFILE_HDFS is
used
|
-| rss.server.tags | - | The
comma-separated list of tags to indicate the shuffle server's attributes. It
will be used as the assignment basis for the coordinator
|
-| rss.server.single.buffer.flush.enabled | true | Whether
single buffer flush when size exceeded rss.server.single.buffer.flush.threshold
|
-| rss.server.single.buffer.flush.threshold | 128M | The
threshold of single shuffle buffer flush
|
-| rss.server.disk.capacity | -1 | Disk
capacity that shuffle server can use. If negative, it will use disk whole space
* ratio
|
-| rss.server.disk.capacity.ratio | 0.9 | When
`rss.server.disk.capacity` is negative, disk whole space * ratio is used
|
-| rss.server.multistorage.fallback.strategy.class | - | The
fallback strategy for `MEMORY_LOCALFILE_HDFS`. Support
`org.apache.uniffle.server.storage.RotateStorageManagerFallbackStrategy`,`org.apache.uniffle.server.storage.LocalStorageManagerFallbackStrategy`
and `org.apache.uniffle.server.storage.HadoopStorageManagerFallbackStrategy`.
If not set,
`org.apache.uniffle.server.storage.HadoopStorageManagerFallbackStrategy` will
be used. |
-| rss.server.leak.shuffledata.check.interval | 3600000 | The
interval of leak shuffle data check (ms)
|
-| rss.server.max.concurrency.of.per-partition.write | 30 | The max
concurrency of single partition writer, the data partition file number is equal
to this value. Default value is 1. This config could improve the writing speed,
especially for huge partition.
|
-| rss.server.max.concurrency.limit.of.per-partition.write | - | The limit for
max concurrency per-partition write specified by client, this won't be enabled
by default.
|
-| rss.metrics.reporter.class | - | The
class of metrics reporter.
|
-| rss.server.multistorage.manager.selector.class |
org.apache.uniffle.server.storage.multi.DefaultStorageManagerSelector | The
manager selector strategy for `MEMORY_LOCALFILE_HDFS`. Default value is
`DefaultStorageManagerSelector`, and another
`HugePartitionSensitiveStorageManagerSelector` will flush only huge partition's
data to cold storage.
[...]
+| Property Name | Default
| Description
[...]
+|---------------------------------------------------------|------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
+| rss.coordinator.quorum | -
| Coordinator quorum
[...]
+| rss.rpc.server.port | -
| RPC port for Shuffle
server, if set zero, grpc server start on random port.
[...]
+| rss.jetty.http.port | -
| Http port for Shuffle
server
[...]
+| rss.server.netty.port | -1
| Netty port for Shuffle
server, if set zero, netty server start on random port.
[...]
+| rss.server.netty.epoll.enable | false
| If enable epoll model
with netty server.
[...]
+| rss.server.netty.accept.thread | 10
| Accept thread count in
netty.
[...]
+| rss.server.netty.worker.thread | 100
| Worker thread count in
netty.
[...]
+| rss.server.netty.connect.backlog | 0
| For netty server,
requested maximum length of the queue of incoming connections.
[...]
+| rss.server.netty.connect.timeout | 5000
| Timeout for connection in
netty.
[...]
+| rss.server.netty.receive.buf | 0
| Receive buffer size
(SO_RCVBUF). Note: the optimal size for receive buffer and send buffer should
be latency * network_bandwidth. Assuming latency = 1ms, network_bandwidth =
10Gbps, buffer size should be ~ 1.25MB. Default is 0, the operating system
automatically estimates the receive buffer size based on default settings.
[...]
+| rss.server.netty.send.buf | 0
| Send buffer size
(SO_SNDBUF).
[...]
+| rss.server.buffer.capacity | -1
| Max memory of buffer
manager for shuffle server. If negative, JVM heap size * buffer.ratio is used
[...]
+| rss.server.buffer.capacity.ratio | 0.8
| when
`rss.server.buffer.capacity`=-1, then the buffer capacity is JVM heap size *
ratio
[...]
+| rss.server.memory.shuffle.highWaterMark.percentage | 75.0
| Threshold of spill data
to storage, percentage of rss.server.buffer.capacity
[...]
+| rss.server.memory.shuffle.lowWaterMark.percentage | 25.0
| Threshold of keep data in
memory, percentage of rss.server.buffer.capacity
[...]
+| rss.server.read.buffer.capacity | -1
| Max size of buffer for
reading data. If negative, JVM heap size * read.buffer.ratio is used
[...]
+| rss.server.read.buffer.capacity.ratio | 0.4
| when
`rss.server.read.buffer.capacity`=-1, then read buffer capacity is JVM heap
size * ratio
[...]
+| rss.server.heartbeat.interval | 10000
| Heartbeat interval to
Coordinator (ms)
[...]
+| rss.server.flush.localfile.threadPool.size | 10
| Thread pool for flush
data to local file
[...]
+| rss.server.flush.hadoop.threadPool.size | 60
| Thread pool for flush
data to hadoop storage
[...]
+| rss.server.commit.timeout | 600000
| Timeout when commit
shuffle data (ms)
[...]
+| rss.storage.type | -
| Supports
MEMORY_LOCALFILE, MEMORY_HDFS, MEMORY_LOCALFILE_HDFS
[...]
+| rss.server.flush.cold.storage.threshold.size | 64M
| The threshold of data
size for LOACALFILE and HADOOP if MEMORY_LOCALFILE_HDFS is used
[...]
+| rss.server.tags | -
| The comma-separated list
of tags to indicate the shuffle server's attributes. It will be used as the
assignment basis for the coordinator
[...]
+| rss.server.single.buffer.flush.enabled | true
| Whether single buffer
flush when size exceeded rss.server.single.buffer.flush.threshold
[...]
+| rss.server.single.buffer.flush.threshold | 128M
| The threshold of single
shuffle buffer flush
[...]
+| rss.server.disk.capacity | -1
| Disk capacity that
shuffle server can use. If negative, it will use disk whole space * ratio
[...]
+| rss.server.disk.capacity.ratio | 0.9
| When
`rss.server.disk.capacity` is negative, disk whole space * ratio is used
[...]
+| rss.server.hybrid.storage.fallback.strategy.class | -
| The fallback strategy for
`MEMORY_LOCALFILE_HDFS`. Support
`org.apache.uniffle.server.storage.RotateStorageManagerFallbackStrategy`,`org.apache.uniffle.server.storage.LocalStorageManagerFallbackStrategy`
and `org.apache.uniffle.server.storage.HadoopStorageManagerFallbackStrategy`.
If not set, `org.apache.uniffle.server.storage.HadoopStorageManagerFallbackSt
[...]
+| rss.server.leak.shuffledata.check.interval | 3600000
| The interval of leak
shuffle data check (ms)
[...]
+| rss.server.max.concurrency.of.per-partition.write | 30
| The max concurrency of
single partition writer, the data partition file number is equal to this value.
Default value is 1. This config could improve the writing speed, especially for
huge partition.
[...]
+| rss.server.max.concurrency.limit.of.per-partition.write | -
| The limit for max
concurrency per-partition write specified by client, this won't be enabled by
default.
[...]
+| rss.metrics.reporter.class | -
| The class of metrics
reporter.
[...]
+| rss.server.hyrbrid.storage.manager.selector.class |
org.apache.uniffle.server.storage.hybrid.DefaultStorageManagerSelector | The
manager selector strategy for `MEMORY_LOCALFILE_HDFS`. Default value is
`DefaultStorageManagerSelector`, and another
`HugePartitionSensitiveStorageManagerSelector` will flush only huge partition's
data to cold storage.
[...]
### Advanced Configurations
|Property Name|Default| Description
|
diff --git
a/integration-test/common/src/test/java/org/apache/uniffle/test/MultiStorageFaultToleranceBase.java
b/integration-test/common/src/test/java/org/apache/uniffle/test/HybridStorageFaultToleranceBase.java
similarity index 98%
rename from
integration-test/common/src/test/java/org/apache/uniffle/test/MultiStorageFaultToleranceBase.java
rename to
integration-test/common/src/test/java/org/apache/uniffle/test/HybridStorageFaultToleranceBase.java
index 3b46519d3..bde7222d2 100644
---
a/integration-test/common/src/test/java/org/apache/uniffle/test/MultiStorageFaultToleranceBase.java
+++
b/integration-test/common/src/test/java/org/apache/uniffle/test/HybridStorageFaultToleranceBase.java
@@ -47,7 +47,7 @@ import org.apache.uniffle.storage.util.StorageType;
import static org.junit.jupiter.api.Assertions.assertTrue;
-public abstract class MultiStorageFaultToleranceBase extends
ShuffleReadWriteBase {
+public abstract class HybridStorageFaultToleranceBase extends
ShuffleReadWriteBase {
private ShuffleServerGrpcClient shuffleServerClient;
private static String REMOTE_STORAGE = HDFS_URI + "rss/multi_storage_fault";
diff --git
a/integration-test/common/src/test/java/org/apache/uniffle/test/MultiStorageHadoopFallbackTest.java
b/integration-test/common/src/test/java/org/apache/uniffle/test/HybridStorageHadoopFallbackTest.java
similarity index 96%
rename from
integration-test/common/src/test/java/org/apache/uniffle/test/MultiStorageHadoopFallbackTest.java
rename to
integration-test/common/src/test/java/org/apache/uniffle/test/HybridStorageHadoopFallbackTest.java
index 1ea9d192e..3c19a6550 100644
---
a/integration-test/common/src/test/java/org/apache/uniffle/test/MultiStorageHadoopFallbackTest.java
+++
b/integration-test/common/src/test/java/org/apache/uniffle/test/HybridStorageHadoopFallbackTest.java
@@ -29,7 +29,7 @@ import org.apache.uniffle.storage.util.StorageType;
import static org.junit.jupiter.api.Assertions.assertEquals;
-public class MultiStorageHadoopFallbackTest extends
MultiStorageFaultToleranceBase {
+public class HybridStorageHadoopFallbackTest extends
HybridStorageFaultToleranceBase {
@BeforeAll
public static void setupServers(@TempDir File tmpDir) throws Exception {
diff --git
a/integration-test/common/src/test/java/org/apache/uniffle/test/MultiStorageLocalFileFallbackTest.java
b/integration-test/common/src/test/java/org/apache/uniffle/test/HybridStorageLocalFileFallbackTest.java
similarity index 90%
rename from
integration-test/common/src/test/java/org/apache/uniffle/test/MultiStorageLocalFileFallbackTest.java
rename to
integration-test/common/src/test/java/org/apache/uniffle/test/HybridStorageLocalFileFallbackTest.java
index 5ed2f3bc1..599adf9c9 100644
---
a/integration-test/common/src/test/java/org/apache/uniffle/test/MultiStorageLocalFileFallbackTest.java
+++
b/integration-test/common/src/test/java/org/apache/uniffle/test/HybridStorageLocalFileFallbackTest.java
@@ -25,14 +25,14 @@ import org.junit.jupiter.api.io.TempDir;
import org.apache.uniffle.coordinator.CoordinatorConf;
import org.apache.uniffle.server.ShuffleServerConf;
+import org.apache.uniffle.server.storage.HybridStorageManager;
import org.apache.uniffle.server.storage.LocalStorageManager;
import org.apache.uniffle.server.storage.LocalStorageManagerFallbackStrategy;
-import org.apache.uniffle.server.storage.MultiStorageManager;
import org.apache.uniffle.storage.common.LocalStorage;
import org.apache.uniffle.storage.common.Storage;
import org.apache.uniffle.storage.util.StorageType;
-public class MultiStorageLocalFileFallbackTest extends
MultiStorageFaultToleranceBase {
+public class HybridStorageLocalFileFallbackTest extends
HybridStorageFaultToleranceBase {
@BeforeAll
public static void setupServers(@TempDir File tmpDir) throws Exception {
@@ -52,7 +52,7 @@ public class MultiStorageLocalFileFallbackTest extends
MultiStorageFaultToleranc
shuffleServerConf.setLong(
ShuffleServerConf.FLUSH_COLD_STORAGE_THRESHOLD_SIZE, 1000L * 1024L *
1024L);
shuffleServerConf.setString(
- ShuffleServerConf.MULTISTORAGE_FALLBACK_STRATEGY_CLASS,
+ ShuffleServerConf.HYBRID_STORAGE_FALLBACK_STRATEGY_CLASS,
LocalStorageManagerFallbackStrategy.class.getCanonicalName());
createAndStartServers(shuffleServerConf, coordinatorConf);
}
@@ -61,7 +61,7 @@ public class MultiStorageLocalFileFallbackTest extends
MultiStorageFaultToleranc
public void makeChaos() {
LocalStorageManager warmStorageManager =
(LocalStorageManager)
- ((MultiStorageManager) shuffleServers.get(0).getStorageManager())
+ ((HybridStorageManager) shuffleServers.get(0).getStorageManager())
.getWarmStorageManager();
for (Storage storage : warmStorageManager.getStorages()) {
LocalStorage localStorage = (LocalStorage) storage;
diff --git
a/integration-test/common/src/test/java/org/apache/uniffle/test/ShuffleServerGrpcTest.java
b/integration-test/common/src/test/java/org/apache/uniffle/test/ShuffleServerGrpcTest.java
index 4fc8a0ea2..ecc8ab999 100644
---
a/integration-test/common/src/test/java/org/apache/uniffle/test/ShuffleServerGrpcTest.java
+++
b/integration-test/common/src/test/java/org/apache/uniffle/test/ShuffleServerGrpcTest.java
@@ -66,7 +66,7 @@ import org.apache.uniffle.server.ShuffleDataFlushEvent;
import org.apache.uniffle.server.ShuffleServerConf;
import org.apache.uniffle.server.ShuffleServerGrpcMetrics;
import org.apache.uniffle.server.ShuffleServerMetrics;
-import org.apache.uniffle.server.storage.MultiStorageManager;
+import org.apache.uniffle.server.storage.HybridStorageManager;
import org.apache.uniffle.storage.util.StorageType;
import static org.junit.jupiter.api.Assertions.assertEquals;
@@ -397,7 +397,7 @@ public class ShuffleServerGrpcTest extends
IntegrationTestBase {
new ShuffleDataFlushEvent(1, appId2, 1, 1, 1, EVENT_THRESHOLD_SIZE +
1, null, null, null);
try {
// can't find storage info with appId2
- ((MultiStorageManager) shuffleServers.get(0).getStorageManager())
+ ((HybridStorageManager) shuffleServers.get(0).getStorageManager())
.getColdStorageManager()
.selectStorage(event2)
.getStoragePath();
diff --git
a/integration-test/spark-common/src/test/java/org/apache/uniffle/test/RepartitionWithHadoopMultiStorageRssTest.java
b/integration-test/spark-common/src/test/java/org/apache/uniffle/test/RepartitionWithHadoopHybridStorageRssTest.java
similarity index 94%
rename from
integration-test/spark-common/src/test/java/org/apache/uniffle/test/RepartitionWithHadoopMultiStorageRssTest.java
rename to
integration-test/spark-common/src/test/java/org/apache/uniffle/test/RepartitionWithHadoopHybridStorageRssTest.java
index da68543af..89a1a56d7 100644
---
a/integration-test/spark-common/src/test/java/org/apache/uniffle/test/RepartitionWithHadoopMultiStorageRssTest.java
+++
b/integration-test/spark-common/src/test/java/org/apache/uniffle/test/RepartitionWithHadoopHybridStorageRssTest.java
@@ -34,10 +34,10 @@ import org.apache.uniffle.coordinator.CoordinatorConf;
import org.apache.uniffle.server.ShuffleServerConf;
import org.apache.uniffle.storage.util.StorageType;
-public class RepartitionWithHadoopMultiStorageRssTest extends RepartitionTest {
+public class RepartitionWithHadoopHybridStorageRssTest extends RepartitionTest
{
private static final Logger LOG =
- LoggerFactory.getLogger(RepartitionWithHadoopMultiStorageRssTest.class);
+ LoggerFactory.getLogger(RepartitionWithHadoopHybridStorageRssTest.class);
@BeforeAll
public static void setupServers(@TempDir File tmpDir) throws Exception {
diff --git
a/integration-test/spark-common/src/test/java/org/apache/uniffle/test/RepartitionWithMemoryMultiStorageRssTest.java
b/integration-test/spark-common/src/test/java/org/apache/uniffle/test/RepartitionWithMemoryHybridStorageRssTest.java
similarity index 96%
rename from
integration-test/spark-common/src/test/java/org/apache/uniffle/test/RepartitionWithMemoryMultiStorageRssTest.java
rename to
integration-test/spark-common/src/test/java/org/apache/uniffle/test/RepartitionWithMemoryHybridStorageRssTest.java
index aba0b410e..32b19378a 100644
---
a/integration-test/spark-common/src/test/java/org/apache/uniffle/test/RepartitionWithMemoryMultiStorageRssTest.java
+++
b/integration-test/spark-common/src/test/java/org/apache/uniffle/test/RepartitionWithMemoryHybridStorageRssTest.java
@@ -31,7 +31,7 @@ import org.apache.uniffle.coordinator.CoordinatorConf;
import org.apache.uniffle.server.ShuffleServerConf;
import org.apache.uniffle.storage.util.StorageType;
-public class RepartitionWithMemoryMultiStorageRssTest extends RepartitionTest {
+public class RepartitionWithMemoryHybridStorageRssTest extends RepartitionTest
{
@BeforeAll
public static void setupServers(@TempDir File tmpDir) throws Exception {
CoordinatorConf coordinatorConf = getCoordinatorConf();
diff --git
a/server/src/main/java/org/apache/uniffle/server/ShuffleServerConf.java
b/server/src/main/java/org/apache/uniffle/server/ShuffleServerConf.java
index 095946b8a..60978ea9a 100644
--- a/server/src/main/java/org/apache/uniffle/server/ShuffleServerConf.java
+++ b/server/src/main/java/org/apache/uniffle/server/ShuffleServerConf.java
@@ -298,19 +298,21 @@ public class ShuffleServerConf extends RssBaseConf {
.withDescription(
"For multistorage, the event size exceed this value, flush data
to cold storage");
- public static final ConfigOption<String> MULTISTORAGE_MANAGER_SELECTOR_CLASS
=
- ConfigOptions.key("rss.server.multistorage.manager.selector.class")
+ public static final ConfigOption<String>
HYBRID_STORAGE_MANAGER_SELECTOR_CLASS =
+ ConfigOptions.key("rss.server.hybrid.storage.manager.selector.class")
.stringType()
-
.defaultValue("org.apache.uniffle.server.storage.multi.DefaultStorageManagerSelector")
+
.defaultValue("org.apache.uniffle.server.storage.hybrid.DefaultStorageManagerSelector")
.withDescription(
"For multistorage, the storage manager selector strategy to
support "
- + "policies of flushing to different storages");
+ + "policies of flushing to different storages")
+
.withDeprecatedKeys("rss.server.multistorage.manager.selector.class");
- public static final ConfigOption<String>
MULTISTORAGE_FALLBACK_STRATEGY_CLASS =
- ConfigOptions.key("rss.server.multistorage.fallback.strategy.class")
+ public static final ConfigOption<String>
HYBRID_STORAGE_FALLBACK_STRATEGY_CLASS =
+ ConfigOptions.key("rss.server.hybrid.storage.fallback.strategy.class")
.stringType()
.noDefaultValue()
- .withDescription("For multistorage, fallback strategy class");
+ .withDescription("For multistorage, fallback strategy class")
+
.withDeprecatedKeys("rss.server.multistorage.fallback.strategy.class");
public static final ConfigOption<Long> FALLBACK_MAX_FAIL_TIMES =
ConfigOptions.key("rss.server.multistorage.fallback.max.fail.times")
diff --git
a/server/src/main/java/org/apache/uniffle/server/storage/MultiStorageManager.java
b/server/src/main/java/org/apache/uniffle/server/storage/HybridStorageManager.java
similarity index 93%
rename from
server/src/main/java/org/apache/uniffle/server/storage/MultiStorageManager.java
rename to
server/src/main/java/org/apache/uniffle/server/storage/HybridStorageManager.java
index 29520c2b1..ae83315f0 100644
---
a/server/src/main/java/org/apache/uniffle/server/storage/MultiStorageManager.java
+++
b/server/src/main/java/org/apache/uniffle/server/storage/HybridStorageManager.java
@@ -35,20 +35,20 @@ import org.apache.uniffle.server.ShuffleDataFlushEvent;
import org.apache.uniffle.server.ShuffleDataReadEvent;
import org.apache.uniffle.server.ShuffleServerConf;
import org.apache.uniffle.server.event.PurgeEvent;
-import org.apache.uniffle.server.storage.multi.StorageManagerSelector;
+import org.apache.uniffle.server.storage.hybrid.StorageManagerSelector;
import org.apache.uniffle.storage.common.Storage;
import org.apache.uniffle.storage.handler.api.ShuffleWriteHandler;
-public class MultiStorageManager implements StorageManager {
+public class HybridStorageManager implements StorageManager {
- private static final Logger LOG =
LoggerFactory.getLogger(MultiStorageManager.class);
+ private static final Logger LOG =
LoggerFactory.getLogger(HybridStorageManager.class);
private final StorageManager warmStorageManager;
private final StorageManager coldStorageManager;
private final Cache<ShuffleDataFlushEvent, StorageManager>
eventOfUnderStorageManagers;
private final StorageManagerSelector storageManagerSelector;
- MultiStorageManager(ShuffleServerConf conf) {
+ HybridStorageManager(ShuffleServerConf conf) {
warmStorageManager = new LocalStorageManager(conf);
coldStorageManager = new HadoopStorageManager(conf);
@@ -73,7 +73,7 @@ public class MultiStorageManager implements StorageManager {
StorageManager warmStorageManager,
StorageManager coldStorageManager)
throws Exception {
- String name =
conf.get(ShuffleServerConf.MULTISTORAGE_MANAGER_SELECTOR_CLASS);
+ String name =
conf.get(ShuffleServerConf.HYBRID_STORAGE_MANAGER_SELECTOR_CLASS);
Class<?> klass = Class.forName(name);
Constructor<?> constructor =
klass.getConstructor(
@@ -92,7 +92,7 @@ public class MultiStorageManager implements StorageManager {
throws Exception {
String name =
conf.getString(
- ShuffleServerConf.MULTISTORAGE_FALLBACK_STRATEGY_CLASS,
+ ShuffleServerConf.HYBRID_STORAGE_FALLBACK_STRATEGY_CLASS,
HadoopStorageManagerFallbackStrategy.class.getCanonicalName());
Class<?> klass = Class.forName(name);
Constructor<?> constructor;
diff --git
a/server/src/main/java/org/apache/uniffle/server/storage/StorageManagerFactory.java
b/server/src/main/java/org/apache/uniffle/server/storage/StorageManagerFactory.java
index d3800932d..f21d6f6fe 100644
---
a/server/src/main/java/org/apache/uniffle/server/storage/StorageManagerFactory.java
+++
b/server/src/main/java/org/apache/uniffle/server/storage/StorageManagerFactory.java
@@ -38,7 +38,7 @@ public class StorageManagerFactory {
return new HadoopStorageManager(conf);
} else if (StorageType.LOCALFILE_HDFS.equals(type)
|| StorageType.MEMORY_LOCALFILE_HDFS.equals(type)) {
- return new MultiStorageManager(conf);
+ return new HybridStorageManager(conf);
} else {
throw new IllegalArgumentException("unknown storageType was found");
}
diff --git
a/server/src/main/java/org/apache/uniffle/server/storage/multi/DefaultStorageManagerSelector.java
b/server/src/main/java/org/apache/uniffle/server/storage/hybrid/DefaultStorageManagerSelector.java
similarity index 97%
rename from
server/src/main/java/org/apache/uniffle/server/storage/multi/DefaultStorageManagerSelector.java
rename to
server/src/main/java/org/apache/uniffle/server/storage/hybrid/DefaultStorageManagerSelector.java
index 50f498237..e9b73d878 100644
---
a/server/src/main/java/org/apache/uniffle/server/storage/multi/DefaultStorageManagerSelector.java
+++
b/server/src/main/java/org/apache/uniffle/server/storage/hybrid/DefaultStorageManagerSelector.java
@@ -15,7 +15,7 @@
* limitations under the License.
*/
-package org.apache.uniffle.server.storage.multi;
+package org.apache.uniffle.server.storage.hybrid;
import org.apache.uniffle.server.ShuffleDataFlushEvent;
import org.apache.uniffle.server.ShuffleServerConf;
diff --git
a/server/src/main/java/org/apache/uniffle/server/storage/multi/FallbackBasedStorageManagerSelector.java
b/server/src/main/java/org/apache/uniffle/server/storage/hybrid/FallbackBasedStorageManagerSelector.java
similarity index 97%
rename from
server/src/main/java/org/apache/uniffle/server/storage/multi/FallbackBasedStorageManagerSelector.java
rename to
server/src/main/java/org/apache/uniffle/server/storage/hybrid/FallbackBasedStorageManagerSelector.java
index 3cddf6787..c1851a751 100644
---
a/server/src/main/java/org/apache/uniffle/server/storage/multi/FallbackBasedStorageManagerSelector.java
+++
b/server/src/main/java/org/apache/uniffle/server/storage/hybrid/FallbackBasedStorageManagerSelector.java
@@ -15,7 +15,7 @@
* limitations under the License.
*/
-package org.apache.uniffle.server.storage.multi;
+package org.apache.uniffle.server.storage.hybrid;
import org.apache.uniffle.server.ShuffleDataFlushEvent;
import
org.apache.uniffle.server.storage.AbstractStorageManagerFallbackStrategy;
diff --git
a/server/src/main/java/org/apache/uniffle/server/storage/multi/HugePartitionSensitiveStorageManagerSelector.java
b/server/src/main/java/org/apache/uniffle/server/storage/hybrid/HugePartitionSensitiveStorageManagerSelector.java
similarity index 97%
rename from
server/src/main/java/org/apache/uniffle/server/storage/multi/HugePartitionSensitiveStorageManagerSelector.java
rename to
server/src/main/java/org/apache/uniffle/server/storage/hybrid/HugePartitionSensitiveStorageManagerSelector.java
index 3e86b8383..062ebd96e 100644
---
a/server/src/main/java/org/apache/uniffle/server/storage/multi/HugePartitionSensitiveStorageManagerSelector.java
+++
b/server/src/main/java/org/apache/uniffle/server/storage/hybrid/HugePartitionSensitiveStorageManagerSelector.java
@@ -15,7 +15,7 @@
* limitations under the License.
*/
-package org.apache.uniffle.server.storage.multi;
+package org.apache.uniffle.server.storage.hybrid;
import org.apache.uniffle.server.ShuffleDataFlushEvent;
import org.apache.uniffle.server.ShuffleServerConf;
diff --git
a/server/src/main/java/org/apache/uniffle/server/storage/multi/StorageManagerSelector.java
b/server/src/main/java/org/apache/uniffle/server/storage/hybrid/StorageManagerSelector.java
similarity index 95%
rename from
server/src/main/java/org/apache/uniffle/server/storage/multi/StorageManagerSelector.java
rename to
server/src/main/java/org/apache/uniffle/server/storage/hybrid/StorageManagerSelector.java
index 55cbd509b..92490b802 100644
---
a/server/src/main/java/org/apache/uniffle/server/storage/multi/StorageManagerSelector.java
+++
b/server/src/main/java/org/apache/uniffle/server/storage/hybrid/StorageManagerSelector.java
@@ -15,7 +15,7 @@
* limitations under the License.
*/
-package org.apache.uniffle.server.storage.multi;
+package org.apache.uniffle.server.storage.hybrid;
import org.apache.uniffle.server.ShuffleDataFlushEvent;
import org.apache.uniffle.server.storage.StorageManager;
diff --git
a/server/src/test/java/org/apache/uniffle/server/ShuffleFlushManagerTest.java
b/server/src/test/java/org/apache/uniffle/server/ShuffleFlushManagerTest.java
index b91e669e2..82974e305 100644
---
a/server/src/test/java/org/apache/uniffle/server/ShuffleFlushManagerTest.java
+++
b/server/src/test/java/org/apache/uniffle/server/ShuffleFlushManagerTest.java
@@ -56,9 +56,9 @@ import org.apache.uniffle.common.util.Constants;
import org.apache.uniffle.server.buffer.ShuffleBufferManager;
import org.apache.uniffle.server.event.AppPurgeEvent;
import org.apache.uniffle.server.storage.HadoopStorageManager;
+import org.apache.uniffle.server.storage.HybridStorageManager;
import org.apache.uniffle.server.storage.LocalStorageManager;
import org.apache.uniffle.server.storage.LocalStorageManagerFallbackStrategy;
-import org.apache.uniffle.server.storage.MultiStorageManager;
import org.apache.uniffle.server.storage.StorageManager;
import org.apache.uniffle.server.storage.StorageManagerFactory;
import org.apache.uniffle.storage.HadoopTestBase;
@@ -646,7 +646,7 @@ public class ShuffleFlushManagerTest extends HadoopTestBase
{
}
@Test
- public void fallbackWrittenWhenMultiStorageManagerEnableTest(@TempDir File
tempDir)
+ public void fallbackWrittenWhenHybridStorageManagerEnableTest(@TempDir File
tempDir)
throws InterruptedException {
shuffleServerConf.setLong(ShuffleServerConf.FLUSH_COLD_STORAGE_THRESHOLD_SIZE,
10000L);
shuffleServerConf.setString(
@@ -655,7 +655,7 @@ public class ShuffleFlushManagerTest extends HadoopTestBase
{
RssBaseConf.RSS_STORAGE_BASE_PATH,
Arrays.asList(tempDir.getAbsolutePath()));
shuffleServerConf.set(ShuffleServerConf.DISK_CAPACITY, 100L);
shuffleServerConf.setString(
- ShuffleServerConf.MULTISTORAGE_FALLBACK_STRATEGY_CLASS,
+ ShuffleServerConf.HYBRID_STORAGE_FALLBACK_STRATEGY_CLASS,
LocalStorageManagerFallbackStrategy.class.getCanonicalName());
StorageManager storageManager =
@@ -685,8 +685,8 @@ public class ShuffleFlushManagerTest extends HadoopTestBase
{
ShuffleDataFlushEvent bigEvent =
new ShuffleDataFlushEvent(1, "1", 1, 1, 1, 100, blocks, null, null);
bigEvent.setUnderStorage(
- ((MultiStorageManager)
storageManager).getWarmStorageManager().selectStorage(event));
- ((MultiStorageManager)
storageManager).getWarmStorageManager().updateWriteMetrics(bigEvent, 0);
+ ((HybridStorageManager)
storageManager).getWarmStorageManager().selectStorage(event));
+ ((HybridStorageManager)
storageManager).getWarmStorageManager().updateWriteMetrics(bigEvent, 0);
event = createShuffleDataFlushEvent(appId, 1, 1, 1, null, 100);
flushManager.addToFlushQueue(event);
@@ -706,7 +706,7 @@ public class ShuffleFlushManagerTest extends HadoopTestBase
{
shuffleServerConf.set(ShuffleServerConf.SERVER_FLUSH_HADOOP_THREAD_POOL_SIZE,
1);
shuffleServerConf.set(ShuffleServerConf.SERVER_FLUSH_LOCALFILE_THREAD_POOL_SIZE,
1);
shuffleServerConf.setString(
- ShuffleServerConf.MULTISTORAGE_FALLBACK_STRATEGY_CLASS,
+ ShuffleServerConf.HYBRID_STORAGE_FALLBACK_STRATEGY_CLASS,
LocalStorageManagerFallbackStrategy.class.getCanonicalName());
StorageManager storageManager =
@@ -739,8 +739,8 @@ public class ShuffleFlushManagerTest extends HadoopTestBase
{
ShuffleDataFlushEvent bigEvent =
new ShuffleDataFlushEvent(1, "1", 1, 1, 1, 100, blocks, null, null);
bigEvent.setUnderStorage(
- ((MultiStorageManager)
storageManager).getWarmStorageManager().selectStorage(event));
- ((MultiStorageManager)
storageManager).getWarmStorageManager().updateWriteMetrics(bigEvent, 0);
+ ((HybridStorageManager)
storageManager).getWarmStorageManager().selectStorage(event));
+ ((HybridStorageManager)
storageManager).getWarmStorageManager().updateWriteMetrics(bigEvent, 0);
event = createShuffleDataFlushEvent(appId, 1, 1, 1, null, 100);
flushManager.addToFlushQueue(event);
diff --git
a/server/src/test/java/org/apache/uniffle/server/storage/MultiStorageManagerTest.java
b/server/src/test/java/org/apache/uniffle/server/storage/HybridStorageManagerTest.java
similarity index 89%
rename from
server/src/test/java/org/apache/uniffle/server/storage/MultiStorageManagerTest.java
rename to
server/src/test/java/org/apache/uniffle/server/storage/HybridStorageManagerTest.java
index 089e3f613..d3e12a284 100644
---
a/server/src/test/java/org/apache/uniffle/server/storage/MultiStorageManagerTest.java
+++
b/server/src/test/java/org/apache/uniffle/server/storage/HybridStorageManagerTest.java
@@ -34,7 +34,7 @@ import org.apache.uniffle.storage.util.StorageType;
import static org.junit.jupiter.api.Assertions.assertTrue;
-public class MultiStorageManagerTest {
+public class HybridStorageManagerTest {
@Test
public void selectStorageManagerTest() {
@@ -44,7 +44,7 @@ public class MultiStorageManagerTest {
conf.setLong(ShuffleServerConf.DISK_CAPACITY, 1024L * 1024L * 1024L);
conf.setString(
ShuffleServerConf.RSS_STORAGE_TYPE.key(),
StorageType.MEMORY_LOCALFILE_HDFS.name());
- MultiStorageManager manager = new MultiStorageManager(conf);
+ HybridStorageManager manager = new HybridStorageManager(conf);
String remoteStorage = "test";
String appId = "selectStorageManagerTest_appId";
manager.registerRemoteStorage(appId, new RemoteStorageInfo(remoteStorage));
@@ -66,19 +66,19 @@ public class MultiStorageManagerTest {
conf.setString(
ShuffleServerConf.RSS_STORAGE_TYPE.key(),
StorageType.MEMORY_LOCALFILE_HDFS.name());
conf.setString(
- ShuffleServerConf.MULTISTORAGE_FALLBACK_STRATEGY_CLASS,
+ ShuffleServerConf.HYBRID_STORAGE_FALLBACK_STRATEGY_CLASS,
RotateStorageManagerFallbackStrategy.class.getCanonicalName());
conf.set(
- ShuffleServerConf.MULTISTORAGE_MANAGER_SELECTOR_CLASS,
-
"org.apache.uniffle.server.storage.multi.HugePartitionSensitiveStorageManagerSelector");
- MultiStorageManager manager = new MultiStorageManager(conf);
+ ShuffleServerConf.HYBRID_STORAGE_MANAGER_SELECTOR_CLASS,
+
"org.apache.uniffle.server.storage.hybrid.HugePartitionSensitiveStorageManagerSelector");
+ HybridStorageManager manager = new HybridStorageManager(conf);
String remoteStorage = "test";
String appId = "selectStorageManagerIfCanNotWriteTest_appId";
manager.registerRemoteStorage(appId, new RemoteStorageInfo(remoteStorage));
/**
* case1: only event owned by huge partition will be flushed to cold
storage when the {@link
- *
org.apache.uniffle.server.storage.multi.StorageManagerSelector.ColdStoragePreferredFactor.HUGE_PARTITION}
+ *
org.apache.uniffle.server.storage.hybrid.StorageManagerSelector.ColdStoragePreferredFactor.HUGE_PARTITION}
* is enabled.
*/
List<ShufflePartitionedBlock> blocks =
@@ -104,9 +104,9 @@ public class MultiStorageManagerTest {
conf.setString(
ShuffleServerConf.RSS_STORAGE_TYPE.key(),
StorageType.MEMORY_LOCALFILE_HDFS.name());
conf.setString(
- ShuffleServerConf.MULTISTORAGE_FALLBACK_STRATEGY_CLASS,
+ ShuffleServerConf.HYBRID_STORAGE_FALLBACK_STRATEGY_CLASS,
RotateStorageManagerFallbackStrategy.class.getCanonicalName());
- MultiStorageManager manager = new MultiStorageManager(conf);
+ HybridStorageManager manager = new HybridStorageManager(conf);
String remoteStorage = "test";
String appId = "selectStorageManagerIfCanNotWriteTest_appId";
manager.registerRemoteStorage(appId, new RemoteStorageInfo(remoteStorage));
