This is an automated email from the ASF dual-hosted git repository.
suneet pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git
The following commit(s) were added to refs/heads/master by this push:
new c410648 fix injection failure of StorageLocationSelectorStrategy
objects (#10363)
c410648 is described below
commit c4106486306df6fadf92994f7d33a88c6d952551
Author: frank chen <[email protected]>
AuthorDate: Wed Dec 9 01:48:31 2020 +0800
fix injection failure of StorageLocationSelectorStrategy objects (#10363)
* fix to allow customer storage location selector strategy
* add test cases to check instance of selector strategy
* update doc
* code format
* resolve code review comments
* inject StorageLocation
* fix CI
* fix mismatched license item reported by CI
* change property path from
druid.segmentCache.locationSelectorStrategy.type to
druid.segmentCache.locationSelector.strategy
* using a helper method to bind to correct property path
---
docs/configuration/index.md | 4 +-
licenses.yaml | 11 +-
server/pom.xml | 4 +
.../org/apache/druid/guice/StorageNodeModule.java | 23 ++++
...stBytesUsedStorageLocationSelectorStrategy.java | 7 +-
...ailableSizeStorageLocationSelectorStrategy.java | 7 +-
.../RandomStorageLocationSelectorStrategy.java | 7 +-
.../RoundRobinStorageLocationSelectorStrategy.java | 11 +-
.../druid/segment/loading/SegmentLoaderConfig.java | 31 +++---
.../loading/SegmentLoaderLocalCacheManager.java | 48 ++++++---
.../loading/StorageLocationSelectorStrategy.java | 2 +-
.../SegmentLoaderLocalCacheManagerTest.java | 22 ++--
.../StorageLocationSelectorStrategyTest.java | 118 +++++++++++++++++++++
13 files changed, 226 insertions(+), 69 deletions(-)
diff --git a/docs/configuration/index.md b/docs/configuration/index.md
index 0755520..640ab0c 100644
--- a/docs/configuration/index.md
+++ b/docs/configuration/index.md
@@ -1409,7 +1409,7 @@ These Historical configurations can be defined in the
`historical/runtime.proper
|Property|Description|Default|
|--------|-----------|-------|
|`druid.segmentCache.locations`|Segments assigned to a Historical process are
first stored on the local file system (in a disk cache) and then served by the
Historical process. These locations define where that local cache resides. This
value cannot be NULL or EMPTY. Here is an example
`druid.segmentCache.locations=[{"path": "/mnt/druidSegments", "maxSize": "10k",
"freeSpacePercent": 1.0}]`. "freeSpacePercent" is optional, if provided then
enforces that much of free disk partition space [...]
-|`druid.segmentCache.locationSelectorStrategy`|The strategy used to select a
location from the configured `druid.segmentCache.locations` for segment
distribution. Possible values are `leastBytesUsed`, `roundRobin`, `random`, or
`mostAvailableSize`. |leastBytesUsed|
+|`druid.segmentCache.locationSelector.strategy`|The strategy used to select a
location from the configured `druid.segmentCache.locations` for segment
distribution. Possible values are `leastBytesUsed`, `roundRobin`, `random`, or
`mostAvailableSize`. |leastBytesUsed|
|`druid.segmentCache.deleteOnRemove`|Delete segment files from cache once a
process is no longer serving a segment.|true|
|`druid.segmentCache.dropSegmentDelayMillis`|How long a process delays before
completely dropping segment.|30000 (30 seconds)|
|`druid.segmentCache.infoDir`|Historical processes keep track of the segments
they are serving so that when the process is restarted they can reload the same
segments without waiting for the Coordinator to reassign. This path defines
where this metadata is kept. Directory will be created if
needed.|${first_location}/info_dir|
@@ -1421,7 +1421,7 @@ These Historical configurations can be defined in the
`historical/runtime.proper
In `druid.segmentCache.locations`, *freeSpacePercent* was added because
*maxSize* setting is only a theoretical limit and assumes that much space will
always be available for storing segments. In case of any druid bug leading to
unaccounted segment files left alone on disk or some other process writing
stuff to disk, This check can start failing segment loading early before
filling up the disk completely and leaving the host usable otherwise.
-In `druid.segmentCache.locationSelectorStrategy`, one of leastBytesUsed,
roundRobin, random, or mostAvailableSize could be specified to represent the
strategy to distribute segments across multiple segment cache locations.
+In `druid.segmentCache.locationSelector.strategy`, one of `leastBytesUsed`,
`roundRobin`, `random`, or `mostAvailableSize` could be specified to represent
the strategy to distribute segments across multiple segment cache locations.
|Strategy|Description|
|--------|-----------|
diff --git a/licenses.yaml b/licenses.yaml
index 90a7ffd..31c3820 100644
--- a/licenses.yaml
+++ b/licenses.yaml
@@ -231,6 +231,7 @@ libraries:
- com.fasterxml.jackson.jaxrs: jackson-jaxrs-json-provider
- com.fasterxml.jackson.jaxrs: jackson-jaxrs-smile-provider
- com.fasterxml.jackson.module: jackson-module-jaxb-annotations
+ - com.fasterxml.jackson.module: jackson-module-guice
notice: |
# Jackson JSON processor
@@ -4126,16 +4127,6 @@ libraries:
---
-name: "Jackson Module: Guice"
-license_category: binary
-module: java-core
-license_name: Apache License version 2.0
-version: 2.6.7
-libraries:
- - com.fasterxml.jackson.module: jackson-module-guice
-
----
-
name: Google APIs Client Library For Java
license_category: binary
module: java-core
diff --git a/server/pom.xml b/server/pom.xml
index caddf48..01b0d03 100644
--- a/server/pom.xml
+++ b/server/pom.xml
@@ -315,6 +315,10 @@
<groupId>io.timeandspace</groupId>
<artifactId>cron-scheduler</artifactId>
</dependency>
+ <dependency>
+ <groupId>com.fasterxml.jackson.module</groupId>
+ <artifactId>jackson-module-guice</artifactId>
+ </dependency>
<!-- Tests -->
<dependency>
diff --git a/server/src/main/java/org/apache/druid/guice/StorageNodeModule.java
b/server/src/main/java/org/apache/druid/guice/StorageNodeModule.java
index d3750a7..a232f6d 100644
--- a/server/src/main/java/org/apache/druid/guice/StorageNodeModule.java
+++ b/server/src/main/java/org/apache/druid/guice/StorageNodeModule.java
@@ -33,11 +33,14 @@ import org.apache.druid.java.util.emitter.EmittingLogger;
import org.apache.druid.query.DruidProcessingConfig;
import org.apache.druid.segment.column.ColumnConfig;
import org.apache.druid.segment.loading.SegmentLoaderConfig;
+import org.apache.druid.segment.loading.StorageLocation;
+import org.apache.druid.segment.loading.StorageLocationSelectorStrategy;
import org.apache.druid.server.DruidNode;
import org.apache.druid.server.coordination.DruidServerMetadata;
import org.apache.druid.server.coordination.ServerType;
import javax.annotation.Nullable;
+import java.util.List;
/**
*/
@@ -52,6 +55,7 @@ public class StorageNodeModule implements Module
{
JsonConfigProvider.bind(binder, "druid.server", DruidServerConfig.class);
JsonConfigProvider.bind(binder, "druid.segmentCache",
SegmentLoaderConfig.class);
+ bindLocationSelectorStrategy(binder);
binder.bind(ServerTypeConfig.class).toProvider(Providers.of(null));
binder.bind(ColumnConfig.class).to(DruidProcessingConfig.class);
@@ -117,4 +121,23 @@ public class StorageNodeModule implements Module
{
return !segmentLoaderConfig.getLocations().isEmpty();
}
+
+ /**
+ * provide a list of StorageLocation
+ * so that it can be injected into objects such as implementations of {@link
StorageLocationSelectorStrategy}
+ */
+ @Provides
+ @LazySingleton
+ public List<StorageLocation> provideStorageLocation(SegmentLoaderConfig
config)
+ {
+ return config.toStorageLocations();
+ }
+
+ /**
+ * a helper method for both storage module and independent unit test cases
+ */
+ public static void bindLocationSelectorStrategy(Binder binder)
+ {
+ JsonConfigProvider.bind(binder, "druid.segmentCache.locationSelector",
StorageLocationSelectorStrategy.class);
+ }
}
diff --git
a/server/src/main/java/org/apache/druid/segment/loading/LeastBytesUsedStorageLocationSelectorStrategy.java
b/server/src/main/java/org/apache/druid/segment/loading/LeastBytesUsedStorageLocationSelectorStrategy.java
index e1e9418..bcc37e3 100644
---
a/server/src/main/java/org/apache/druid/segment/loading/LeastBytesUsedStorageLocationSelectorStrategy.java
+++
b/server/src/main/java/org/apache/druid/segment/loading/LeastBytesUsedStorageLocationSelectorStrategy.java
@@ -19,6 +19,8 @@
package org.apache.druid.segment.loading;
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
import com.google.common.collect.Ordering;
import java.util.Comparator;
@@ -34,9 +36,10 @@ public class LeastBytesUsedStorageLocationSelectorStrategy
implements StorageLoc
private static final Ordering<StorageLocation> ORDERING =
Ordering.from(Comparator
.comparingLong(StorageLocation::currSizeBytes));
- private List<StorageLocation> storageLocations;
+ private final List<StorageLocation> storageLocations;
- public LeastBytesUsedStorageLocationSelectorStrategy(List<StorageLocation>
storageLocations)
+ @JsonCreator
+ public LeastBytesUsedStorageLocationSelectorStrategy(@JacksonInject final
List<StorageLocation> storageLocations)
{
this.storageLocations = storageLocations;
}
diff --git
a/server/src/main/java/org/apache/druid/segment/loading/MostAvailableSizeStorageLocationSelectorStrategy.java
b/server/src/main/java/org/apache/druid/segment/loading/MostAvailableSizeStorageLocationSelectorStrategy.java
index 4790f90..4e4a84c 100644
---
a/server/src/main/java/org/apache/druid/segment/loading/MostAvailableSizeStorageLocationSelectorStrategy.java
+++
b/server/src/main/java/org/apache/druid/segment/loading/MostAvailableSizeStorageLocationSelectorStrategy.java
@@ -19,6 +19,8 @@
package org.apache.druid.segment.loading;
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
import com.google.common.collect.Ordering;
import java.util.Comparator;
@@ -35,9 +37,10 @@ public class
MostAvailableSizeStorageLocationSelectorStrategy implements Storage
.comparingLong(StorageLocation::availableSizeBytes)
.reversed());
- private List<StorageLocation> storageLocations;
+ private final List<StorageLocation> storageLocations;
- public
MostAvailableSizeStorageLocationSelectorStrategy(List<StorageLocation>
storageLocations)
+ @JsonCreator
+ public MostAvailableSizeStorageLocationSelectorStrategy(@JacksonInject final
List<StorageLocation> storageLocations)
{
this.storageLocations = storageLocations;
}
diff --git
a/server/src/main/java/org/apache/druid/segment/loading/RandomStorageLocationSelectorStrategy.java
b/server/src/main/java/org/apache/druid/segment/loading/RandomStorageLocationSelectorStrategy.java
index 85df860..ed0114a 100644
---
a/server/src/main/java/org/apache/druid/segment/loading/RandomStorageLocationSelectorStrategy.java
+++
b/server/src/main/java/org/apache/druid/segment/loading/RandomStorageLocationSelectorStrategy.java
@@ -19,6 +19,9 @@
package org.apache.druid.segment.loading;
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+
import java.util.ArrayList;
import java.util.Collections;
import java.util.Iterator;
@@ -33,7 +36,8 @@ public class RandomStorageLocationSelectorStrategy implements
StorageLocationSel
private final List<StorageLocation> storageLocations;
- public RandomStorageLocationSelectorStrategy(List<StorageLocation>
storageLocations)
+ @JsonCreator
+ public RandomStorageLocationSelectorStrategy(@JacksonInject final
List<StorageLocation> storageLocations)
{
this.storageLocations = storageLocations;
}
@@ -45,5 +49,4 @@ public class RandomStorageLocationSelectorStrategy implements
StorageLocationSel
Collections.shuffle(copyLocation);
return copyLocation.iterator();
}
-
}
diff --git
a/server/src/main/java/org/apache/druid/segment/loading/RoundRobinStorageLocationSelectorStrategy.java
b/server/src/main/java/org/apache/druid/segment/loading/RoundRobinStorageLocationSelectorStrategy.java
index 5e702c9..95e57f5 100644
---
a/server/src/main/java/org/apache/druid/segment/loading/RoundRobinStorageLocationSelectorStrategy.java
+++
b/server/src/main/java/org/apache/druid/segment/loading/RoundRobinStorageLocationSelectorStrategy.java
@@ -19,6 +19,9 @@
package org.apache.druid.segment.loading;
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+
import java.util.Iterator;
import java.util.List;
import java.util.NoSuchElementException;
@@ -32,11 +35,11 @@ import java.util.concurrent.atomic.AtomicInteger;
*/
public class RoundRobinStorageLocationSelectorStrategy implements
StorageLocationSelectorStrategy
{
-
private final List<StorageLocation> storageLocations;
private final AtomicInteger startIndex = new AtomicInteger(0);
- public RoundRobinStorageLocationSelectorStrategy(List<StorageLocation>
storageLocations)
+ @JsonCreator
+ public RoundRobinStorageLocationSelectorStrategy(@JacksonInject
List<StorageLocation> storageLocations)
{
this.storageLocations = storageLocations;
}
@@ -44,7 +47,8 @@ public class RoundRobinStorageLocationSelectorStrategy
implements StorageLocatio
@Override
public Iterator<StorageLocation> getLocations()
{
- return new Iterator<StorageLocation>() {
+ return new Iterator<StorageLocation>()
+ {
private final int numStorageLocations = storageLocations.size();
private int remainingIterations = numStorageLocations;
@@ -73,5 +77,4 @@ public class RoundRobinStorageLocationSelectorStrategy
implements StorageLocatio
}
};
}
-
}
diff --git
a/server/src/main/java/org/apache/druid/segment/loading/SegmentLoaderConfig.java
b/server/src/main/java/org/apache/druid/segment/loading/SegmentLoaderConfig.java
index edd2c0d..b2cf075 100644
---
a/server/src/main/java/org/apache/druid/segment/loading/SegmentLoaderConfig.java
+++
b/server/src/main/java/org/apache/druid/segment/loading/SegmentLoaderConfig.java
@@ -20,7 +20,6 @@
package org.apache.druid.segment.loading;
import com.fasterxml.jackson.annotation.JsonProperty;
-import com.google.common.annotations.VisibleForTesting;
import com.google.common.collect.Lists;
import org.apache.druid.utils.JvmUtils;
@@ -28,6 +27,7 @@ import java.io.File;
import java.util.Collections;
import java.util.List;
import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
/**
*
@@ -55,9 +55,6 @@ public class SegmentLoaderConfig
@JsonProperty("numBootstrapThreads")
private Integer numBootstrapThreads = null;
- @JsonProperty("locationSelectorStrategy")
- private StorageLocationSelectorStrategy locationSelectorStrategy;
-
@JsonProperty
private File infoDir = null;
@@ -101,15 +98,6 @@ public class SegmentLoaderConfig
return numBootstrapThreads == null ? numLoadingThreads :
numBootstrapThreads;
}
- public StorageLocationSelectorStrategy
getStorageLocationSelectorStrategy(List<StorageLocation> storageLocations)
- {
- if (locationSelectorStrategy == null) {
- // default strategy if no strategy is specified in the config
- locationSelectorStrategy = new
LeastBytesUsedStorageLocationSelectorStrategy(storageLocations);
- }
- return locationSelectorStrategy;
- }
-
public File getInfoDir()
{
if (infoDir == null) {
@@ -140,11 +128,19 @@ public class SegmentLoaderConfig
return retVal;
}
- @VisibleForTesting
- SegmentLoaderConfig
withStorageLocationSelectorStrategy(StorageLocationSelectorStrategy strategy)
+ /**
+ * Convert StorageLocationConfig objects to StorageLocation objects
+ *
+ * Note: {@link #getLocations} is called instead of variable access because
some testcases overrides this method
+ */
+ public List<StorageLocation> toStorageLocations()
{
- this.locationSelectorStrategy = strategy;
- return this;
+ return this.getLocations()
+ .stream()
+ .map(locationConfig -> new
StorageLocation(locationConfig.getPath(),
+
locationConfig.getMaxSize(),
+
locationConfig.getFreeSpacePercent()))
+ .collect(Collectors.toList());
}
@Override
@@ -154,7 +150,6 @@ public class SegmentLoaderConfig
"locations=" + locations +
", deleteOnRemove=" + deleteOnRemove +
", dropSegmentDelayMillis=" + dropSegmentDelayMillis +
- ", locationSelectorStrategy=" + locationSelectorStrategy +
", infoDir=" + infoDir +
'}';
}
diff --git
a/server/src/main/java/org/apache/druid/segment/loading/SegmentLoaderLocalCacheManager.java
b/server/src/main/java/org/apache/druid/segment/loading/SegmentLoaderLocalCacheManager.java
index 80e8dc6..2aa2038 100644
---
a/server/src/main/java/org/apache/druid/segment/loading/SegmentLoaderLocalCacheManager.java
+++
b/server/src/main/java/org/apache/druid/segment/loading/SegmentLoaderLocalCacheManager.java
@@ -30,9 +30,9 @@ import org.apache.druid.segment.IndexIO;
import org.apache.druid.segment.Segment;
import org.apache.druid.timeline.DataSegment;
+import javax.annotation.Nonnull;
import java.io.File;
import java.io.IOException;
-import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import java.util.concurrent.ConcurrentHashMap;
@@ -82,24 +82,48 @@ public class SegmentLoaderLocalCacheManager implements
SegmentLoader
@Inject
public SegmentLoaderLocalCacheManager(
IndexIO indexIO,
+ List<StorageLocation> locations,
SegmentLoaderConfig config,
+ @Nonnull StorageLocationSelectorStrategy strategy,
@Json ObjectMapper mapper
)
{
this.indexIO = indexIO;
this.config = config;
this.jsonMapper = mapper;
- this.locations = new ArrayList<>();
- for (StorageLocationConfig locationConfig : config.getLocations()) {
- locations.add(
- new StorageLocation(
- locationConfig.getPath(),
- locationConfig.getMaxSize(),
- locationConfig.getFreeSpacePercent()
- )
- );
- }
- this.strategy = config.getStorageLocationSelectorStrategy(locations);
+ this.locations = locations;
+ this.strategy = strategy;
+ log.info("Using storage location strategy: [%s]",
this.strategy.getClass().getSimpleName());
+ }
+
+ @VisibleForTesting
+ SegmentLoaderLocalCacheManager(
+ IndexIO indexIO,
+ SegmentLoaderConfig config,
+ @Nonnull StorageLocationSelectorStrategy strategy,
+ @Json ObjectMapper mapper
+ )
+ {
+ this(indexIO, config.toStorageLocations(), config, strategy, mapper);
+ }
+
+ /**
+ * creates instance with default storage location selector strategy
+ *
+ * This ctor is mainly for test cases, including test cases in other modules
+ */
+ public SegmentLoaderLocalCacheManager(
+ IndexIO indexIO,
+ SegmentLoaderConfig config,
+ @Json ObjectMapper mapper
+ )
+ {
+ this.indexIO = indexIO;
+ this.config = config;
+ this.jsonMapper = mapper;
+ this.locations = config.toStorageLocations();
+ this.strategy = new
LeastBytesUsedStorageLocationSelectorStrategy(locations);
+ log.info("Using storage location strategy: [%s]",
this.strategy.getClass().getSimpleName());
}
@Override
diff --git
a/server/src/main/java/org/apache/druid/segment/loading/StorageLocationSelectorStrategy.java
b/server/src/main/java/org/apache/druid/segment/loading/StorageLocationSelectorStrategy.java
index adc7388..57977d6 100644
---
a/server/src/main/java/org/apache/druid/segment/loading/StorageLocationSelectorStrategy.java
+++
b/server/src/main/java/org/apache/druid/segment/loading/StorageLocationSelectorStrategy.java
@@ -34,7 +34,7 @@ import java.util.Iterator;
* https://github.com/apache/druid/pull/8038#discussion_r325520829 of PR
https://github
* .com/apache/druid/pull/8038 for more details.
*/
-@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "type", defaultImpl =
+@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "strategy", defaultImpl =
LeastBytesUsedStorageLocationSelectorStrategy.class)
@JsonSubTypes(value = {
@JsonSubTypes.Type(name = "leastBytesUsed", value =
LeastBytesUsedStorageLocationSelectorStrategy.class),
diff --git
a/server/src/test/java/org/apache/druid/segment/loading/SegmentLoaderLocalCacheManagerTest.java
b/server/src/test/java/org/apache/druid/segment/loading/SegmentLoaderLocalCacheManagerTest.java
index 3130696..a01aef4 100644
---
a/server/src/test/java/org/apache/druid/segment/loading/SegmentLoaderLocalCacheManagerTest.java
+++
b/server/src/test/java/org/apache/druid/segment/loading/SegmentLoaderLocalCacheManagerTest.java
@@ -422,9 +422,8 @@ public class SegmentLoaderLocalCacheManagerTest
manager = new SegmentLoaderLocalCacheManager(
TestHelper.getTestIndexIO(),
- new
SegmentLoaderConfig().withLocations(locationConfigs).withStorageLocationSelectorStrategy(
- new RoundRobinStorageLocationSelectorStrategy(locations)
- ),
+ new SegmentLoaderConfig().withLocations(locationConfigs),
+ new RoundRobinStorageLocationSelectorStrategy(locations),
jsonMapper
);
final File segmentSrcFolder = tmpFolder.newFolder("segmentSrcFolder");
@@ -669,21 +668,12 @@ public class SegmentLoaderLocalCacheManagerTest
locationConfigs.add(locationConfig2);
locationConfigs.add(locationConfig3);
- List<StorageLocation> locations = new ArrayList<>();
- for (StorageLocationConfig locConfig : locationConfigs) {
- locations.add(
- new StorageLocation(
- locConfig.getPath(),
- locConfig.getMaxSize(),
- null
- )
- );
- }
+ SegmentLoaderConfig segmentLoaderConfig = new
SegmentLoaderConfig().withLocations(locationConfigs);
+
manager = new SegmentLoaderLocalCacheManager(
TestHelper.getTestIndexIO(),
- new
SegmentLoaderConfig().withLocations(locationConfigs).withStorageLocationSelectorStrategy(
- new RandomStorageLocationSelectorStrategy(locations)
- ),
+ new SegmentLoaderConfig().withLocations(locationConfigs),
+ new
RandomStorageLocationSelectorStrategy(segmentLoaderConfig.toStorageLocations()),
jsonMapper
);
diff --git
a/server/src/test/java/org/apache/druid/segment/loading/StorageLocationSelectorStrategyTest.java
b/server/src/test/java/org/apache/druid/segment/loading/StorageLocationSelectorStrategyTest.java
index 82cbe70..a2bfb5b 100644
---
a/server/src/test/java/org/apache/druid/segment/loading/StorageLocationSelectorStrategyTest.java
+++
b/server/src/test/java/org/apache/druid/segment/loading/StorageLocationSelectorStrategyTest.java
@@ -19,16 +19,31 @@
package org.apache.druid.segment.loading;
+import com.fasterxml.jackson.module.guice.ObjectMapperModule;
+import com.google.inject.Binder;
+import com.google.inject.Guice;
+import com.google.inject.Injector;
+import com.google.inject.Module;
+import com.google.inject.Provides;
+import org.apache.druid.guice.DruidGuiceExtensions;
+import org.apache.druid.guice.JsonConfigProvider;
+import org.apache.druid.guice.JsonConfigurator;
+import org.apache.druid.guice.LazySingleton;
+import org.apache.druid.guice.StorageNodeModule;
+import org.apache.druid.jackson.DefaultObjectMapper;
import org.junit.Assert;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
+import javax.validation.Validation;
+import javax.validation.Validator;
import java.io.File;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Iterator;
import java.util.List;
+import java.util.Properties;
public class StorageLocationSelectorStrategyTest
{
@@ -256,4 +271,107 @@ public class StorageLocationSelectorStrategyTest
Assert.assertEquals("The next element of the iterator should point to path
local_storage_folder_1",
localStorageFolder1, loc3.getPath());
}
+
+ @Test
+ public void testDefaultSelectorStrategyConfig()
+ {
+ //no druid.segmentCache.locationSelector.strategy specified, the default
will be used
+ final Properties props = new Properties();
+ props.setProperty("druid.segmentCache.locations", "[{\"path\":
\"/tmp/druid/indexCache\"}]");
+
+ StorageLocationSelectorStrategy strategy =
makeInjectorWithProperties(props).getInstance(StorageLocationSelectorStrategy.class);
+ Assert.assertEquals(LeastBytesUsedStorageLocationSelectorStrategy.class,
+ strategy.getClass());
+ Assert.assertEquals("/tmp/druid/indexCache",
strategy.getLocations().next().getPath().getAbsolutePath());
+ }
+
+ @Test
+ public void testRoundRobinSelectorStrategyConfig()
+ {
+ final Properties props = new Properties();
+ props.setProperty("druid.segmentCache.locations", "[{\"path\":
\"/tmp/druid/indexCache\"}]");
+ props.setProperty("druid.segmentCache.locationSelector.strategy",
"roundRobin");
+
+ Injector injector = makeInjectorWithProperties(props);
+ StorageLocationSelectorStrategy strategy =
injector.getInstance(StorageLocationSelectorStrategy.class);
+
+ Assert.assertEquals(RoundRobinStorageLocationSelectorStrategy.class,
+ strategy.getClass());
+ Assert.assertEquals("/tmp/druid/indexCache",
strategy.getLocations().next().getPath().getAbsolutePath());
+ }
+
+ @Test
+ public void testLeastBytesUsedSelectorStrategyConfig()
+ {
+ final Properties props = new Properties();
+ props.setProperty("druid.segmentCache.locations", "[{\"path\":
\"/tmp/druid/indexCache\"}]");
+ props.setProperty("druid.segmentCache.locationSelector.strategy",
"leastBytesUsed");
+
+ Injector injector = makeInjectorWithProperties(props);
+ StorageLocationSelectorStrategy strategy =
injector.getInstance(StorageLocationSelectorStrategy.class);
+
+ Assert.assertEquals(LeastBytesUsedStorageLocationSelectorStrategy.class,
+ strategy.getClass());
+ Assert.assertEquals("/tmp/druid/indexCache",
strategy.getLocations().next().getPath().getAbsolutePath());
+ }
+
+ @Test
+ public void testRandomSelectorStrategyConfig()
+ {
+ final Properties props = new Properties();
+ props.setProperty("druid.segmentCache.locations", "[{\"path\":
\"/tmp/druid/indexCache\"}]");
+ props.setProperty("druid.segmentCache.locationSelector.strategy",
"random");
+
+ Injector injector = makeInjectorWithProperties(props);
+ StorageLocationSelectorStrategy strategy =
injector.getInstance(StorageLocationSelectorStrategy.class);
+
+ Assert.assertEquals(RandomStorageLocationSelectorStrategy.class,
+ strategy.getClass());
+ Assert.assertEquals("/tmp/druid/indexCache",
strategy.getLocations().next().getPath().getAbsolutePath());
+ }
+
+ @Test
+ public void testMostAvailableSizeSelectorStrategyConfig()
+ {
+ final Properties props = new Properties();
+ props.setProperty("druid.segmentCache.locationSelector.strategy",
"mostAvailableSize");
+ props.setProperty("druid.segmentCache.locations", "[{\"path\":
\"/tmp/druid/indexCache\"}]");
+
+ Injector injector = makeInjectorWithProperties(props);
+ StorageLocationSelectorStrategy strategy =
injector.getInstance(StorageLocationSelectorStrategy.class);
+
+ Assert.assertEquals(MostAvailableSizeStorageLocationSelectorStrategy.class,
+ strategy.getClass());
+ Assert.assertEquals("/tmp/druid/indexCache",
strategy.getLocations().next().getPath().getAbsolutePath());
+ }
+
+ private Injector makeInjectorWithProperties(final Properties props)
+ {
+ return Guice.createInjector(
+ new Module()
+ {
+ @Override
+ public void configure(Binder binder)
+ {
+ //ObjectMapperModule introduce Guice injector for jackson
+ binder.install(new ObjectMapperModule()
+ .withObjectMapper(new DefaultObjectMapper()));
+ binder.install(new DruidGuiceExtensions());
+
+
binder.bind(Validator.class).toInstance(Validation.buildDefaultValidatorFactory().getValidator());
+ binder.bind(JsonConfigurator.class).in(LazySingleton.class);
+ binder.bind(Properties.class).toInstance(props);
+
+ JsonConfigProvider.bind(binder, "druid.segmentCache",
SegmentLoaderConfig.class);
+ StorageNodeModule.bindLocationSelectorStrategy(binder);
+ }
+
+ @Provides
+ public List provideStorageLocation(SegmentLoaderConfig
segmentLoader)
+ {
+ return segmentLoader.toStorageLocations();
+ }
+ }
+ );
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
