This is an automated email from the ASF dual-hosted git repository.
FrankChen021 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git
The following commit(s) were added to refs/heads/master by this push:
new 5abae391109 fix: standalone Overlord ignores
druid.coordinator.asOverlord.enabled (#20781)
5abae391109 is described below
commit 5abae3911099910abfb28ed2663e7fc839744b49
Author: Frank Chen <[email protected]>
AuthorDate: Sat Oct 31 08:06:15 2026 +0800
fix: standalone Overlord ignores druid.coordinator.asOverlord.enabled
(#20781)
A standalone Overlord decided whether it was embedded in a Coordinator by
reading druid.coordinator.asOverlord.enabled. When that Coordinator-only
property reached the Overlord, for example through a shared
runtime.properties, the Overlord never started the segment metadata cache,
and with druid.manager.segments.useIncrementalCache=always every segment
write hung.
- CliOverlord binds an OverlordMode enum (STANDALONE /
EMBEDDED_IN_COORDINATOR) from how the process was started. DruidOverlord
and OverlordCompactionScheduler use it instead of
CoordinatorOverlordServiceConfig.
- Remove CoordinatorOverlordServiceConfig and the unused
druid.coordinator.asOverlord.overlordService property from the docs and
example configs.
---
docs/configuration/index.md | 3 +-
.../auto/coordinator-overlord/runtime.properties | 1 -
.../master/coordinator-overlord/runtime.properties | 1 -
.../large/coordinator-overlord/runtime.properties | 1 -
.../medium/coordinator-overlord/runtime.properties | 1 -
.../coordinator-overlord/runtime.properties | 1 -
.../coordinator-overlord/runtime.properties | 1 -
.../small/coordinator-overlord/runtime.properties | 1 -
.../xlarge/coordinator-overlord/runtime.properties | 1 -
.../compact/OverlordCompactionScheduler.java | 6 +-
.../druid/indexing/overlord/DruidOverlord.java | 9 +-
.../druid/indexing/overlord/OverlordMode.java | 48 ++++++++++
.../compact/OverlordCompactionSchedulerTest.java | 12 +--
.../druid/indexing/overlord/DruidOverlordTest.java | 105 +++++++++++++++++++++
.../druid/indexing/overlord/http/OverlordTest.java | 4 +-
.../CoordinatorOverlordServiceConfig.java | 64 -------------
.../CoordinatorOverlordServiceConfigTest.java | 39 --------
.../java/org/apache/druid/cli/CliOverlord.java | 5 +-
.../java/org/apache/druid/cli/CliOverlordTest.java | 20 ++++
19 files changed, 192 insertions(+), 131 deletions(-)
diff --git a/docs/configuration/index.md b/docs/configuration/index.md
index de05cec50d4..9a4fdeb6342 100644
--- a/docs/configuration/index.md
+++ b/docs/configuration/index.md
@@ -752,8 +752,7 @@ These Coordinator static configurations can be defined in
the `coordinator/runti
|`druid.coordinator.balancer.diskNormalized.utilizationThreshold`|Only used
when `druid.coordinator.balancer.strategy` is `diskNormalized`. Projected disk
utilization difference that the strategy ignores before applying the
exponential utilization penalty. A value of `0.05` means candidates within 5
percentage points of the least-used candidate are ordered by normal cost, while
candidates outside that band are penalized exponentially by the excess
utilization difference. Must be in `(0.0 [...]
|`druid.coordinator.loadqueuepeon.http.repeatDelay`|The start and repeat delay
(in milliseconds) for the load queue peon, which manages the load/drop queue of
segments for any server.|1 minute|
|`druid.coordinator.loadqueuepeon.http.batchSize`|Number of segment load/drop
requests to batch in one HTTP request. Note that it must be smaller than or
equal to the `druid.segmentCache.numLoadingThreads` config on Historical
service. If this value is not configured, the coordinator uses the value of the
`numLoadingThreads` for the respective server. |
`druid.segmentCache.numLoadingThreads` |
-|`druid.coordinator.asOverlord.enabled`|Boolean value for whether this
Coordinator service should act like an Overlord as well. This configuration
allows users to simplify a Druid cluster by not having to deploy any standalone
Overlord services. If set to true, then Overlord console is available at
`http://coordinator-host:port/console.html` and be sure to set
`druid.coordinator.asOverlord.overlordService` also.|false|
-|`druid.coordinator.asOverlord.overlordService`| Required, if
`druid.coordinator.asOverlord.enabled` is `true`. This must be same value as
`druid.service` on standalone Overlord services and
`druid.selectors.indexing.serviceName` on Middle Managers.|NULL|
+|`druid.coordinator.asOverlord.enabled`|Boolean value for whether this
Coordinator service should act like an Overlord as well. This configuration
allows users to simplify a Druid cluster by not having to deploy any standalone
Overlord services. If set to true, then Overlord console is available at
`http://coordinator-host:port/console.html`.|false|
##### Data management
diff --git a/examples/conf/druid/auto/coordinator-overlord/runtime.properties
b/examples/conf/druid/auto/coordinator-overlord/runtime.properties
index ed892f8641d..6254ccffc6b 100644
--- a/examples/conf/druid/auto/coordinator-overlord/runtime.properties
+++ b/examples/conf/druid/auto/coordinator-overlord/runtime.properties
@@ -29,7 +29,6 @@ druid.manager.segments.pollDuration=PT5S
# Run the overlord service in the coordinator process
druid.coordinator.asOverlord.enabled=true
-druid.coordinator.asOverlord.overlordService=druid/overlord
druid.indexer.queue.startDelay=PT5S
diff --git
a/examples/conf/druid/cluster/master/coordinator-overlord/runtime.properties
b/examples/conf/druid/cluster/master/coordinator-overlord/runtime.properties
index ed892f8641d..6254ccffc6b 100644
--- a/examples/conf/druid/cluster/master/coordinator-overlord/runtime.properties
+++ b/examples/conf/druid/cluster/master/coordinator-overlord/runtime.properties
@@ -29,7 +29,6 @@ druid.manager.segments.pollDuration=PT5S
# Run the overlord service in the coordinator process
druid.coordinator.asOverlord.enabled=true
-druid.coordinator.asOverlord.overlordService=druid/overlord
druid.indexer.queue.startDelay=PT5S
diff --git
a/examples/conf/druid/single-server/large/coordinator-overlord/runtime.properties
b/examples/conf/druid/single-server/large/coordinator-overlord/runtime.properties
index ed892f8641d..6254ccffc6b 100644
---
a/examples/conf/druid/single-server/large/coordinator-overlord/runtime.properties
+++
b/examples/conf/druid/single-server/large/coordinator-overlord/runtime.properties
@@ -29,7 +29,6 @@ druid.manager.segments.pollDuration=PT5S
# Run the overlord service in the coordinator process
druid.coordinator.asOverlord.enabled=true
-druid.coordinator.asOverlord.overlordService=druid/overlord
druid.indexer.queue.startDelay=PT5S
diff --git
a/examples/conf/druid/single-server/medium/coordinator-overlord/runtime.properties
b/examples/conf/druid/single-server/medium/coordinator-overlord/runtime.properties
index ed892f8641d..6254ccffc6b 100644
---
a/examples/conf/druid/single-server/medium/coordinator-overlord/runtime.properties
+++
b/examples/conf/druid/single-server/medium/coordinator-overlord/runtime.properties
@@ -29,7 +29,6 @@ druid.manager.segments.pollDuration=PT5S
# Run the overlord service in the coordinator process
druid.coordinator.asOverlord.enabled=true
-druid.coordinator.asOverlord.overlordService=druid/overlord
druid.indexer.queue.startDelay=PT5S
diff --git
a/examples/conf/druid/single-server/micro-quickstart/coordinator-overlord/runtime.properties
b/examples/conf/druid/single-server/micro-quickstart/coordinator-overlord/runtime.properties
index ed892f8641d..6254ccffc6b 100644
---
a/examples/conf/druid/single-server/micro-quickstart/coordinator-overlord/runtime.properties
+++
b/examples/conf/druid/single-server/micro-quickstart/coordinator-overlord/runtime.properties
@@ -29,7 +29,6 @@ druid.manager.segments.pollDuration=PT5S
# Run the overlord service in the coordinator process
druid.coordinator.asOverlord.enabled=true
-druid.coordinator.asOverlord.overlordService=druid/overlord
druid.indexer.queue.startDelay=PT5S
diff --git
a/examples/conf/druid/single-server/nano-quickstart/coordinator-overlord/runtime.properties
b/examples/conf/druid/single-server/nano-quickstart/coordinator-overlord/runtime.properties
index ed892f8641d..6254ccffc6b 100644
---
a/examples/conf/druid/single-server/nano-quickstart/coordinator-overlord/runtime.properties
+++
b/examples/conf/druid/single-server/nano-quickstart/coordinator-overlord/runtime.properties
@@ -29,7 +29,6 @@ druid.manager.segments.pollDuration=PT5S
# Run the overlord service in the coordinator process
druid.coordinator.asOverlord.enabled=true
-druid.coordinator.asOverlord.overlordService=druid/overlord
druid.indexer.queue.startDelay=PT5S
diff --git
a/examples/conf/druid/single-server/small/coordinator-overlord/runtime.properties
b/examples/conf/druid/single-server/small/coordinator-overlord/runtime.properties
index ed892f8641d..6254ccffc6b 100644
---
a/examples/conf/druid/single-server/small/coordinator-overlord/runtime.properties
+++
b/examples/conf/druid/single-server/small/coordinator-overlord/runtime.properties
@@ -29,7 +29,6 @@ druid.manager.segments.pollDuration=PT5S
# Run the overlord service in the coordinator process
druid.coordinator.asOverlord.enabled=true
-druid.coordinator.asOverlord.overlordService=druid/overlord
druid.indexer.queue.startDelay=PT5S
diff --git
a/examples/conf/druid/single-server/xlarge/coordinator-overlord/runtime.properties
b/examples/conf/druid/single-server/xlarge/coordinator-overlord/runtime.properties
index ed892f8641d..6254ccffc6b 100644
---
a/examples/conf/druid/single-server/xlarge/coordinator-overlord/runtime.properties
+++
b/examples/conf/druid/single-server/xlarge/coordinator-overlord/runtime.properties
@@ -29,7 +29,6 @@ druid.manager.segments.pollDuration=PT5S
# Run the overlord service in the coordinator process
druid.coordinator.asOverlord.enabled=true
-druid.coordinator.asOverlord.overlordService=druid/overlord
druid.indexer.queue.startDelay=PT5S
diff --git
a/indexing-service/src/main/java/org/apache/druid/indexing/compact/OverlordCompactionScheduler.java
b/indexing-service/src/main/java/org/apache/druid/indexing/compact/OverlordCompactionScheduler.java
index 647b5774513..d7d244ecb53 100644
---
a/indexing-service/src/main/java/org/apache/druid/indexing/compact/OverlordCompactionScheduler.java
+++
b/indexing-service/src/main/java/org/apache/druid/indexing/compact/OverlordCompactionScheduler.java
@@ -31,6 +31,7 @@ import org.apache.druid.indexer.TaskStatus;
import org.apache.druid.indexing.common.actions.TaskActionClientFactory;
import org.apache.druid.indexing.common.task.Task;
import org.apache.druid.indexing.overlord.GlobalTaskLockbox;
+import org.apache.druid.indexing.overlord.OverlordMode;
import org.apache.druid.indexing.overlord.TaskMaster;
import org.apache.druid.indexing.overlord.TaskQueryTool;
import org.apache.druid.indexing.overlord.TaskRunner;
@@ -53,7 +54,6 @@ import
org.apache.druid.server.compaction.CompactionStatusTracker;
import org.apache.druid.server.coordinator.AutoCompactionSnapshot;
import org.apache.druid.server.coordinator.ClusterCompactionConfig;
import org.apache.druid.server.coordinator.CompactionConfigValidationResult;
-import org.apache.druid.server.coordinator.CoordinatorOverlordServiceConfig;
import org.apache.druid.server.coordinator.DataSourceCompactionConfig;
import org.apache.druid.server.coordinator.DruidCompactionConfig;
import org.apache.druid.server.coordinator.duty.CompactSegments;
@@ -154,7 +154,7 @@ public class OverlordCompactionScheduler implements
CompactionScheduler
SegmentsMetadataManagerConfig segmentManagerConfig,
Supplier<DruidCompactionConfig> compactionConfigSupplier,
CompactionStatusTracker statusTracker,
- CoordinatorOverlordServiceConfig coordinatorOverlordServiceConfig,
+ OverlordMode overlordMode,
TaskActionClientFactory taskActionClientFactory,
DruidInputSourceFactory druidInputSourceFactory,
ScheduledExecutorFactory executorFactory,
@@ -179,7 +179,7 @@ public class OverlordCompactionScheduler implements
CompactionScheduler
this.executor = executorFactory.create(1, "CompactionScheduler-%s");
this.statusTracker = statusTracker;
this.shouldPollSegments = segmentManager != null
- && !coordinatorOverlordServiceConfig.isEnabled();
+ && overlordMode == OverlordMode.STANDALONE;
this.overlordClient = new LocalOverlordClient(taskMaster, taskQueryTool,
objectMapper);
this.brokerClient = brokerClient;
this.activeSupervisors = new ConcurrentHashMap<>();
diff --git
a/indexing-service/src/main/java/org/apache/druid/indexing/overlord/DruidOverlord.java
b/indexing-service/src/main/java/org/apache/druid/indexing/overlord/DruidOverlord.java
index bcded57e0ff..228d019da25 100644
---
a/indexing-service/src/main/java/org/apache/druid/indexing/overlord/DruidOverlord.java
+++
b/indexing-service/src/main/java/org/apache/druid/indexing/overlord/DruidOverlord.java
@@ -39,7 +39,6 @@ import
org.apache.druid.java.util.common.lifecycle.LifecycleStop;
import org.apache.druid.java.util.emitter.EmittingLogger;
import org.apache.druid.java.util.emitter.service.ServiceEmitter;
import org.apache.druid.metadata.segment.cache.SegmentMetadataCache;
-import org.apache.druid.server.coordinator.CoordinatorOverlordServiceConfig;
import java.util.Set;
import java.util.concurrent.atomic.AtomicReference;
@@ -57,7 +56,7 @@ public class DruidOverlord
private final DruidLeaderSelector overlordLeaderSelector;
private final DruidLeaderSelector.Listener leadershipListener;
private final SegmentMetadataCache segmentMetadataCache;
- private final CoordinatorOverlordServiceConfig
coordinatorOverlordServiceConfig;
+ private final OverlordMode overlordMode;
private final ReentrantLock giant = new ReentrantLock(true);
@@ -86,7 +85,7 @@ public class DruidOverlord
final TaskStorage taskStorage,
final TaskActionClientFactory taskActionClientFactory,
final TaskRunnerFactory runnerFactory,
- final CoordinatorOverlordServiceConfig coordinatorOverlordServiceConfig,
+ final OverlordMode overlordMode,
final ServiceEmitter emitter,
final SupervisorManager supervisorManager,
final OverlordDutyExecutor overlordDutyExecutor,
@@ -100,7 +99,7 @@ public class DruidOverlord
{
this.overlordLeaderSelector = overlordLeaderSelector;
this.segmentMetadataCache = segmentMetadataCache;
- this.coordinatorOverlordServiceConfig = coordinatorOverlordServiceConfig;
+ this.overlordMode = overlordMode;
this.leadershipListener = new DruidLeaderSelector.Listener()
{
@@ -309,7 +308,7 @@ public class DruidOverlord
private boolean isStandalone()
{
- return !coordinatorOverlordServiceConfig.isEnabled();
+ return overlordMode == OverlordMode.STANDALONE;
}
}
diff --git
a/indexing-service/src/main/java/org/apache/druid/indexing/overlord/OverlordMode.java
b/indexing-service/src/main/java/org/apache/druid/indexing/overlord/OverlordMode.java
new file mode 100644
index 00000000000..2c78b292733
--- /dev/null
+++
b/indexing-service/src/main/java/org/apache/druid/indexing/overlord/OverlordMode.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.overlord;
+
+/// Deployment mode of the Overlord in the current process.
+///
+/// The mode is bound by `CliOverlord#getModules(boolean)`, which already
knows how
+/// the process was started: as the `overlord` service, or with the Overlord
modules
+/// loaded into a Coordinator by `CliCoordinator`.
+///
+/// Overlord code must use this binding rather than
`druid.coordinator.asOverlord.enabled`
+/// to find out how it is deployed. That property is meant for the Coordinator
only, and a
+/// standalone Overlord may still see it set to `true`, for example when it
shares a
+/// `runtime.properties` file with the Coordinator.
+///
+/// Components that the Overlord shares with the Coordinator in the same
process use the mode
+/// to decide who owns their lifecycle:
+///
+/// - [DruidOverlord] starts and stops the segment metadata cache only when
[#STANDALONE].
+/// - `OverlordCompactionScheduler` polls segments itself only when
[#STANDALONE].
+public enum OverlordMode
+{
+ /// The Overlord runs as its own service, started by `CliOverlord`. It owns
the lifecycle
+ /// of every component it uses.
+ STANDALONE,
+
+ /// The Overlord runs inside a Coordinator process
(`druid.coordinator.asOverlord.enabled=true`
+ /// on the Coordinator). Components shared with the Coordinator, such as the
segment metadata cache
+ /// and segment polling, are started and stopped by the Coordinator.
+ EMBEDDED_IN_COORDINATOR
+}
diff --git
a/indexing-service/src/test/java/org/apache/druid/indexing/compact/OverlordCompactionSchedulerTest.java
b/indexing-service/src/test/java/org/apache/druid/indexing/compact/OverlordCompactionSchedulerTest.java
index 94d9c16e7da..2b9207b4a78 100644
---
a/indexing-service/src/test/java/org/apache/druid/indexing/compact/OverlordCompactionSchedulerTest.java
+++
b/indexing-service/src/test/java/org/apache/druid/indexing/compact/OverlordCompactionSchedulerTest.java
@@ -45,6 +45,7 @@ import org.apache.druid.indexing.common.task.CompactionTask;
import org.apache.druid.indexing.common.task.Task;
import org.apache.druid.indexing.overlord.GlobalTaskLockbox;
import org.apache.druid.indexing.overlord.HeapMemoryTaskStorage;
+import org.apache.druid.indexing.overlord.OverlordMode;
import org.apache.druid.indexing.overlord.Segments;
import org.apache.druid.indexing.overlord.TaskMaster;
import org.apache.druid.indexing.overlord.TaskQueryTool;
@@ -76,7 +77,6 @@ import org.apache.druid.server.compaction.Table;
import org.apache.druid.server.coordinator.AutoCompactionSnapshot;
import org.apache.druid.server.coordinator.ClusterCompactionConfig;
import org.apache.druid.server.coordinator.CompactionConfigValidationResult;
-import org.apache.druid.server.coordinator.CoordinatorOverlordServiceConfig;
import org.apache.druid.server.coordinator.CreateDataSegments;
import org.apache.druid.server.coordinator.DataSourceCompactionConfig;
import org.apache.druid.server.coordinator.DruidCompactionConfig;
@@ -123,7 +123,7 @@ public class OverlordCompactionSchedulerTest
private static final DateTime JAN_20 = DateTimes.of("2025-01-20");
private AtomicReference<ClusterCompactionConfig> compactionConfig;
- private CoordinatorOverlordServiceConfig coordinatorOverlordServiceConfig;
+ private OverlordMode overlordMode;
private TaskMaster taskMaster;
private TaskQueue taskQueue;
@@ -178,7 +178,7 @@ public class OverlordCompactionSchedulerTest
segmentsMetadataManager = segmentStorage.getManager();
compactionConfig = new AtomicReference<>(new ClusterCompactionConfig(1.0,
100, null, true, null, null));
- coordinatorOverlordServiceConfig = new
CoordinatorOverlordServiceConfig(false, null);
+ overlordMode = OverlordMode.STANDALONE;
taskActionClientFactory = task -> new TaskActionClient()
{
@@ -224,7 +224,7 @@ public class OverlordCompactionSchedulerTest
new SegmentsMetadataManagerConfig(null, null, null),
() ->
DruidCompactionConfig.empty().withClusterConfig(compactionConfig.get()),
new CompactionStatusTracker(),
- coordinatorOverlordServiceConfig,
+ overlordMode,
taskActionClientFactory,
new DruidInputSourceFactory(
TestIndex.INDEX_IO,
@@ -333,7 +333,7 @@ public class OverlordCompactionSchedulerTest
@Test
public void test_enableSupervisors_inStandaloneMode_enablesSegmentPolling()
{
- coordinatorOverlordServiceConfig = new
CoordinatorOverlordServiceConfig(false, null);
+ overlordMode = OverlordMode.STANDALONE;
initScheduler();
verifySegmentPolling(true);
@@ -342,7 +342,7 @@ public class OverlordCompactionSchedulerTest
@Test
public void test_enableSupervisors_inCoordinatorMode_disablesSegmentPolling()
{
- coordinatorOverlordServiceConfig = new
CoordinatorOverlordServiceConfig(true, "overlord");
+ overlordMode = OverlordMode.EMBEDDED_IN_COORDINATOR;
initScheduler();
verifySegmentPolling(false);
diff --git
a/indexing-service/src/test/java/org/apache/druid/indexing/overlord/DruidOverlordTest.java
b/indexing-service/src/test/java/org/apache/druid/indexing/overlord/DruidOverlordTest.java
new file mode 100644
index 00000000000..013afa1b82c
--- /dev/null
+++
b/indexing-service/src/test/java/org/apache/druid/indexing/overlord/DruidOverlordTest.java
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.overlord;
+
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.indexing.common.actions.SegmentAllocationQueue;
+import org.apache.druid.indexing.common.actions.TaskActionClientFactory;
+import org.apache.druid.indexing.common.task.NoopTaskContextEnricher;
+import org.apache.druid.indexing.overlord.config.DefaultTaskConfig;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.indexing.overlord.config.TaskQueueConfig;
+import org.apache.druid.indexing.overlord.duty.OverlordDutyExecutor;
+import org.apache.druid.indexing.overlord.supervisor.SupervisorManager;
+import org.apache.druid.jackson.DefaultObjectMapper;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.metadata.segment.cache.SegmentMetadataCache;
+import org.easymock.EasyMock;
+import org.junit.jupiter.api.Test;
+
+import java.util.Set;
+
+public class DruidOverlordTest
+{
+ @Test
+ public void testStandaloneOverlordStartsAndStopsSegmentMetadataCache()
+ {
+ final SegmentMetadataCache segmentMetadataCache =
EasyMock.createStrictMock(SegmentMetadataCache.class);
+ segmentMetadataCache.start();
+ EasyMock.expectLastCall().once();
+ segmentMetadataCache.stop();
+ EasyMock.expectLastCall().once();
+ EasyMock.replay(segmentMetadataCache);
+
+ final DruidOverlord overlord = createOverlord(OverlordMode.STANDALONE,
segmentMetadataCache);
+ overlord.start();
+ overlord.stop();
+
+ EasyMock.verify(segmentMetadataCache);
+ }
+
+ @Test
+ public void
testOverlordEmbeddedInCoordinatorDoesNotStartSegmentMetadataCache()
+ {
+ // The Coordinator owns the lifecycle of the shared cache when the
Overlord is embedded in it
+ final SegmentMetadataCache segmentMetadataCache =
EasyMock.createStrictMock(SegmentMetadataCache.class);
+ EasyMock.replay(segmentMetadataCache);
+
+ final DruidOverlord overlord =
createOverlord(OverlordMode.EMBEDDED_IN_COORDINATOR, segmentMetadataCache);
+ overlord.start();
+ overlord.stop();
+
+ EasyMock.verify(segmentMetadataCache);
+ }
+
+ private static DruidOverlord createOverlord(
+ final OverlordMode overlordMode,
+ final SegmentMetadataCache segmentMetadataCache
+ )
+ {
+ return new DruidOverlord(
+ niceMock(TaskMaster.class),
+ new TaskLockConfig(),
+ new TaskQueueConfig(null, null, null, null, null, null),
+ new DefaultTaskConfig(),
+ niceMock(GlobalTaskLockbox.class),
+ niceMock(TaskStorage.class),
+ niceMock(TaskActionClientFactory.class),
+ niceMock(TaskRunnerFactory.class),
+ overlordMode,
+ niceMock(ServiceEmitter.class),
+ niceMock(SupervisorManager.class),
+ niceMock(OverlordDutyExecutor.class),
+ niceMock(DruidLeaderSelector.class),
+ niceMock(SegmentAllocationQueue.class),
+ segmentMetadataCache,
+ Set.of(),
+ new DefaultObjectMapper(),
+ new NoopTaskContextEnricher()
+ );
+ }
+
+ private static <T> T niceMock(final Class<T> clazz)
+ {
+ final T mock = EasyMock.createNiceMock(clazz);
+ EasyMock.replay(mock);
+ return mock;
+ }
+}
diff --git
a/indexing-service/src/test/java/org/apache/druid/indexing/overlord/http/OverlordTest.java
b/indexing-service/src/test/java/org/apache/druid/indexing/overlord/http/OverlordTest.java
index 1d9805b4b13..654dcc43c44 100644
---
a/indexing-service/src/test/java/org/apache/druid/indexing/overlord/http/OverlordTest.java
+++
b/indexing-service/src/test/java/org/apache/druid/indexing/overlord/http/OverlordTest.java
@@ -52,6 +52,7 @@ import org.apache.druid.indexing.overlord.GlobalTaskLockbox;
import org.apache.druid.indexing.overlord.HeapMemoryTaskStorage;
import org.apache.druid.indexing.overlord.IndexerMetadataStorageAdapter;
import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.OverlordMode;
import org.apache.druid.indexing.overlord.TaskMaster;
import org.apache.druid.indexing.overlord.TaskQueryTool;
import org.apache.druid.indexing.overlord.TaskRunner;
@@ -77,7 +78,6 @@ import
org.apache.druid.java.util.emitter.service.ServiceEmitter;
import org.apache.druid.metadata.segment.cache.SegmentMetadataCache;
import org.apache.druid.segment.TestHelper;
import org.apache.druid.server.DruidNode;
-import org.apache.druid.server.coordinator.CoordinatorOverlordServiceConfig;
import org.apache.druid.server.metrics.NoopServiceEmitter;
import org.apache.druid.server.security.AuthConfig;
import org.apache.druid.server.security.AuthTestUtils;
@@ -254,7 +254,7 @@ public class OverlordTest
taskStorage,
taskActionClientFactory,
taskRunnerFactory,
- new CoordinatorOverlordServiceConfig(null, null),
+ OverlordMode.STANDALONE,
serviceEmitter,
supervisorManager,
EasyMock.createNiceMock(OverlordDutyExecutor.class),
diff --git
a/server/src/main/java/org/apache/druid/server/coordinator/CoordinatorOverlordServiceConfig.java
b/server/src/main/java/org/apache/druid/server/coordinator/CoordinatorOverlordServiceConfig.java
deleted file mode 100644
index 03a20bbc1ff..00000000000
---
a/server/src/main/java/org/apache/druid/server/coordinator/CoordinatorOverlordServiceConfig.java
+++ /dev/null
@@ -1,64 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.druid.server.coordinator;
-
-import com.fasterxml.jackson.annotation.JsonProperty;
-import com.google.common.base.Preconditions;
-import org.apache.druid.common.config.Configs;
-
-/**
- */
-public class CoordinatorOverlordServiceConfig
-{
- @JsonProperty
- private final boolean enabled;
-
- @JsonProperty
- private final String overlordService;
-
- public CoordinatorOverlordServiceConfig(
- @JsonProperty("enabled") Boolean enabled,
- @JsonProperty("overlordService") String overlordService
- )
- {
- this.enabled = Configs.valueOrDefault(enabled, false);
- this.overlordService = overlordService;
-
- if (this.enabled) {
- Preconditions.checkArgument(
- this.overlordService != null,
- "'overlordService' must be specified when running Coordinator as
Overlord."
- );
- }
- }
-
- /**
- * @return true if Coordinator and Overlord are running as a single process.
- */
- public boolean isEnabled()
- {
- return enabled;
- }
-
- public String getOverlordService()
- {
- return overlordService;
- }
-}
diff --git
a/server/src/test/java/org/apache/druid/server/coordinator/CoordinatorOverlordServiceConfigTest.java
b/server/src/test/java/org/apache/druid/server/coordinator/CoordinatorOverlordServiceConfigTest.java
deleted file mode 100644
index 05cfc13525d..00000000000
---
a/server/src/test/java/org/apache/druid/server/coordinator/CoordinatorOverlordServiceConfigTest.java
+++ /dev/null
@@ -1,39 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.druid.server.coordinator;
-
-import org.junit.jupiter.api.Assertions;
-import org.junit.jupiter.api.Test;
-
-public class CoordinatorOverlordServiceConfigTest
-{
- @Test
- public void testOverlordServiceIsRequiredIfEnabled()
- {
- IllegalArgumentException e = Assertions.assertThrows(
- IllegalArgumentException.class,
- () -> new CoordinatorOverlordServiceConfig(true, null)
- );
- Assertions.assertEquals(
- "'overlordService' must be specified when running Coordinator as
Overlord.",
- e.getMessage()
- );
- }
-}
diff --git a/services/src/main/java/org/apache/druid/cli/CliOverlord.java
b/services/src/main/java/org/apache/druid/cli/CliOverlord.java
index 9b81a4a987b..e3243abe429 100644
--- a/services/src/main/java/org/apache/druid/cli/CliOverlord.java
+++ b/services/src/main/java/org/apache/druid/cli/CliOverlord.java
@@ -79,6 +79,7 @@ import
org.apache.druid.indexing.overlord.HeapMemoryTaskStorage;
import org.apache.druid.indexing.overlord.IndexerMetadataStorageAdapter;
import org.apache.druid.indexing.overlord.LeaderOverlordService;
import org.apache.druid.indexing.overlord.MetadataTaskStorage;
+import org.apache.druid.indexing.overlord.OverlordMode;
import org.apache.druid.indexing.overlord.TaskMaster;
import org.apache.druid.indexing.overlord.TaskQueryTool;
import org.apache.druid.indexing.overlord.TaskRunnerFactory;
@@ -125,7 +126,6 @@ import
org.apache.druid.segment.realtime.ChatHandlerProvider;
import org.apache.druid.segment.realtime.appenderator.AppenderatorsManager;
import
org.apache.druid.segment.realtime.appenderator.DummyForInjectionAppenderatorsManager;
import org.apache.druid.server.compaction.CompactionStatusTracker;
-import org.apache.druid.server.coordinator.CoordinatorOverlordServiceConfig;
import org.apache.druid.server.coordinator.DruidCompactionConfig;
import org.apache.druid.server.http.RedirectFilter;
import org.apache.druid.server.http.RedirectInfo;
@@ -223,7 +223,8 @@ public class CliOverlord extends ServerRunnable
binder.bind(CompactionStatusTracker.class).in(LazySingleton.class);
}
- JsonConfigProvider.bind(binder, "druid.coordinator.asOverlord",
CoordinatorOverlordServiceConfig.class);
+ binder.bind(OverlordMode.class)
+ .toInstance(standalone ? OverlordMode.STANDALONE :
OverlordMode.EMBEDDED_IN_COORDINATOR);
JsonConfigProvider.bind(binder, "druid.indexer.queue",
TaskQueueConfig.class);
JsonConfigProvider.bind(binder, "druid.indexer.tasklock",
TaskLockConfig.class);
JsonConfigProvider.bind(binder, "druid.indexer.task",
TaskConfig.class);
diff --git a/services/src/test/java/org/apache/druid/cli/CliOverlordTest.java
b/services/src/test/java/org/apache/druid/cli/CliOverlordTest.java
index 790f6658d92..c8b7e5ceec9 100644
--- a/services/src/test/java/org/apache/druid/cli/CliOverlordTest.java
+++ b/services/src/test/java/org/apache/druid/cli/CliOverlordTest.java
@@ -22,6 +22,7 @@ package org.apache.druid.cli;
import com.google.inject.Injector;
import org.apache.druid.discovery.NodeRole;
import org.apache.druid.guice.StartupInjectorBuilder;
+import org.apache.druid.indexing.overlord.OverlordMode;
import org.apache.druid.metadata.SegmentsMetadataManager;
import org.apache.druid.metadata.segment.SqlSegmentsMetadataManagerV2;
import org.apache.druid.metadata.segment.cache.HeapMemorySegmentMetadataCache;
@@ -52,6 +53,25 @@ public class CliOverlordTest
Assertions.assertTrue(segmentsMetadataManager instanceof
SqlSegmentsMetadataManagerV2);
}
+ @Test
+ public void testStandaloneOverlordIgnoresCoordinatorAsOverlordConfig()
+ {
+ // The Overlord may share a runtime.properties with the Coordinator, e.g.
the
+ // "coordinator-overlord" conf directory of the Docker image
+ System.setProperty("druid.coordinator.asOverlord.enabled", "true");
+ try {
+ final Injector injector = new
StartupInjectorBuilder().forServer().build();
+ final CliOverlord overlord = new CliOverlord();
+ injector.injectMembers(overlord);
+
+ final Injector overlordInjector =
overlord.makeInjector(Set.of(NodeRole.OVERLORD));
+ Assertions.assertEquals(OverlordMode.STANDALONE,
overlordInjector.getInstance(OverlordMode.class));
+ }
+ finally {
+ System.clearProperty("druid.coordinator.asOverlord.enabled");
+ }
+ }
+
@Test
public void testGetDefaultMaxConcurrentRequests()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
