This is an automated email from the ASF dual-hosted git repository.
kfaraz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git
The following commit(s) were added to refs/heads/master by this push:
new 8c75ad649d9 Remove "experimental" label from compaction supervisors
and MSQ compaction engine (#19113)
8c75ad649d9 is described below
commit 8c75ad649d9b8157f1a166da92c6509f2174c033
Author: Kashif Faraz <[email protected]>
AuthorDate: Wed Mar 11 09:40:24 2026 +0530
Remove "experimental" label from compaction supervisors and MSQ compaction
engine (#19113)
Changes:
- Remove experimental tag from compaction supervisors and MSQ compaction
engine
- Update docs and remove mention of old properties
- Override default engine in provided in the supervisor spec
---
docs/data-management/automatic-compaction.md | 17 +++++++----------
.../IngestionBackwardCompatibilityDockerTest.java | 10 ++++++++++
.../testing/embedded/indexing/IngestionSmokeTest.java | 10 ++++++++++
.../compact/CompactionConfigBasedJobTemplate.java | 8 +++++++-
4 files changed, 34 insertions(+), 11 deletions(-)
diff --git a/docs/data-management/automatic-compaction.md
b/docs/data-management/automatic-compaction.md
index f82c66d70fe..e0ed7818cab 100644
--- a/docs/data-management/automatic-compaction.md
+++ b/docs/data-management/automatic-compaction.md
@@ -52,7 +52,7 @@ The automatic compaction system uses the following syntax:
}
```
-:::info[Experimental]
+:::info
The MSQ task engine is available as a compaction engine when you run automatic
compaction as a compaction supervisor. For more information, see
[Auto-compaction using compaction
supervisors](#auto-compaction-using-compaction-supervisors).
@@ -230,10 +230,6 @@ The following auto-compaction configuration compacts
updates the `wikipedia` seg
## Auto-compaction using compaction supervisors
-:::info[Experimental]
-Compaction supervisors are experimental. For production use, we recommend
[auto-compaction using Coordinator
duties](#auto-compaction-using-coordinator-duties).
-:::
-
You can run automatic compaction using compaction supervisors on the Overlord
rather than Coordinator duties. Compaction supervisors provide the following
benefits over Coordinator duties:
* Can use the supervisor framework to get information about the
auto-compaction, such as status or state
@@ -305,7 +301,7 @@ curl --location --request POST
'http://localhost:8081/druid/indexer/v1/superviso
}'
```
-Note that if you omit `spec.engine`, Druid uses the default compaction engine.
You can control the default compaction engine with the
`druid.supervisor.compaction.engine` Overlord runtime property. If
`spec.engine` and `druid.supervisor.compaction.engine` are omitted, Druid
defaults to the native engine.
+Note that if you omit `spec.engine`, Druid uses the default compaction engine.
You can control the default compaction engine by setting `engine` in the
[compaction dynamic
config](../api-reference/automatic-compaction-api.md#update-cluster-level-compaction-config).
If `spec.engine` and `engine` in compaction dynamic config are omitted, Druid
defaults to the `native` engine.
To stop the automatic compaction task, suspend or terminate the supervisor
through the UI or API.
@@ -313,10 +309,11 @@ To stop the automatic compaction task, suspend or
terminate the supervisor throu
The MSQ task engine is available as a compaction engine if you configure
auto-compaction to use compaction supervisors. To use the MSQ task engine for
automatic compaction, make sure the following requirements are met:
-* In your Overlord runtime properties, set the following properties:
- * `druid.supervisor.compaction.enabled` to `true` so that compaction tasks
can be run as a supervisor task.
- * Optionally, set `druid.supervisor.compaction.engine` to `msq` to specify
the MSQ task engine as the default compaction engine. If you don't do this,
you'll need to set `spec.engine` to `msq` for each compaction supervisor spec
where you want to use the MSQ task engine.
-* Have at least two compaction task slots available or set
`compactionConfig.taskContext.maxNumTasks` to two or more. The MSQ task engine
requires at least two tasks to run, one controller task and one worker task.
+* Enable [incremental segment metadata
caching](../configuration/index.md#metadata-retrieval) on the Overlord.
+* Enable [Auto-compaction using compaction
supervisors](#auto-compaction-using-compaction-supervisors).
+* Update the [compaction dynamic
config](../api-reference/automatic-compaction-api.md#update-cluster-level-compaction-config)
and set `engine` to `msq` to use the MSQ task engine as the default compaction
engine for all compaction supervisors.
+ * Alternatively, you may override the default engine by setting
`spec.engine` to `msq` in the compaction supervisor for the relevant datasource.
+* Have at least two compaction task slots available or set
`spec.taskContext.maxNumTasks` to two or more. The MSQ task engine requires at
least two tasks to run, one controller task and one worker task.
You can use [MSQ task engine context
parameters](../multi-stage-query/reference.md#context-parameters) in
`spec.taskContext` when configuring your datasource for automatic compaction,
such as setting the maximum number of tasks using the
`spec.taskContext.maxNumTasks` parameter. Some of the MSQ task engine context
parameters overlap with automatic compaction parameters. When these settings
overlap, set one or the other.
diff --git
a/embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/IngestionBackwardCompatibilityDockerTest.java
b/embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/IngestionBackwardCompatibilityDockerTest.java
index 757aa886a81..45b2ee7d2ed 100644
---
a/embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/IngestionBackwardCompatibilityDockerTest.java
+++
b/embedded-tests/src/test/java/org/apache/druid/testing/embedded/docker/IngestionBackwardCompatibilityDockerTest.java
@@ -110,4 +110,14 @@ public class IngestionBackwardCompatibilityDockerTest
extends IngestionSmokeTest
throw new RuntimeException(e);
}
}
+
+ @Override
+ protected void waitForNextCoordinatorCacheSync()
+ {
+ // Wait for full Coordinator cache sync (Druid 31 does not support
incremental cache)
+ eventCollector.latchableEmitter().waitForNextEvent(
+ event -> event.hasMetricName("segment/poll/time")
+ .hasService("druid/coordinator")
+ );
+ }
}
diff --git
a/embedded-tests/src/test/java/org/apache/druid/testing/embedded/indexing/IngestionSmokeTest.java
b/embedded-tests/src/test/java/org/apache/druid/testing/embedded/indexing/IngestionSmokeTest.java
index 0c71c63f451..e6b81cf4924 100644
---
a/embedded-tests/src/test/java/org/apache/druid/testing/embedded/indexing/IngestionSmokeTest.java
+++
b/embedded-tests/src/test/java/org/apache/druid/testing/embedded/indexing/IngestionSmokeTest.java
@@ -114,6 +114,7 @@ public class IngestionSmokeTest extends
EmbeddedClusterTestBase
.addResource(new PostgreSQLMetadataResource())
.addResource(new MinIOStorageResource())
.addResource(kafkaServer)
+ .addCommonProperty("druid.manager.segments.useIncrementalCache",
"always")
.addCommonProperty("druid.emitter", "http")
.addCommonProperty("druid.emitter.http.recipientBaseUrl",
eventCollector.getMetricsUrl())
.addCommonProperty("druid.emitter.http.flushMillis", "500")
@@ -408,12 +409,21 @@ public class IngestionSmokeTest extends
EmbeddedClusterTestBase
.hasService("druid/broker")
.hasDimension(DruidMetrics.DATASOURCE, dataSource)
);
+ waitForNextCoordinatorCacheSync();
eventCollector.latchableEmitter().waitForNextEvent(
event -> event.hasMetricName("segment/metadataCache/sync/time")
.hasService("druid/broker")
);
}
+ protected void waitForNextCoordinatorCacheSync()
+ {
+ eventCollector.latchableEmitter().waitForNextEvent(
+ event -> event.hasMetricName("segment/metadataCache/sync/time")
+ .hasService("druid/coordinator")
+ );
+ }
+
/**
* Verifies the total number of used segments in {@link #dataSource}.
*/
diff --git
a/indexing-service/src/main/java/org/apache/druid/indexing/compact/CompactionConfigBasedJobTemplate.java
b/indexing-service/src/main/java/org/apache/druid/indexing/compact/CompactionConfigBasedJobTemplate.java
index 25643231b2c..1fa48f2b621 100644
---
a/indexing-service/src/main/java/org/apache/druid/indexing/compact/CompactionConfigBasedJobTemplate.java
+++
b/indexing-service/src/main/java/org/apache/druid/indexing/compact/CompactionConfigBasedJobTemplate.java
@@ -20,8 +20,10 @@
package org.apache.druid.indexing.compact;
import org.apache.druid.client.indexing.ClientCompactionTaskQuery;
+import org.apache.druid.common.config.Configs;
import org.apache.druid.error.DruidException;
import org.apache.druid.error.InvalidInput;
+import org.apache.druid.indexer.CompactionEngine;
import org.apache.druid.indexing.input.DruidInputSource;
import org.apache.druid.java.util.common.Intervals;
import org.apache.druid.java.util.common.granularity.Granularity;
@@ -95,10 +97,14 @@ public class CompactionConfigBasedJobTemplate implements
CompactionJobTemplate
// Allow template-specific customization of the config per candidate
DataSourceCompactionConfig finalConfig =
configOptimizer.optimizeConfig(config, candidate, params);
+ final CompactionEngine engine = Configs.valueOrDefault(
+ finalConfig.getEngine(),
+ params.getClusterCompactionConfig().getEngine()
+ );
ClientCompactionTaskQuery taskPayload =
CompactSegments.createCompactionTask(
candidate,
finalConfig,
- params.getClusterCompactionConfig().getEngine(),
+ engine,
indexingStateFingerprint,
params.getClusterCompactionConfig().isStoreCompactionStatePerSegment()
);
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
