This is an automated email from the ASF dual-hosted git repository.
brile pushed a commit to branch 35.0.0
in repository https://gitbox.apache.org/repos/asf/druid.git
The following commit(s) were added to refs/heads/35.0.0 by this push:
new 1cbd5d0f1f5 docs: msq not an extension anymore (#18725) (#18726)
1cbd5d0f1f5 is described below
commit 1cbd5d0f1f50ebce2c95710c3b4c0eec3e376b7f
Author: 317brian <[email protected]>
AuthorDate: Fri Nov 7 08:22:41 2025 -0800
docs: msq not an extension anymore (#18725) (#18726)
---
docs/data-management/automatic-compaction.md | 1 -
docs/multi-stage-query/concepts.md | 15 +++------------
docs/multi-stage-query/examples.md | 7 ++-----
docs/multi-stage-query/index.md | 16 ++++------------
docs/multi-stage-query/known-issues.md | 6 ------
docs/multi-stage-query/reference.md | 6 ------
docs/multi-stage-query/security.md | 6 ------
docs/querying/query-from-deep-storage.md | 2 --
8 files changed, 9 insertions(+), 50 deletions(-)
diff --git a/docs/data-management/automatic-compaction.md
b/docs/data-management/automatic-compaction.md
index 1a0803bafb2..97d7074586c 100644
--- a/docs/data-management/automatic-compaction.md
+++ b/docs/data-management/automatic-compaction.md
@@ -307,7 +307,6 @@ To stop the automatic compaction task, suspend or terminate
the supervisor throu
The MSQ task engine is available as a compaction engine if you configure
auto-compaction to use compaction supervisors. To use the MSQ task engine for
automatic compaction, make sure the following requirements are met:
-* [Load the MSQ task engine
extension](../multi-stage-query/index.md#load-the-extension).
* In your Overlord runtime properties, set the following properties:
* `druid.supervisor.compaction.enabled` to `true` so that compaction tasks
can be run as a supervisor task.
* Optionally, set `druid.supervisor.compaction.engine` to `msq` to specify
the MSQ task engine as the default compaction engine. If you don't do this,
you'll need to set `spec.engine` to `msq` for each compaction supervisor spec
where you want to use the MSQ task engine.
diff --git a/docs/multi-stage-query/concepts.md
b/docs/multi-stage-query/concepts.md
index dee6f80bdd4..cbee68eb8b2 100644
--- a/docs/multi-stage-query/concepts.md
+++ b/docs/multi-stage-query/concepts.md
@@ -23,21 +23,12 @@ sidebar_label: "Key concepts"
~ under the License.
-->
-:::info
- This page describes SQL-based batch ingestion using the
[`druid-multi-stage-query`](../multi-stage-query/index.md)
- extension, new in Druid 24.0. Refer to the [ingestion
methods](../ingestion/index.md#batch) table to determine which
- ingestion method is right for you.
-:::
+ This page describes SQL-based batch ingestion using the [multi-stage query
(MSQ) task engine](../multi-stage-query/index.md). Refer to the [ingestion
methods](../ingestion/index.md#batch) table to determine which ingestion method
is right for you.
## Multi-stage query task engine
-The `druid-multi-stage-query` extension adds a multi-stage query (MSQ) task
engine that executes SQL statements as batch
-tasks in the indexing service, which execute on [Middle
Managers](../design/architecture.md#druid-services).
-[INSERT](reference.md#insert) and [REPLACE](reference.md#replace) tasks publish
-[segments](../design/storage.md) just like [all other forms of batch
-ingestion](../ingestion/index.md#batch). Each query occupies at least two task
slots while running: one controller task,
-and at least one worker task. As an experimental feature, the MSQ task engine
also supports running SELECT queries as
-batch tasks. The behavior and result format of plain SELECT (without INSERT or
REPLACE) is subject to change.
+The MSQ task engine executes SQL statements as batch tasks in the indexing
service, which execute on [Middle
Managers](../design/architecture.md#druid-services).
+[INSERT](reference.md#insert) and [REPLACE](reference.md#replace) tasks
publish [segments](../design/storage.md) just like [all other forms of batch
ingestion](../ingestion/index.md#batch). Each query occupies at least two task
slots while running: one controller task, and at least one worker task. As an
experimental feature, the MSQ task engine also supports running SELECT queries
as batch tasks. The behavior and result format of plain SELECT (without INSERT
or REPLACE) is subject to change.
You can execute SQL statements using the MSQ task engine through the **Query**
view in the [web
console](../operations/web-console.md) or through the [`/druid/v2/sql/task`
API](../api-reference/sql-ingestion-api.md).
diff --git a/docs/multi-stage-query/examples.md
b/docs/multi-stage-query/examples.md
index d81e874d1f7..797af2ed351 100644
--- a/docs/multi-stage-query/examples.md
+++ b/docs/multi-stage-query/examples.md
@@ -23,11 +23,8 @@ sidebar_label: Examples
~ under the License.
-->
-:::info
- This page describes SQL-based batch ingestion using the
[`druid-multi-stage-query`](../multi-stage-query/index.md)
- extension, new in Druid 24.0. Refer to the [ingestion
methods](../ingestion/index.md#batch) table to determine which
- ingestion method is right for you.
-:::
+
+This page describes SQL-based batch ingestion using the [multi-stage query
task engine](../multi-stage-query/index.md) (MSQ task engine). Refer to the
[ingestion methods](../ingestion/index.md#batch) table to determine which
ingestion method is right for you.
These example queries show you some of the things you can do when modifying
queries for your use case. Copy the example queries into the **Query** view of
the web console and run them to see what they do.
diff --git a/docs/multi-stage-query/index.md b/docs/multi-stage-query/index.md
index 39bb40748a1..63f48c4e4ca 100644
--- a/docs/multi-stage-query/index.md
+++ b/docs/multi-stage-query/index.md
@@ -24,14 +24,9 @@ description: Introduces multi-stage query architecture and
its task engine
~ under the License.
-->
-:::info
- This page describes SQL-based batch ingestion using the
[`druid-multi-stage-query`](../multi-stage-query/index.md)
- extension, new in Druid 24.0. Refer to the [ingestion
methods](../ingestion/index.md#batch) table to determine which
- ingestion method is right for you.
-:::
+This page describes SQL-based batch ingestion using the [multi-stage query
(MSQ) task engine](../multi-stage-query/index.md). Refer to the [ingestion
methods](../ingestion/index.md#batch) table to determine which ingestion method
is right for you.
-Apache Druid supports SQL-based ingestion using the bundled
[`druid-multi-stage-query` extension](#load-the-extension).
-This extension adds a [multi-stage query task engine for
SQL](concepts.md#multi-stage-query-task-engine) that allows running SQL
+Apache® Druid supports SQL-based ingestion using MSQ task engine that allows
running SQL
[INSERT](concepts.md#load-data-with-insert) and
[REPLACE](concepts.md#overwrite-data-with-replace) statements as batch tasks.
As an experimental feature,
the task engine also supports running `SELECT` queries as batch tasks.
@@ -59,12 +54,9 @@ transformation: creating new tables based on queries of
other tables.
- **Shuffle**: Workers exchange data between themselves on a per-partition
basis in a process called
shuffling. During a shuffle, each output partition is sorted by a clustering
key.
-## Load the extension
+## External resource types
-To add the extension to an existing cluster, add `druid-multi-stage-query` to
`druid.extensions.loadlist` in your
-`common.runtime.properties` file.
-
-For more information about how to load an extension, see [Loading
extensions](../configuration/extensions.md#loading-extensions).
+The MSQ task engine supports reading and writing data from external sources
through the EXTERN function.
To use [EXTERN](reference.md#extern-function), you need READ permission on the
resource named "EXTERNAL" of the resource type
"EXTERNAL". If you encounter a 403 error when trying to use `EXTERN`, verify
that you have the correct permissions.
diff --git a/docs/multi-stage-query/known-issues.md
b/docs/multi-stage-query/known-issues.md
index 76649d5ab8c..7b32b9486ce 100644
--- a/docs/multi-stage-query/known-issues.md
+++ b/docs/multi-stage-query/known-issues.md
@@ -23,12 +23,6 @@ sidebar_label: Known issues
~ under the License.
-->
-:::info
- This page describes SQL-based batch ingestion using the
[`druid-multi-stage-query`](../multi-stage-query/index.md)
- extension, new in Druid 24.0. Refer to the [ingestion
methods](../ingestion/index.md#batch) table to determine which
- ingestion method is right for you.
-:::
-
## Multi-stage query task runtime
- Fault tolerance is partially implemented. Workers get relaunched when they
are killed unexpectedly. The controller does not get relaunched if it is killed
unexpectedly.
diff --git a/docs/multi-stage-query/reference.md
b/docs/multi-stage-query/reference.md
index a3ecd01abe6..ab007d71f6a 100644
--- a/docs/multi-stage-query/reference.md
+++ b/docs/multi-stage-query/reference.md
@@ -23,12 +23,6 @@ sidebar_label: Reference
~ under the License.
-->
-:::info
- This page describes SQL-based batch ingestion using the
[`druid-multi-stage-query`](../multi-stage-query/index.md)
- extension, new in Druid 24.0. Refer to the [ingestion
methods](../ingestion/index.md#batch) table to determine which
- ingestion method is right for you.
-:::
-
## SQL reference
This topic is a reference guide for the multi-stage query architecture in
Apache Druid. For examples of real-world
diff --git a/docs/multi-stage-query/security.md
b/docs/multi-stage-query/security.md
index d98695ebedc..77acafc29f5 100644
--- a/docs/multi-stage-query/security.md
+++ b/docs/multi-stage-query/security.md
@@ -23,12 +23,6 @@ sidebar_label: Security
~ under the License.
-->
-:::info
- This page describes SQL-based batch ingestion using the
[`druid-multi-stage-query`](../multi-stage-query/index.md)
- extension, new in Druid 24.0. Refer to the [ingestion
methods](../ingestion/index.md#batch) table to determine which
- ingestion method is right for you.
-:::
-
All authenticated users can use the multi-stage query task engine (MSQ task
engine) through the UI and API if the
extension is loaded. However, without additional permissions, users are not
able to issue queries that read or write
Druid datasources or external data. The permission needed depends on what the
user is trying to do.
diff --git a/docs/querying/query-from-deep-storage.md
b/docs/querying/query-from-deep-storage.md
index ba34d116a3b..6e3f545e27e 100644
--- a/docs/querying/query-from-deep-storage.md
+++ b/docs/querying/query-from-deep-storage.md
@@ -26,8 +26,6 @@ Druid can query segments that are only stored in deep
storage. Running a query f
## Prerequisites
-Query from deep storage requires the Multi-stage query (MSQ) task engine. Load
the extension for it if you don't already have it enabled before you begin. See
[enable MSQ](../multi-stage-query/index.md#load-the-extension) for more
information.
-
To be queryable, your datasource must meet one of the following conditions:
- At least one segment from the datasource is loaded onto a Historical service
for Druid to plan the query. This segment can be any segment from the
datasource. You can verify that a datasource has at least one segment on a
Historical service if it's visible in the Druid console.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
