This is an automated email from the ASF dual-hosted git repository.
victoria pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git
The following commit(s) were added to refs/heads/master by this push:
new 9fe7c01c16 Automatic compaction API documentation refactor (#14740)
9fe7c01c16 is described below
commit 9fe7c01c16c7c7079e615157047b1498a6165811
Author: Nhi Pham <[email protected]>
AuthorDate: Mon Aug 21 11:34:41 2023 -0700
Automatic compaction API documentation refactor (#14740)
Co-authored-by: Katya Macedo <[email protected]>
Co-authored-by: Katya Macedo <[email protected]>
Co-authored-by: 317brian <[email protected]>
---
docs/api-reference/automatic-compaction-api.md | 781 +++++++++++++++++++++++--
docs/data-management/automatic-compaction.md | 8 +-
docs/design/coordinator.md | 2 +-
website/.spelling | 1 +
4 files changed, 745 insertions(+), 47 deletions(-)
diff --git a/docs/api-reference/automatic-compaction-api.md
b/docs/api-reference/automatic-compaction-api.md
index ea0f824190..d917cee42e 100644
--- a/docs/api-reference/automatic-compaction-api.md
+++ b/docs/api-reference/automatic-compaction-api.md
@@ -23,69 +23,766 @@ sidebar_label: Automatic compaction
~ under the License.
-->
-This document describes status and configuration API endpoints for [automatic
compaction](../data-management/automatic-compaction.md) in Apache Druid.
+This topic describes the status and configuration API endpoints for [automatic
compaction](../data-management/automatic-compaction.md) in Apache Druid. You
can configure automatic compaction in the Druid web console or API.
-## Automatic compaction status
+In this topic, `http://ROUTER_IP:ROUTER_PORT` is a placeholder for your Router
service address and port. Replace it with the information for your deployment.
For example, use `http://localhost:8888` for quickstart deployments.
-`GET /druid/coordinator/v1/compaction/progress?dataSource={dataSource}`
+## Manage automatic compaction
-Returns the total size of segments awaiting compaction for the given
dataSource. The specified dataSource must have [automatic
compaction](../data-management/automatic-compaction.md) enabled.
+### Create or update automatic compaction configuration
-`GET /druid/coordinator/v1/compaction/status`
+Creates or updates the automatic compaction configuration for a datasource.
Pass the automatic compaction as a JSON object in the request body.
-Returns the status and statistics from the auto-compaction run of all
dataSources which have auto-compaction enabled in the latest run. The response
payload includes a list of `latestStatus` objects. Each `latestStatus`
represents the status for a dataSource (which has/had auto-compaction enabled).
+The automatic compaction configuration requires only the `dataSource`
property. Druid fills all other properties with default values if not
specified. See [Automatic compaction dynamic
configuration](../configuration/index.md#automatic-compaction-dynamic-configuration)
for configuration details.
-The `latestStatus` object has the following keys:
-* `dataSource`: name of the datasource for this status information
-* `scheduleStatus`: auto-compaction scheduling status. Possible values are
`NOT_ENABLED` and `RUNNING`. Returns `RUNNING ` if the dataSource has an active
auto-compaction config submitted. Otherwise, returns `NOT_ENABLED`.
-* `bytesAwaitingCompaction`: total bytes of this datasource waiting to be
compacted by the auto-compaction (only consider intervals/segments that are
eligible for auto-compaction)
-* `bytesCompacted`: total bytes of this datasource that are already compacted
with the spec set in the auto-compaction config
-* `bytesSkipped`: total bytes of this datasource that are skipped (not
eligible for auto-compaction) by the auto-compaction
-* `segmentCountAwaitingCompaction`: total number of segments of this
datasource waiting to be compacted by the auto-compaction (only consider
intervals/segments that are eligible for auto-compaction)
-* `segmentCountCompacted`: total number of segments of this datasource that
are already compacted with the spec set in the auto-compaction config
-* `segmentCountSkipped`: total number of segments of this datasource that are
skipped (not eligible for auto-compaction) by the auto-compaction
-* `intervalCountAwaitingCompaction`: total number of intervals of this
datasource waiting to be compacted by the auto-compaction (only consider
intervals/segments that are eligible for auto-compaction)
-* `intervalCountCompacted`: total number of intervals of this datasource that
are already compacted with the spec set in the auto-compaction config
-* `intervalCountSkipped`: total number of intervals of this datasource that
are skipped (not eligible for auto-compaction) by the auto-compaction
+Note that this endpoint returns an HTTP `200 OK` message code even if the
datasource name does not exist.
-`GET /druid/coordinator/v1/compaction/status?dataSource={dataSource}`
+#### URL
-Similar to the API `/druid/coordinator/v1/compaction/status` above but filters
response to only return information for the dataSource given.
-The dataSource must have auto-compaction enabled.
+<code class="postAPI">POST</code>
<code>/druid/coordinator/v1/config/compaction</code>
-## Automatic compaction configuration
+#### Responses
-`GET /druid/coordinator/v1/config/compaction`
+<!--DOCUSAURUS_CODE_TABS-->
-Returns all automatic compaction configs.
+<!--200 SUCCESS-->
-`GET /druid/coordinator/v1/config/compaction/{dataSource}`
+*Successfully submitted auto compaction configuration*
-Returns an automatic compaction config of a dataSource.
+<!--END_DOCUSAURUS_CODE_TABS-->
-`GET
/druid/coordinator/v1/config/compaction/{dataSource}/history?interval={interval}&count={count}`
+---
+#### Sample request
+
+The following example creates an automatic compaction configuration for the
datasource `wikipedia_hour`, which was ingested with `HOUR` segment
granularity. This automatic compaction configuration performs compaction on
`wikipedia_hour`, resulting in compacted segments that represent a day interval
of data.
+
+In this example:
+
+* `wikipedia_hour` is a datasource with `HOUR` segment granularity.
+* `skipOffsetFromLatest` is set to `PT0S`, meaning that no data is skipped.
+* `partitionsSpec` is set to the default `dynamic`, allowing Druid to
dynamically determine the optimal partitioning strategy.
+* `type` is set to `index_parallel`, meaning that parallel indexing is used.
+* `segmentGranularity` is set to `DAY`, meaning that each compacted segment is
a day of data.
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--cURL-->
+
+```shell
+curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/config/compaction"\
+--header 'Content-Type: application/json' \
+--data '{
+ "dataSource": "wikipedia_hour",
+ "skipOffsetFromLatest": "PT0S",
+ "tuningConfig": {
+ "partitionsSpec": {
+ "type": "dynamic"
+ },
+ "type": "index_parallel"
+ },
+ "granularitySpec": {
+ "segmentGranularity": "DAY"
+ }
+}'
+```
+
+<!--HTTP-->
+
+```HTTP
+POST /druid/coordinator/v1/config/compaction HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+Content-Type: application/json
+Content-Length: 281
+
+{
+ "dataSource": "wikipedia_hour",
+ "skipOffsetFromLatest": "PT0S",
+ "tuningConfig": {
+ "partitionsSpec": {
+ "type": "dynamic"
+ },
+ "type": "index_parallel"
+ },
+ "granularitySpec": {
+ "segmentGranularity": "DAY"
+ }
+}
+```
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+#### Sample response
+
+A successful request returns an HTTP `200 OK` message code and an empty
response body.
+
+
+### Remove automatic compaction configuration
+
+Removes the automatic compaction configuration for a datasource. This updates
the compaction status of the datasource to "Not enabled."
+
+#### URL
+
+<code class="deleteAPI">DELETE</code>
<code>/druid/coordinator/v1/config/compaction/:dataSource</code>
+
+#### Responses
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--200 SUCCESS-->
+
+*Successfully deleted automatic compaction configuration*
+
+<!--404 NOT FOUND-->
+
+*Datasource does not have automatic compaction or invalid datasource name*
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+---
+
+
+#### Sample request
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--cURL-->
+
+```shell
+curl --request DELETE
"http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/config/compaction/wikipedia_hour";
+```
+
+<!--HTTP-->
+
+```HTTP
+DELETE /druid/coordinator/v1/config/compaction/wikipedia_hour HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+```
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+#### Sample response
+
+A successful request returns an HTTP `200 OK` message code and an empty
response body.
+
+### Update capacity for compaction tasks
+
+Updates the capacity for compaction tasks. The minimum number of compaction
tasks is 1 and the maximum is 2147483647.
+
+Note that while the max compaction tasks can theoretically be set to
2147483647, the practical limit is determined by the available cluster capacity
and is capped at 10% of the cluster's total capacity.
+
+#### URL
+
+<code class="postAPI">POST</code>
<code>/druid/coordinator/v1/config/compaction/taskslots</code>
+
+#### Query parameters
+
+To limit the maximum number of compaction tasks, use the optional query
parameters `ratio` and `max`:
+
+* `ratio` (optional)
+ * Type: Float
+ * Default: 0.1
+ * Limits the ratio of the total task slots to compaction task slots.
+* `max` (optional)
+ * Type: Int
+ * Default: 2147483647
+ * Limits the maximum number of task slots for compaction tasks.
+
+#### Responses
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--200 SUCCESS-->
+
+*Successfully updated compaction configuration*
+
+<!--404 NOT FOUND-->
+
+*Invalid `max` value*
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+---
+
+#### Sample request
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--cURL-->
+
+```shell
+curl --request POST
"http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/config/compaction/taskslots?ratio=0.2&max=250000";
+```
+
+<!--HTTP-->
+
+```HTTP
+POST /druid/coordinator/v1/config/compaction/taskslots?ratio=0.2&max=250000
HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+```
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+#### Sample response
+
+A successful request returns an HTTP `200 OK` message code and an empty
response body.
+
+## View automatic compaction configuration
+
+### Get all automatic compaction configurations
+
+Retrieves all automatic compaction configurations. Returns a
`compactionConfigs` object containing the active automatic compaction
configurations of all datasources.
+
+You can use this endpoint to retrieve `compactionTaskSlotRatio` and
`maxCompactionTaskSlots` values for managing resource allocation of compaction
tasks.
+
+#### URL
+
+<code class="getAPI">GET</code>
<code>/druid/coordinator/v1/config/compaction</code>
+
+#### Responses
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--200 SUCCESS-->
+
+*Successfully retrieved automatic compaction configurations*
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+---
+
+#### Sample request
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--cURL-->
+
+```shell
+curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/config/compaction";
+```
+
+<!--HTTP-->
+
+```HTTP
+GET /druid/coordinator/v1/config/compaction HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+```
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+#### Sample response
+
+<details>
+ <summary>Click to show sample response</summary>
+
+```json
+{
+ "compactionConfigs": [
+ {
+ "dataSource": "wikipedia_hour",
+ "taskPriority": 25,
+ "inputSegmentSizeBytes": 100000000000000,
+ "maxRowsPerSegment": null,
+ "skipOffsetFromLatest": "PT0S",
+ "tuningConfig": {
+ "maxRowsInMemory": null,
+ "appendableIndexSpec": null,
+ "maxBytesInMemory": null,
+ "maxTotalRows": null,
+ "splitHintSpec": null,
+ "partitionsSpec": {
+ "type": "dynamic",
+ "maxRowsPerSegment": 5000000,
+ "maxTotalRows": null
+ },
+ "indexSpec": null,
+ "indexSpecForIntermediatePersists": null,
+ "maxPendingPersists": null,
+ "pushTimeout": null,
+ "segmentWriteOutMediumFactory": null,
+ "maxNumConcurrentSubTasks": null,
+ "maxRetry": null,
+ "taskStatusCheckPeriodMs": null,
+ "chatHandlerTimeout": null,
+ "chatHandlerNumRetries": null,
+ "maxNumSegmentsToMerge": null,
+ "totalNumMergeTasks": null,
+ "maxColumnsToMerge": null,
+ "type": "index_parallel",
+ "forceGuaranteedRollup": false
+ },
+ "granularitySpec": {
+ "segmentGranularity": "DAY",
+ "queryGranularity": null,
+ "rollup": null
+ },
+ "dimensionsSpec": null,
+ "metricsSpec": null,
+ "transformSpec": null,
+ "ioConfig": null,
+ "taskContext": null
+ },
+ {
+ "dataSource": "wikipedia",
+ "taskPriority": 25,
+ "inputSegmentSizeBytes": 100000000000000,
+ "maxRowsPerSegment": null,
+ "skipOffsetFromLatest": "PT0S",
+ "tuningConfig": {
+ "maxRowsInMemory": null,
+ "appendableIndexSpec": null,
+ "maxBytesInMemory": null,
+ "maxTotalRows": null,
+ "splitHintSpec": null,
+ "partitionsSpec": {
+ "type": "dynamic",
+ "maxRowsPerSegment": 5000000,
+ "maxTotalRows": null
+ },
+ "indexSpec": null,
+ "indexSpecForIntermediatePersists": null,
+ "maxPendingPersists": null,
+ "pushTimeout": null,
+ "segmentWriteOutMediumFactory": null,
+ "maxNumConcurrentSubTasks": null,
+ "maxRetry": null,
+ "taskStatusCheckPeriodMs": null,
+ "chatHandlerTimeout": null,
+ "chatHandlerNumRetries": null,
+ "maxNumSegmentsToMerge": null,
+ "totalNumMergeTasks": null,
+ "maxColumnsToMerge": null,
+ "type": "index_parallel",
+ "forceGuaranteedRollup": false
+ },
+ "granularitySpec": {
+ "segmentGranularity": "DAY",
+ "queryGranularity": null,
+ "rollup": null
+ },
+ "dimensionsSpec": null,
+ "metricsSpec": null,
+ "transformSpec": null,
+ "ioConfig": null,
+ "taskContext": null
+ }
+ ],
+ "compactionTaskSlotRatio": 0.1,
+ "maxCompactionTaskSlots": 2147483647,
+ "useAutoScaleSlots": false
+}
+```
+</details>
+
+### Get automatic compaction configuration
+
+Retrieves the automatic compaction configuration for a datasource.
-Returns the history of the automatic compaction config for a dataSource.
Optionally accepts `interval` and `count`
-query string parameters to filter by interval and limit the number of results
respectively. If the dataSource does not
-exist or there is no compaction history for the dataSource, an empty list is
returned.
+#### URL
+
+<code class="getAPI">GET</code>
<code>/druid/coordinator/v1/config/compaction/:dataSource</code>
+
+#### Responses
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--200 SUCCESS-->
+
+*Successfully retrieved configuration for datasource*
+
+<!--404 NOT FOUND-->
+
+*Invalid datasource or datasource does not have automatic compaction enabled*
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+---
+
+#### Sample request
+
+The following example retrieves the automatic compaction configuration for
datasource `wikipedia_hour`.
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--cURL-->
+
+```shell
+curl
"http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/config/compaction/wikipedia_hour";
+```
+
+<!--HTTP-->
+
+```HTTP
+GET /druid/coordinator/v1/config/compaction/wikipedia_hour HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+```
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+#### Sample response
+
+<details>
+ <summary>Click to show sample response</summary>
+
+```json
+{
+ "dataSource": "wikipedia_hour",
+ "taskPriority": 25,
+ "inputSegmentSizeBytes": 100000000000000,
+ "maxRowsPerSegment": null,
+ "skipOffsetFromLatest": "PT0S",
+ "tuningConfig": {
+ "maxRowsInMemory": null,
+ "appendableIndexSpec": null,
+ "maxBytesInMemory": null,
+ "maxTotalRows": null,
+ "splitHintSpec": null,
+ "partitionsSpec": {
+ "type": "dynamic",
+ "maxRowsPerSegment": 5000000,
+ "maxTotalRows": null
+ },
+ "indexSpec": null,
+ "indexSpecForIntermediatePersists": null,
+ "maxPendingPersists": null,
+ "pushTimeout": null,
+ "segmentWriteOutMediumFactory": null,
+ "maxNumConcurrentSubTasks": null,
+ "maxRetry": null,
+ "taskStatusCheckPeriodMs": null,
+ "chatHandlerTimeout": null,
+ "chatHandlerNumRetries": null,
+ "maxNumSegmentsToMerge": null,
+ "totalNumMergeTasks": null,
+ "maxColumnsToMerge": null,
+ "type": "index_parallel",
+ "forceGuaranteedRollup": false
+ },
+ "granularitySpec": {
+ "segmentGranularity": "DAY",
+ "queryGranularity": null,
+ "rollup": null
+ },
+ "dimensionsSpec": null,
+ "metricsSpec": null,
+ "transformSpec": null,
+ "ioConfig": null,
+ "taskContext": null
+}
+```
+</details>
+
+### Get automatic compaction configuration history
+
+Retrieves the history of the automatic compaction configuration for a
datasource. Returns an empty list if the datasource does not exist or there is
no compaction history for the datasource.
The response contains a list of objects with the following keys:
-* `globalConfig`: A json object containing automatic compaction config that
applies to the entire cluster.
-* `compactionConfig`: A json object containing the automatic compaction config
for the datasource.
-* `auditInfo`: A json object that contains information about the change made -
like `author`, `comment` and `ip`.
+* `globalConfig`: A JSON object containing automatic compaction configuration
that applies to the entire cluster.
+* `compactionConfig`: A JSON object containing the automatic compaction
configuration for the datasource.
+* `auditInfo`: A JSON object containing information about the change made,
such as `author`, `comment` or `ip`.
* `auditTime`: The date and time when the change was made.
-`POST
/druid/coordinator/v1/config/compaction/taskslots?ratio={someRatio}&max={someMaxSlots}`
+#### URL
+
+<code class="getAPI">GET</code>
<code>/druid/coordinator/v1/config/compaction/:dataSource/history</code>
+
+#### Query parameters
+* `interval` (optional)
+ * Type: ISO-8601
+ * Limits the results within a specified interval. Use `/` as the delimiter
for the interval string.
+* `count` (optional)
+ * Type: Int
+ * Limits the number of results.
+
+#### Responses
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--200 SUCCESS-->
+
+*Successfully retrieved configuration history*
+
+<!--400 BAD REQUEST-->
+
+*Invalid `count` value*
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+---
+
+#### Sample request
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--cURL-->
+
+```shell
+curl
"http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/config/compaction/wikipedia_hour/history";
+```
+
+<!--HTTP-->
+
+```HTTP
+GET /druid/coordinator/v1/config/compaction/wikipedia_hour/history HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+```
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+#### Sample response
+
+<details>
+ <summary>Click to show sample response</summary>
+
+```json
+[
+ {
+ "globalConfig": {
+ "compactionTaskSlotRatio": 0.1,
+ "maxCompactionTaskSlots": 2147483647,
+ "useAutoScaleSlots": false
+ },
+ "compactionConfig": {
+ "dataSource": "wikipedia_hour",
+ "taskPriority": 25,
+ "inputSegmentSizeBytes": 100000000000000,
+ "maxRowsPerSegment": null,
+ "skipOffsetFromLatest": "P1D",
+ "tuningConfig": null,
+ "granularitySpec": {
+ "segmentGranularity": "DAY",
+ "queryGranularity": null,
+ "rollup": null
+ },
+ "dimensionsSpec": null,
+ "metricsSpec": null,
+ "transformSpec": null,
+ "ioConfig": null,
+ "taskContext": null
+ },
+ "auditInfo": {
+ "author": "",
+ "comment": "",
+ "ip": "127.0.0.1"
+ },
+ "auditTime": "2023-07-31T18:15:19.302Z"
+ },
+ {
+ "globalConfig": {
+ "compactionTaskSlotRatio": 0.1,
+ "maxCompactionTaskSlots": 2147483647,
+ "useAutoScaleSlots": false
+ },
+ "compactionConfig": {
+ "dataSource": "wikipedia_hour",
+ "taskPriority": 25,
+ "inputSegmentSizeBytes": 100000000000000,
+ "maxRowsPerSegment": null,
+ "skipOffsetFromLatest": "PT0S",
+ "tuningConfig": {
+ "maxRowsInMemory": null,
+ "appendableIndexSpec": null,
+ "maxBytesInMemory": null,
+ "maxTotalRows": null,
+ "splitHintSpec": null,
+ "partitionsSpec": {
+ "type": "dynamic",
+ "maxRowsPerSegment": 5000000,
+ "maxTotalRows": null
+ },
+ "indexSpec": null,
+ "indexSpecForIntermediatePersists": null,
+ "maxPendingPersists": null,
+ "pushTimeout": null,
+ "segmentWriteOutMediumFactory": null,
+ "maxNumConcurrentSubTasks": null,
+ "maxRetry": null,
+ "taskStatusCheckPeriodMs": null,
+ "chatHandlerTimeout": null,
+ "chatHandlerNumRetries": null,
+ "maxNumSegmentsToMerge": null,
+ "totalNumMergeTasks": null,
+ "maxColumnsToMerge": null,
+ "type": "index_parallel",
+ "forceGuaranteedRollup": false
+ },
+ "granularitySpec": {
+ "segmentGranularity": "DAY",
+ "queryGranularity": null,
+ "rollup": null
+ },
+ "dimensionsSpec": null,
+ "metricsSpec": null,
+ "transformSpec": null,
+ "ioConfig": null,
+ "taskContext": null
+ },
+ "auditInfo": {
+ "author": "",
+ "comment": "",
+ "ip": "127.0.0.1"
+ },
+ "auditTime": "2023-07-31T18:16:16.362Z"
+ }
+]
+```
+</details>
+
+## View automatic compaction status
+
+### Get segments awaiting compaction
+
+Returns the total size of segments awaiting compaction for a given datasource.
Returns a 404 response if a datasource does not have automatic compaction
enabled.
+
+#### URL
+
+<code class="getAPI">GET</code>
<code>/druid/coordinator/v1/compaction/progress?dataSource=:dataSource</code>
+
+#### Query parameter
+* `dataSource` (required)
+ * Type: String
+ * Name of the datasource for this status information.
+
+#### Responses
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--200 SUCCESS-->
+
+*Successfully retrieved segment size awaiting compaction*
+
+<!--404 NOT FOUND-->
+
+*Unknown datasource name or datasource does not have automatic compaction
enabled*
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+---
+
+#### Sample request
+
+The following example retrieves the remaining segments to be compacted for
datasource `wikipedia_hour`.
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--cURL-->
+
+```shell
+curl
"http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/compaction/progress?dataSource=wikipedia_hour";
+```
+
+<!--HTTP-->
+
+```HTTP
+GET /druid/coordinator/v1/compaction/progress?dataSource=wikipedia_hour
HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+```
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+#### Sample response
+
+<details>
+ <summary>Click to show sample response</summary>
+
+```json
+{
+ "remainingSegmentSize": 7615837
+}
+```
+</details>
+
+
+### Get compaction status and statistics
+
+Retrieves an array of `latestStatus` objects representing the status and
statistics from the latest automatic compaction run for all datasources with
automatic compaction enabled.
+
+The `latestStatus` object has the following properties:
+* `dataSource`: Name of the datasource for this status information.
+* `scheduleStatus`: Automatic compaction scheduling status. Possible values
are `NOT_ENABLED` and `RUNNING`. Returns `RUNNING ` if the datasource has an
active automatic compaction configuration submitted. Otherwise, returns
`NOT_ENABLED`.
+* `bytesAwaitingCompaction`: Total bytes of this datasource waiting to be
compacted by the automatic compaction (only consider intervals/segments that
are eligible for automatic compaction).
+* `bytesCompacted`: Total bytes of this datasource that are already compacted
with the spec set in the automatic compaction configuration.
+* `bytesSkipped`: Total bytes of this datasource that are skipped (not
eligible for automatic compaction) by the automatic compaction.
+* `segmentCountAwaitingCompaction`: Total number of segments of this
datasource waiting to be compacted by the automatic compaction (only consider
intervals/segments that are eligible for automatic compaction).
+* `segmentCountCompacted`: Total number of segments of this datasource that
are already compacted with the spec set in the automatic compaction
configuration.
+* `segmentCountSkipped`: Total number of segments of this datasource that are
skipped (not eligible for automatic compaction) by the automatic compaction.
+* `intervalCountAwaitingCompaction`: Total number of intervals of this
datasource waiting to be compacted by the automatic compaction (only consider
intervals/segments that are eligible for automatic compaction).
+* `intervalCountCompacted`: Total number of intervals of this datasource that
are already compacted with the spec set in the automatic compaction
configuration.
+* `intervalCountSkipped`: Total number of intervals of this datasource that
are skipped (not eligible for automatic compaction) by the automatic compaction.
+
+#### URL
+
+<code class="getAPI">GET</code>
<code>/druid/coordinator/v1/compaction/status</code>
+
+#### Query parameters
+* `dataSource` (optional)
+ * Type: String
+ * Filter the result by name of a specific datasource.
+
+#### Responses
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--200 SUCCESS-->
+
+*Successfully retrieved `latestStatus` object*
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+---
+#### Sample request
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--cURL-->
+
+```shell
+curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/compaction/status";
+```
+
+<!--HTTP-->
-Update the capacity for compaction tasks. `ratio` and `max` are used to limit
the max number of compaction tasks.
-They mean the ratio of the total task slots to the compaction task slots and
the maximum number of task slots for compaction tasks, respectively. The actual
max number of compaction tasks is `min(max, ratio * total task slots)`.
-Note that `ratio` and `max` are optional and can be omitted. If they are
omitted, default values (0.1 and unbounded)
-will be set for them.
+```HTTP
+GET /druid/coordinator/v1/compaction/status HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+```
-`POST /druid/coordinator/v1/config/compaction`
+<!--END_DOCUSAURUS_CODE_TABS-->
-Creates or updates the [automatic
compaction](../data-management/automatic-compaction.md) config for a
dataSource. See [Automatic compaction dynamic
configuration](../configuration/index.md#automatic-compaction-dynamic-configuration)
for configuration details.
+#### Sample response
-`DELETE /druid/coordinator/v1/config/compaction/{dataSource}`
+<details>
+ <summary>Click to show sample response</summary>
-Removes the automatic compaction config for a dataSource.
+```json
+{
+ "latestStatus": [
+ {
+ "dataSource": "wikipedia_api",
+ "scheduleStatus": "RUNNING",
+ "bytesAwaitingCompaction": 0,
+ "bytesCompacted": 0,
+ "bytesSkipped": 64133616,
+ "segmentCountAwaitingCompaction": 0,
+ "segmentCountCompacted": 0,
+ "segmentCountSkipped": 8,
+ "intervalCountAwaitingCompaction": 0,
+ "intervalCountCompacted": 0,
+ "intervalCountSkipped": 1
+ },
+ {
+ "dataSource": "wikipedia_hour",
+ "scheduleStatus": "RUNNING",
+ "bytesAwaitingCompaction": 0,
+ "bytesCompacted": 5998634,
+ "bytesSkipped": 0,
+ "segmentCountAwaitingCompaction": 0,
+ "segmentCountCompacted": 1,
+ "segmentCountSkipped": 0,
+ "intervalCountAwaitingCompaction": 0,
+ "intervalCountCompacted": 1,
+ "intervalCountSkipped": 0
+ }
+ ]
+}
+```
+</details>
diff --git a/docs/data-management/automatic-compaction.md
b/docs/data-management/automatic-compaction.md
index cdf78369e9..1a27176986 100644
--- a/docs/data-management/automatic-compaction.md
+++ b/docs/data-management/automatic-compaction.md
@@ -59,10 +59,10 @@ To disable auto-compaction for a datasource, click
**Delete** from the **Compact
### Compaction configuration API
-Use the [Automatic compaction
API](../api-reference/automatic-compaction-api.md#automatic-compaction-status)
to configure automatic compaction.
+Use the [Automatic compaction
API](../api-reference/automatic-compaction-api.md#manage-automatic-compaction)
to configure automatic compaction.
To enable auto-compaction for a datasource, create a JSON object with the
desired auto-compaction settings.
See [Configure automatic compaction](#configure-automatic-compaction) for the
syntax of an auto-compaction spec.
-Send the JSON object as a payload in a [`POST`
request](../api-reference/automatic-compaction-api.md#automatic-compaction-configuration)
to `/druid/coordinator/v1/config/compaction`.
+Send the JSON object as a payload in a [`POST`
request](../api-reference/automatic-compaction-api.md#create-or-update-automatic-compaction-configuration)
to `/druid/coordinator/v1/config/compaction`.
The following example configures auto-compaction for the `wikipedia`
datasource:
```sh
@@ -76,7 +76,7 @@ curl --location --request POST
'http://localhost:8081/druid/coordinator/v1/confi
}'
```
-To disable auto-compaction for a datasource, send a [`DELETE`
request](../api-reference/automatic-compaction-api.md#automatic-compaction-configuration)
to `/druid/coordinator/v1/config/compaction/{dataSource}`. Replace
`{dataSource}` with the name of the datasource for which to disable
auto-compaction. For example:
+To disable auto-compaction for a datasource, send a [`DELETE`
request](../api-reference/automatic-compaction-api.md#remove-automatic-compaction-configuration)
to `/druid/coordinator/v1/config/compaction/{dataSource}`. Replace
`{dataSource}` with the name of the datasource for which to disable
auto-compaction. For example:
```sh
curl --location --request DELETE
'http://localhost:8081/druid/coordinator/v1/config/compaction/wikipedia'
@@ -152,7 +152,7 @@ After the Coordinator has initiated auto-compaction, you
can view compaction sta
In the web console, the Datasources view displays auto-compaction statistics.
The Tasks view shows the task information for compaction tasks that were
triggered by the automatic compaction system.
-To get statistics by API, send a [`GET`
request](../api-reference/automatic-compaction-api.md#automatic-compaction-status)
to `/druid/coordinator/v1/compaction/status`. To filter the results to a
particular datasource, pass the datasource name as a query parameter to the
request—for example,
`/druid/coordinator/v1/compaction/status?dataSource=wikipedia`.
+To get statistics by API, send a [`GET`
request](../api-reference/automatic-compaction-api.md#view-automatic-compaction-status)
to `/druid/coordinator/v1/compaction/status`. To filter the results to a
particular datasource, pass the datasource name as a query parameter to the
request—for example,
`/druid/coordinator/v1/compaction/status?dataSource=wikipedia`.
## Examples
diff --git a/docs/design/coordinator.md b/docs/design/coordinator.md
index 625a0cadfb..8e4536271e 100644
--- a/docs/design/coordinator.md
+++ b/docs/design/coordinator.md
@@ -92,7 +92,7 @@ Once some segments are found, it issues a [compaction
task](../ingestion/tasks.m
The maximum number of running compaction tasks is `min(sum of worker capacity
* slotRatio, maxSlots)`.
Note that even if `min(sum of worker capacity * slotRatio, maxSlots) = 0`, at
least one compaction task is always submitted
if the compaction is enabled for a dataSource.
-See [Automatic compaction configuration
API](../api-reference/automatic-compaction-api.md#automatic-compaction-configuration)
and [Automatic compaction
configuration](../configuration/index.md#automatic-compaction-dynamic-configuration)
to enable and configure automatic compaction.
+See [Automatic compaction configuration
API](../api-reference/automatic-compaction-api.md#manage-automatic-compaction)
and [Automatic compaction
configuration](../configuration/index.md#automatic-compaction-dynamic-configuration)
to enable and configure automatic compaction.
Compaction tasks might fail due to the following reasons:
diff --git a/website/.spelling b/website/.spelling
index 200aec5914..32f7786b1b 100644
--- a/website/.spelling
+++ b/website/.spelling
@@ -2313,5 +2313,6 @@ terminateAll
selfDiscovered
loadstatus
isLeader
+taskslots
loadstatus
sqlQueryId
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
