ektravel commented on code in PR #14579: URL: https://github.com/apache/druid/pull/14579#discussion_r1265933262
########## docs/api-reference/supervisor-api.md: ########## @@ -23,89 +23,3075 @@ sidebar_label: Supervisors ~ under the License. --> -This document describes the API endpoints to manage and monitor supervisors for Apache Druid. +This document describes the API endpoints to manage and monitor Supervisors for Apache Druid. -## Supervisors +In this document, `http://<SERVICE_IP>:<SERVICE_PORT>` is a placeholder for the server address of deployment and the service port. For example, on the quickstart configuration, replace `http://<ROUTER_IP>:<ROUTER_PORT>` with `http://localhost:8888`. -`GET /druid/indexer/v1/supervisor` +## Supervisor information -Returns a list of strings of the currently active supervisor ids. +### Get an array of active Supervisors -`GET /druid/indexer/v1/supervisor?full` +#### URL -Returns a list of objects of the currently active supervisors. +<code className="getAPI">GET</code> `/druid/indexer/v1/supervisor` + +Retrieves an array of active Supervisors. Returns a response object that includes an array of strings representing the names of the active Supervisors. If there are no active Supervisors, it returns an empty array. + +#### Responses + +<!--DOCUSAURUS_CODE_TABS--> + +<!--200 SUCCESS--> +<br/> +*Successfully retrieved array of active Supervisor IDs* + +<!--END_DOCUSAURUS_CODE_TABS--> + +--- + +#### Sample request + +<!--DOCUSAURUS_CODE_TABS--> + +<!--cURL--> +```shell +curl "http://<ROUTER_IP>:<ROUTER_PORT>/druid/indexer/v1/supervisor" +``` +<!--HTTP--> +```HTTP +GET /druid/indexer/v1/supervisor HTTP/1.1 +Host: http://<ROUTER_IP>:<ROUTER_PORT> +``` +<!--END_DOCUSAURUS_CODE_TABS--> + +#### Sample response + +<details> + <summary>Click to show sample response</summary> + + ```json + [ + "wikipedia_stream", + "social_media" + ] + ``` +</details> + +### Get an array of active Supervisor objects + +#### URL + +<code className="getAPI">GET</code> `/druid/indexer/v1/supervisor?full` + +Retrieves an array of active Supervisor objects. Each object has properties relevant to the Supervisor: `id`, `state`, `detailedState`, `healthy`, and `spec`. + +See the following table for details on properties within the response object: |Field|Type|Description| |---|---|---| -|`id`|String|supervisor unique identifier| -|`state`|String|basic state of the supervisor. Available states:`UNHEALTHY_SUPERVISOR`, `UNHEALTHY_TASKS`, `PENDING`, `RUNNING`, `SUSPENDED`, `STOPPING`. Check [Kafka Docs](../development/extensions-core/kafka-supervisor-operations.md) for details.| -|`detailedState`|String|supervisor specific state. See documentation of specific supervisor for details: [Kafka](../development/extensions-core/kafka-ingestion.md) or [Kinesis](../development/extensions-core/kinesis-ingestion.md)| -|`healthy`|Boolean|true or false indicator of overall supervisor health| -|`spec`|SupervisorSpec|JSON specification of supervisor| +|`id`|String|Supervisor unique identifier| +|`state`|String|The basic state of the supervisor. Available states:`UNHEALTHY_SUPERVISOR`, `UNHEALTHY_TASKS`, `PENDING`, `RUNNING`, `SUSPENDED`, `STOPPING`. Check [Kafka Docs](../development/extensions-core/kafka-supervisor-operations.md) for details.| +|`detailedState`|String|The Supervisor specific state. See documentation of specific supervisor for details: [Kafka](../development/extensions-core/kafka-ingestion.md) or [Kinesis](../development/extensions-core/kinesis-ingestion.md)| +|`healthy`|Boolean|True or false indicator of overall Supervisor health| +|`spec`|SupervisorSpec|JSON specification of the Supervisor| + +#### Responses + +<!--DOCUSAURUS_CODE_TABS--> + +<!--200 SUCCESS--> +<br/> +*Successfully retrieved active Supervisor objects* + +<!--END_DOCUSAURUS_CODE_TABS--> + +--- + +#### Sample request + +<!--DOCUSAURUS_CODE_TABS--> + +<!--cURL--> +```shell +curl "http://<ROUTER_IP>:<ROUTER_PORT>/druid/indexer/v1/supervisor?full=null" +``` +<!--HTTP--> +```HTTP +GET /druid/indexer/v1/supervisor?full=null HTTP/1.1 +Host: http://<ROUTER_IP>:<ROUTER_PORT> +``` +<!--END_DOCUSAURUS_CODE_TABS--> + +#### Sample response + +<details> + <summary>Click to show sample response</summary> + + ```json + [ + { + "id": "wikipedia_stream", + "state": "RUNNING", + "detailedState": "CONNECTING_TO_STREAM", + "healthy": true, + "spec": { + "type": "kafka", + "spec": { + "dataSchema": { + "dataSource": "wikipedia_stream", + "timestampSpec": { + "column": "__time", + "format": "iso", + "missingValue": null + }, + "dimensionsSpec": { + "dimensions": [ + { + "type": "string", + "name": "username", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + }, + { + "type": "string", + "name": "post_title", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + }, + { + "type": "long", + "name": "views", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "long", + "name": "upvotes", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "long", + "name": "comments", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "string", + "name": "edited", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + } + ], + "dimensionExclusions": [ + "__time" + ], + "includeAllDimensions": false, + "useSchemaDiscovery": false + }, + "metricsSpec": [], + "granularitySpec": { + "type": "uniform", + "segmentGranularity": "HOUR", + "queryGranularity": { + "type": "none" + }, + "rollup": false, + "intervals": [] + }, + "transformSpec": { + "filter": null, + "transforms": [] + } + }, + "ioConfig": { + "topic": "social_media", + "inputFormat": { + "type": "json", + "keepNullColumns": false, + "assumeNewlineDelimited": false, + "useJsonNodeReader": false + }, + "replicas": 1, + "taskCount": 1, + "taskDuration": "PT3600S", + "consumerProperties": { + "bootstrap.servers": "localhost:9042" + }, + "autoScalerConfig": null, + "pollTimeout": 100, + "startDelay": "PT5S", + "period": "PT30S", + "useEarliestOffset": true, + "completionTimeout": "PT1800S", + "lateMessageRejectionPeriod": null, + "earlyMessageRejectionPeriod": null, + "lateMessageRejectionStartDateTime": null, + "configOverrides": null, + "idleConfig": null, + "stream": "social_media", + "useEarliestSequenceNumber": true + }, + "tuningConfig": { + "type": "kafka", + "appendableIndexSpec": { + "type": "onheap", + "preserveExistingMetrics": false + }, + "maxRowsInMemory": 150000, + "maxBytesInMemory": 0, + "skipBytesInMemoryOverheadCheck": false, + "maxRowsPerSegment": 5000000, + "maxTotalRows": null, + "intermediatePersistPeriod": "PT10M", + "maxPendingPersists": 0, + "indexSpec": { + "bitmap": { + "type": "roaring" + }, + "dimensionCompression": "lz4", + "stringDictionaryEncoding": { + "type": "utf8" + }, + "metricCompression": "lz4", + "longEncoding": "longs" + }, + "indexSpecForIntermediatePersists": { + "bitmap": { + "type": "roaring" + }, + "dimensionCompression": "lz4", + "stringDictionaryEncoding": { + "type": "utf8" + }, + "metricCompression": "lz4", + "longEncoding": "longs" + }, + "reportParseExceptions": false, + "handoffConditionTimeout": 0, + "resetOffsetAutomatically": false, + "segmentWriteOutMediumFactory": null, + "workerThreads": null, + "chatThreads": null, + "chatRetries": 8, + "httpTimeout": "PT10S", + "shutdownTimeout": "PT80S", + "offsetFetchPeriod": "PT30S", + "intermediateHandoffPeriod": "P2147483647D", + "logParseExceptions": false, + "maxParseExceptions": 2147483647, + "maxSavedParseExceptions": 0, + "skipSequenceNumberAvailabilityCheck": false, + "repartitionTransitionDuration": "PT120S" + } + }, + "dataSchema": { + "dataSource": "wikipedia_stream", + "timestampSpec": { + "column": "__time", + "format": "iso", + "missingValue": null + }, + "dimensionsSpec": { + "dimensions": [ + { + "type": "string", + "name": "username", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + }, + { + "type": "string", + "name": "post_title", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + }, + { + "type": "long", + "name": "views", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "long", + "name": "upvotes", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "long", + "name": "comments", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "string", + "name": "edited", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + } + ], + "dimensionExclusions": [ + "__time" + ], + "includeAllDimensions": false, + "useSchemaDiscovery": false + }, + "metricsSpec": [], + "granularitySpec": { + "type": "uniform", + "segmentGranularity": "HOUR", + "queryGranularity": { + "type": "none" + }, + "rollup": false, + "intervals": [] + }, + "transformSpec": { + "filter": null, + "transforms": [] + } + }, + "tuningConfig": { + "type": "kafka", + "appendableIndexSpec": { + "type": "onheap", + "preserveExistingMetrics": false + }, + "maxRowsInMemory": 150000, + "maxBytesInMemory": 0, + "skipBytesInMemoryOverheadCheck": false, + "maxRowsPerSegment": 5000000, + "maxTotalRows": null, + "intermediatePersistPeriod": "PT10M", + "maxPendingPersists": 0, + "indexSpec": { + "bitmap": { + "type": "roaring" + }, + "dimensionCompression": "lz4", + "stringDictionaryEncoding": { + "type": "utf8" + }, + "metricCompression": "lz4", + "longEncoding": "longs" + }, + "indexSpecForIntermediatePersists": { + "bitmap": { + "type": "roaring" + }, + "dimensionCompression": "lz4", + "stringDictionaryEncoding": { + "type": "utf8" + }, + "metricCompression": "lz4", + "longEncoding": "longs" + }, + "reportParseExceptions": false, + "handoffConditionTimeout": 0, + "resetOffsetAutomatically": false, + "segmentWriteOutMediumFactory": null, + "workerThreads": null, + "chatThreads": null, + "chatRetries": 8, + "httpTimeout": "PT10S", + "shutdownTimeout": "PT80S", + "offsetFetchPeriod": "PT30S", + "intermediateHandoffPeriod": "P2147483647D", + "logParseExceptions": false, + "maxParseExceptions": 2147483647, + "maxSavedParseExceptions": 0, + "skipSequenceNumberAvailabilityCheck": false, + "repartitionTransitionDuration": "PT120S" + }, + "ioConfig": { + "topic": "social_media", + "inputFormat": { + "type": "json", + "keepNullColumns": false, + "assumeNewlineDelimited": false, + "useJsonNodeReader": false + }, + "replicas": 1, + "taskCount": 1, + "taskDuration": "PT3600S", + "consumerProperties": { + "bootstrap.servers": "localhost:9042" + }, + "autoScalerConfig": null, + "pollTimeout": 100, + "startDelay": "PT5S", + "period": "PT30S", + "useEarliestOffset": true, + "completionTimeout": "PT1800S", + "lateMessageRejectionPeriod": null, + "earlyMessageRejectionPeriod": null, + "lateMessageRejectionStartDateTime": null, + "configOverrides": null, + "idleConfig": null, + "stream": "social_media", + "useEarliestSequenceNumber": true + }, + "context": null, + "suspended": false + }, + "suspended": false + }, + { + "id": "social_media", + "state": "RUNNING", + "detailedState": "RUNNING", + "healthy": true, + "spec": { + "type": "kafka", + "spec": { + "dataSchema": { + "dataSource": "social_media", + "timestampSpec": { + "column": "__time", + "format": "iso", + "missingValue": null + }, + "dimensionsSpec": { + "dimensions": [ + { + "type": "string", + "name": "username", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + }, + { + "type": "string", + "name": "post_title", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + }, + { + "type": "long", + "name": "views", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "long", + "name": "upvotes", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "long", + "name": "comments", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "string", + "name": "edited", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + } + ], + "dimensionExclusions": [ + "__time" + ], + "includeAllDimensions": false, + "useSchemaDiscovery": false + }, + "metricsSpec": [], + "granularitySpec": { + "type": "uniform", + "segmentGranularity": "HOUR", + "queryGranularity": { + "type": "none" + }, + "rollup": false, + "intervals": [] + }, + "transformSpec": { + "filter": null, + "transforms": [] + } + }, + "ioConfig": { + "topic": "social_media", + "inputFormat": { + "type": "json", + "keepNullColumns": false, + "assumeNewlineDelimited": false, + "useJsonNodeReader": false + }, + "replicas": 1, + "taskCount": 1, + "taskDuration": "PT3600S", + "consumerProperties": { + "bootstrap.servers": "localhost:9094" + }, + "autoScalerConfig": null, + "pollTimeout": 100, + "startDelay": "PT5S", + "period": "PT30S", + "useEarliestOffset": true, + "completionTimeout": "PT1800S", + "lateMessageRejectionPeriod": null, + "earlyMessageRejectionPeriod": null, + "lateMessageRejectionStartDateTime": null, + "configOverrides": null, + "idleConfig": null, + "stream": "social_media", + "useEarliestSequenceNumber": true + }, + "tuningConfig": { + "type": "kafka", + "appendableIndexSpec": { + "type": "onheap", + "preserveExistingMetrics": false + }, + "maxRowsInMemory": 150000, + "maxBytesInMemory": 0, + "skipBytesInMemoryOverheadCheck": false, + "maxRowsPerSegment": 5000000, + "maxTotalRows": null, + "intermediatePersistPeriod": "PT10M", + "maxPendingPersists": 0, + "indexSpec": { + "bitmap": { + "type": "roaring" + }, + "dimensionCompression": "lz4", + "stringDictionaryEncoding": { + "type": "utf8" + }, + "metricCompression": "lz4", + "longEncoding": "longs" + }, + "indexSpecForIntermediatePersists": { + "bitmap": { + "type": "roaring" + }, + "dimensionCompression": "lz4", + "stringDictionaryEncoding": { + "type": "utf8" + }, + "metricCompression": "lz4", + "longEncoding": "longs" + }, + "reportParseExceptions": false, + "handoffConditionTimeout": 0, + "resetOffsetAutomatically": false, + "segmentWriteOutMediumFactory": null, + "workerThreads": null, + "chatThreads": null, + "chatRetries": 8, + "httpTimeout": "PT10S", + "shutdownTimeout": "PT80S", + "offsetFetchPeriod": "PT30S", + "intermediateHandoffPeriod": "P2147483647D", + "logParseExceptions": false, + "maxParseExceptions": 2147483647, + "maxSavedParseExceptions": 0, + "skipSequenceNumberAvailabilityCheck": false, + "repartitionTransitionDuration": "PT120S" + } + }, + "dataSchema": { + "dataSource": "social_media", + "timestampSpec": { + "column": "__time", + "format": "iso", + "missingValue": null + }, + "dimensionsSpec": { + "dimensions": [ + { + "type": "string", + "name": "username", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + }, + { + "type": "string", + "name": "post_title", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + }, + { + "type": "long", + "name": "views", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "long", + "name": "upvotes", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "long", + "name": "comments", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "string", + "name": "edited", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + } + ], + "dimensionExclusions": [ + "__time" + ], + "includeAllDimensions": false, + "useSchemaDiscovery": false + }, + "metricsSpec": [], + "granularitySpec": { + "type": "uniform", + "segmentGranularity": "HOUR", + "queryGranularity": { + "type": "none" + }, + "rollup": false, + "intervals": [] + }, + "transformSpec": { + "filter": null, + "transforms": [] + } + }, + "tuningConfig": { + "type": "kafka", + "appendableIndexSpec": { + "type": "onheap", + "preserveExistingMetrics": false + }, + "maxRowsInMemory": 150000, + "maxBytesInMemory": 0, + "skipBytesInMemoryOverheadCheck": false, + "maxRowsPerSegment": 5000000, + "maxTotalRows": null, + "intermediatePersistPeriod": "PT10M", + "maxPendingPersists": 0, + "indexSpec": { + "bitmap": { + "type": "roaring" + }, + "dimensionCompression": "lz4", + "stringDictionaryEncoding": { + "type": "utf8" + }, + "metricCompression": "lz4", + "longEncoding": "longs" + }, + "indexSpecForIntermediatePersists": { + "bitmap": { + "type": "roaring" + }, + "dimensionCompression": "lz4", + "stringDictionaryEncoding": { + "type": "utf8" + }, + "metricCompression": "lz4", + "longEncoding": "longs" + }, + "reportParseExceptions": false, + "handoffConditionTimeout": 0, + "resetOffsetAutomatically": false, + "segmentWriteOutMediumFactory": null, + "workerThreads": null, + "chatThreads": null, + "chatRetries": 8, + "httpTimeout": "PT10S", + "shutdownTimeout": "PT80S", + "offsetFetchPeriod": "PT30S", + "intermediateHandoffPeriod": "P2147483647D", + "logParseExceptions": false, + "maxParseExceptions": 2147483647, + "maxSavedParseExceptions": 0, + "skipSequenceNumberAvailabilityCheck": false, + "repartitionTransitionDuration": "PT120S" + }, + "ioConfig": { + "topic": "social_media", + "inputFormat": { + "type": "json", + "keepNullColumns": false, + "assumeNewlineDelimited": false, + "useJsonNodeReader": false + }, + "replicas": 1, + "taskCount": 1, + "taskDuration": "PT3600S", + "consumerProperties": { + "bootstrap.servers": "localhost:9094" + }, + "autoScalerConfig": null, + "pollTimeout": 100, + "startDelay": "PT5S", + "period": "PT30S", + "useEarliestOffset": true, + "completionTimeout": "PT1800S", + "lateMessageRejectionPeriod": null, + "earlyMessageRejectionPeriod": null, + "lateMessageRejectionStartDateTime": null, + "configOverrides": null, + "idleConfig": null, + "stream": "social_media", + "useEarliestSequenceNumber": true + }, + "context": null, + "suspended": false + }, + "suspended": false + } + ] + ``` +</details> -`GET /druid/indexer/v1/supervisor?state=true` +### Get Supervisor state -Returns a list of objects of the currently active supervisors and their current state. +<code className="getAPI">GET</code> `/druid/indexer/v1/supervisor?state=true` +Retrieves an array of objects of the currently active Supervisors and their current state. If there are no active Supervisors, it returns an empty array. + +See the following table for details on properties within the response object: |Field|Type|Description| |---|---|---| -|`id`|String|supervisor unique identifier| -|`state`|String|basic state of the supervisor. Available states: `UNHEALTHY_SUPERVISOR`, `UNHEALTHY_TASKS`, `PENDING`, `RUNNING`, `SUSPENDED`, `STOPPING`. Check [Kafka Docs](../development/extensions-core/kafka-supervisor-operations.md) for details.| -|`detailedState`|String|supervisor specific state. See documentation of the specific supervisor for details: [Kafka](../development/extensions-core/kafka-ingestion.md) or [Kinesis](../development/extensions-core/kinesis-ingestion.md)| -|`healthy`|Boolean|true or false indicator of overall supervisor health| -|`suspended`|Boolean|true or false indicator of whether the supervisor is in suspended state| +|`id`|String|Supervisor unique identifier| +|`state`|String|The basic state of the Supervisor. Available states: `UNHEALTHY_SUPERVISOR`, `UNHEALTHY_TASKS`, `PENDING`, `RUNNING`, `SUSPENDED`, `STOPPING`. Check [Kafka Docs](../development/extensions-core/kafka-supervisor-operations.md) for details.| +|`detailedState`|String|Supervisor specific state. See documentation of the specific Supervisor for details: [Kafka](../development/extensions-core/kafka-ingestion.md) or [Kinesis](../development/extensions-core/kinesis-ingestion.md)| +|`healthy`|Boolean|True or false indicator of overall Supervisor health| +|`suspended`|Boolean|True or false indicator of whether the Supervisor is in suspended state| + +#### Responses + +<!--DOCUSAURUS_CODE_TABS--> + +<!--200 SUCCESS--> +<br/> +*Successfully retrieved Supervisor state objects* + +<!--END_DOCUSAURUS_CODE_TABS--> + +--- + +#### Sample request + +<!--DOCUSAURUS_CODE_TABS--> + +<!--cURL--> +```shell +curl "http://<ROUTER_IP>:<ROUTER_PORT>/druid/indexer/v1/supervisor?state=true" +``` +<!--HTTP--> +```HTTP +GET /druid/indexer/v1/supervisor?state=true HTTP/1.1 +Host: http://<ROUTER_IP>:<ROUTER_PORT> +``` +<!--END_DOCUSAURUS_CODE_TABS--> + +#### Sample response + +<details> + <summary>Click to show sample response</summary> + + ```json + [ + { + "id": "wikipedia_stream", + "state": "UNHEALTHY_SUPERVISOR", + "detailedState": "UNABLE_TO_CONNECT_TO_STREAM", + "healthy": false, + "suspended": false + }, + { + "id": "social_media", + "state": "RUNNING", + "detailedState": "RUNNING", + "healthy": true, + "suspended": false + } + ] + ``` + +</details> + +### Get Supervisor specification + +#### URL + +<code className="getAPI">GET</code> `/druid/indexer/v1/supervisor/{supervisorId}` + +Retrieves the current spec for the Supervisor with the provided ID. The returned supervisor spec specifies the `dataSchema`, `ioConfig`, and `tuningConfig`. Review Comment: ```suggestion Retrieves the specification for a single supervisor. The returned specification includes the `dataSchema`, `ioConfig`, and `tuningConfig` objects. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
