ektravel commented on code in PR #14579: URL: https://github.com/apache/druid/pull/14579#discussion_r1270910560
########## docs/api-reference/supervisor-api.md: ########## @@ -23,89 +23,3189 @@ sidebar_label: Supervisors ~ under the License. --> -This document describes the API endpoints to manage and monitor supervisors for Apache Druid. +This topic describes the API endpoints to manage and monitor supervisors for Apache Druid. -## Supervisors +In this topic, `http://ROUTER_IP:ROUTER_PORT` is a place holder for your Router service address and port. Replace it with the information for your deployment. For example, use `http://localhost:8888` for quickstart deployments. -`GET /druid/indexer/v1/supervisor` +## Supervisor information -Returns a list of strings of the currently active supervisor ids. +The following table lists the properties of a supervisor object: -`GET /druid/indexer/v1/supervisor?full` +|Property|Type|Description| +|---|---|---| +|`id`|String|Unique identifier.| +|`state`|String|Generic state of the supervisor. Available states:`UNHEALTHY_SUPERVISOR`, `UNHEALTHY_TASKS`, `PENDING`, `RUNNING`, `SUSPENDED`, `STOPPING`. See [Apache Kafka operations](../development/extensions-core/kafka-supervisor-operations.md) for details.| +|`detailedState`|String|Detailed state of the supervisor. This property contains a more descriptive, implementation-specific state that may provide more insight into the supervisor's activities than the `state` property. See [Apache Kafka ingestion](../development/extensions-core/kafka-ingestion.md) and [Amazon Kinesis ingestion](../development/extensions-core/kinesis-ingestion.md) for supervisor-specific states.| +|`healthy`|Boolean|Supervisor health indicator.| +|`spec`|Object|Container object for the supervisor configuration.| +|`suspended`|Boolean|Indicates whether the supervisor is in a suspended state.| -Returns a list of objects of the currently active supervisors. +### Get an array of active supervisor IDs -|Field|Type|Description| -|---|---|---| -|`id`|String|supervisor unique identifier| -|`state`|String|basic state of the supervisor. Available states:`UNHEALTHY_SUPERVISOR`, `UNHEALTHY_TASKS`, `PENDING`, `RUNNING`, `SUSPENDED`, `STOPPING`. Check [Kafka Docs](../development/extensions-core/kafka-supervisor-operations.md) for details.| -|`detailedState`|String|supervisor specific state. See documentation of specific supervisor for details: [Kafka](../development/extensions-core/kafka-ingestion.md) or [Kinesis](../development/extensions-core/kinesis-ingestion.md)| -|`healthy`|Boolean|true or false indicator of overall supervisor health| -|`spec`|SupervisorSpec|JSON specification of supervisor| +Returns an array of strings representing the names of active supervisors. If there are no active supervisors, it returns an empty array. -`GET /druid/indexer/v1/supervisor?state=true` +#### URL -Returns a list of objects of the currently active supervisors and their current state. +<code class="getAPI">GET</code> <code>/druid/indexer/v1/supervisor</code> -|Field|Type|Description| -|---|---|---| -|`id`|String|supervisor unique identifier| -|`state`|String|basic state of the supervisor. Available states: `UNHEALTHY_SUPERVISOR`, `UNHEALTHY_TASKS`, `PENDING`, `RUNNING`, `SUSPENDED`, `STOPPING`. Check [Kafka Docs](../development/extensions-core/kafka-supervisor-operations.md) for details.| -|`detailedState`|String|supervisor specific state. See documentation of the specific supervisor for details: [Kafka](../development/extensions-core/kafka-ingestion.md) or [Kinesis](../development/extensions-core/kinesis-ingestion.md)| -|`healthy`|Boolean|true or false indicator of overall supervisor health| -|`suspended`|Boolean|true or false indicator of whether the supervisor is in suspended state| +#### Responses + +<!--DOCUSAURUS_CODE_TABS--> + +<!--200 SUCCESS--> + +<br/> + +*Successfully retrieved array of active supervisor IDs* + +<!--END_DOCUSAURUS_CODE_TABS--> + +--- + +#### Sample request + +<!--DOCUSAURUS_CODE_TABS--> + +<!--cURL--> + +```shell +curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/supervisor"; +``` + +<!--HTTP--> + +```HTTP +GET /druid/indexer/v1/supervisor HTTP/1.1 +Host: http://ROUTER_IP:ROUTER_PORT +``` + +<!--END_DOCUSAURUS_CODE_TABS--> + +#### Sample response + +<details> + <summary>Click to show sample response</summary> + + ```json + [ + "wikipedia_stream", + "social_media" + ] + ``` +</details> + +### Get an array of active supervisor objects + +Retrieves an array of active supervisor objects. If there are no active supervisors, it returns an empty array. For reference on the supervisor object properties, see the following [table](#supervisor-information). + +#### URL + +<code class="getAPI">GET</code> <code>/druid/indexer/v1/supervisor?full</code> + +#### Responses + +<!--DOCUSAURUS_CODE_TABS--> + +<!--200 SUCCESS--> + +<br/> + +*Successfully retrieved supervisor objects* + +<!--END_DOCUSAURUS_CODE_TABS--> + +--- + +#### Sample request + +<!--DOCUSAURUS_CODE_TABS--> + +<!--cURL--> + +```shell +curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/supervisor?full=null"; +``` + +<!--HTTP--> + +```HTTP +GET /druid/indexer/v1/supervisor?full=null HTTP/1.1 +Host: http://ROUTER_IP:ROUTER_PORT +``` + +<!--END_DOCUSAURUS_CODE_TABS--> + +#### Sample response + +<details> + <summary>Click to show sample response</summary> + + ```json + [ + { + "id": "wikipedia_stream", + "state": "RUNNING", + "detailedState": "CONNECTING_TO_STREAM", + "healthy": true, + "spec": { + "type": "kafka", + "spec": { + "dataSchema": { + "dataSource": "wikipedia_stream", + "timestampSpec": { + "column": "__time", + "format": "iso", + "missingValue": null + }, + "dimensionsSpec": { + "dimensions": [ + { + "type": "string", + "name": "username", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + }, + { + "type": "string", + "name": "post_title", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + }, + { + "type": "long", + "name": "views", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "long", + "name": "upvotes", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "long", + "name": "comments", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "string", + "name": "edited", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + } + ], + "dimensionExclusions": [ + "__time" + ], + "includeAllDimensions": false, + "useSchemaDiscovery": false + }, + "metricsSpec": [], + "granularitySpec": { + "type": "uniform", + "segmentGranularity": "HOUR", + "queryGranularity": { + "type": "none" + }, + "rollup": false, + "intervals": [] + }, + "transformSpec": { + "filter": null, + "transforms": [] + } + }, + "ioConfig": { + "topic": "social_media", + "inputFormat": { + "type": "json", + "keepNullColumns": false, + "assumeNewlineDelimited": false, + "useJsonNodeReader": false + }, + "replicas": 1, + "taskCount": 1, + "taskDuration": "PT3600S", + "consumerProperties": { + "bootstrap.servers": "localhost:9042" + }, + "autoScalerConfig": null, + "pollTimeout": 100, + "startDelay": "PT5S", + "period": "PT30S", + "useEarliestOffset": true, + "completionTimeout": "PT1800S", + "lateMessageRejectionPeriod": null, + "earlyMessageRejectionPeriod": null, + "lateMessageRejectionStartDateTime": null, + "configOverrides": null, + "idleConfig": null, + "stream": "social_media", + "useEarliestSequenceNumber": true + }, + "tuningConfig": { + "type": "kafka", + "appendableIndexSpec": { + "type": "onheap", + "preserveExistingMetrics": false + }, + "maxRowsInMemory": 150000, + "maxBytesInMemory": 0, + "skipBytesInMemoryOverheadCheck": false, + "maxRowsPerSegment": 5000000, + "maxTotalRows": null, + "intermediatePersistPeriod": "PT10M", + "maxPendingPersists": 0, + "indexSpec": { + "bitmap": { + "type": "roaring" + }, + "dimensionCompression": "lz4", + "stringDictionaryEncoding": { + "type": "utf8" + }, + "metricCompression": "lz4", + "longEncoding": "longs" + }, + "indexSpecForIntermediatePersists": { + "bitmap": { + "type": "roaring" + }, + "dimensionCompression": "lz4", + "stringDictionaryEncoding": { + "type": "utf8" + }, + "metricCompression": "lz4", + "longEncoding": "longs" + }, + "reportParseExceptions": false, + "handoffConditionTimeout": 0, + "resetOffsetAutomatically": false, + "segmentWriteOutMediumFactory": null, + "workerThreads": null, + "chatThreads": null, + "chatRetries": 8, + "httpTimeout": "PT10S", + "shutdownTimeout": "PT80S", + "offsetFetchPeriod": "PT30S", + "intermediateHandoffPeriod": "P2147483647D", + "logParseExceptions": false, + "maxParseExceptions": 2147483647, + "maxSavedParseExceptions": 0, + "skipSequenceNumberAvailabilityCheck": false, + "repartitionTransitionDuration": "PT120S" + } + }, + "dataSchema": { + "dataSource": "wikipedia_stream", + "timestampSpec": { + "column": "__time", + "format": "iso", + "missingValue": null + }, + "dimensionsSpec": { + "dimensions": [ + { + "type": "string", + "name": "username", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + }, + { + "type": "string", + "name": "post_title", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + }, + { + "type": "long", + "name": "views", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "long", + "name": "upvotes", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "long", + "name": "comments", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "string", + "name": "edited", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + } + ], + "dimensionExclusions": [ + "__time" + ], + "includeAllDimensions": false, + "useSchemaDiscovery": false + }, + "metricsSpec": [], + "granularitySpec": { + "type": "uniform", + "segmentGranularity": "HOUR", + "queryGranularity": { + "type": "none" + }, + "rollup": false, + "intervals": [] + }, + "transformSpec": { + "filter": null, + "transforms": [] + } + }, + "tuningConfig": { + "type": "kafka", + "appendableIndexSpec": { + "type": "onheap", + "preserveExistingMetrics": false + }, + "maxRowsInMemory": 150000, + "maxBytesInMemory": 0, + "skipBytesInMemoryOverheadCheck": false, + "maxRowsPerSegment": 5000000, + "maxTotalRows": null, + "intermediatePersistPeriod": "PT10M", + "maxPendingPersists": 0, + "indexSpec": { + "bitmap": { + "type": "roaring" + }, + "dimensionCompression": "lz4", + "stringDictionaryEncoding": { + "type": "utf8" + }, + "metricCompression": "lz4", + "longEncoding": "longs" + }, + "indexSpecForIntermediatePersists": { + "bitmap": { + "type": "roaring" + }, + "dimensionCompression": "lz4", + "stringDictionaryEncoding": { + "type": "utf8" + }, + "metricCompression": "lz4", + "longEncoding": "longs" + }, + "reportParseExceptions": false, + "handoffConditionTimeout": 0, + "resetOffsetAutomatically": false, + "segmentWriteOutMediumFactory": null, + "workerThreads": null, + "chatThreads": null, + "chatRetries": 8, + "httpTimeout": "PT10S", + "shutdownTimeout": "PT80S", + "offsetFetchPeriod": "PT30S", + "intermediateHandoffPeriod": "P2147483647D", + "logParseExceptions": false, + "maxParseExceptions": 2147483647, + "maxSavedParseExceptions": 0, + "skipSequenceNumberAvailabilityCheck": false, + "repartitionTransitionDuration": "PT120S" + }, + "ioConfig": { + "topic": "social_media", + "inputFormat": { + "type": "json", + "keepNullColumns": false, + "assumeNewlineDelimited": false, + "useJsonNodeReader": false + }, + "replicas": 1, + "taskCount": 1, + "taskDuration": "PT3600S", + "consumerProperties": { + "bootstrap.servers": "localhost:9042" + }, + "autoScalerConfig": null, + "pollTimeout": 100, + "startDelay": "PT5S", + "period": "PT30S", + "useEarliestOffset": true, + "completionTimeout": "PT1800S", + "lateMessageRejectionPeriod": null, + "earlyMessageRejectionPeriod": null, + "lateMessageRejectionStartDateTime": null, + "configOverrides": null, + "idleConfig": null, + "stream": "social_media", + "useEarliestSequenceNumber": true + }, + "context": null, + "suspended": false + }, + "suspended": false + }, + { + "id": "social_media", + "state": "RUNNING", + "detailedState": "RUNNING", + "healthy": true, + "spec": { + "type": "kafka", + "spec": { + "dataSchema": { + "dataSource": "social_media", + "timestampSpec": { + "column": "__time", + "format": "iso", + "missingValue": null + }, + "dimensionsSpec": { + "dimensions": [ + { + "type": "string", + "name": "username", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + }, + { + "type": "string", + "name": "post_title", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + }, + { + "type": "long", + "name": "views", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "long", + "name": "upvotes", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "long", + "name": "comments", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "string", + "name": "edited", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + } + ], + "dimensionExclusions": [ + "__time" + ], + "includeAllDimensions": false, + "useSchemaDiscovery": false + }, + "metricsSpec": [], + "granularitySpec": { + "type": "uniform", + "segmentGranularity": "HOUR", + "queryGranularity": { + "type": "none" + }, + "rollup": false, + "intervals": [] + }, + "transformSpec": { + "filter": null, + "transforms": [] + } + }, + "ioConfig": { + "topic": "social_media", + "inputFormat": { + "type": "json", + "keepNullColumns": false, + "assumeNewlineDelimited": false, + "useJsonNodeReader": false + }, + "replicas": 1, + "taskCount": 1, + "taskDuration": "PT3600S", + "consumerProperties": { + "bootstrap.servers": "localhost:9094" + }, + "autoScalerConfig": null, + "pollTimeout": 100, + "startDelay": "PT5S", + "period": "PT30S", + "useEarliestOffset": true, + "completionTimeout": "PT1800S", + "lateMessageRejectionPeriod": null, + "earlyMessageRejectionPeriod": null, + "lateMessageRejectionStartDateTime": null, + "configOverrides": null, + "idleConfig": null, + "stream": "social_media", + "useEarliestSequenceNumber": true + }, + "tuningConfig": { + "type": "kafka", + "appendableIndexSpec": { + "type": "onheap", + "preserveExistingMetrics": false + }, + "maxRowsInMemory": 150000, + "maxBytesInMemory": 0, + "skipBytesInMemoryOverheadCheck": false, + "maxRowsPerSegment": 5000000, + "maxTotalRows": null, + "intermediatePersistPeriod": "PT10M", + "maxPendingPersists": 0, + "indexSpec": { + "bitmap": { + "type": "roaring" + }, + "dimensionCompression": "lz4", + "stringDictionaryEncoding": { + "type": "utf8" + }, + "metricCompression": "lz4", + "longEncoding": "longs" + }, + "indexSpecForIntermediatePersists": { + "bitmap": { + "type": "roaring" + }, + "dimensionCompression": "lz4", + "stringDictionaryEncoding": { + "type": "utf8" + }, + "metricCompression": "lz4", + "longEncoding": "longs" + }, + "reportParseExceptions": false, + "handoffConditionTimeout": 0, + "resetOffsetAutomatically": false, + "segmentWriteOutMediumFactory": null, + "workerThreads": null, + "chatThreads": null, + "chatRetries": 8, + "httpTimeout": "PT10S", + "shutdownTimeout": "PT80S", + "offsetFetchPeriod": "PT30S", + "intermediateHandoffPeriod": "P2147483647D", + "logParseExceptions": false, + "maxParseExceptions": 2147483647, + "maxSavedParseExceptions": 0, + "skipSequenceNumberAvailabilityCheck": false, + "repartitionTransitionDuration": "PT120S" + } + }, + "dataSchema": { + "dataSource": "social_media", + "timestampSpec": { + "column": "__time", + "format": "iso", + "missingValue": null + }, + "dimensionsSpec": { + "dimensions": [ + { + "type": "string", + "name": "username", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + }, + { + "type": "string", + "name": "post_title", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + }, + { + "type": "long", + "name": "views", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "long", + "name": "upvotes", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "long", + "name": "comments", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "string", + "name": "edited", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + } + ], + "dimensionExclusions": [ + "__time" + ], + "includeAllDimensions": false, + "useSchemaDiscovery": false + }, + "metricsSpec": [], + "granularitySpec": { + "type": "uniform", + "segmentGranularity": "HOUR", + "queryGranularity": { + "type": "none" + }, + "rollup": false, + "intervals": [] + }, + "transformSpec": { + "filter": null, + "transforms": [] + } + }, + "tuningConfig": { + "type": "kafka", + "appendableIndexSpec": { + "type": "onheap", + "preserveExistingMetrics": false + }, + "maxRowsInMemory": 150000, + "maxBytesInMemory": 0, + "skipBytesInMemoryOverheadCheck": false, + "maxRowsPerSegment": 5000000, + "maxTotalRows": null, + "intermediatePersistPeriod": "PT10M", + "maxPendingPersists": 0, + "indexSpec": { + "bitmap": { + "type": "roaring" + }, + "dimensionCompression": "lz4", + "stringDictionaryEncoding": { + "type": "utf8" + }, + "metricCompression": "lz4", + "longEncoding": "longs" + }, + "indexSpecForIntermediatePersists": { + "bitmap": { + "type": "roaring" + }, + "dimensionCompression": "lz4", + "stringDictionaryEncoding": { + "type": "utf8" + }, + "metricCompression": "lz4", + "longEncoding": "longs" + }, + "reportParseExceptions": false, + "handoffConditionTimeout": 0, + "resetOffsetAutomatically": false, + "segmentWriteOutMediumFactory": null, + "workerThreads": null, + "chatThreads": null, + "chatRetries": 8, + "httpTimeout": "PT10S", + "shutdownTimeout": "PT80S", + "offsetFetchPeriod": "PT30S", + "intermediateHandoffPeriod": "P2147483647D", + "logParseExceptions": false, + "maxParseExceptions": 2147483647, + "maxSavedParseExceptions": 0, + "skipSequenceNumberAvailabilityCheck": false, + "repartitionTransitionDuration": "PT120S" + }, + "ioConfig": { + "topic": "social_media", + "inputFormat": { + "type": "json", + "keepNullColumns": false, + "assumeNewlineDelimited": false, + "useJsonNodeReader": false + }, + "replicas": 1, + "taskCount": 1, + "taskDuration": "PT3600S", + "consumerProperties": { + "bootstrap.servers": "localhost:9094" + }, + "autoScalerConfig": null, + "pollTimeout": 100, + "startDelay": "PT5S", + "period": "PT30S", + "useEarliestOffset": true, + "completionTimeout": "PT1800S", + "lateMessageRejectionPeriod": null, + "earlyMessageRejectionPeriod": null, + "lateMessageRejectionStartDateTime": null, + "configOverrides": null, + "idleConfig": null, + "stream": "social_media", + "useEarliestSequenceNumber": true + }, + "context": null, + "suspended": false + }, + "suspended": false + } + ] + ``` +</details> + +### Get an array of supervisor states + +Retrieves an array of objects with the currently active supervisors and their current state. If there are no active supervisors, it returns an empty array. For reference on the supervisor object properties, see the following [table](#supervisor-information). + +#### URL + +<code class="getAPI">GET</code> <code>/druid/indexer/v1/supervisor?state=true</code> + +#### Responses + +<!--DOCUSAURUS_CODE_TABS--> + +<!--200 SUCCESS--> + +<br/> + +*Successfully retrieved supervisor state objects.* + +<!--END_DOCUSAURUS_CODE_TABS--> + +--- + +#### Sample request + +<!--DOCUSAURUS_CODE_TABS--> + +<!--cURL--> + +```shell +curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/supervisor?state=true"; +``` + +<!--HTTP--> + +```HTTP +GET /druid/indexer/v1/supervisor?state=true HTTP/1.1 +Host: http://ROUTER_IP:ROUTER_PORT +``` + +<!--END_DOCUSAURUS_CODE_TABS--> + +#### Sample response + +<details> + <summary>Click to show sample response</summary> + + ```json + [ + { + "id": "wikipedia_stream", + "state": "UNHEALTHY_SUPERVISOR", + "detailedState": "UNABLE_TO_CONNECT_TO_STREAM", + "healthy": false, + "suspended": false + }, + { + "id": "social_media", + "state": "RUNNING", + "detailedState": "RUNNING", + "healthy": true, + "suspended": false + } + ] + ``` + +</details> + +### Get supervisor specification + +Retrieves the specification for a single supervisor. The returned specification includes the `dataSchema`, `ioConfig`, and `tuningConfig` objects. + +#### URL + +<code class="getAPI">GET</code> <code>/druid/indexer/v1/supervisor/:supervisorId</code> + +#### Responses + +<!--DOCUSAURUS_CODE_TABS--> + +<!--200 SUCCESS--> + +<br/> + +*Successfully retrieved supervisor spec* + +<!--404 NOT FOUND--> + +<br/> + +*Invalid supervisor ID* + +<!--END_DOCUSAURUS_CODE_TABS--> + +--- + +#### Sample request + +The following example shows how to retrieve the specification of a supervisor with the name `wikipedia_stream`. + +<!--DOCUSAURUS_CODE_TABS--> + +<!--cURL--> + +```shell +curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/supervisor/wikipedia_stream"; +``` + +<!--HTTP--> + +```HTTP +GET /druid/indexer/v1/supervisor/wikipedia_stream HTTP/1.1 +Host: http://ROUTER_IP:ROUTER_PORT +``` + +<!--END_DOCUSAURUS_CODE_TABS--> + + +#### Sample response + +<details> + <summary>Click to show sample response</summary> -`GET /druid/indexer/v1/supervisor/<supervisorId>` + ```json +{ + "type": "kafka", + "spec": { + "dataSchema": { + "dataSource": "social_media", + "timestampSpec": { + "column": "__time", + "format": "iso", + "missingValue": null + }, + "dimensionsSpec": { + "dimensions": [ + { + "type": "string", + "name": "username", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + }, + { + "type": "string", + "name": "post_title", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + }, + { + "type": "long", + "name": "views", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "long", + "name": "upvotes", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "long", + "name": "comments", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "string", + "name": "edited", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + } + ], + "dimensionExclusions": [ + "__time" + ], + "includeAllDimensions": false, + "useSchemaDiscovery": false + }, + "metricsSpec": [], + "granularitySpec": { + "type": "uniform", + "segmentGranularity": "HOUR", + "queryGranularity": { + "type": "none" + }, + "rollup": false, + "intervals": [] + }, + "transformSpec": { + "filter": null, + "transforms": [] + } + }, + "ioConfig": { + "topic": "social_media", + "inputFormat": { + "type": "json", + "keepNullColumns": false, + "assumeNewlineDelimited": false, + "useJsonNodeReader": false + }, + "replicas": 1, + "taskCount": 1, + "taskDuration": "PT3600S", + "consumerProperties": { + "bootstrap.servers": "localhost:9094" + }, + "autoScalerConfig": null, + "pollTimeout": 100, + "startDelay": "PT5S", + "period": "PT30S", + "useEarliestOffset": true, + "completionTimeout": "PT1800S", + "lateMessageRejectionPeriod": null, + "earlyMessageRejectionPeriod": null, + "lateMessageRejectionStartDateTime": null, + "configOverrides": null, + "idleConfig": null, + "stream": "social_media", + "useEarliestSequenceNumber": true + }, + "tuningConfig": { + "type": "kafka", + "appendableIndexSpec": { + "type": "onheap", + "preserveExistingMetrics": false + }, + "maxRowsInMemory": 150000, + "maxBytesInMemory": 0, + "skipBytesInMemoryOverheadCheck": false, + "maxRowsPerSegment": 5000000, + "maxTotalRows": null, + "intermediatePersistPeriod": "PT10M", + "maxPendingPersists": 0, + "indexSpec": { + "bitmap": { + "type": "roaring" + }, + "dimensionCompression": "lz4", + "stringDictionaryEncoding": { + "type": "utf8" + }, + "metricCompression": "lz4", + "longEncoding": "longs" + }, + "indexSpecForIntermediatePersists": { + "bitmap": { + "type": "roaring" + }, + "dimensionCompression": "lz4", + "stringDictionaryEncoding": { + "type": "utf8" + }, + "metricCompression": "lz4", + "longEncoding": "longs" + }, + "reportParseExceptions": false, + "handoffConditionTimeout": 0, + "resetOffsetAutomatically": false, + "segmentWriteOutMediumFactory": null, + "workerThreads": null, + "chatThreads": null, + "chatRetries": 8, + "httpTimeout": "PT10S", + "shutdownTimeout": "PT80S", + "offsetFetchPeriod": "PT30S", + "intermediateHandoffPeriod": "P2147483647D", + "logParseExceptions": false, + "maxParseExceptions": 2147483647, + "maxSavedParseExceptions": 0, + "skipSequenceNumberAvailabilityCheck": false, + "repartitionTransitionDuration": "PT120S" + } + }, + "dataSchema": { + "dataSource": "social_media", + "timestampSpec": { + "column": "__time", + "format": "iso", + "missingValue": null + }, + "dimensionsSpec": { + "dimensions": [ + { + "type": "string", + "name": "username", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + }, + { + "type": "string", + "name": "post_title", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + }, + { + "type": "long", + "name": "views", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "long", + "name": "upvotes", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "long", + "name": "comments", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "string", + "name": "edited", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + } + ], + "dimensionExclusions": [ + "__time" + ], + "includeAllDimensions": false, + "useSchemaDiscovery": false + }, + "metricsSpec": [], + "granularitySpec": { + "type": "uniform", + "segmentGranularity": "HOUR", + "queryGranularity": { + "type": "none" + }, + "rollup": false, + "intervals": [] + }, + "transformSpec": { + "filter": null, + "transforms": [] + } + }, + "tuningConfig": { + "type": "kafka", + "appendableIndexSpec": { + "type": "onheap", + "preserveExistingMetrics": false + }, + "maxRowsInMemory": 150000, + "maxBytesInMemory": 0, + "skipBytesInMemoryOverheadCheck": false, + "maxRowsPerSegment": 5000000, + "maxTotalRows": null, + "intermediatePersistPeriod": "PT10M", + "maxPendingPersists": 0, + "indexSpec": { + "bitmap": { + "type": "roaring" + }, + "dimensionCompression": "lz4", + "stringDictionaryEncoding": { + "type": "utf8" + }, + "metricCompression": "lz4", + "longEncoding": "longs" + }, + "indexSpecForIntermediatePersists": { + "bitmap": { + "type": "roaring" + }, + "dimensionCompression": "lz4", + "stringDictionaryEncoding": { + "type": "utf8" + }, + "metricCompression": "lz4", + "longEncoding": "longs" + }, + "reportParseExceptions": false, + "handoffConditionTimeout": 0, + "resetOffsetAutomatically": false, + "segmentWriteOutMediumFactory": null, + "workerThreads": null, + "chatThreads": null, + "chatRetries": 8, + "httpTimeout": "PT10S", + "shutdownTimeout": "PT80S", + "offsetFetchPeriod": "PT30S", + "intermediateHandoffPeriod": "P2147483647D", + "logParseExceptions": false, + "maxParseExceptions": 2147483647, + "maxSavedParseExceptions": 0, + "skipSequenceNumberAvailabilityCheck": false, + "repartitionTransitionDuration": "PT120S" + }, + "ioConfig": { + "topic": "social_media", + "inputFormat": { + "type": "json", + "keepNullColumns": false, + "assumeNewlineDelimited": false, + "useJsonNodeReader": false + }, + "replicas": 1, + "taskCount": 1, + "taskDuration": "PT3600S", + "consumerProperties": { + "bootstrap.servers": "localhost:9094" + }, + "autoScalerConfig": null, + "pollTimeout": 100, + "startDelay": "PT5S", + "period": "PT30S", + "useEarliestOffset": true, + "completionTimeout": "PT1800S", + "lateMessageRejectionPeriod": null, + "earlyMessageRejectionPeriod": null, + "lateMessageRejectionStartDateTime": null, + "configOverrides": null, + "idleConfig": null, + "stream": "social_media", + "useEarliestSequenceNumber": true + }, + "context": null, + "suspended": false +} + ``` +</details> -Returns the current spec for the supervisor with the provided ID. +### Get supervisor status -`GET /druid/indexer/v1/supervisor/<supervisorId>/status` +Retrieves the current status report for a single supervisor. The report contains the state of the supervisor tasks and an array of recently thrown exceptions. -Returns the current status of the supervisor with the provided ID. +For additional information about the status report, see the topic for each streaming ingestion methods: +* [Amazon Kinesis](../development/extensions-core/kinesis-ingestion.md#getting-supervisor-status-report) +* [Apache Kafka](../development/extensions-core/kafka-supervisor-operations.md#getting-supervisor-status-report) -`GET/druid/indexer/v1/supervisor/history` +#### URL -Returns an audit history of specs for all supervisors (current and past). +<code class="getAPI">GET</code> <code>/druid/indexer/v1/supervisor/:supervisorId/status</code> -`GET /druid/indexer/v1/supervisor/<supervisorId>/history` +#### Responses -Returns an audit history of specs for the supervisor with the provided ID. +<!--DOCUSAURUS_CODE_TABS--> -`POST /druid/indexer/v1/supervisor` +<!--200 SUCCESS--> -Create a new supervisor or update an existing one. +<br/> + +*Successfully retrieved supervisor status* + +<!--404 NOT FOUND--> + +<br/> + +*Invalid supervisor ID* + +<!--END_DOCUSAURUS_CODE_TABS--> + +--- + +#### Sample request + +The following example shows how to retrieve the status of a supervisor with the name `social_media`. + +<!--DOCUSAURUS_CODE_TABS--> + +<!--cURL--> + +```shell +curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/supervisor/social_media/status"; +``` + +<!--HTTP--> + +```HTTP +GET /druid/indexer/v1/supervisor/social_media/status HTTP/1.1 +Host: http://ROUTER_IP:ROUTER_PORT +``` + +<!--END_DOCUSAURUS_CODE_TABS--> + +#### Sample response + +<details> + <summary>Click to show sample response</summary> + + ```json + { + "id": "social_media", + "generationTime": "2023-07-05T23:24:43.934Z", + "payload": { + "dataSource": "social_media", + "stream": "social_media", + "partitions": 1, + "replicas": 1, + "durationSeconds": 3600, + "activeTasks": [ + { + "id": "index_kafka_social_media_ab72ae4127c591c_flcbhdlh", + "startingOffsets": { + "0": 3176381 + }, + "startTime": "2023-07-05T23:21:39.321Z", + "remainingSeconds": 3415, + "type": "ACTIVE", + "currentOffsets": { + "0": 3296632 + }, + "lag": { + "0": 3 + } + } + ], + "publishingTasks": [], + "latestOffsets": { + "0": 3296635 + }, + "minimumLag": { + "0": 3 + }, + "aggregateLag": 3, + "offsetsLastUpdated": "2023-07-05T23:24:30.212Z", + "suspended": false, + "healthy": true, + "state": "RUNNING", + "detailedState": "RUNNING", + "recentErrors": [] + } + } + ``` +</details> + +## Audit history + +An audit history provides a comprehensive log of events, including supervisor configuration, creation, suspension, and modification history. + +### Get audit history for all supervisors + +Retrieve an audit history of specs for all supervisors. + +#### URL + +<code class="getAPI">GET</code> <code>/druid/indexer/v1/supervisor/history</code> + +#### Responses + +<!--DOCUSAURUS_CODE_TABS--> + +<!--200 SUCCESS--> + +<br/> + +*Successfully retrieved audit history* + +<!--END_DOCUSAURUS_CODE_TABS--> + +--- + +#### Sample request + +<!--DOCUSAURUS_CODE_TABS--> + +<!--cURL--> + +```shell +curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/supervisor/history"; +``` + +<!--HTTP--> + +```HTTP +GET /druid/indexer/v1/supervisor/history HTTP/1.1 +Host: http://ROUTER_IP:ROUTER_PORT +``` + +<!--END_DOCUSAURUS_CODE_TABS--> + +#### Sample response + +<details> + <summary>Click to show sample response</summary> + + ```json +{ + "social_media": [ + { + "spec": { + "type": "kafka", + "spec": { + "dataSchema": { + "dataSource": "social_media", + "timestampSpec": { + "column": "__time", + "format": "iso", + "missingValue": null + }, + "dimensionsSpec": { + "dimensions": [ + { + "type": "string", + "name": "username", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + }, + { + "type": "string", + "name": "post_title", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + }, + { + "type": "long", + "name": "views", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "long", + "name": "upvotes", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "long", + "name": "comments", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "string", + "name": "edited", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + } + ], + "dimensionExclusions": [ + "__time" + ], + "includeAllDimensions": false, + "useSchemaDiscovery": false + }, + "metricsSpec": [], + "granularitySpec": { + "type": "uniform", + "segmentGranularity": "HOUR", + "queryGranularity": { + "type": "none" + }, + "rollup": false, + "intervals": [] + }, + "transformSpec": { + "filter": null, + "transforms": [] + } + }, + "ioConfig": { + "topic": "social_media", + "inputFormat": { + "type": "json", + "keepNullColumns": false, + "assumeNewlineDelimited": false, + "useJsonNodeReader": false + }, + "replicas": 1, + "taskCount": 1, + "taskDuration": "PT3600S", + "consumerProperties": { + "bootstrap.servers": "localhost:9094" + }, + "autoScalerConfig": null, + "pollTimeout": 100, + "startDelay": "PT5S", + "period": "PT30S", + "useEarliestOffset": true, + "completionTimeout": "PT1800S", + "lateMessageRejectionPeriod": null, + "earlyMessageRejectionPeriod": null, + "lateMessageRejectionStartDateTime": null, + "configOverrides": null, + "idleConfig": null, + "stream": "social_media", + "useEarliestSequenceNumber": true + }, + "tuningConfig": { + "type": "kafka", + "appendableIndexSpec": { + "type": "onheap", + "preserveExistingMetrics": false + }, + "maxRowsInMemory": 150000, + "maxBytesInMemory": 0, + "skipBytesInMemoryOverheadCheck": false, + "maxRowsPerSegment": 5000000, + "maxTotalRows": null, + "intermediatePersistPeriod": "PT10M", + "maxPendingPersists": 0, + "indexSpec": { + "bitmap": { + "type": "roaring" + }, + "dimensionCompression": "lz4", + "stringDictionaryEncoding": { + "type": "utf8" + }, + "metricCompression": "lz4", + "longEncoding": "longs" + }, + "indexSpecForIntermediatePersists": { + "bitmap": { + "type": "roaring" + }, + "dimensionCompression": "lz4", + "stringDictionaryEncoding": { + "type": "utf8" + }, + "metricCompression": "lz4", + "longEncoding": "longs" + }, + "reportParseExceptions": false, + "handoffConditionTimeout": 0, + "resetOffsetAutomatically": false, + "segmentWriteOutMediumFactory": null, + "workerThreads": null, + "chatThreads": null, + "chatRetries": 8, + "httpTimeout": "PT10S", + "shutdownTimeout": "PT80S", + "offsetFetchPeriod": "PT30S", + "intermediateHandoffPeriod": "P2147483647D", + "logParseExceptions": false, + "maxParseExceptions": 2147483647, + "maxSavedParseExceptions": 0, + "skipSequenceNumberAvailabilityCheck": false, + "repartitionTransitionDuration": "PT120S" + } + }, + "dataSchema": { + "dataSource": "social_media", + "timestampSpec": { + "column": "__time", + "format": "iso", + "missingValue": null + }, + "dimensionsSpec": { + "dimensions": [ + { + "type": "string", + "name": "username", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + }, + { + "type": "string", + "name": "post_title", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + }, + { + "type": "long", + "name": "views", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "long", + "name": "upvotes", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "long", + "name": "comments", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": false + }, + { + "type": "string", + "name": "edited", + "multiValueHandling": "SORTED_ARRAY", + "createBitmapIndex": true + } + ], + "dimensionExclusions": [ + "__time" + ], + "includeAllDimensions": false, + "useSchemaDiscovery": false + }, + "metricsSpec": [], + "granularitySpec": { + "type": "uniform", + "segmentGranularity": "HOUR", + "queryGranularity": { + "type": "none" + }, + "rollup": false, + "intervals": [] + }, + "transformSpec": { + "filter": null, + "transforms": [] + } + }, + "tuningConfig": { + "type": "kafka", + "appendableIndexSpec": { + "type": "onheap", + "preserveExistingMetrics": false + }, + "maxRowsInMemory": 150000, + "maxBytesInMemory": 0, + "skipBytesInMemoryOverheadCheck": false, + "maxRowsPerSegment": 5000000, + "maxTotalRows": null, + "intermediatePersistPeriod": "PT10M", + "maxPendingPersists": 0, + "indexSpec": { + "bitmap": { + "type": "roaring" + }, + "dimensionCompression": "lz4", + "stringDictionaryEncoding": { + "type": "utf8" + }, + "metricCompression": "lz4", + "longEncoding": "longs" + }, + "indexSpecForIntermediatePersists": { + "bitmap": { + "type": "roaring" + }, + "dimensionCompression": "lz4", + "stringDictionaryEncoding": { + "type": "utf8" + }, + "metricCompression": "lz4", + "longEncoding": "longs" + }, + "reportParseExceptions": false, + "handoffConditionTimeout": 0, + "resetOffsetAutomatically": false, + "segmentWriteOutMediumFactory": null, + "workerThreads": null, + "chatThreads": null, + "chatRetries": 8, + "httpTimeout": "PT10S", + "shutdownTimeout": "PT80S", + "offsetFetchPeriod": "PT30S", + "intermediateHandoffPeriod": "P2147483647D", + "logParseExceptions": false, + "maxParseExceptions": 2147483647, + "maxSavedParseExceptions": 0, + "skipSequenceNumberAvailabilityCheck": false, + "repartitionTransitionDuration": "PT120S" + }, + "ioConfig": { + "topic": "social_media", + "inputFormat": { + "type": "json", + "keepNullColumns": false, + "assumeNewlineDelimited": false, + "useJsonNodeReader": false + }, + "replicas": 1, + "taskCount": 1, + "taskDuration": "PT3600S", + "consumerProperties": { + "bootstrap.servers": "localhost:9094" + }, + "autoScalerConfig": null, + "pollTimeout": 100, + "startDelay": "PT5S", + "period": "PT30S", + "useEarliestOffset": true, + "completionTimeout": "PT1800S", + "lateMessageRejectionPeriod": null, + "earlyMessageRejectionPeriod": null, + "lateMessageRejectionStartDateTime": null, + "configOverrides": null, + "idleConfig": null, + "stream": "social_media", + "useEarliestSequenceNumber": true + }, + "context": null, + "suspended": false + }, + "version": "2023-07-03T18:51:02.970Z" + } + ] +} + ``` +</details> + +### Get audit history for specific supervisor Review Comment: ```suggestion ### Get audit history for a specific supervisor ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
