zhangyue19921010 commented on a change in pull request #10524:
URL: https://github.com/apache/druid/pull/10524#discussion_r580789397
##########
File path: docs/development/extensions-core/kafka-ingestion.md
##########
@@ -146,6 +146,26 @@ A sample supervisor spec is shown below:
|`lateMessageRejectionStartDateTime`|ISO8601 DateTime|Configure tasks to
reject messages with timestamps earlier than this date time; for example if
this is set to `2016-01-01T11:00Z` and the supervisor creates a task at
*2016-01-01T12:00Z*, messages with timestamps earlier than *2016-01-01T11:00Z*
will be dropped. This may help prevent concurrency issues if your data stream
has late messages and you have multiple pipelines that need to operate on the
same segments (e.g. a realtime and a nightly batch ingestion pipeline).|no
(default == none)|
|`lateMessageRejectionPeriod`|ISO8601 Period|Configure tasks to reject
messages with timestamps earlier than this period before the task was created;
for example if this is set to `PT1H` and the supervisor creates a task at
*2016-01-01T12:00Z*, messages with timestamps earlier than *2016-01-01T11:00Z*
will be dropped. This may help prevent concurrency issues if your data stream
has late messages and you have multiple pipelines that need to operate on the
same segments (e.g. a realtime and a nightly batch ingestion pipeline). Please
note that only one of `lateMessageRejectionPeriod` or
`lateMessageRejectionStartDateTime` can be specified.|no (default == none)|
|`earlyMessageRejectionPeriod`|ISO8601 Period|Configure tasks to reject
messages with timestamps later than this period after the task reached its
taskDuration; for example if this is set to `PT1H`, the taskDuration is set to
`PT1H` and the supervisor creates a task at *2016-01-01T12:00Z*, messages with
timestamps later than *2016-01-01T14:00Z* will be dropped. **Note:** Tasks
sometimes run past their task duration, for example, in cases of supervisor
failover. Setting earlyMessageRejectionPeriod too low may cause messages to be
dropped unexpectedly whenever a task runs past its originally configured task
duration.|no (default == none)|
+|`autoscalerConfig`|Object|`autoscalerConfig` to specify how to auto scale the
number of Kafka ingest tasks based on Lag metrics. ONLY supported for Kafka
indexing as of now. See [Tasks Autoscaler Properties](#Tasks Autoscaler
Properties) for details.|no (default == null)|
+
+#### Tasks Autoscaler Properties
+
+| Property | Description | Default |
+| ------------- | ------------- | ------------- |
+| `enableTaskAutoscaler` | whether enable this feature or not. Set false or
ignored here will disable `autoscaler` even though `autoscalerConfig` is not
null| false |
+| `metricsCollectionIntervalMillis` | Define the frequency of lag points
collection. | 30000 |
+| `metricsCollectionRangeMillis` | The total time window of lag collection,
Use with `metricsCollectionIntervalMillis`，it means that in the recent
`metricsCollectionRangeMill`, collect lag metric points every
`metricsCollectionIntervalMillis`. | 600000 |
+| `scaleOutThreshold` | The Threshold of scale out action | 6000000 |
+| `triggerScaleOutThresholdFrequency` | If `triggerScaleOutThresholdFrequency`
percent of lag points are higher than `scaleOutThreshold`, then do scale out
action. | 0.3 |
+| `scaleInThreshold` | The Threshold of scale in action | 1000000 |
+| `triggerScaleInThresholdFrequency` | If `triggerScaleInThresholdFrequency`
percent of lag points are lower than `scaleOutThreshold`, then do scale in
action. | 0.9 |
+| `dynamicCheckStartDelayMillis` | Number of milliseconds after supervisor
starts when first check scale logic. | 300000 |
+| `dynamicCheckPeriod` | the frequency of checking whether to do scale action
| 60000 |
Review comment:
Thanks && Changed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
