This is an automated email from the ASF dual-hosted git repository.
brile pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git
The following commit(s) were added to refs/heads/master by this push:
new c8cbbbc4d2e Document stopTaskCount supervisor property (#18165)
c8cbbbc4d2e is described below
commit c8cbbbc4d2e8c77a0b7a25eaa8b502e858c69203
Author: Jill Osborne <[email protected]>
AuthorDate: Wed Jul 2 22:30:18 2025 +0100
Document stopTaskCount supervisor property (#18165)
---
docs/ingestion/supervisor.md | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/docs/ingestion/supervisor.md b/docs/ingestion/supervisor.md
index b5b38df66dc..50e8857b00d 100644
--- a/docs/ingestion/supervisor.md
+++ b/docs/ingestion/supervisor.md
@@ -64,6 +64,7 @@ For configuration properties specific to Kafka and Kinesis,
see [Kafka I/O confi
|`lateMessageRejectionStartDateTime`|ISO 8601 date time|Configures tasks to
reject messages with timestamps earlier than this date time. For example, if
this property is set to `2016-01-01T11:00Z` and the supervisor creates a task
at `2016-01-01T12:00Z`, Druid drops messages with timestamps earlier than
`2016-01-01T11:00Z`. This can prevent concurrency issues if your data stream
has late messages and you have multiple pipelines that need to operate on the
same segments, such as a realtim [...]
|`lateMessageRejectionPeriod`|ISO 8601 period|Configures tasks to reject
messages with timestamps earlier than this period before the task was created.
For example, if this property is set to `PT1H` and the supervisor creates a
task at `2016-01-01T12:00Z`, Druid drops messages with timestamps earlier than
`2016-01-01T11:00Z`. This may help prevent concurrency issues if your data
stream has late messages and you have multiple pipelines that need to operate
on the same segments, such as a [...]
|`earlyMessageRejectionPeriod`|ISO 8601 period|Configures tasks to reject
messages with timestamps later than this period after the task reached its task
duration. For example, if this property is set to `PT1H`, the task duration is
set to `PT1H` and the supervisor creates a task at `2016-01-01T12:00Z`, Druid
drops messages with timestamps later than `2016-01-01T14:00Z`. Tasks sometimes
run past their task duration, such as in cases of supervisor failover.|No||
+|`stopTaskCount`|Integer|Limits the number of ingestion tasks Druid can cycle
at any given time. If not set, Druid can cycle all tasks at the same time. If
set to a value less than `taskCount`, your cluster needs fewer available slots
to run the supervisor. You can save costs by scaling down your ingestion tier,
but this can lead to slower cycle times and lag. See
[`stopTaskCount`](#stoptaskcount) for more information.|No|`taskCount` value|
#### Task autoscaler
@@ -185,6 +186,14 @@ The following example shows a supervisor spec with
`lagBased` autoscaler:
```
</details>
+#### `stopTaskCount`
+
+Before you set `stopTaskCount`, note the following:
+
+- Some operations require all tasks to cycle at the same time, for example
changes to the supervisor spec and change to the number of Kafka partitions.
These operations can cause lag without sufficient task slot capacity.
+- The [task autoscaler](#task-autoscaler) ignores `stopTaskCount` when
shutting down tasks in response to a task count change. The task autoscaler
needs to redistribute partitions across tasks, which requires all tasks to be
shut down.
+- If you set `stopTaskCount` to a value less than `taskCount`, Druid cycles
the longest running tasks first, then other tasks up to the value set.
+
### Tuning configuration
The `tuningConfig` object is optional. If you don't specify the `tuningConfig`
object, Druid uses the default configuration settings.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
