vtlim commented on code in PR #16600:
URL: https://github.com/apache/druid/pull/16600#discussion_r1644840615
##########
docs/development/extensions-contrib/k8s-jobs.md:
##########
@@ -217,59 +492,112 @@ data:
druid.peon.mode=remote
druid.indexer.task.encapsulatedTask=true
```
-#### Dynamic Pod Template Selection Config
-The Dynamic Pod Template Selection feature enhances the K8s extension by
enabling more flexible and dynamic selection of pod templates based on task
properties. This process is governed by the `PodTemplateSelectStrategy`. Below
are the two strategies implemented:
+</details>
-|Property|Description|Default|
-|--------|-----------|-------|
-|`TaskTypePodTemplateSelectStrategy`| This strategy selects pod templates
based on task type for execution purposes, implementing the behavior that maps
templates to specific task types. | true |
-|`SelectorBasedPodTemplateSelectStrategy`| This strategy evaluates a series of
selectors, known as `selectors`, which are aligned with potential task
properties. | false |
+#### Pod template selection
+
+The pod template adapapter can select which pod template should be used for a
task using the [task runner execution config](#dynamic-config)
-`SelectorBasedPodTemplateSelectStrategy`, the strategy implementing this new
feature, is based on conditional `selectors` that match against top-level keys
from the task payload. Currently, it supports matching based on task context
tags, task type, and dataSource. These selectors are ordered in the dynamic
configuration, with the first selector given the highest priority during the
evaluation process. This means that the selection process uses these ordered
conditions to determine a task’s Pod template. The first matching condition
immediately determines the Pod template, thereby prioritizing certain
configurations over others. If no selector matches, it will fall back to an
optional `defaultKey` if configured; if there is still no match, it will use
the `base` template.
+##### Select based on task type
-Example Configuration:
+The `TaskTypePodTemplateSelectStrategy` strategy selects pod templates based
on task type for execution purposes,
+implementing the behavior that maps templates to specific task types. This is
the default pod template selection
+strategy. To explicitly select this strategy, set the
`podTemplateSelectStrategy` in the dynamic execution config to
+
+```json
+{ "type": "default" }
+```
+
+Task specific pod templates can be specified as the runtime property
+`druid.indexer.runner.k8s.podTemplate.{taskType}:
/path/to/taskSpecificPodSpec.yaml` where {taskType} is the name of the
+task type. For example, `index_parallel`.
+
+If you are trying to use the default image's environment variable parsing
feature to set runtime properties, you need to add a extra escape underscore
when specifying pod templates.
+For example, set the environment variable
`druid_indexer_runner_k8s_podTemplate_index__kafka` when you set the runtime
property `druid.indexer.runner.k8s.podTemplate.index_kafka`
-We define two template keys in the configuration—`low-throughput` and
`medium-throughput`—each associated with specific task conditions and arranged
in a priority order.
-- Low Throughput Template: This is the first template evaluated and has the
highest priority. Tasks that have a context tag
`billingCategory=streaming_ingestion` and a datasource of `wikipedia` will be
classified under the `low-throughput` template. This classification directs
such tasks to utilize a predefined pod template optimized for low throughput
requirements.
+The following example shows a configuration for task-based pod template
selection:
-- Medium Throughput Template: If a task does not meet the low-throughput
criteria, the system will then evaluate it against the next selector in order.
In this example, if the task type is index_kafka, it will fall into the
`medium-throughput` template.
+```properties
+druid.indexer.runner.k8s.podTemplate.base=/path/to/basePodSpec.yaml
+druid.indexer.runner.k8s.podTemplate.index_kafka=/path/to/kafkaPodSpec.yaml
```
+
+##### Select based on one or more conditions
+
+The `SelectorBasedPodTemplateSelectStrategy` strategy evaluates a series of
criteria within `selectors` to determine
+which pod template to use to run the task. Pod templates are configured in
the runtime properties like
+`druid.indexer.runner.k8s.podTemplate.<selectionKey>=...`.
+
+```json
+{
+ "type": "selectorBased",
+ "selectors": [
+ {
+ "selectionKey": "podSpec1",
+ "context.tags":
+ {
+ "userProvidedTag": ["tag1", "tag2"]
+ },
+ "dataSource": ["wikipedia"]
+ },
+ {
+ "selectionKey": "podSpec2",
+ "type": ["index_kafka"]
+ }
+ ]
+}
+```
+
+Selectors are processed in order. Druid selects the template based on the
first matching selector. If a task does not
+match any selector in the list, it will use the `base` pod template.
+
+For a task to match a selector, all the conditions within the selector must
match. A selector can match on
+- `type`: Type of the task
Review Comment:
```suggestion
- `type`: Type of the task.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
