georgew5656 commented on code in PR #14674:
URL: https://github.com/apache/druid/pull/14674#discussion_r1279345659
##########
docs/development/extensions-contrib/k8s-jobs.md:
##########
@@ -28,85 +28,45 @@ Consider this an [EXPERIMENTAL](../experimental.md) feature
mostly because it ha
## How it works
-The K8s extension builds a pod spec using the specified pod adapter, the
default implementation takes the podSpec of your `Overlord` pod and creates a
kubernetes job from this podSpec. Thus if you have sidecars such as Splunk or
Istio it can optionally launch a task as a K8s job. All jobs are natively
restorable, they are decoupled from the druid deployment, thus restarting pods
or doing upgrades has no affect on tasks in flight. They will continue to run
and when the overlord comes back up it will start tracking them again.
+The K8s extension builds a pod spec from each task using the specified pod
adapter. All jobs are natively restorable, they are decoupled from the druid
deployment, thus restarting pods or doing upgrades has no affect on tasks in
flight. They will continue to run and when the overlord comes back up it will
start tracking them again.
-## Pod Adapters
-The logic defining how the pod template is built for your kubernetes job
depends on which pod adapter you have specified.
-
-### Overlord Single Container Pod Adapter
-The overlord single container pod adapter takes the podSpec of your `Overlord`
pod and creates a kubernetes job from this podSpec. This is the default pod
adapter implementation, to explicitly enable it you can specify the runtime
property `druid.indexer.runner.k8s.adapter.type: overlordSingleContainer`
-
-### Overlord Multi Container Pod Adapter
-The overlord multi container pod adapter takes the podSpec of your `Overlord`
pod and creates a kubernetes job from this podSpec. It uses kubexit to manage
dependency ordering between the main container that runs your druid peon and
other sidecars defined in the `Overlord` pod spec. To enable this pod adapter
you can specify the runtime property `druid.indexer.runner.k8s.adapter.type:
overlordMultiContainer`
-
-### Custom Template Pod Adapter
-The custom template pod adapter allows you to specify a pod template file per
task type. This adapter requires you to specify a `base` pod spec which will
be used in the case that a task specific pod spec has not been defined. To
enable this pod adapter you can specify the runtime property
`druid.indexer.runner.k8s.adapter.type: customTemplateAdapter`
-
-The base pod template must be specified as the runtime property
`druid.indexer.runner.k8s.podTemplate.base: /path/to/basePodSpec.yaml`
-Task specific pod templates must be specified as the runtime property
`druid.indexer.runner.k8s.podTemplate.{taskType}:
/path/to/taskSpecificPodSpec.yaml` where {taskType} is the name of the task
type i.e `index_parallel`
## Configuration
To use this extension please make sure to
[include](../../configuration/extensions.md#loading-extensions)`druid-kubernetes-overlord-extensions`
in the extensions load list for your overlord process.
-The extension uses the task queue to limit how many concurrent tasks (K8s
jobs) are in flight so it is required you have a reasonable value for
`druid.indexer.queue.maxSize`. Additionally set the variable
`druid.indexer.runner.namespace` to the namespace in which you are running
druid.
+The extension uses `druid.indexer.runner.capacity` to limit the number of k8s
jobs in flight. You should set this to a reasonable value. Additionally set the
variable `druid.indexer.runner.namespace` to the namespace in which you are
running druid.
Review Comment:
do you think "the same number as the total # of mm capacity slots" is a okay
recommendation here? maybe leaving a note that having this too high may cause
some memory issues due to # of threads?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
