georgew5656 commented on code in PR #14674: URL: https://github.com/apache/druid/pull/14674#discussion_r1277681298
########## extensions-contrib/kubernetes-overlord-extensions/README.md: ########## @@ -0,0 +1,222 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +# Kubernetes Task Scheduling Extension +The Kubernetes Task Scheduling extension allows a Druid cluster running on Kubernetes to schedule +its tasks as Kubernetes Jobs instead of sending them to workers (middle managers or indexers). + +## How the Kubernetes Task Scheduler works. +The Kubernetes Task Scheduler replaces the HTTP or Zookeeper based task runner (which both send tasks to workers) on the overlord. +Based on which TaskAdapter (`druid.indexer.runner.k8s.adapter.type`) is selected, every task that the overlord runs is transformed into a Kubernetes Job that runs +the Druid peon process for that task and then exits. The overlord then watches the K8s API for the Job's success or failure. + +## Configuration + +## Enabling +1. Create a role binding for the overlord's service account that provides the needed permissions for interacting with Kubernetes. + ``` + kind: Role + apiVersion: rbac.authorization.k8s.io/v1 + metadata: + namespace: <druid-namespace> + name: druid-k8s-task-scheduler + rules: + - apiGroups: ["batch"] + resources: ["jobs"] + verbs: ["get", "watch", "list", "delete", "create"] + - apiGroups: [""] + resources: ["pods", "pods/log"] + verbs: ["get", "watch", "list", "delete", "create"] + --- + kind: RoleBinding + apiVersion: rbac.authorization.k8s.io/v1 + metadata: + name: druid-k8s-binding + namespace: <druid-namespace> + subjects: + - kind: ServiceAccount + name: <druid-overlord-k8s-service-account> + namespace: <druid-namespace> + roleRef: + kind: Role + name: druid-k8s-task-scheduler + apiGroup: rbac.authorization.k8s.io + ``` + +2. Configure the overlord's runtime properties to use the Kubernetes task scheduler. + ``` + druid.indexer.runner.type=k8s + druid.indexer.task.encapsulatedTask=true + druid.indexer.runner.namespace=<druid-namespace> + druid.indexer.runner.capacity=100 (number of concurrent jobs that can be sent to kubernetes at once) + ``` + +3. Choose a Task Adapter (determines how tasks will be converted to a Kubernetes Job) + #### Option 1. Copy the overlord's pod template to use as the base job template + This adapter takes the overlord's own pod specification and injects needed information (like the task id) on top of it. Review Comment: IMO this method is best used for testing the scheduling capability of the K8s runner or for quick startup (it's easier to run than having to inject a pod template) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
