abhishekagarwal87 commented on code in PR #14156: URL: https://github.com/apache/druid/pull/14156#discussion_r1192020779
########## extensions-contrib/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/KubernetesTaskRunner.java: ########## @@ -89,174 +69,142 @@ /** * Runs tasks as k8s jobs using the "internal peon" verb. - * One additional feature of this class is that kubernetes is the source of truth, so if you launch a task - * shutdown druid, bring up druid, the task will keep running and the state will be updated when the cluster - * comes back. Thus while no tasks are technically restorable, all tasks once launched will run in isolation to the - * extent possible without requiring the overlord consistently up during their lifetime. + * The KubernetesTaskRunner runs tasks by transforming the task spec into a K8s Job spec based + * on the TaskAdapter it is configured with. The KubernetesTaskRunner has a pool of threads + * (configurable with the capacity configuration) to track the jobs (1 thread tracks 1 job). + * + * Each thread calls down to the KubernetesPeonLifecycle class to submit the Job to K8s and then + * waits for the lifecycle class to report back with the Job's status (success/failure). + * + * If there are not enough threads in the thread pool to execute and wait for a job, then the + * task is put in a queue and left in WAITING state until another task completes. + * + * When the KubernetesTaskRunner comes up it attempts to restore its internal mapping of tasks + * from Kubernetes by listing running jobs and calling join on each job, which spawns a thread to + * wait for the fabric8 client library to report back, similar to what happens when a new + * job is run. Review Comment: Nice. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
