nlippis commented on PR #14156: URL: https://github.com/apache/druid/pull/14156#issuecomment-1523900915
> this is mostly for my own understanding, but as I understand there is a alternative solution where you could still have getKnownTasks directly query k8s to get the list of peon jobs and then join each of them to be more in line with the original "use k8s for everything" idea? > > I think I like this current system better since it still gets info from k8s but doesn't have to do so many call each time tasks are listed, but just wanted to confirm this was a conscious choice Reading from state local to the `KubernetesTaskRunner` (in this case the `tasks` hashmap) instead of directly calling K8S was required. When the tasks are queued in the thread queue (not running) there won't be any corresponding K8S job so making a direct call in `getKnownTasks()` won't return all tasks that the task runner knows about. While there is a solution that calls both K8S and checks state local to the task runner it is complicated to get that code right and in the end is unnecessary. Another unstated goal of this approach is to improve performance when the task runner is managing hundreds of tasks. Reading from local state improves performance of in many ways. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
