nishantmonu51 commented on pull request #10910: URL: https://github.com/apache/druid/pull/10910#issuecomment-810050804
> > > letting the overlord create k8s pods is a huge change including API, Task scheduling model and etc. > > > > > > That's too bad, since one of the original goals of the TaskRunner interface is that it could be used to allow the Overlord to launch tasks directly as YARN applications. (At the time, YARN was more popular than K8S 🙂) > > I guess it didn't live up to this goal, if you found it easier to have the MMs launch K8S pods. > > Anyway, I'm not a K8S expert, but this seems like a very interesting idea. > > Hi @gianm Thanks for your attention. Maybe I didn’t make my point clear. > In my opinion, I just follow the current workflow as overlord -> middlemanger -> peon > The difference between ForkingTaskRunner and new K8sForkingTaskRunner(something like `ThreadingTaskRunner` for `CliIndexer`) is that middlemanger launch peon as pod in Kubernetes rather than a child process on the same machine. > > Because of the same workflow mentioned above, we don't need to consider potential api, task life control or other changes. So that I believe it's easier to have the MMs launch K8S pods :) I think it's possible and a valid use-case to use ForkingTaskRunner/RemoteTaskRunner/K8sForkingTaskRunner directly on the overlord, it's just a configuration change on the overlord unless the k8sForkingTaskRunner depends on something specific from the MiddleManager. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
