There are more and more tasks running on k8s. Some tasks run in k8s in the form of independent images, in addition to spark, flick on k8s, etc. Dolphinscheduler does not seem to support their scheduling at present.
By introducing https://github.com/fabric8io/kubernetes-client To manage the scheduling of k8s tasks. Spark, flink on k8s will also make much higher use of resources. Support for this type can be the same as for yarn. 1. Introduce k8s client 2. Use namespace to control resource and permission management, similar to yarn queue 3. Development based on k8s using independent images or other types of tasks 4. The task management method is the same as that of dolphin scheduler, only add k8s type jobs The above is the preliminary idea of the previous. Our team currently has such using scenario and has enough energy to participate in the development. We hope to discuss the significance and ways of realization. Best Wishes?? qianli2022
