Sounds good to me and good to see you open start this feature. I have two questions: If use client submits, we need to do first some design on the UI side or API side, which is more important and involves user interaction. Second, we may also need to do some processing for the task status acquisition, and do the corresponding API status acquisition according to different submission platforms.
It is best to have a related design diagram for these two points, and the other is how to better adapt the plug-in architecture. Best wishes?? Calvin Kirs On 01/11/2022 14:04??????<[email protected]> wrote?? There are more and more tasks running on k8s. Some tasks run in k8s in the form of independent images, in addition to spark, flick on k8s, etc. Dolphinscheduler does not seem to support their scheduling at present. By introducing https://github.com/fabric8io/kubernetes-client To manage the scheduling of k8s tasks. Spark, flink on k8s will also make much higher use of resources. Support for this type can be the same as for yarn. 1. Introduce k8s client 2. Use namespace to control resource and permission management, similar to yarn queue 3. Development based on k8s using independent images or other types of tasks 4. The task management method is the same as that of dolphin scheduler, only add k8s type jobs The above is the preliminary idea of the previous. Our team currently has such using scenario and has enough energy to participate in the development. We hope to discuss the significance and ways of realization. Best Wishes?? qianli2022
