Mighten opened a new issue, #16478: URL: https://github.com/apache/dolphinscheduler/issues/16478
### Search before asking - [X] I had searched in the [DSIP](https://github.com/apache/dolphinscheduler/issues/14102) and found no similar DSIP. ### Motivation Supporting **user-customized K8s YAML tasks** has the following benefits: - **Flexibility:** Unlike the existing K8s low-code job with limited functionality, YAML tasks provide users with the flexibility to define sophisticated task instances in DolphinScheduler, similar to how custom JSON does in DataX. - **Workflow Customization:** Users can integrate operational and maintenance processes into DolphinScheduler using YAML for complex workflows. - **Configuration Requirements:** The current K8s low-code job does not meet users' in-depth needs, particularly for tasks involving *multiple pods* or specific *configurations* like *environment variables* and *tolerations*; in contrast, K8s YAML tasks do. In short, by enabling user-customized YAML tasks, *DolphinScheduler* can better support a wide range of Kubernetes-based workflows and operational requirements. ### Design Detail ### 2.1 Design Overview The following is a Swimlane Diagram showing how this k8s YAML task is *embedded* into Apache DolphinScheduler:  *Figure 2-1(1). Design Overview* 1. **User** starts a Web page to edit and save K8s YAML Workflow. 2. **UI** provides an editor for user to input YAML in Custom Template mode. 3. **API Server** encapsulates command and hands it over to *Master*. 4. **Master** splits the workflow DAG and dispatches tasks to **Worker**. 5. **Worker** picks the appropriate *task executor* and *operation*. E.g., for k8s Pod YAML, Worker picks *YAML Task Executor*, and then picks *Pod Operation*. 6. **Worker** reports status to *Master*. 7. **User** reviews k8s YAML task *log* in the Task Instance Window. ### 2.2 Frontend Design The frontend adds support for user-customized k8s YAML tasks while remaining compatible with the original k8s low-code jobs.  *Figure 2-2(1). Design Overview* 1. **The Web UI layouts** When the user switches on the Custom Template, the Low-code k8s Job fields should *hide* and YAML editor should *appear* (or vice versa), similar to the JSON Custom Template in the *DataX* plugin. This feature, as shown in Figure 2-2(1), is implemented using the *Vue component span*, which is controlled by *reactive variables* (such as `yamlEditorSpan`) in the file `dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-k8s.ts`. 2. **The Request body** When the user switches to Custom Template mode, the request body should include only YAML-related fields (`customConfig` and `yamlContent`), and all previously hidden fields should not be sent. This feature is implemented using the `taskParams` in the file `dolphinscheduler-ui/src/views/projects/task/components/node/format-data.ts` 3. **i18n/locales** Apache DolphinScheduler is an international software and should support multiple languages. The text on the Web UI are retrieved from variables defined in the file `dolphinscheduler-ui/src/locales/{en_US, zh_CN}/project.ts`. And for user-customized k8s YAML tasks, there are three key variables to consider: - `k8s_custom_template`: the label for the switch to enable user-customized k8s YAML tasks. - `k8s_yaml_template`: the label for the text editor used to input user YAML. - `k8s_yaml_empty_tips`: the warning message displayed when a user tries to submit empty YAML This feature is implemented by invoking `t('project.node.${variable_name}')` (such as `t('project.node.k8s_yaml_template')`) in the file `dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-k8s.ts`. ### 2.3 Backend Design The backend design describes the process of how the worker executes user-customized k8s YAML tasks. As shown in Figure 2-3(1), we can see how user-customized k8s YAML Pod tasks are related to the original k8s low-code jobs.  *Figure 2-3(1). Backend Design Overview* After the worker checks the parameters, `K8sYamlTaskExecutor` is loaded for the current user-customized k8s YAML Pod task. Once the YAML is parsed into `HasMetadata`, its `kind` field is used to assign `abstractK8sOperation` as `K8sPodOperation` for executing the YAML Pod task. 1. **K8s Task Executors**  *Figure 2-3(2). K8s Task Executors* Three k8s task executor are involved, as shown in Figure 2-3(2): - `AbstractK8sTaskExecutor` is an abstract class that represents a k8s task executor. - `K8sTaskExecutor` is a concrete class that *extends* `AbstractK8sTaskExecutor` to represent a low-code executor - `K8sYamlTaskExecutor` is a concrete class that *extends* `AbstractK8sTaskExecutor` to represent a user-customized k8s YAML task executor. 2. **K8s Operation handler**  *Figure 2-3(3). K8s Operation Handlers* Two operation handlers are involved, as shown in Figure 2-3(3): - `AbstractK8sOperation` is an interface representing all k8s resource operations. - `K8sPodOperation` is a concrete class that implements `AbstractK8sOperation` to handle *Pod* operations ### 2.4 Usecase Design A typical use case for a k8s YAML task includes *uploading* YAML, *online* workflows, and *starting* workflows, similar to k8s low-code jobs, unless users switch to the *Custom Template* option to fill in YAML.  *Figure 2-4(1). Usecase Design* 1. The user **edits** a k8s YAML node in a workflow 2. If the *Custom Template* is activated and YAML content is not blank, the user may **online** this whole workflow 3. If the workflow is *online*, the user may **start** the workflow and review the logs generated during the execution of the workflow. ### Compatibility, Deprecation, and Migration Plan ### 3.1 Compatibility Plan The user-customized k8s YAML feature requires only `customConfig` to be activated, By default, the value is 0, which applies to the existing k8s low-code jobs. **Action List:** Extension of operations for the k8s YAML task: - [x] Pod #15301 - [ ] ConfigMaps - [ ] ... The remainder of this section will demonstrate the *flexibility* and *compatibility* of this design by using the example of introducing `Configmaps`: ```java this.k8sYamlType = K8sYamlType.valueOf(this.metadata.getKind()); generateOperation(); ``` After parsing with `YamlUtils::load`, the `kind` field acquired by `this.metadata.getKind()` will be `ConfigMaps`. Then, `this.k8sYamlType` is determined and used to generate the corresponding operations: ```java private void generateOperation() { switch (k8sYamlType) { case Pod: abstractK8sOperation = new K8sPodOperation(k8sUtils.getClient()); break; case ConfigMaps: abstractK8sOperation = new K8sConfigmapsOperation(k8sUtils.getClient()); break; default: throw new TaskException( String.format("K8sYamlTaskExecutor do not support type %s", k8sYamlType.name())); } } ``` Consequently, `generateOperation()` will set `this.abstractK8sOperation` to a new instance of `K8sConfigmapsOperation`. Next, we can implement K8sConfigmapsOperation to handle the ConfigMaps operations. ### 3.2 Deprecation Plan N/A for now, *waiting for community opinions*. ### 3.3 Migration Plan N/A for now, *waiting for community opinions*. ### Test Plan ### 4.1 Overview The User-customized k8s YAML task feature allows users to submit YAML task to k8s, including *Pod*, *ConfigMaps*, and other resources. This test plan aims to ensure that the feature functions as expected and meets user requirements. ### 4.2 Scope 1. **YAML Pod** | Test Case #| Name | Action | Expectation | | :--------: | :----------------| :------------------------- | :------------------------------------ | | 1 | UI Display | Edit YAML, save and reopen | The YAML content stays up-to-date. | | 2 | UI Validation | try to submit empty YAML | The UI modal dialog intercepts empty YAML. | | 3 | Online Workflow | Save workflow, and online | The User successfully brings the workflow online. | | 4 | Dryrun Workflow | Run workflow as dryrun mode| The Master successfully dry runs this task. | | 5 | Test Workflow | Run workflow as test mode | The Worker successfully tests this task. | | 6 | Run Workflow | Run workflow | The Worker successfully runs this task. | ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
