Radeity opened a new pull request, #13550: URL: https://github.com/apache/dolphinscheduler/pull/13550
<!--Thanks very much for contributing to Apache DolphinScheduler. Please review https://dolphinscheduler.apache.org/en-us/community/development/pull-request.html before opening a pull request.--> ## Purpose of the pull request - This PR closes https://github.com/apache/dolphinscheduler/issues/13511. ## Brief change log Now in DS, use `Yarn` as application manager by default, e.g. directly executes `yarn application -kill <appId>` for canceling application. For using different resource manager such as `Kubernetes`, this pr makes following changes: - Create `ApplicationManager`, implement `KubernetesApplicationManager` and `YarnApplicationManager`, design as SPI way. Only support **kill application** and **query application status** now. - Modify am-related configuration and function name, e.g. change `killYarnJobWhenTaskFailover` to `killApplicationWhenTaskFailover`. - Add Kubernetes application(pod) status check when submit process exit. Because when deploying in cluster mode on k8s and failures happen inside k8s cluster, submit process will exit normally, which will show wrong status of task execution. Support submit Spark task directly on Kubernetes(on native Kubernetes, not Spark Operator). - Modify Spark's resource related configuration to accommodate Spark on K8S deploy mode, e.g. change `--driver-cores` to `--conf spark.driver.cores`. - Generate master URL based on whether user choose cluster namespace. <img width="597" alt="image" src="https://user-images.githubusercontent.com/45198818/218303053-171dfe6a-93c2-4f6f-bcd1-78fefe7935d1.png";> - Add `--conf spark.kubernetes.driver.label.dolphinscheduler-label=<processInstanceId>_<taskInstanceId>` automatically for killing driver pod. - **NOTICE** that Spark doesn't support any configuration to delete driver pod (or i don't that configuration), DS should better not automatically delete it, because pod's logs might be useful. ## Verify this pull request -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
![https://user-images.githubusercontent.com/45198818/218303053-171dfe6a-93c2-4f6f-bcd1-78fefe7935d1.png"](https://user-images.githubusercontent.com/45198818/218303053-171dfe6a-93c2-4f6f-bcd1-78fefe7935d1.png"){kind=link}