yuchaoran2011 opened a new pull request #253: URL: https://github.com/apache/incubator-yunikorn-k8shim/pull/253
### What is this PR for? Improved the integration with Spark operator plugin. The areas of improvement as as follows: * Make sure that individual pods fall under the responsibility of the general plugin. Make the Spark operator plugin manage the SparkApplication CRD only * Make sure that the general and operator plugins see the same application ID for the same SparkApplication. The ID is the string generated by the Spark K8s backend that starts with "spark-" * Make sure that when a SparkApplication completes or fails, cleanup is performed for all placeholders if there's any that remain at that time * When cleaning up placeholders, there are often cases where the placeholder doesn't exist anymore and that will cause the deletion to fail. Do not add the placeholder to the orphan pod list in this case because there's no need to retry deletion. * Added and updated some logging to aid monitoring and debugging ### What type of PR is it? * [ ] - Bug Fix * [ ] - Improvement * [ ] - Feature * [ ] - Documentation * [ ] - Hot Fix * [ ] - Refactoring ### Todos * This is only part of the work for the referenced JIRA. A separate PR will be made on https://github.com/GoogleCloudPlatform/spark-on-k8s-operator to complete the integration. ### What is the Jira issue? * https://issues.apache.org/jira/browse/YUNIKORN-558 (Integrate YuniKorn with Spark K8S Operator) ### How should this be tested? All the existing unit tests passed. Not quite sure how to unit test this feature specifically. But probably we can enhance the integration tests to cover it. We have been running this patch for relatively large Spark operator workloads for a few weeks and it has been stable for us. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
