theesen opened a new pull request #21092: URL: https://github.com/apache/airflow/pull/21092
**SparkKubernetesOperator deletes any previous Spark Application with the same name** **Current issue**: If a Spark App is being launched you either have to template the name inside the yaml file (making it unique with a timestamp) or need to delete the previous run of the Spark App in order to prevent failure. Especially for newcomers to the topic this quickly leads to errors by design of the operator. **Open Questions**: I am wondering if we should delete the K8 Spark Application after a successful run as well from inside the Operator. Or at least provide a flag to enable this. _Downsides_: Harder to debug since container is gone in case of errors (or we keep the container if it failed) _Upsides_: Less polluted K8 Cluster with old Spark Apps. I would love to get some input on these thoughts. **Minor Updates**: As far as I could tell the docstring was not correct. You can only pass a String or Dict as `application_file` . The name of the parameter would need to be changed from my point of view as well. But since this is my first PR here, I did not want to introduce a breaking change to an Operator right away. But if someone makes sure I am doing this the right way, I am happy to adjust. closes: #16290 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
