fenglu-g commented on issue #4064: AIRFLOW-3149 Support dataproc cluster deletion on ERROR URL: https://github.com/apache/incubator-airflow/pull/4064#issuecomment-434850291 Sorry for the late reply. Thank you @dossett for the detailed explanation. The root cause seems to be that DataprocClusterCreateOperator is not idempotent. Similar to what you have described, how about we re-factor the operator based on the following logic? - check existence of dataproc cluster - if yes and in "running" state, no-op. - else, delete the cluster, and attempt create again. So we don't have to export this additional delete_cluster_on_error?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
