olchas commented on a change in pull request #9593: URL: https://github.com/apache/airflow/pull/9593#discussion_r449062072
########## File path: airflow/providers/google/cloud/operators/dataproc.py ########## @@ -502,32 +506,79 @@ def __init__(self, self.timeout = timeout self.metadata = metadata self.gcp_conn_id = gcp_conn_id + self.delete_on_error = delete_on_error + + def _create_cluster(self, hook): + operation = hook.create_cluster( + project_id=self.project_id, + region=self.region, + cluster=self.cluster, + request_id=self.request_id, + retry=self.retry, + timeout=self.timeout, + metadata=self.metadata, + ) + cluster = operation.result() + self.log.info("Cluster created.") + return cluster + + def _delete_cluster(self, hook): + self.log.info("Deleting the cluster") + hook.delete_cluster( + region=self.region, + cluster_name=self.cluster_name, + project_id=self.project_id, + ) + self.log.info("Cluster %s deleted", self.cluster_name) Review comment: I wonder if it would not be a good idea to raise an exception here. It just seems weird to me that an operator that is supposed to create a cluster ends up deleting one instead and even returns a reference to no-longer-existing cluster. Raising an exception would also allow a second attempt to create the cluster on retry (if retries apply, of course). ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org