vincbeck opened a new pull request, #27820:
URL: https://github.com/apache/airflow/pull/27820
Add retry option in `RedshiftDeleteClusterOperator` to retry when an
operation is running in the cluster. When an operation is running in the
cluster, the deletion fails and throw an exception like below. This option
allows users to retry in such scenario.
```
INFO [0m airflow.task:taskinstance.py:1278 Executing
<Task(RedshiftDeleteClusterOperator): delete_cluster> on 2021-01-01
00:00:00+00:00
INFO [0m airflow.task:taskinstance.py:1487 Exporting the following env
vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=example_redshift_to_s3
AIRFLOW_CTX_TASK_ID=delete_cluster
AIRFLOW_CTX_EXECUTION_DATE=2021-01-01T00:00:00+00:00
AIRFLOW_CTX_TRY_NUMBER=1
AIRFLOW_CTX_DAG_RUN_ID=backfill__2021-01-01T00:00:00+00:00
INFO airflow.hooks.base:base.py:71 Using connection ID 'aws_default' for
task execution.
INFO botocore.credentials:credentials.py:1180 Found credentials in
environment variables.
ERROR airflow.task:taskinstance.py:1746 Task failed with exception
Traceback (most recent call last):
File
"/opt/airflow/airflow/providers/amazon/aws/operators/redshift_cluster.py", line
492, in execute
final_cluster_snapshot_identifier=self.final_cluster_snapshot_identifier,
File
"/opt/airflow/airflow/providers/amazon/aws/hooks/redshift_cluster.py", line
112, in delete_cluster
FinalClusterSnapshotIdentifier=final_cluster_snapshot_identifier,
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line
495, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line
914, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.InvalidClusterStateFault: An error occurred
(InvalidClusterState) when calling the DeleteCluster operation: There is an
operation running on the Cluster. Please try to delete it at a later time.
INFO airflow.task:taskinstance.py:1301 Marking task as FAILED.
dag_id=example_redshift_to_s3, task_id=delete_cluster,
execution_date=20210101T000000, start_date=20221113T120403,
end_date=20221113T120403
ERROR airflow.executors.debug_executor.DebugExecutor:debug_executor.py:85
Failed to execute task: An error occurred (InvalidClusterState) when calling
the DeleteCluster operation: There is an operation running on the Cluster.
Please try to delete it at a later time..
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
