seanghaeli opened a new pull request, #69574:
URL: https://github.com/apache/airflow/pull/69574
## Why
`RedshiftDeleteClusterOperator` retries on `InvalidClusterStateFault` when
the cluster is mid-transition (pausing/resuming/resizing), but the retry budget
was **hardcoded** to `10 attempts * 15s = 2.5 minutes`. Real transitions
routinely take longer (a pause is minutes; a classic resize ~6+ min), so the
loop exhausts and re-raises before the cluster becomes deletable — the task
fails and the cluster **leaks** (keeps running/billing until manually deleted).
The error seen mid-transition:
```
InvalidClusterState: Unable to delete the cluster ... You can only delete
clusters with
ACTIVE, INCOMPATIBLE_HSM, INSUFFICIENT_VPCE, HARDWARE_FAILURE,
INCOMPATIBLE_NETWORK,
PAUSED, INACCESSIBLE_KMS_KEY, INSUFFICIENT_CAPACITY, INCOMPATIBLE_RESTORE
lifecycles.
```
## What
Fixed for both execution modes:
**Non-deferrable mode:** raise the synchronous busy-retry budget to `60 *
15s = ~15 min` so it outlasts a transition. Still fail-loud once exhausted.
**Deferrable mode:** previously the synchronous retry loop ran *before* the
defer, blocking the worker for the whole budget (defeating the purpose of
deferrable mode). Now the operator attempts the delete once; on
`InvalidClusterStateFault` it **defers** to a new
`RedshiftClusterSettledTrigger` — an `AwsBaseWaiterTrigger` backed by a custom
`cluster_deletable` waiter that treats transitional states as retry and fires
once the cluster reaches a deletable lifecycle. A callback then re-issues the
delete and defers to the existing `RedshiftDeleteClusterTrigger`
(`cluster_deleted`); on a race it re-defers to the settle trigger. No worker is
blocked at any point.
Bounded by the existing `poll_interval` / `max_attempts` params — no new
configuration knobs.
### Consistency with the codebase
Mirrors the deferrable re-defer pattern established for EKS in #32355
(`EksCreateClusterOperator.deferrable_create_cluster_next`), and
`RedshiftClusterSettledTrigger` uses `AwsBaseWaiterTrigger` + a custom waiter
like the other five Redshift triggers and the base-class direction of #35278.
## Testing
Unit tests (24 passed): sync busy-retry defaults / exhaust-then-raise /
succeed-on-second-attempt / sync-mode-still-uses-loop; deferrable
defer-to-settle, re-issue, re-defer-on-race, error-event-raises, already-gone
short-circuit; `RedshiftClusterSettledTrigger` serialization +
`cluster_deletable` waiter.
### End-to-end before/after — real AWS (Breeze, `airflow dags test`),
identical ~6.5 min classic-resize transient
**Non-deferrable — BEFORE (stock `main`, 2.5-min budget) → FAILS, cluster
leaks:**
```
20:32:04 Cluster in resizing state, unable to delete. 9 attempts remaining.
... (15s apart, 9→1) ...
20:34:22 botocore.errorfactory.InvalidClusterStateFault: Unable to delete
the cluster rs-e2e-sync ...
DagRun ... state=failed TASK_EXIT=1
```
**Non-deferrable — AFTER (this PR, 15-min budget) → SUCCEEDS, cluster
deleted:**
```
20:58:33 Cluster in resizing state, unable to delete. 49 attempts
remaining. <-- past stock's 10-cap
21:01:07 Cluster in resizing state, unable to delete. 39 attempts remaining.
21:01:22 new_state=success DagRun ... state=success TASK_EXIT=0
```
**Deferrable — BEFORE (stock `main`) → blocks worker 2.5 min then FAILS
(never defers):**
```
20:51:53 Cluster in resizing state, unable to delete. 9 attempts remaining.
... 9→1 ...
20:54:11 new_state=failed DagRun ... state=failed TASK_EXIT=1
```
**Deferrable — AFTER (this PR) → async wait via `cluster_deletable` waiter,
then delete → SUCCEEDS:**
```
21:02:10 Cluster rs-e2e-async is busy; deferring until it settles into a
deletable state.
21:02:10 Pausing task as DEFERRED.
21:02:11 Waiting for redshift cluster to settle into a deletable state:
['resizing']
... waiter treats 'resizing' as retry for ~10 min, task stays DEFERRED
(worker not blocked) ...
21:12:38 (settled) delete re-issued and accepted -> defer to cluster_deleted
21:15:10 Redshift Cluster deletion in progress: ['deleting']
21:16:10 Cluster deleted successfully DagRun ... state=success
TASK_EXIT=0
```
AWS-side status timeline confirmed for both: `resizing → available →
deleting → ClusterNotFound`. Both clusters fully deleted, no leak. (task logs
are the verification artifact; no UI screenshots — headless. Classic resize was
used as the transient because pause/resume settle too fast to exceed stock's
2.5-min budget; any `InvalidClusterStateFault`-producing operation exercises
the same path.)
---
Generated-by: Claude Code (Opus)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
