amoghrajesh opened a new issue, #71485: URL: https://github.com/apache/airflow/issues/71485
`ResumableJobMixin`'s crash-recovery decision logic (reconnect / already-succeeded / terminal-resubmit) lives entirely inside `execute_resumable`, inlined together with the synchronous polling that follows it. There's no way for a caller to get just the decision without also getting blocked on `poll_until_complete`. This matters for any operator that supports both `deferrable=True` and durable execution, `GlueJobOperator` included. On the deferrable path, the operator calls `submit_job` directly from `execute()`, bypassing the mixin's `task_state_store` read entirely — nothing is ever persisted there for a deferred run. So a deferrable retry has no cheap way to check "is there already a run I can reconnect to" and instead always falls through to the operator's own bootstrap fallback (for Glue, a full paginated `get_job_runs` scan), even though the run id could have been persisted to `task_state_store` before `defer()` and read back cheaply on retry. Fixing this requires the mixin to expose its reconnect-decision step (read stored id, call `get_job_status`, apply `is_job_active`/`is_job_succeeded`) as something callable on its own, separate from the polling loop in `execute_resumable`. Once that exists, a deferrable operator could: 1. Call the decision step before `defer()`. 2. If nothing to reconnect to, submit fresh and persist the id to `task_state_store`. 3. On a deferrable retry, call the decision step again before deciding whether to resubmit. This would let deferrable retries reconnect via the cheap store lookup instead of an operator-specific bootstrap scan, for every current and future `ResumableJobMixin` port that supports `deferrable=True`. Raised during review of #71211 (kaxil, https://github.com/apache/airflow/pull/71211#discussion_r3764425769 — see comment thread for full context). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
