amoghrajesh opened a new issue, #71490:
URL: https://github.com/apache/airflow/issues/71490
`GlueJobOperator`'s handling of a Glue job run ending in `STOPPED` (e.g.
cancelled manually from the AWS console, or by this operator's own `on_kill`)
is inconsistent across code paths -- three different answers exist for the same
state today:
| Path | Where | `STOPPED` classified as | Result |
|---|---|---|---|
| Deferrable, non-verbose |
`providers/amazon/src/airflow/providers/amazon/aws/waiters/glue.json`, waiter
`job_complete` | Explicit `"state": "failure"` acceptor | Task fails |
| Deferrable, verbose | `GlueJobCompleteTrigger.run()`'s hand-rolled polling
loop (`providers/amazon/src/airflow/providers/amazon/aws/triggers/glue.py`) |
Bundled with `SUCCEEDED`: `if job_run_state in ("SUCCEEDED", "STOPPED"): ...
"status": "success"` | Task succeeds |
| Synchronous | `GlueJobOperator.poll_until_complete`
(`JOB_RUN_SUCCESS_STATES = ("SUCCEEDED",)`) | Same bucket as `FAILED`/`TIMEOUT`
| Task fails |
The synchronous path was made to agree with the deferrable non-verbose
path's waiter as part of #71211 (closing a "self-inflicted stop silently
reported as success" gap). The deferrable *verbose* path was not touched and
still disagrees with both -- it's a separate, pre-existing bug in the trigger's
custom polling loop, not something #71211 introduced.
Fix direction: make the verbose branch of `GlueJobCompleteTrigger.run()`
treat `STOPPED` as a failure, matching the waiter used by the non-verbose
branch and the operator's own synchronous handling.
Raised during review of #71211:
https://github.com/apache/airflow/pull/71211#discussion_r3764425797
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
