Leondon9 opened a new pull request, #73702: URL: https://github.com/apache/airflow/pull/73702
This makes `GlueJobOperator`'s `stop_job_run_on_kill` work on the deferrable path: when a deferred Glue task is killed (for example, cleared while running), the trigger now stops the running Glue job run instead of leaving it running. ## Motivation Today `stop_job_run_on_kill` is only honoured by the operator's synchronous `on_kill`, which does not run for a deferred task (the task lives in the triggerer, not on a worker). So a cleared deferrable Glue task left its Spark job running. `EmrServerlessStartJobOperator`'s trigger already cancels the AWS job on kill, so the two Spark-submitting operators behaved differently on the same user action. This brings Glue into line. `GlueJobCompleteTrigger` now honours `stop_job_run_on_kill`: - **Airflow 3.3+** — via the trigger's `on_kill()` hook, which the triggerer calls only for user-initiated kills. - **Older Airflow** — via a `CancelledError` handler in `run()`, guarded by a task-state check (`safe_to_cancel`) so that a triggerer restart does not stop a still-healthy run. The behaviour is opt-in and unchanged by default (`stop_job_run_on_kill=False`). ## Context This is the second of two PRs aligning EMR Serverless and Glue deferrable behaviour so both Spark-submitting operators expose the same knobs: - durable reconnect on EMR Serverless (mirrors Glue's `durable`) — #73701. - this PR: `stop_job_run_on_kill` on Glue's deferrable path (mirrors EMR's `cancel_on_kill`). Paired with `durable`, `stop_job_run_on_kill=True` gives "stop the old run, submit fresh" on clear; leaving it `False` keeps the reconnect-on-clear behaviour. ## Testing - New trigger unit tests: serialization, `on_kill` (enabled/disabled), `run()` cancellation (safe → stop, unsafe → skip, 3.3+ sentinel → skip), and the `safe_to_cancel` task-state check. - New operator test: the deferred trigger receives `stop_job_run_on_kill`. - Full `test_glue.py` trigger + operator suites green (138 passed). - Provider `mypy` and other Docker-dependent prek hooks were not run locally (Docker unavailable); CI will validate them. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 4.8) Generated-by: Claude Code (Opus 4.8) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
