Leondon9 commented on code in PR #73701:
URL: https://github.com/apache/airflow/pull/73701#discussion_r4371089611
##########
providers/amazon/src/airflow/providers/amazon/aws/operators/emr.py:
##########
@@ -1369,7 +1386,24 @@ def __init__(
self.job_id: str | None = None
self.deferrable = deferrable
self.enable_application_ui_links = enable_application_ui_links
- self.cancel_on_kill = cancel_on_kill
+ # In deferrable mode durable reconnects to the still-running job on
clear, while cancel_on_kill
+ # (which only applies to the deferrable trigger) kills it: the two are
mutually exclusive there.
+ # Because the worker re-executes before the trigger's on_kill runs, a
durable retry would
+ # reconnect to a run that on_kill is about to cancel and then fail, so
reject the contradiction
+ # and keep cancel_on_kill off when durable is enabled. Synchronous
runs have no such race and
+ # cancel_on_kill does not apply to them, so the value is left
untouched there.
+ if durable and deferrable:
Review Comment:
One more thing worth flagging: since #72100, clearing a task discards its
task state unless `keep_task_state` is set, so on 3.4+ `durable` reconnects on
retries and on clears that keep task state, while a plain clear submits a new
run. With `cancel_on_kill` off (the `durable` default) the old run then keeps
going. I've documented that in the docstring rather than work around it, since
it follows the "clearing starts over" behaviour, but let me know if you'd
prefer a different default there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]