Hey All,

Wanted to call out a new hook we added to BaseTrigger — `on_kill()`, PR:
https://github.com/apache/airflow/pull/65590.

*Why does it exist?*

If your trigger polls an external job (BigQuery, Databricks, Dataproc,
etc.), there was no clean way to cancel that job when
a user kills the deferred task, either by marking it as success, failure,
or clearing it. `cleanup()` wasn't the right place —
it fires on every trigger exit including triggerer restarts and rolling
deploys, so putting cancellation there would kill in-flight
work on every redeploy.

`on_kill()` solves this. It only fires when a user explicitly acts on the
task. Not on restart, not on redistribution.

*How it works*

The triggerer passes a private sentinel via `asyncio.Task.cancel(msg)` to
distinguish user kills from other exits — no DB
round-trip needed. There's a configurable timeout (*[triggerer]
on_kill_timeout, default 30s*) so a slow external API call
cannot block the triggerer.

Databricks is updated as a reference:
https://github.com/apache/airflow/pull/65672

Both `DatabricksExecutionTrigger` and
`DatabricksSQLStatementExecutionTrigger` now implement it.

*Who benefits from this?*

If you are someone building or maintaining a trigger that manages external
work, you can consider adopting to this hook
and implementing it for your trigger class, Airflow 3.3 will launch this.

No compat guard is needed, Airflow < 3.2 the method just never gets called.

*Next steps*

I pulled some data from the repo and these are candidates for this
migration.

Google Provider: DataprocSubmitTrigger and DataprocSubmitJobDirectTrigger
in the google provider. Currently catches
CancelledError in run() and calls hook.cancel_job(job_id) after checking
safe_to_cancel().

Amazon Provider: EmrServerlessStartJobTrigger. Same pattern. CancelledError
handler with safe_to_cancel() calling
hook.conn.cancel_job_run(). Straightforward migration.

CNCF provider: KubernetesPodTrigger in cncf/kubernetes. Has an
on_kill_action parameter controlling whether to
delete/patch the pod. The cancellation logic lives in a CancelledError
handler. Move it to on_kill().

Happy to help with reviews for these.


Thanks & Regards,
Amogh Desai

Reply via email to