ramitkataria commented on code in PR #75363:
URL: https://github.com/apache/airflow/pull/75363#discussion_r4608220581
##########
airflow-core/src/airflow/jobs/triggerer_job_runner.py:
##########
@@ -1685,6 +1744,14 @@ async def create_triggers(self):
else:
trigger_name = f"ID {trigger_id}"
trigger_instance = trigger_class(**deserialised_kwargs)
+ if workload.dag_run_data:
+ from airflow.triggers.callback import CallbackTrigger
+
+ # Same as setting task_instance on task-bound triggers
above.
+ if isinstance(trigger_instance, CallbackTrigger):
+ trigger_instance._callback_context =
self._build_callback_context(
Review Comment:
If building the context raises here (params validation, for example), the
trigger lands in `failed_triggers`, but `Trigger.submit_failure` only handles
task instances. The callback is never marked failed and the trigger gets
re-created. Could we fail the callback like the missing Dag run case does, with
a test on its state?
##########
task-sdk/src/airflow/sdk/definitions/context.py:
##########
@@ -76,6 +77,8 @@ class Context(TypedDict, total=False):
dag_run: DagRunProtocol
data_interval_end: NotRequired[DateTime | None]
data_interval_start: NotRequired[DateTime | None]
+ # Only in deadline callbacks: ``{"id": ..., "deadline_time": ...}``.
+ deadline: NotRequired[dict[str, Any]]
Review Comment:
This makes `deadline` a reserved `@task` argument name:
`.expand(deadline=...)` and `.partial(deadline=...)` now raise at parse time,
and a required `deadline` param becomes optional. A task context never has this
key, so could we leave it out of `KNOWN_CONTEXT_KEYS` the way `loop` is?
Since the context shape is already changing here, it might also be a good
moment to make `deadline` a small typed object instead of `dict[str, Any]`, so
it reads like `dag_run` and can gain fields later.
##########
airflow-core/newsfragments/75363.significant.rst:
##########
@@ -0,0 +1,10 @@
+Deadline Alert callbacks get their context when they run, and their kwargs are
templated
+
+The context passed to Deadline Alert callbacks is now built where the callback
runs, the same way as a task's
+context, without the task fields, instead of being stored by the scheduler
when the deadline is missed. Its
+``dag_run`` entry is now a Dag run object, as in a task's context, rather than
a dict of the REST API Dag run
+fields, so read its fields as attributes (``context["dag_run"].dag_id``), and
``context["deadline"]["deadline_time"]``
Review Comment:
Nit: Notifier templates that used `dag_run.dag_run_id` need `dag_run.run_id`
now, and they render with `StrictUndefined`. Might be worth a clause here, up
to you.
##########
task-sdk/src/airflow/sdk/execution_time/context.py:
##########
@@ -1793,3 +1794,125 @@ def context_get_outlet_events(context: Context) ->
OutletEventAccessorsProtocol:
except KeyError:
outlet_events = context["outlet_events"] = OutletEventAccessors()
return outlet_events
+
+
+def build_dag_run_context(
+ dag_run: DagRunProtocol, dag: DAG | None = None, *, multi_team: bool =
False
+) -> Context:
+ """
+ Build the part of a task's template context that comes from its Dag run.
+
+ ``RuntimeTaskInstance.get_template_context`` adds the task keys to it, and
deadline callbacks,
+ which have a Dag run but no task, use it as their context. With ``dag``,
it also has ``dag``
+ and the Dag's ``params`` with the Dag run's ``conf`` applied.
+ """
+ from airflow.sdk.definitions.param import process_params
+ from airflow.sdk.execution_time.comms import
AssetEventDagRunReferenceResult
+ from airflow.sdk.plugins_manager import integrate_macros_plugins
+ from airflow.sdk.timezone import coerce_datetime
+
+ integrate_macros_plugins()
+
+ context: Context = {
+ "dag_run": dag_run,
+ "run_id": dag_run.run_id,
+ "partition_key": dag_run.partition_key,
+ "partition_date": coerce_datetime(dag_run.partition_date),
+ "triggering_asset_events": TriggeringAssetEventsAccessor.build(
Review Comment:
Small gap with tasks: `triggering_asset_events` is always empty for
callbacks, because only `ti_run` loads `consumed_asset_events`. `team_name` is
also None, so team-owned macros fail under `multi_team`, and the triggerer's
`dag_run` lacks `triggering_user_name` and `note`. I'm happy with either
loading these or listing them in the docs as not available yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]