dabla commented on code in PR #75659:
URL: https://github.com/apache/airflow/pull/75659#discussion_r4626541981
##########
airflow-core/docs/authoring-and-scheduling/deferring.rst:
##########
@@ -301,6 +303,21 @@ If you want to defer your task directly to the triggerer
without going into the
* ``next_kwargs``: Additional keyword arguments to pass to the ``next_method``
when it is called.
* ``timeout``: (Optional) A timedelta that specifies a timeout after which
this deferral will fail, and fail the task instance. Defaults to ``None``,
which means no timeout.
+The scheduler defers these tasks in the same step where it queues other tasks,
so pool slots,
+``max_active_tasks``, ``max_active_tis_per_dag`` and
``max_active_tis_per_dagrun`` must allow it.
+They do not use executor slots. A task is deferred one scheduler loop after it
becomes ``SCHEDULED``,
Review Comment:
**[nit]** "They do not use executor slots" is only half true now.
These tasks don't take a slot from a specific executor, but the deferral now
happens inside the critical section, which is gated on executor capacity: it is
skipped when all executors are full, and `max_tis` is bounded by
`core.parallelism`. So a saturated executor does delay them (see [3]). Either
fix the gate or say so here.
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -1051,6 +1092,9 @@ def _select_task_instances_to_queue(
stats.gauge("scheduler.tasks.starving", num_starving_tasks_total)
stats.gauge("scheduler.tasks.executable", len(executable_tis))
+ for task_instance in tis_to_defer:
+ task_instance.defer_task(session=session)
Review Comment:
**[blocker]** `start_from_trigger` tasks are no longer deferred while the
executors are full.
Before this PR, `schedule_tis()` deferred them whatever the executor's load
was. Now deferral only happens in `_select_task_instances_to_queue()`, and the
scheduler never gets there when workers are saturated:
- `_do_scheduling` skips the critical section entirely when
`sum(executor.slots_available) <= 0` ([line
2353](https://github.com/apache/airflow/blob/e74b432a3f9056cbdfa00b2a4abfdd7941ca6c19/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L2353),
"All executors are full, skipping critical section").
- `_critical_section_enqueue_task_instances` returns 0 when
`core.parallelism - slots_occupied <= 0` ([line
1412](https://github.com/apache/airflow/blob/e74b432a3f9056cbdfa00b2a4abfdd7941ca6c19/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L1412)).
- When it does run, `max_tis` still comes from free executor capacity, and
deferred TIs now count against it (`len(executable_tis) + len(tis_to_defer) >=
max_tis`).
Scenario: `parallelism=32`, 32 long-running tasks occupy the workers, and a
Dag with 100 `DateTimeSensorAsync(start_from_trigger=True)` tasks in a pool
with free slots. All 100 stay `SCHEDULED` until a worker frees up, and then
they go out at most `max_tis` per loop. Running without a worker is the whole
point of `start_from_trigger`, and the docs in this PR say they "do not use
executor slots".
The deferral needs a budget that doesn't depend on executor capacity: for
example, let the critical section run when there are `start_from_trigger`
candidates even with full executors, and keep a separate counter for
`tis_to_defer` instead of sharing `max_tis`. Please also add a test with
`slots_available = 0` on the executor.
##########
airflow-core/src/airflow/ti_deps/dependencies_states.py:
##########
@@ -23,17 +23,23 @@
TaskInstanceState.QUEUED,
}
+# States counted towards max_active_tasks of a Dag run. A deferred task
instance holds no worker slot but
+# is still in flight, so it takes a max_active_tasks slot until it finishes.
+DAG_RUN_ACTIVE_STATES = {
Review Comment:
**[warning]** This reverses a deliberate decision for every deferrable task,
not only `start_from_trigger`, and it's bundled with the pool fix.
#61769 made deferred TIs count towards
`max_active_tis_per_dag`/`_per_dagrun` and kept them out of `max_active_tasks`
on purpose. It added
`test_find_executable_task_instances_deferred_does_not_affect_max_active_tasks`,
whose docstring reads "max_active_tasks is about worker-level parallelism,
while deferred tasks don't consume worker slots". This PR renames that test and
flips its assertion. Every Dag that relies on many deferrable sensors running
at once (the newsfragment's own example) loses concurrency after upgrading,
with no setting to get the old behaviour back except raising `max_active_tasks`.
That may well be the right call for #40528, but it's a behaviour change with
its own trade-off, separate from #75497 (start_from_trigger ignores pools),
which is a clear bug. I'd suggest splitting it into its own PR, so the pool fix
can land without waiting on agreement about what `max_active_tasks` means, and
so the title says what changes for users.
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -1051,6 +1092,9 @@ def _select_task_instances_to_queue(
stats.gauge("scheduler.tasks.starving", num_starving_tasks_total)
stats.gauge("scheduler.tasks.executable", len(executable_tis))
+ for task_instance in tis_to_defer:
+ task_instance.defer_task(session=session)
Review Comment:
**[warning]** One TI that fails `defer_task()` now blocks queuing for every
Dag.
In `schedule_tis()`, an exception from `defer_task()` was contained per Dag
run: `_schedule_all_dag_runs` catches `Exception` and logs "Error scheduling
DAG run" ([line
3402](https://github.com/apache/airflow/blob/e74b432a3f9056cbdfa00b2a4abfdd7941ca6c19/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L3402)).
Here it runs inside the critical section with no guard. If it raises (for
example while `Trigger.encrypt_kwargs` serializes the `trigger_kwargs` with
serde and Fernet), the exception leaves
`_critical_section_enqueue_task_instances`, the whole selection for every Dag
rolls back, and the scheduler loop dies. On restart the same `SCHEDULED` TI is
picked again, so you get a crash loop that a single Dag causes.
Guard each `defer_task()` call: log it, and fail that TI (or at least starve
it) so the rest of the selection still gets queued. Per "Errors should never
pass silently" ([PEP 20](https://peps.python.org/pep-0020/)), the failure
should land on the TI that caused it, not on the scheduler.
Side note: with `multi_team` on, `defer_task()` runs
`DagModel.get_team_name` once per TI, inside the lock, even though this method
already resolved `dag_id_to_team_name` for the batch.
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -612,6 +609,37 @@ def _debug_dump(self, signum: int, frame: FrameType |
None) -> None:
self.log.info("\n\t".join(map(repr, callstack)))
self.log.info("-" * 80)
+ def _set_task_if_started_from_trigger(
+ self,
+ task_instance: TI,
+ *,
+ serialized_dags: dict[tuple[str, str], SerializedDAG | None],
+ session: Session,
+ ) -> bool:
+ """
+ Set the task of ``task_instance`` and return True if the triggerer
starts it, without the executor.
+
+ :param serialized_dags: the Dags already read, by Dag run: read once
per Dag run, not per task.
+
+ :meta private:
+ """
+ # A task resumed by its trigger has next_method set: it must run its
next method on the executor
+ if task_instance.next_method is not None:
+ return False
+ dag_run_key = (task_instance.dag_id, task_instance.run_id)
+ if dag_run_key not in serialized_dags:
+ serialized_dags[dag_run_key] =
self.scheduler_dag_bag.get_dag_for_run(
Review Comment:
**[warning]** This loads the serialized Dag for every selected TI inside the
critical section, and repeats the lookup in
`_task_concurrency_allows_execution`.
The selection loop deliberately skips that load ([line
989](https://github.com/apache/airflow/blob/e74b432a3f9056cbdfa00b2a4abfdd7941ca6c19/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L989):
"Many DAGs do not define task concurrency limits, so avoid loading the
serialized DAG unless required"). With this change every TI that passes the
pool and `max_active_tasks` checks pays for it, deferrable or not. The cost is
real, not just an in-memory cache hit:
- For a run with no `bundle_version` (any non-versioned bundle, e.g. the
local one), `DBDagBag._version_from_dag_run` runs
`DagVersion.get_latest_version` on every call. That's one extra query per Dag
run per selection pass, while the pool rows and the advisory lock are held.
- On a cache miss the Dag is deserialized inside the lock, and past the
revalidation window the `dag_hash` check adds another query.
It also duplicates the `get_dag_for_run` + `has_task` + `get_task` sequence
of `_task_concurrency_allows_execution`, which already ran for the same TI when
the Dag has task concurrency limits. Two lookups of the same task with
different error handling (the new one has no `try/except` for a Dag that fails
to load) will drift apart.
Suggestion: gate on a cheap Dag-level flag the way
`has_task_concurrency_limits` does (computed at parse time, so only Dags that
have a `start_from_trigger` task pay for the lookup), and resolve the task once
per TI for both checks. Numbers for `scheduler.critical_section_duration`
before and after, on a deployment with many Dag runs, would show what this
costs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]