luc-pimentel opened a new pull request, #75028: URL: https://github.com/apache/airflow/pull/75028
A task with `depends_on_past=True` and `wait_for_past_depends_before_skipping=True` is skipped right away when an upstream `ShortCircuitOperator` or branch operator decides to skip it, even when its task instance in the previous run failed. The parameter's docstring says the task stays unscheduled until the previous run's task instance has succeeded or been skipped, and only then is skipped. Two things stop the parameter from working. `SkipMixin` raises `DownstreamTasksSkipped` for every direct downstream task, so the execution API writes `SKIPPED` before any scheduler dependency runs. And the three deps that know about the flag (`NotPreviouslySkippedDep`, `TriggerRuleDep`, `PrevDagrunDep`) only read `DepContext.wait_for_past_depends_before_skipping`, which nothing sets in Airflow 3; the task attribute is never read, and `SerializedMappedOperator` does not expose it. With this change, `SkipMixin.skip` and `skip_all_except` leave the direct downstream tasks that set both flags out of the state change. They stay in the XCom decision, so `NotPreviouslySkippedDep` skips them once the past depends are met. The deps read the task attribute as well as the context flag, and `SerializedMappedOperator` gets the property like `wait_for_downstream`. `PrevDagrunDep` now writes the `past_depends_met` XCom through the scheduler's session: with a fresh session it committed inside the scheduling loop and crashed the scheduler, which nobody hit because the path was unreachable until now. A few notes on scope. A task that sets only one of the two flags keeps today's immediate skip, as the docstring says, and `PrevDagrunDep` marks a task without `depends_on_past` as met at once. Tasks reached through another skipped task (`ignore_downstream_trigger_rules=True`) are still skipped at once. A task waiting this way keeps its Dag run running, exactly as `depends_on_past` already does for a failed previous run. With a new task SDK and an older scheduler, the task left out of the state change is skipped on the next scheduling pass. The standard provider's `SkipMixin` copy is not touched; on Airflow 3 the compat shim resolves the task SDK class. Tested on 3.3.2 and main with `airflow standalone` on SQLite and two Dags of the issue's shape, `ShortCircuitOperator` >> `PythonOperator(depends_on_past=True)`, one with `wait_for_past_depends_before_skipping=True` and a control without it. Run 1 fails the downstream task, run 2 short-circuits: | Build | Dag | Run 2, 60 s after the short circuit | After run 1's task is marked success | | --- | --- | --- | --- | | 3.3.2 and main | flag on | `skipped` | n/a | | 3.3.2 and main | control | `skipped` | n/a | | main with this change | flag on | `None` (waiting) | `skipped` within 10 s | | main with this change | control | `skipped` | `skipped` | The new tests fail without this change: the scheduling decision in `test_dagrun.py` for a short-circuit and a branch decision (the flagged task stays `None` while the previous run's task is failed and is skipped once it succeeded; the control is skipped at once), the two `SkipMixin` lists, the three deps reading the task attribute, and the serialized mapped operator property. One existing assertion in `test_prev_dagrun_dep.py` gains the `session` argument. The touched test modules pass under Breeze on SQLite, and prek passes on the changed files. #73470 took the `SkipMixin` side of this and was closed because the scheduler side was missing. The rule itself comes from the parameter's design in discussion #26460. closes: #55146 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Fable 5.1) Generated-by: Claude Code (Fable 5.1) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
