hussein-awala opened a new pull request, #74592: URL: https://github.com/apache/airflow/pull/74592
<!-- SPDX-License-Identifier: Apache-2.0 --> ## Summary Fix stale Dag-version associations on waiting task instances when an unversioned Dag changes during an active run. For a local, unversioned bundle, a downstream task can start after its upstream task but still display an older Dag version. The upstream task's version reference is updated while the waiting downstream task's reference is left unchanged. ## Root cause For runs without a pinned bundle version, `_verify_integrity_if_dag_changed()` updates unfinished task instances using: ```python TI.state.in_(State.unfinished) ``` Although `State.unfinished` includes Python `None`, SQL `IN (..., NULL)` does not match rows whose state is `NULL`. Task instances waiting for their upstream dependencies can therefore be excluded from the version update, while running or deferred tasks receive the latest version. Subsequent integrity checks can return early once the latest version is associated with any task instance or task-instance history record in the run, allowing the stale association to persist. ## Generic reproduction 1. Create an unversioned Dag containing `upstream >> downstream` and start a run. 2. Leave `upstream` running or deferred, with `downstream` still in `state=None`. 3. Update the Dag and persist a new serialized Dag version. 4. Run the scheduler's scheduling pass. 5. Observe that `upstream` references the new version but `downstream` retains the old one. Expected: both unfinished task instances reference the new Dag version, including the downstream task before it becomes schedulable. ## Changes - Explicitly include `TI.state.is_(None)` in the existing unfinished-task update filter. - Add a database-backed, parametrized regression test for running and deferred upstream tasks. - Verify that the downstream task receives the new version while waiting and retains it when scheduled after upstream success. The fix preserves the existing Dag/run scope and versioned-bundle guard. It changes task-instance version metadata; it does not add source-file versioning to local bundles or retroactively repair previously inconsistent runs. A displayed Dag version alone is not proof of the source revision executed by a worker using an unversioned bundle. ## Validation Reproduced on a local `main` checkout at `ef9feefd83`: - Before the fix: both new regression cases failed at the downstream version assertion. - After the fix: all five version-integrity tests passed through Breeze: ```bash breeze run pytest airflow-core/tests/unit/jobs/test_scheduler_job.py -k verify_integrity -vs ``` The publication branch was then based on newer upstream `main`. Fast pre-commit checks, including the airflow-core mypy hook, and applicable manual checks passed for the two changed files. Ruff formatting/linting and `git diff --check` also passed. **Draft validation limitation:** rerunning the tests in the newer-main worktree was blocked before pytest started by a local Breeze/Docker build-context mismatch. The broader test suites and documentation build listed by `breeze verify` have not been run. This draft does not claim a passing test rerun on the newer base; CI validation is still pending. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — pi (OpenAI GPT-6 Astra) Generated-by: pi (OpenAI GPT-6 Astra) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. --- Drafted-by: pi (OpenAI GPT-6 Astra) (no human review before posting) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
