hkc-8010 opened a new pull request, #66484: URL: https://github.com/apache/airflow/pull/66484
Fix dag-processor present/orphan checks so versioned callback files are not treated as removed solely because the queued callback entry includes `bundle_version`. This is distinct from #66301 / #66474. Those changes addressed stale serialized DAG metadata when only the bundle version changed. This fix is later in the callback pipeline: the scheduler creates the `DagCallbackRequest`, the dag processor fetches and queues it, and the manager must not purge or orphan that callback file before `_execute_dag_callbacks` has a chance to run. The change keeps `DagFileInfo` equality and hashing version-aware, but makes manager "file presence" checks compare a stable key of: - `bundle_name` - `rel_path` That is applied in: - `purge_removed_files_from_queue` - `terminate_orphan_processes` - `remove_orphaned_file_stats` Tests added: - preserve a versioned callback queue entry when the same unversioned file is present - preserve a versioned callback processor when the same unversioned file is present - preserve versioned callback file stats in the same scenario Validation: - `pytest airflow-core/tests/unit/dag_processing/test_manager.py -q` - `breeze testing core-tests --backend postgres --python 3.10 --db-reset -- airflow-core/tests/unit/dag_processing/test_manager.py -q` - `breeze testing core-tests --backend postgres --python 3.10 --downgrade-pendulum --db-reset -- airflow-core/tests/unit/dag_processing/test_manager.py -q` - `breeze testing core-tests --backend sqlite --python 3.10 --force-lowest-dependencies --db-reset -- airflow-core/tests/unit/dag_processing/test_manager.py -q` closes: #66483 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (Codex) Generated-by: Codex following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
