nailo2c opened a new pull request, #60734: URL: https://github.com/apache/airflow/pull/60734
Closes: #52040 # Why 1. Original issue: When using GitDagBundle with `supports_versioning=True`, Airflow DAG Processor fails to execute DAG callbacks because it cannot find the materialized code under `versions/<commit_hash>/`. 2. Additional concern: The DAG processor should honor `stale_bundle_cleanup_*` to avoid accumulating versioned bundles. # How 1. Initialize versioned bundles in `_add_callback_to_queue`. 2. Wrap callback execution in `BundleVersionLock` to update usage and avoid cleanup races. 3. Add `_cleanup_stale_bundle_versions` in `DagFileProcessorManager._run_parsing_loop` to periodically check and remove stale bundle versions. # What Before the fixes, `_execute_dag_callbacks` raised errors and the callback logs did not appear in the DAG processor logs. Here is my remote dag repo: https://github.com/nailo2c/my_repo/blob/main/example_git_bundle_success_dag.py <img width="1279" height="230" alt="issue-52040-reproduce-1" src="https://github.com/user-attachments/assets/9e6c5572-223c-47bb-8e58-7035c7a86090"; /> <img width="1323" height="77" alt="issue-52040-reproduce-2" src="https://github.com/user-attachments/assets/3380aef2-9223-4000-aeb0-b989f7610f43"; /> <br><br> After the fixes, `_execute_dag_callbacks` completes without errors, and callbacks continue to work after committing new versions. ```console aaron.chen@Aarons-MacBook-Air-3 airflow_2 % kubectl exec -n airflow airflow-dag-processor-fd9f96f64-nzc7h -c dag-processor -- grep -R "SUCCESS CALLBACK" /opt/airflow/logs/dag_processor /opt/airflow/logs/dag_processor/2026-01-17/git_bundle/example_git_bundle_success_dag.py.log:{"timestamp":"2026-01-17T18:47:18.871227Z","level":"warning","event":">>> SUCCESS CALLBACK TRIGGERED VERSION 2 <<<","logger":"root","filename":"example_git_bundle_success_dag.py","lineno":7} /opt/airflow/logs/dag_processor/2026-01-17/git_bundle/example_git_bundle_success_dag.py.log:{"timestamp":"2026-01-17T19:32:06.676855Z","level":"warning","event":">>> SUCCESS CALLBACK TRIGGERED VERSION 3 <<<","logger":"root","filename":"example_git_bundle_success_dag.py","lineno":7} ``` Both `on_success_callback` and `on_failure_callback` behave as expected. --- ##### Was generative AI tooling used to co-author this PR? <!-- If generative AI tooling has been used in the process of authoring this PR, please change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by". --> - [X] Yes (please specify the tool below) GPT 5.2 <!-- Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --> --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
