bnspopi opened a new pull request, #77313: URL: https://github.com/apache/airflow/pull/77313
`DagBundlesManager.parse_config` keys bundles by name and never compares where they read their files from. If the same Git repository, branch and subdir is configured twice, once with an SSH URL and once with an HTTPS URL, Airflow accepts both as independent bundles. The Dag processor then clones and parses everything twice, every import error is reported twice, and each Dag keeps moving from one bundle to the other. That is what #77241 reports. This adds the detection described in the issue, as a warning only: - `BaseDagBundle.source_key(**kwargs)` is a new classmethod that says where a bundle with those kwargs reads from. It returns `None` by default, which means "not known", so existing and third-party bundles are unaffected. - `LocalDagBundle` returns its resolved directory. - `GitDagBundle` returns the repository as `host/path` together with the tracking ref and the subdir. The user, port and `.git` suffix are dropped and the host is lower-cased, so `[email protected]:org/repo.git`, `ssh://[email protected]:22/org/repo.git` and `https://[email protected]/org/repo` give the same key. When `repo_url` is not in the kwargs (it comes from the connection) or is a local path, the key is `None` and the bundle is not compared. - `DagBundlesManager.parse_config` logs a warning naming the bundles that share a key. - `dag-bundles.rst` and the duplicate dag id FAQ entry now say that two bundles must not read the same source. Left out on purpose: - The `[dag_processor] reject_duplicate_bundle_sources` flag that would turn the warning into an error. It is a new config option and a separate decision. - Nested subdirs of one repository (`subdir=""` and `subdir="dags"`). The issue notes this can be intentional. - The warning is logged every time a `DagBundlesManager` parses the config, so it appears in the log of each component that loads the bundle configuration. @basantjamwal proposed this same plan in the issue. I am happy to close this in favour of their PR if they already have one in progress. Tests, run on Linux with SQLite: - `test_dag_bundle_manager.py`, `test_local.py`, `test_base.py`: 107 passed (5 new cases). - `providers/git/tests/unit/git/bundles/test_git.py`: 142 passed (11 new cases). - Without the source changes, 14 of the new cases fail. related: #77241 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (Claude Code) Generated-by: Claude Code following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
