vishnuprakashpalanisamy8892 opened a new issue, #77241:
URL: https://github.com/apache/airflow/issues/77241
### Under which category would you file this issue?
Airflow Core
### Apache Airflow version
3.4.0
### What happened and how to reproduce it?
### Issue description
`DagBundlesManager.parse_config` keys bundles by `name` only. Nothing
compares where the bundles read their files from. So if the same Git
repository, branch and subdir are configured twice, once with an SSH URL and
once with an HTTPS URL, Airflow accepts both bundles and treats them as two
independent DAG sources:
- The DAG processor clones/fetches the repository twice and parses every
file twice.
- Import errors are stored per `(bundle_name, filename)`, so every broken
file is reported twice in the UI.
- `dag_id` is global, so each DAG is stored once and `DagModel.bundle_name`
is whichever bundle parsed it last. Ownership keeps switching between the two
bundles from one parse loop to the next.
- 3.4.0 adds a `DUPLICATE_DAG_ID` DagWarning (#68791) to every affected DAG,
so the UI fills with warnings that all have the same root cause.
- From reading the code (not yet observed): `SerializedDagModel.write_dag`
only reuses a `DagVersion` when the bundle name is unchanged, so once a DAG has
task instances, every ownership switch writes a new `DagVersion`.
This is easy to hit when the bundle list is generated (for example by a
Kubernetes operator or a deployment UI), because SSH and HTTPS look like two
different repositories unless the URLs are normalized.
Related: #76551 and its PR #76556 handle the general case of the same
`dag_id` in two *different* bundles (keep the first owner, import error for the
second). This issue is about the narrower, earlier case: two bundle entries
that are really **the same source**. Even with #76556 merged, this
configuration still clones and parses everything twice, and the second bundle
ends up with an import error for every DAG. That can be caught at config-parse
time, before any parsing happens.
### Steps to reproduce
1. Use any Git repository with DAGs under `dags/` on branch `main`
(placeholder: `example-org/example-dags` below).
2. Create a Git connection `git_ssh` whose `extra` has `{"key_file":
"/path/to/id_ed25519"}` with read access to the repository.
3. Configure two bundles that point at the same repository, branch and
subdir, one over SSH and one over HTTPS:
```ini
[dag_processor]
dag_bundle_config_list = [
{
"name": "dags_ssh",
"classpath": "airflow.providers.git.bundles.git.GitDagBundle",
"kwargs": {
"git_conn_id": "git_ssh",
"repo_url": "[email protected]:example-org/example-dags.git",
"tracking_ref": "main",
"subdir": "dags"
}
},
{
"name": "dags_https",
"classpath": "airflow.providers.git.bundles.git.GitDagBundle",
"kwargs": {
"repo_url": "https://github.com/example-org/example-dags.git",
"tracking_ref": "main",
"subdir": "dags"
}
}
]
```
4. Start the DAG processor, scheduler and API server. Both bundles are
loaded without any warning (`DAG bundles loaded: dags_ssh, dags_https`).
5. Wait for a few parse loops, then compare DAG ownership and import errors:
```bash
airflow dags list -o json > dags.json
airflow dags list-import-errors -o json > errors.json
python3 - <<'PY'
import json
from collections import Counter
dags = json.load(open("dags.json"))
errs = json.load(open("errors.json"))
print("dags", len(dags), Counter(d["bundle_name"] for d in dags))
print("import_errors", len(errs), Counter(e["bundle_name"] for e in errs))
print("distinct_error_files", len({e["filepath"] for e in errs}))
PY
```
### Observed (real deployment, one repository, branch `main`, subdir `dags/`)
| | Value |
| --- | --- |
| DAGs (each stored once) | 41 |
| DAGs owned by the SSH / HTTPS bundle at that moment | 28 / 13 |
| Import errors in the UI | 32 |
| Distinct files behind those errors | 16 (every one reported by both
bundles, same error text) |
Running the script again later gives a different SSH/HTTPS split, because
ownership goes to whichever bundle parsed the file last.
### Where this comes from (3.4.0)
-
[`DagBundlesManager.parse_config`](https://github.com/apache/airflow/blob/3.4.0/airflow-core/src/airflow/dag_processing/bundles/manager.py#L315-L344)
stores `self._bundle_config[bundle_config.name]` and never looks at the bundle
kwargs.
-
[`_build_duplicate_dag_id_warnings`](https://github.com/apache/airflow/blob/3.4.0/airflow-core/src/airflow/dag_processing/collection.py#L314-L350)
only records a warning ("…which is now overwritten by this file").
- [Import
errors](https://github.com/apache/airflow/blob/3.4.0/airflow-core/src/airflow/dag_processing/collection.py#L395-L445)
are keyed by `(bundle_name, filename)`.
-
[`dag-bundles.rst`](https://github.com/apache/airflow/blob/3.4.0/airflow-core/docs/administration-and-deployment/dag-bundles.rst#L170)
says bundles can be combined but says nothing about two bundles reading the
same source. The [duplicate dag_id
FAQ](https://github.com/apache/airflow/blob/3.4.0/airflow-core/docs/faq.rst#L422)
asks for unique dag_ids but does not mention this cause.
### What you think should happen instead?
The same repository + branch + subdir should be one DAG source, no matter
whether it is reached over SSH or HTTPS. Airflow should at least tell the
operator that two configured bundles read the same files, instead of silently
parsing them twice and letting dag_id ownership flip.
Suggested, in order of how small and safe the change is:
1. **Docs:** in `dag-bundles.rst`, next to "You can combine multiple bundle
types", state that two bundles must not read the same source, and link the
duplicate-dag_id FAQ. Add this cause to the FAQ entry.
2. **Opt-in detection at config parse time:** add an optional classmethod on
`BaseDagBundle`, e.g. `source_key(**kwargs) -> Hashable | None` (default
`None`). `GitDagBundle` returns `(normalized repo_url, tracking_ref,
subdir.strip("/"))`, where normalization maps `git@host:path`,
`ssh://user@host:port/path` and `https://user@host/path(.git)` to `host/path`
(lower-cased host, no user, port or `.git`). It can reuse the existing
`GitDagBundle._convert_git_ssh_url_to_https`. `DagBundlesManager.parse_config`
logs a warning when two bundles return the same key, and raises
`AirflowConfigException` when a new flag such as `[dag_processor]
reject_duplicate_bundle_sources = True` is set. Call the hook through `getattr`
so newer providers keep working with older cores. If `repo_url` comes only from
the connection, the key is unknown at parse time and the check is skipped.
The general duplicate-dag_id ownership fix is tracked separately in #76551 /
#76556 and complements this one.
### Operating System
Linux container (Python 3.12). Not OS-specific: the behaviour comes from
bundle config parsing.
### Deployment
Other 3rd-party Helm chart
### Apache Airflow Provider(s)
git
### Versions of Apache Airflow Providers
apache-airflow-providers-git==1.0.1
### Official Helm Chart version
Not Applicable
### Kubernetes Version
Not Applicable (not specific to Kubernetes; reproducible with any deployment
that has two Git bundles)
### Helm Chart configuration
Not Applicable
### Docker Image customizations
Custom image with Airflow 3.4.0 and apache-airflow-providers-git==1.0.1. Not
relevant to the issue: the code paths involved are the upstream ones linked
above.
### Anything else?
- Happens every time both bundles are configured; it is not intermittent.
Only the SSH/HTTPS split of DAG ownership changes between runs.
- Not a regression in a specific release: 2.x had a single `dags_folder`, so
a repository could not be loaded twice; 3.0–3.3 have bundles with no
cross-bundle check; 3.4.0 adds the `DUPLICATE_DAG_ID` warning but still
overwrites.
- The same pattern applies to nested subdirs of one repository and branch
(`subdir=""` and `subdir="dags"`), which also parse files twice. That one can
be intentional, so it is probably a warning at most.
- Workaround: configure each repository/branch/subdir only once.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]