vishnuprakashpalanisamy8892 opened a new issue, #77241:
URL: https://github.com/apache/airflow/issues/77241

   ### Under which category would you file this issue?
   
   Airflow Core
   
   ### Apache Airflow version
   
   3.4.0
   
   ### What happened and how to reproduce it?
   
   ### Issue description
   
   `DagBundlesManager.parse_config` keys bundles by `name` only. Nothing 
compares where the bundles read their files from. So if the same Git 
repository, branch and subdir are configured twice, once with an SSH URL and 
once with an HTTPS URL, Airflow accepts both bundles and treats them as two 
independent DAG sources:
   
   - The DAG processor clones/fetches the repository twice and parses every 
file twice.
   - Import errors are stored per `(bundle_name, filename)`, so every broken 
file is reported twice in the UI.
   - `dag_id` is global, so each DAG is stored once and `DagModel.bundle_name` 
is whichever bundle parsed it last. Ownership keeps switching between the two 
bundles from one parse loop to the next.
   - 3.4.0 adds a `DUPLICATE_DAG_ID` DagWarning (#68791) to every affected DAG, 
so the UI fills with warnings that all have the same root cause.
   - From reading the code (not yet observed): `SerializedDagModel.write_dag` 
only reuses a `DagVersion` when the bundle name is unchanged, so once a DAG has 
task instances, every ownership switch writes a new `DagVersion`.
   
   This is easy to hit when the bundle list is generated (for example by a 
Kubernetes operator or a deployment UI), because SSH and HTTPS look like two 
different repositories unless the URLs are normalized.
   
   Related: #76551 and its PR #76556 handle the general case of the same 
`dag_id` in two *different* bundles (keep the first owner, import error for the 
second). This issue is about the narrower, earlier case: two bundle entries 
that are really **the same source**. Even with #76556 merged, this 
configuration still clones and parses everything twice, and the second bundle 
ends up with an import error for every DAG. That can be caught at config-parse 
time, before any parsing happens.
   
   ### Steps to reproduce
   
   1. Use any Git repository with DAGs under `dags/` on branch `main` 
(placeholder: `example-org/example-dags` below).
   2. Create a Git connection `git_ssh` whose `extra` has `{"key_file": 
"/path/to/id_ed25519"}` with read access to the repository.
   3. Configure two bundles that point at the same repository, branch and 
subdir, one over SSH and one over HTTPS:
   
   ```ini
   [dag_processor]
   dag_bundle_config_list = [
       {
         "name": "dags_ssh",
         "classpath": "airflow.providers.git.bundles.git.GitDagBundle",
         "kwargs": {
           "git_conn_id": "git_ssh",
           "repo_url": "[email protected]:example-org/example-dags.git",
           "tracking_ref": "main",
           "subdir": "dags"
         }
       },
       {
         "name": "dags_https",
         "classpath": "airflow.providers.git.bundles.git.GitDagBundle",
         "kwargs": {
           "repo_url": "https://github.com/example-org/example-dags.git";,
           "tracking_ref": "main",
           "subdir": "dags"
         }
       }
   ]
   ```
   
   4. Start the DAG processor, scheduler and API server. Both bundles are 
loaded without any warning (`DAG bundles loaded: dags_ssh, dags_https`).
   5. Wait for a few parse loops, then compare DAG ownership and import errors:
   
   ```bash
   airflow dags list -o json > dags.json
   airflow dags list-import-errors -o json > errors.json
   python3 - <<'PY'
   import json
   from collections import Counter
   dags = json.load(open("dags.json"))
   errs = json.load(open("errors.json"))
   print("dags", len(dags), Counter(d["bundle_name"] for d in dags))
   print("import_errors", len(errs), Counter(e["bundle_name"] for e in errs))
   print("distinct_error_files", len({e["filepath"] for e in errs}))
   PY
   ```
   
   ### Observed (real deployment, one repository, branch `main`, subdir `dags/`)
   
   | | Value |
   | --- | --- |
   | DAGs (each stored once) | 41 |
   | DAGs owned by the SSH / HTTPS bundle at that moment | 28 / 13 |
   | Import errors in the UI | 32 |
   | Distinct files behind those errors | 16 (every one reported by both 
bundles, same error text) |
   
   Running the script again later gives a different SSH/HTTPS split, because 
ownership goes to whichever bundle parsed the file last.
   
   ### Where this comes from (3.4.0)
   
   - 
[`DagBundlesManager.parse_config`](https://github.com/apache/airflow/blob/3.4.0/airflow-core/src/airflow/dag_processing/bundles/manager.py#L315-L344)
 stores `self._bundle_config[bundle_config.name]` and never looks at the bundle 
kwargs.
   - 
[`_build_duplicate_dag_id_warnings`](https://github.com/apache/airflow/blob/3.4.0/airflow-core/src/airflow/dag_processing/collection.py#L314-L350)
 only records a warning ("…which is now overwritten by this file").
   - [Import 
errors](https://github.com/apache/airflow/blob/3.4.0/airflow-core/src/airflow/dag_processing/collection.py#L395-L445)
 are keyed by `(bundle_name, filename)`.
   - 
[`dag-bundles.rst`](https://github.com/apache/airflow/blob/3.4.0/airflow-core/docs/administration-and-deployment/dag-bundles.rst#L170)
 says bundles can be combined but says nothing about two bundles reading the 
same source. The [duplicate dag_id 
FAQ](https://github.com/apache/airflow/blob/3.4.0/airflow-core/docs/faq.rst#L422)
 asks for unique dag_ids but does not mention this cause.
   
   ### What you think should happen instead?
   
   The same repository + branch + subdir should be one DAG source, no matter 
whether it is reached over SSH or HTTPS. Airflow should at least tell the 
operator that two configured bundles read the same files, instead of silently 
parsing them twice and letting dag_id ownership flip.
   
   Suggested, in order of how small and safe the change is:
   
   1. **Docs:** in `dag-bundles.rst`, next to "You can combine multiple bundle 
types", state that two bundles must not read the same source, and link the 
duplicate-dag_id FAQ. Add this cause to the FAQ entry.
   2. **Opt-in detection at config parse time:** add an optional classmethod on 
`BaseDagBundle`, e.g. `source_key(**kwargs) -> Hashable | None` (default 
`None`). `GitDagBundle` returns `(normalized repo_url, tracking_ref, 
subdir.strip("/"))`, where normalization maps `git@host:path`, 
`ssh://user@host:port/path` and `https://user@host/path(.git)` to `host/path` 
(lower-cased host, no user, port or `.git`). It can reuse the existing 
`GitDagBundle._convert_git_ssh_url_to_https`. `DagBundlesManager.parse_config` 
logs a warning when two bundles return the same key, and raises 
`AirflowConfigException` when a new flag such as `[dag_processor] 
reject_duplicate_bundle_sources = True` is set. Call the hook through `getattr` 
so newer providers keep working with older cores. If `repo_url` comes only from 
the connection, the key is unknown at parse time and the check is skipped.
   
   The general duplicate-dag_id ownership fix is tracked separately in #76551 / 
#76556 and complements this one.
   
   ### Operating System
   
   Linux container (Python 3.12). Not OS-specific: the behaviour comes from 
bundle config parsing.
   
   ### Deployment
   
   Other 3rd-party Helm chart
   
   ### Apache Airflow Provider(s)
   
   git
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-git==1.0.1
   
   ### Official Helm Chart version
   
   Not Applicable
   
   ### Kubernetes Version
   
   Not Applicable (not specific to Kubernetes; reproducible with any deployment 
that has two Git bundles)
   
   ### Helm Chart configuration
   
   Not Applicable
   
   ### Docker Image customizations
   
   Custom image with Airflow 3.4.0 and apache-airflow-providers-git==1.0.1. Not 
relevant to the issue: the code paths involved are the upstream ones linked 
above.
   
   ### Anything else?
   
   - Happens every time both bundles are configured; it is not intermittent. 
Only the SSH/HTTPS split of DAG ownership changes between runs.
   - Not a regression in a specific release: 2.x had a single `dags_folder`, so 
a repository could not be loaded twice; 3.0–3.3 have bundles with no 
cross-bundle check; 3.4.0 adds the `DUPLICATE_DAG_ID` warning but still 
overwrites.
   - The same pattern applies to nested subdirs of one repository and branch 
(`subdir=""` and `subdir="dags"`), which also parse files twice. That one can 
be intentional, so it is probably a warning at most.
   - Workaround: configure each repository/branch/subdir only once.
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [I] DAG bundles: SSH ... vishnuprakashpalanisamy8892 (via GitHub) via commits

Reply via email to