basantjamwal commented on issue #77241:
URL: https://github.com/apache/airflow/issues/77241#issuecomment-6980839273
Hi, I'd like to work on this.
**Reproduced** on the latest `apache/airflow` Docker image (`standalone`)
with two `LocalDagBundle`s (`a`, `b`) pointing at the same folder:
- `DAG bundles loaded: a, b` is logged with no warning.
- Both bundles parse the same files (`a good.py` and `b good.py` in the
processing stats).
- `dag_id` ownership flips on every parse loop (every ~30s):
```
Duplicate dag_id detected dag_id=good_dag existing_bundle=b
incoming_bundle=a
Duplicate dag_id detected dag_id=good_dag existing_bundle=a
incoming_bundle=b
```
- A broken file is reported twice in `airflow dags list-import-errors`, once
for bundle `a` and once for `b`, with the same error text.
**Plan** (starting small):
1. Docs: note in `dag-bundles.rst` and the duplicate dag_id FAQ that two
bundles must not read the same source.
2. Add an optional `source_key()` hook on `BaseDagBundle`. `GitDagBundle`
would return a normalized `(repo_url, tracking_ref, subdir)`, with SSH and
HTTPS URLs mapped to the same key. `parse_config` would log a warning when two
bundles share a key.
I'd leave the `reject_duplicate_bundle_sources` flag for a follow-up if
maintainers want it.
Does this approach sound good, and should the docs change go in its own PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]