Hi Airflow community, I would like feedback on separating Dag discovery paths from Python import roots. I had previously raised a Github discussion: https://github.com/apache/airflow/discussions/70313 and received comments regarding starting a dev-list thread
A DAG bundle can contain DAG files together with shared Python modules, configuration files, and other resources. Airflow currently uses BaseDagBundle.path for two purposes: 1. Scan for Dag files. 2. Add a directory to sys.path. These paths can differ in a monorepo. For our use case too with repository like: ``` repository/ ├── project/ │ ├── lib/ │ └── dags/ │ └── example_dag.py ``` The DAG may import shared code using: >> from project.lib import some_function Airflow automatically adds BaseDagBundle.path to sys.path. In this example, it adds only `repository/project/dags`. However, an import such as `from project.lib import some_function` requires `repository` to be on sys.path. Airflow may need to scan only a Dag subdirectory, while Python imports start at the repository root. Using the repository root for discovery can make Airflow scan many unrelated files. Using only the Dag directory can make valid imports fail. Thus, the proposal is to decouple import root with bundle path. For the same, I have raised the PR: https://github.com/apache/airflow/pull/73388 The proposed change adds BaseDagBundle.import_root: - import_root defaults to path. - Existing bundle providers keep their current behavior. - GitDagBundle uses the repository root as import_root when path points to a subdirectory. - Airflow uses the same import root during Dag parsing, task execution, and callback execution. - The import root must belong to the same initialized bundle checkout. The change is backward compatible for LocalDagBundle, S3DagBundle, GCSDagBundle, and custom providers that do not override import_root. Comments from the discussion thread have also been addressed in the PR. I would appreciate feedback on these questions: 1. Is Airflow community aligned with this change. 2. If yes, is import_root the correct public API for this use case. and any other feedbacks is appreciated! Several other users have also requested this: https://github.com/apache/airflow/discussions/61901 So this can be a value add for Airflow community! Thanks, Sameer
