Hi all, Thank you for the feedback on this proposal. Regarding some of the feedback:
> You should checkout Git at the root and refer to your Dags as from project.dags... and utilities as from project.lib.... > This should not introduce several roots. > Being free to choose roots can create confusion, especially with implicit namespaces. I have revised the PR: https://github.com/apache/airflow/pull/73388. Now, I am only adding the specified import root in the sys.path bundle.path = <repo>/team/dags bundle.import_root = <repo> -> sys.path += bundle.path only We scan bundle.discovery_path only for DAGs This way, we only enforce one root as per the feedback.(Infact, this is the same pattern being followed in monorepo.) > Generally, the "dag" folder where dags are should be the root of the Python imports, and they should come as a single commit hash. This is ensured as part of PR now too. In GitDagBundle, both the discovery path and import root are resolved within the same checkout for the selected commit. > Standard Python projects using pyproject.toml and uv or Hatch workspaces would provide a standard package structure. I agree that standard Python packaging is the preferred model when a repository can adopt it. The proposal is not intended to replace that model or detect multiple Python projects. It provides an explicit option for existing monorepos that currently use repository-rooted imports. Regarding Amogh's feedback on using .airflowignore I agree that it technically provides both efficient discovery and repository-rooted imports. However, adding .airflowingore at repository root is not a feasible solution in monorepos (or shared repositories) since It would introduce Airflow specific construct for many teams. Also, regarding adding explicit .airflowignore in `lib`, I understand the structure of the repository. We have 90+ directories from where DAG authors can import dependencies, and thus adding .airflowignore isn't a feasible option as that is owned by different teams. The aim of this functionality proposed is to decouple bundle_path with module import, and user wont be needing to maintain airflowignore. It makes the discovery path and single Python import root explicit without requiring changes to repository. Thanks, Sameer Raj On Mon, Sep 21, 2026 at 4:27 PM Amogh Desai <[email protected]> wrote: > Hi Sameer, > > I have a correction to the .airflowignore approach: it does not just filter > the results but it prunes the ignored > directories out of the *os.walk* itself. > > So setting the bundle path to the repository root and ignoring project/lib/ > gives you both halves: > 1. `from project.lib import some_function` resolves > 2. Airflow never walks the ignored trees. > > Ash's variant works too: `*` in project/lib/.airflowignore, which keeps the > ignore file next to the code it covers > instead of at the monorepo root. > > Thanks & Regards, > Amogh > > On Sun, Sep 20, 2026 at 7:48 PM Jarek Potiuk <[email protected]> wrote: > > > I am not sure this is good idea. Generally "dag" folder where dags are > > **should** be root of the Python imports - and they **should** come as a > > single commit hash exported for example when you use Git Bundle. > > > > So I would rather say that If you need to get Dag + It's utility > libraries > > for example you should checkout Git at the root and refer to your Dags as > > "from project.dags.example_dag import Dag" and "from > > project.lib.example_library import ExampleUtil" - rather than have > several > > roots. > > > > It creates a lot of confusion otherwise when you are "free" to choose > where > > the "roots" for your Python packages should be - especially when you > > include implicit namespaces. > > > > Unless of course we decide to use standard Python project settings (minus > > workspaces) with pyproject.toml and (optionally with uv/hatch > workspaces). > > For example I would be quite receptive to this: > > > > ``` > > repository/ > > ├── project/ > > │ ├───── lib1/ > > │ │ ├── pyproject.toml > > | | └── src/ > > | | └─ toplevelutilpackage1/ > > | | └─ lib1_util.py > > │ ├───── lib2/ > > │ │ ├── pyproject.toml > > | | ├── src/ > > | | | └─ toplevelutilpackage2/ > > | | | └─ secondlevelutilpackage/ > > | | | └─ > > lib2_util.py > > | | └── tests/ > > | | └─ toplevelutilpackage2/ > > | | └─ secondlevelutilpackage/ > > | | > └─ > > tests_lib2_util.py > > │ └── dags/ > > | ├── pyproject.toml > > | ├── src/ > > | | └─ topleveldagpackage > > | | └─ secondleveldagpackage > > | | └─ > > example_dag1.py > > | └── tests/ > > | └─ topleveldagpackage > > | └─ secondleveldagpackage > > | └─ > > test_example_dag1.py > > ``` > > > > Then, the set of ROOTS for Python could be detected and used in a fully > > standard way: > > > > * from toplevelutilpackage1.lib1_util import Util1 > > * from toplevelutilpackage2.secondlevelutilpackage.lib2_util import Util2 > > * from topleveldagpackage.secondleveldagpackage.example_dag1 import Dag1 > > > > This would allow all tooling/IDEs to work appropriately, adhere to modern > > Python standards, and follow the standard setup for tests etc. > > > > J. > > > > > > On Sat, Sep 19, 2026 at 10:48 PM Sameer Raj <[email protected]> wrote: > > > > > Hi Airflow community, > > > > > > I would like feedback on separating Dag discovery paths from Python > > import > > > roots. > > > I had previously raised a Github discussion: > > > https://github.com/apache/airflow/discussions/70313 > > > and received comments regarding starting a dev-list thread > > > > > > A DAG bundle can contain DAG files together with shared Python modules, > > > configuration files, and other resources. > > > Airflow currently uses BaseDagBundle.path for two purposes: > > > 1. Scan for Dag files. > > > 2. Add a directory to sys.path. > > > > > > These paths can differ in a monorepo. For our use case too with > > repository > > > like: > > > ``` > > > repository/ > > > ├── project/ > > > │ ├── lib/ > > > │ └── dags/ > > > │ └── example_dag.py > > > ``` > > > The DAG may import shared code using: > > > >> from project.lib import some_function > > > > > > Airflow automatically adds BaseDagBundle.path to sys.path. In this > > example, > > > it adds only `repository/project/dags`. However, an import such as > `from > > > project.lib import some_function` requires `repository` to be on > > sys.path. > > > > > > Airflow may need to scan only a Dag subdirectory, while Python imports > > > start at the repository root. Using the repository root for discovery > can > > > make Airflow scan many unrelated files. Using only the Dag directory > can > > > make valid imports fail. > > > > > > Thus, the proposal is to decouple import root with bundle path. > > > > > > For the same, I have raised the PR: > > > https://github.com/apache/airflow/pull/73388 > > > > > > The proposed change adds BaseDagBundle.import_root: > > > > > > - import_root defaults to path. > > > - Existing bundle providers keep their current behavior. > > > - GitDagBundle uses the repository root as import_root when path points > > to > > > a subdirectory. > > > - Airflow uses the same import root during Dag parsing, task execution, > > and > > > callback execution. > > > - The import root must belong to the same initialized bundle checkout. > > > > > > The change is backward compatible for LocalDagBundle, S3DagBundle, > > > GCSDagBundle, and custom providers that do not override import_root. > > > Comments from the discussion thread have also been addressed in the PR. > > > > > > I would appreciate feedback on these questions: > > > 1. Is Airflow community aligned with this change. > > > 2. If yes, is import_root the correct public API for this use case. > > > > > > and any other feedbacks is appreciated! > > > > > > Several other users have also requested this: > > > https://github.com/apache/airflow/discussions/61901 > > > So this can be a value add for Airflow community! > > > > > > Thanks, > > > Sameer > > > > > >
