Hi Jarek,
Thank you for the reply. I think that I need to clarify two assumptions
about the proposal.
> We are trying to implement an Airflow solution to handle the repository's
non-Python standard layout.
This is not the problem that the PR tries to solve.
The repository uses one normal Python import root. The repository root
contains the top-level Python packages. Python can import these packages
when the repository root is in sys.path.
For example:
from project.dags.example_dag import Dag
from project.lib.example_library import ExampleUtil
Python does not require a pyproject.toml file for these imports. A
pyproject.toml file is needed when we want to build or install the code as
a Python distribution.
Even in below example repository which has pyproject.toml at root:
repository/
├── pyproject.toml
├── src/
│ └── project/
│ └── lib/
└── dags/
└── dag.py
If Airflow uses repository/dags as bundle.path, it adds only that directory
to sys.path. Python still cannot import project.lib.
The problem is specific to Airflow. Airflow currently uses one path for two
different purposes:
1. Find Dag files.
2. Add the same directory to sys.path (assuming imports are anchored from
that directory).
These paths do not always have the same scope. And with this PR, I am
proposing to decouple this.
For example:
Bundle path: <checkout>/project/dags
Python root: <checkout>
The revised PR adds only the Python root to sys.path. It does not add
several roots. Both paths remain in the same versioned checkout.
> I would very much prefer the solution where what is submitted to Airflow
auto-detects the roots.
The proposal does not require Airflow to detect several roots. There is
only one Python root, and the bundle configuration selects it explicitly.
If no value is specificed, it fallbacks to current behaviour.
The proposal therefore does not try to correct a non-standard Python
layout. It separates Dag discovery from one explicit Python import root.
Both remain part of the same bundle and the same commit.
Thanks,
Sameer Raj
On Mon, 21 Sept 2026 at 22:54, Jarek Potiuk <[email protected]> wrote:
> Why can't you just add the root to your configuration? That seems like we
> are trying to implement an Airflow solution to handle the repository's
> non-Python standard layout. Otherwise, why don't you have pyproject.toml at
> the root and use the python-standard way?
>
> In pyproject.toml in the "repository" folder in your repo would be both
> standard and support for it could be implemented in Airflow.
>
> I would very much prefer the solution where what is submitted to Airflow
> auto-detects the roots - and you just configure your repo to use
> "standards". And I thin it's a very strange argument with .airflowignore.
> Yes it needs to talk to other teams, and i understand it might be a bit
> "red-tap-y" in some organisations, but you repo is **ALREADY** airflow
> specific - it has a "dag" folder.
>
> J.
>
>
> On Mon, Sep 21, 2026 at 7:09 PM Sameer Raj <[email protected]> wrote:
>
> > Hi all,
> > Thank you for the feedback on this proposal.
> >
> > Regarding some of the feedback:
> >
> > > You should checkout Git at the root and refer to your Dags as from
> > project.dags... and utilities as from project.lib....
> > > This should not introduce several roots.
> > > Being free to choose roots can create confusion, especially with
> implicit
> > namespaces.
> >
> > I have revised the PR: https://github.com/apache/airflow/pull/73388.
> > Now, I am only adding the specified import root in the sys.path
> >
> > bundle.path = <repo>/team/dags
> > bundle.import_root = <repo>
> > -> sys.path += bundle.path only
> >
> > We scan bundle.discovery_path only for DAGs
> > This way, we only enforce one root as per the feedback.(Infact, this is
> the
> > same pattern being followed in monorepo.)
> >
> > > Generally, the "dag" folder where dags are should be the root of the
> > Python imports, and they should come as a single commit hash.
> >
> > This is ensured as part of PR now too. In GitDagBundle, both the
> discovery
> > path and import root are resolved within the same checkout for the
> selected
> > commit.
> >
> > > Standard Python projects using pyproject.toml and uv or Hatch
> workspaces
> > would provide a standard package structure.
> > I agree that standard Python packaging is the preferred model when a
> > repository can adopt it. The proposal is not intended to replace that
> model
> > or detect multiple Python projects. It provides an explicit option for
> > existing monorepos that currently use repository-rooted imports.
> >
> > Regarding Amogh's feedback on using .airflowignore
> >
> > I agree that it technically provides both efficient discovery and
> > repository-rooted imports. However, adding .airflowingore at repository
> > root is not a feasible solution in monorepos (or shared repositories)
> since
> > It would introduce Airflow specific construct for many teams. Also,
> > regarding adding explicit .airflowignore in `lib`, I understand the
> > structure of the repository. We have 90+ directories from where DAG
> authors
> > can import dependencies, and thus adding .airflowignore isn't a feasible
> > option as that is owned by different teams.
> >
> > The aim of this functionality proposed is to decouple bundle_path with
> > module import, and user wont be needing to maintain airflowignore. It
> makes
> > the discovery path and single Python import root explicit without
> requiring
> > changes to repository.
> >
> > Thanks,
> > Sameer Raj
> >
> >
> > On Mon, Sep 21, 2026 at 4:27 PM Amogh Desai <[email protected]>
> wrote:
> >
> > > Hi Sameer,
> > >
> > > I have a correction to the .airflowignore approach: it does not just
> > filter
> > > the results but it prunes the ignored
> > > directories out of the *os.walk* itself.
> > >
> > > So setting the bundle path to the repository root and ignoring
> > project/lib/
> > > gives you both halves:
> > > 1. `from project.lib import some_function` resolves
> > > 2. Airflow never walks the ignored trees.
> > >
> > > Ash's variant works too: `*` in project/lib/.airflowignore, which keeps
> > the
> > > ignore file next to the code it covers
> > > instead of at the monorepo root.
> > >
> > > Thanks & Regards,
> > > Amogh
> > >
> > > On Sun, Sep 20, 2026 at 7:48 PM Jarek Potiuk <[email protected]> wrote:
> > >
> > > > I am not sure this is good idea. Generally "dag" folder where dags
> are
> > > > **should** be root of the Python imports - and they **should** come
> as
> > a
> > > > single commit hash exported for example when you use Git Bundle.
> > > >
> > > > So I would rather say that If you need to get Dag + It's utility
> > > libraries
> > > > for example you should checkout Git at the root and refer to your
> Dags
> > as
> > > > "from project.dags.example_dag import Dag" and "from
> > > > project.lib.example_library import ExampleUtil" - rather than have
> > > several
> > > > roots.
> > > >
> > > > It creates a lot of confusion otherwise when you are "free" to choose
> > > where
> > > > the "roots" for your Python packages should be - especially when you
> > > > include implicit namespaces.
> > > >
> > > > Unless of course we decide to use standard Python project settings
> > (minus
> > > > workspaces) with pyproject.toml and (optionally with uv/hatch
> > > workspaces).
> > > > For example I would be quite receptive to this:
> > > >
> > > > ```
> > > > repository/
> > > > ├── project/
> > > > │ ├───── lib1/
> > > > │ │ ├── pyproject.toml
> > > > | | └── src/
> > > > | | └─ toplevelutilpackage1/
> > > > | | └─ lib1_util.py
> > > > │ ├───── lib2/
> > > > │ │ ├── pyproject.toml
> > > > | | ├── src/
> > > > | | | └─ toplevelutilpackage2/
> > > > | | | └─
> > secondlevelutilpackage/
> > > > | | |
> > └─
> > > > lib2_util.py
> > > > | | └── tests/
> > > > | | └─ toplevelutilpackage2/
> > > > | | └─
> > secondlevelutilpackage/
> > > > | |
> > > └─
> > > > tests_lib2_util.py
> > > > │ └── dags/
> > > > | ├── pyproject.toml
> > > > | ├── src/
> > > > | | └─ topleveldagpackage
> > > > | | └─ secondleveldagpackage
> > > > | | └─
> > > > example_dag1.py
> > > > | └── tests/
> > > > | └─ topleveldagpackage
> > > > | └─ secondleveldagpackage
> > > > | └─
> > > > test_example_dag1.py
> > > > ```
> > > >
> > > > Then, the set of ROOTS for Python could be detected and used in a
> fully
> > > > standard way:
> > > >
> > > > * from toplevelutilpackage1.lib1_util import Util1
> > > > * from toplevelutilpackage2.secondlevelutilpackage.lib2_util import
> > Util2
> > > > * from topleveldagpackage.secondleveldagpackage.example_dag1 import
> > Dag1
> > > >
> > > > This would allow all tooling/IDEs to work appropriately, adhere to
> > modern
> > > > Python standards, and follow the standard setup for tests etc.
> > > >
> > > > J.
> > > >
> > > >
> > > > On Sat, Sep 19, 2026 at 10:48 PM Sameer Raj <[email protected]>
> > wrote:
> > > >
> > > > > Hi Airflow community,
> > > > >
> > > > > I would like feedback on separating Dag discovery paths from Python
> > > > import
> > > > > roots.
> > > > > I had previously raised a Github discussion:
> > > > > https://github.com/apache/airflow/discussions/70313
> > > > > and received comments regarding starting a dev-list thread
> > > > >
> > > > > A DAG bundle can contain DAG files together with shared Python
> > modules,
> > > > > configuration files, and other resources.
> > > > > Airflow currently uses BaseDagBundle.path for two purposes:
> > > > > 1. Scan for Dag files.
> > > > > 2. Add a directory to sys.path.
> > > > >
> > > > > These paths can differ in a monorepo. For our use case too with
> > > > repository
> > > > > like:
> > > > > ```
> > > > > repository/
> > > > > ├── project/
> > > > > │ ├── lib/
> > > > > │ └── dags/
> > > > > │ └── example_dag.py
> > > > > ```
> > > > > The DAG may import shared code using:
> > > > > >> from project.lib import some_function
> > > > >
> > > > > Airflow automatically adds BaseDagBundle.path to sys.path. In this
> > > > example,
> > > > > it adds only `repository/project/dags`. However, an import such as
> > > `from
> > > > > project.lib import some_function` requires `repository` to be on
> > > > sys.path.
> > > > >
> > > > > Airflow may need to scan only a Dag subdirectory, while Python
> > imports
> > > > > start at the repository root. Using the repository root for
> discovery
> > > can
> > > > > make Airflow scan many unrelated files. Using only the Dag
> directory
> > > can
> > > > > make valid imports fail.
> > > > >
> > > > > Thus, the proposal is to decouple import root with bundle path.
> > > > >
> > > > > For the same, I have raised the PR:
> > > > > https://github.com/apache/airflow/pull/73388
> > > > >
> > > > > The proposed change adds BaseDagBundle.import_root:
> > > > >
> > > > > - import_root defaults to path.
> > > > > - Existing bundle providers keep their current behavior.
> > > > > - GitDagBundle uses the repository root as import_root when path
> > points
> > > > to
> > > > > a subdirectory.
> > > > > - Airflow uses the same import root during Dag parsing, task
> > execution,
> > > > and
> > > > > callback execution.
> > > > > - The import root must belong to the same initialized bundle
> > checkout.
> > > > >
> > > > > The change is backward compatible for LocalDagBundle, S3DagBundle,
> > > > > GCSDagBundle, and custom providers that do not override
> import_root.
> > > > > Comments from the discussion thread have also been addressed in the
> > PR.
> > > > >
> > > > > I would appreciate feedback on these questions:
> > > > > 1. Is Airflow community aligned with this change.
> > > > > 2. If yes, is import_root the correct public API for this use case.
> > > > >
> > > > > and any other feedbacks is appreciated!
> > > > >
> > > > > Several other users have also requested this:
> > > > > https://github.com/apache/airflow/discussions/61901
> > > > > So this can be a value add for Airflow community!
> > > > >
> > > > > Thanks,
> > > > > Sameer
> > > > >
> > > >
> > >
> >
On Mon, Sep 21, 2026 at 10:54 PM Jarek Potiuk <[email protected]> wrote:
> Why can't you just add the root to your configuration? That seems like we
> are trying to implement an Airflow solution to handle the repository's
> non-Python standard layout. Otherwise, why don't you have pyproject.toml at
> the root and use the python-standard way?
>
> In pyproject.toml in the "repository" folder in your repo would be both
> standard and support for it could be implemented in Airflow.
>
> I would very much prefer the solution where what is submitted to Airflow
> auto-detects the roots - and you just configure your repo to use
> "standards". And I thin it's a very strange argument with .airflowignore.
> Yes it needs to talk to other teams, and i understand it might be a bit
> "red-tap-y" in some organisations, but you repo is **ALREADY** airflow
> specific - it has a "dag" folder.
>
> J.
>
>
> On Mon, Sep 21, 2026 at 7:09 PM Sameer Raj <[email protected]> wrote:
>
> > Hi all,
> > Thank you for the feedback on this proposal.
> >
> > Regarding some of the feedback:
> >
> > > You should checkout Git at the root and refer to your Dags as from
> > project.dags... and utilities as from project.lib....
> > > This should not introduce several roots.
> > > Being free to choose roots can create confusion, especially with
> implicit
> > namespaces.
> >
> > I have revised the PR: https://github.com/apache/airflow/pull/73388.
> > Now, I am only adding the specified import root in the sys.path
> >
> > bundle.path = <repo>/team/dags
> > bundle.import_root = <repo>
> > -> sys.path += bundle.path only
> >
> > We scan bundle.discovery_path only for DAGs
> > This way, we only enforce one root as per the feedback.(Infact, this is
> the
> > same pattern being followed in monorepo.)
> >
> > > Generally, the "dag" folder where dags are should be the root of the
> > Python imports, and they should come as a single commit hash.
> >
> > This is ensured as part of PR now too. In GitDagBundle, both the
> discovery
> > path and import root are resolved within the same checkout for the
> selected
> > commit.
> >
> > > Standard Python projects using pyproject.toml and uv or Hatch
> workspaces
> > would provide a standard package structure.
> > I agree that standard Python packaging is the preferred model when a
> > repository can adopt it. The proposal is not intended to replace that
> model
> > or detect multiple Python projects. It provides an explicit option for
> > existing monorepos that currently use repository-rooted imports.
> >
> > Regarding Amogh's feedback on using .airflowignore
> >
> > I agree that it technically provides both efficient discovery and
> > repository-rooted imports. However, adding .airflowingore at repository
> > root is not a feasible solution in monorepos (or shared repositories)
> since
> > It would introduce Airflow specific construct for many teams. Also,
> > regarding adding explicit .airflowignore in `lib`, I understand the
> > structure of the repository. We have 90+ directories from where DAG
> authors
> > can import dependencies, and thus adding .airflowignore isn't a feasible
> > option as that is owned by different teams.
> >
> > The aim of this functionality proposed is to decouple bundle_path with
> > module import, and user wont be needing to maintain airflowignore. It
> makes
> > the discovery path and single Python import root explicit without
> requiring
> > changes to repository.
> >
> > Thanks,
> > Sameer Raj
> >
> >
> > On Mon, Sep 21, 2026 at 4:27 PM Amogh Desai <[email protected]>
> wrote:
> >
> > > Hi Sameer,
> > >
> > > I have a correction to the .airflowignore approach: it does not just
> > filter
> > > the results but it prunes the ignored
> > > directories out of the *os.walk* itself.
> > >
> > > So setting the bundle path to the repository root and ignoring
> > project/lib/
> > > gives you both halves:
> > > 1. `from project.lib import some_function` resolves
> > > 2. Airflow never walks the ignored trees.
> > >
> > > Ash's variant works too: `*` in project/lib/.airflowignore, which keeps
> > the
> > > ignore file next to the code it covers
> > > instead of at the monorepo root.
> > >
> > > Thanks & Regards,
> > > Amogh
> > >
> > > On Sun, Sep 20, 2026 at 7:48 PM Jarek Potiuk <[email protected]> wrote:
> > >
> > > > I am not sure this is good idea. Generally "dag" folder where dags
> are
> > > > **should** be root of the Python imports - and they **should** come
> as
> > a
> > > > single commit hash exported for example when you use Git Bundle.
> > > >
> > > > So I would rather say that If you need to get Dag + It's utility
> > > libraries
> > > > for example you should checkout Git at the root and refer to your
> Dags
> > as
> > > > "from project.dags.example_dag import Dag" and "from
> > > > project.lib.example_library import ExampleUtil" - rather than have
> > > several
> > > > roots.
> > > >
> > > > It creates a lot of confusion otherwise when you are "free" to choose
> > > where
> > > > the "roots" for your Python packages should be - especially when you
> > > > include implicit namespaces.
> > > >
> > > > Unless of course we decide to use standard Python project settings
> > (minus
> > > > workspaces) with pyproject.toml and (optionally with uv/hatch
> > > workspaces).
> > > > For example I would be quite receptive to this:
> > > >
> > > > ```
> > > > repository/
> > > > ├── project/
> > > > │ ├───── lib1/
> > > > │ │ ├── pyproject.toml
> > > > | | └── src/
> > > > | | └─ toplevelutilpackage1/
> > > > | | └─ lib1_util.py
> > > > │ ├───── lib2/
> > > > │ │ ├── pyproject.toml
> > > > | | ├── src/
> > > > | | | └─ toplevelutilpackage2/
> > > > | | | └─
> > secondlevelutilpackage/
> > > > | | |
> > └─
> > > > lib2_util.py
> > > > | | └── tests/
> > > > | | └─ toplevelutilpackage2/
> > > > | | └─
> > secondlevelutilpackage/
> > > > | |
> > > └─
> > > > tests_lib2_util.py
> > > > │ └── dags/
> > > > | ├── pyproject.toml
> > > > | ├── src/
> > > > | | └─ topleveldagpackage
> > > > | | └─ secondleveldagpackage
> > > > | | └─
> > > > example_dag1.py
> > > > | └── tests/
> > > > | └─ topleveldagpackage
> > > > | └─ secondleveldagpackage
> > > > | └─
> > > > test_example_dag1.py
> > > > ```
> > > >
> > > > Then, the set of ROOTS for Python could be detected and used in a
> fully
> > > > standard way:
> > > >
> > > > * from toplevelutilpackage1.lib1_util import Util1
> > > > * from toplevelutilpackage2.secondlevelutilpackage.lib2_util import
> > Util2
> > > > * from topleveldagpackage.secondleveldagpackage.example_dag1 import
> > Dag1
> > > >
> > > > This would allow all tooling/IDEs to work appropriately, adhere to
> > modern
> > > > Python standards, and follow the standard setup for tests etc.
> > > >
> > > > J.
> > > >
> > > >
> > > > On Sat, Sep 19, 2026 at 10:48 PM Sameer Raj <[email protected]>
> > wrote:
> > > >
> > > > > Hi Airflow community,
> > > > >
> > > > > I would like feedback on separating Dag discovery paths from Python
> > > > import
> > > > > roots.
> > > > > I had previously raised a Github discussion:
> > > > > https://github.com/apache/airflow/discussions/70313
> > > > > and received comments regarding starting a dev-list thread
> > > > >
> > > > > A DAG bundle can contain DAG files together with shared Python
> > modules,
> > > > > configuration files, and other resources.
> > > > > Airflow currently uses BaseDagBundle.path for two purposes:
> > > > > 1. Scan for Dag files.
> > > > > 2. Add a directory to sys.path.
> > > > >
> > > > > These paths can differ in a monorepo. For our use case too with
> > > > repository
> > > > > like:
> > > > > ```
> > > > > repository/
> > > > > ├── project/
> > > > > │ ├── lib/
> > > > > │ └── dags/
> > > > > │ └── example_dag.py
> > > > > ```
> > > > > The DAG may import shared code using:
> > > > > >> from project.lib import some_function
> > > > >
> > > > > Airflow automatically adds BaseDagBundle.path to sys.path. In this
> > > > example,
> > > > > it adds only `repository/project/dags`. However, an import such as
> > > `from
> > > > > project.lib import some_function` requires `repository` to be on
> > > > sys.path.
> > > > >
> > > > > Airflow may need to scan only a Dag subdirectory, while Python
> > imports
> > > > > start at the repository root. Using the repository root for
> discovery
> > > can
> > > > > make Airflow scan many unrelated files. Using only the Dag
> directory
> > > can
> > > > > make valid imports fail.
> > > > >
> > > > > Thus, the proposal is to decouple import root with bundle path.
> > > > >
> > > > > For the same, I have raised the PR:
> > > > > https://github.com/apache/airflow/pull/73388
> > > > >
> > > > > The proposed change adds BaseDagBundle.import_root:
> > > > >
> > > > > - import_root defaults to path.
> > > > > - Existing bundle providers keep their current behavior.
> > > > > - GitDagBundle uses the repository root as import_root when path
> > points
> > > > to
> > > > > a subdirectory.
> > > > > - Airflow uses the same import root during Dag parsing, task
> > execution,
> > > > and
> > > > > callback execution.
> > > > > - The import root must belong to the same initialized bundle
> > checkout.
> > > > >
> > > > > The change is backward compatible for LocalDagBundle, S3DagBundle,
> > > > > GCSDagBundle, and custom providers that do not override
> import_root.
> > > > > Comments from the discussion thread have also been addressed in the
> > PR.
> > > > >
> > > > > I would appreciate feedback on these questions:
> > > > > 1. Is Airflow community aligned with this change.
> > > > > 2. If yes, is import_root the correct public API for this use case.
> > > > >
> > > > > and any other feedbacks is appreciated!
> > > > >
> > > > > Several other users have also requested this:
> > > > > https://github.com/apache/airflow/discussions/61901
> > > > > So this can be a value add for Airflow community!
> > > > >
> > > > > Thanks,
> > > > > Sameer
> > > > >
> > > >
> > >
> >
>
