GitHub user uplsh580 created a discussion: [Question] Bundle-specific Python
Path Isolation in Airflow 3.x Git Bundles
### [Question] Bundle-specific Python Path Isolation in Airflow 3.x Git Bundles
**Environment:**
- Airflow Version: 3.1.7
- Deployment: Git Bundles
- Setup: Multi-tenant environment where each Git Repository (Bundle) belongs to
a specific team.
---
**Context:**
We are operating a multi-tenant Airflow environment. Each team manages their
own Git repository, which is deployed as a Git Bundle. The directory structure
is versioned by commit IDs:
`bundle_root/{bundle_name}/version/{commit_id}/`
**Assumptions:**
- **One Bundle = One Team:** All DAGs and libraries within a specific
`{bundle_name}` are maintained by the same team.
- **Internal Shared Library:** Each bundle has an internal library folder
(e.g., `airflow_lib/`) at the bundle root, intended to be shared across all
DAGs within that specific bundle and version.
**The Directory Structure:**
```text
{BUNDLE_ROOT}/{bundle_name}/version/{commit_id}/
├── airflow_lib/ # Team-specific internal library (Sibling to dags/)
│ ├── __init__.py
│ ├── constants/
│ └── util/
├── dags/ # DAGs folder
│ ├── team_dag_1.py
│ └── sub_dir/ # Nested DAGs
│ └── team_dag_2.py
└── README.md
**The Problem:**
When the DAG Processor parses these DAGs or the Worker executes tasks, we face
a ModuleNotFoundError: No module named 'airflow_lib'.
Currently, Airflow only includes the dags/ directory in sys.path. Since
airflow_lib sits at the bundle root, it is not reachable by default.
**Key Challenges:**
1. Namespace Collisions across Teams: While one team owns a bundle, different
teams might use identical names for their internal libraries (e.g.,
airflow_lib, utils, common). Therefore, we cannot add all bundle roots to a
global PYTHONPATH.
2. Strict Versioning: A DAG must only import the airflow_lib that belongs to
its specific {commit_id} to ensure consistency.
3. Automated Path Injection: We have hundreds of DAGs across various teams.
Manually adding sys.path.append to every DAG file is not a scalable or clean
solution.
**Questions:**
1. Does Airflow 3.x/AIP-66 provide a native mechanism to automatically add the
Bundle Root to the Python path, scoped only to the DAGs within that specific
bundle?
2. Is there a way to configure a "Base Path" for imports at the Bundle level so
that the DAG Processor and Workers can resolve sibling packages correctly?
3. If this is not yet supported natively, what is the recommended
"Airflow-native" way to inject this path dynamically without polluting the
global environment or modifying every DAG file?
We are looking for an architectural pattern that respects bundle isolation
while allowing for shared internal code. Any advice would be greatly
appreciated!
GitHub link: https://github.com/apache/airflow/discussions/61901
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
