1fanwang opened a new pull request, #70354: URL: https://github.com/apache/airflow/pull/70354
### Why A DAG bundle defines its *source* (a path and a version) but not the *environment* its files are parsed in. When one DAG processor serves multiple bundles, every bundle is parsed in that processor's single Python environment. Bundles from different teams that need different dependencies (or different versions of the same one) can't each be parsed against their own — a DAG that imports a team-only or version-pinned package builds fine for the team but fails `DagBag` import when a shared processor parses it. Today the only isolation is running a DAG processor per environment (`--bundle-name` scoping), which costs a process per dependency set. ### What Add an opt-in `BaseDagBundle.parse_pythonpath` (empty by default). When set, the DAG processor prepends those entries to `sys.path` in the per-file parsing subprocess for that bundle only, so a single processor can parse bundles against their own dependencies (for example each bundle's own virtualenv `site-packages`). Bundles that declare nothing parse exactly as before. - `BaseDagBundle.parse_pythonpath` property (default `[]`). - `DagFileParseRequest.bundle_pythonpath`, threaded from `DagFileProcessorManager._create_process` through `DagFileProcessorProcess.start`. - `_parse_file` prepends it to `sys.path` before parsing. ### Tests `_parse_file` parses a DAG importing a module present only on the bundle's `parse_pythonpath`, and reproduces the import failure without it; plus `parse_pythonpath` default and override. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: GitHub Copilot CLI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
