sjyangkevin opened a new pull request, #74542:
URL: https://github.com/apache/airflow/pull/74542
### Why
Dags packaged as zip archives could not use template files shipped inside
the archive. For a zipped Dag, `dag.folder` points into the archive (for
example `.../dags.zip`), and Jinja's `FileSystemLoader` only reads real
directories. A templated field such as `"query": "test_sql/test.sql"` therefore
failed every run with:
```
jinja2.exceptions.TemplateNotFound: 'test_sql/test.sql' not found in search
path: '.../test.zip'
```
At parse time the same lookup failed silently (only logged), and pointing
`template_searchpath` into the archive failed the same way.
`create_template_env()` now reads search-path entries that are, or point
inside, a zip archive directly from the archive. Normal directories keep
exactly the same `FileSystemLoader` as before. Both template environment
builders (`DAG.get_template_env()` and the operator-level
`render_template_as_native_obj` override) go through this function.
closes: #59310
related: #60999
### How zip entries are handled
- The archive is found by walking up to the nearest existing path and
checking it with `zipfile.is_zipfile()`, so detection does not depend on the
`.zip` extension or its case.
- Members are read in memory by exact name; nothing is extracted to disk.
- Template names are validated with Jinja's `split_template_path`, as in
`FileSystemLoader`, so they cannot leave the search root. Search-path prefixes
containing `..` are not treated as archive paths.
- Search-path order is preserved across zip and directory entries, and a
missing template still raises `TemplateNotFound` with the `not found in search
path(s): ...` message.
- An unreadable archive raises instead of falling through to a later search
path.
- The archive is opened per lookup, so no file handle is shared across the
task supervisor's fork.
### Testing
- 18 new test cases. 9 fail without this change: loading from a zip
(including `DAGS.ZIP`), zip-first ordering, both template environment builders
with a `.sql` path nested in a dict, and parse-time `resolve_template_files()`
on a Dag imported by the zip importer. The other 9 pass on `main` by design:
they pin unchanged behaviour for non-zip search paths, the not-found message,
and containment of template names. Each was checked to fail when the
corresponding check in the new loader is removed.
- End-to-end in Breeze, using `airflow dags test` and the UI from `breeze
start-airflow`, with zipped Dags using a top-level template file, a `.sql` path
nested in a dict, `template_searchpath` inside the archive, and `{% include
%}`. All failed with `TemplateNotFound` before this change and succeed after
it. Unzipped copies of the same Dags succeed both before and after.
- The zip attached to the issue now gets past template rendering and reaches
`BigQueryInsertJobOperator.execute()`. The BigQuery call itself was not run (no
GCP credentials), and this change does not touch it.
- prek pre-commit and manual stages pass, including `mypy-task-sdk`. From
`breeze verify`: `task-sdk-tests` (3656 passed), core non-DB tests and core DB
tests pass.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code (Opus 5.5) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
