Aditya-XR opened a new pull request, #74573: URL: https://github.com/apache/airflow/pull/74573
On 3.3, Dag discovery decides whether a non-`.py` file is a zipped Dag with `zipfile.is_zipfile()` alone, without looking at the file name. `.jar`, `.whl`, `.egg`, `.docx`, `.xlsx` and plenty of other formats are zip containers, so any of them sitting in a Dags folder gets picked up as a Dag file. The Dag processor then queues it for parsing on every loop, the parse finds no importer for that extension and returns nothing, and the file shows up in parsing stats and `airflow dags report` with 0 Dags. #71125 has a report of this from a real deployment. This goes straight to `v3-3-test` because there is nothing to cherry-pick: on `main` and `v3-4-test` the importer-based discovery already matches on file extension only. **Change:** a non-`.py` file is only treated as a zipped Dag when its name ends in `.zip` (any case) and it is a valid zip archive. Applied to both discovery paths: - `find_dag_file_paths` in `airflow/utils/file.py`, used by the Dag processor via `list_py_file_paths` - `PythonDagImporter.list_dag_files`, used by `DagBag.collect_dags` This makes the directory walk agree with the rest of 3.3, where the importer registry already picks an importer by file extension. **Behaviour notes** - `.zip` and `.ZIP` archives are discovered exactly as before, and an invalid `.zip` is still skipped. - A zip archive whose name doesn't end in `.zip` (a `bundle.zip.bak` backup, a file with no extension, ...) is no longer discovered. Dags inside such files couldn't be loaded before either, because no importer is registered for those extensions, so nothing that works today stops working. - `import_file` and `_get_observed_filelocs` still use `is_zipfile`, but they only ever see files that discovery returned, so I left them alone to keep the patch small. **Testing** - New parametrized tests for both discovery paths, each case a real archive containing a Dag-like file: `.zip`, `.ZIP`, `.jar`, `.whl`, `.egg`, `.docx`, `.zip.bak` and no extension. The six non-`.zip` cases fail on current `v3-3-test` and pass with the change. - `airflow-core/tests/unit/utils/test_file.py` and `airflow-core/tests/unit/dag_processing/`: 466 passed (450 existing + 16 new). The same 450 pass on the base branch. - `prek` on the changed files passes, including mypy for airflow-core. - Checked by hand with a Dags folder holding `.zip` and `.ZIP` bundles, a plain `.py` Dag, a directory named `folder.zip/`, a corrupt `.zip`, and a `.jar`, `.whl`, `.docx` and `.zip.bak` that each contain a Dag. Before, discovery returned 8 entries including all four non-`.zip` archives; after, only the `.py`, the two archives and the file inside `folder.zip/`. The set of loaded Dags is the same before and after. related: #71125 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5.5) Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
