dilnazanlid opened a new pull request, #74020:
URL: https://github.com/apache/airflow/pull/74020
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
* **Project Board:** [AIP-85 Dag Importer (Project
#672)](https://github.com/orgs/apache/projects/672)
* 4th PR for rewiring the Dag processing process with `sdk/importers`:
------------
TheDag processor manager still found files with `list_py_file_paths`, which
only returns `.py` and `.zip` files. Files handled by a custom importer
configured in `dag_importer_configs` were never queued for parsing. Discovery
now uses the importer registry, so anything a configured importer can handle is
picked up.
Some importers return definitions nested inside a file, such as members of
an archive. The manager still queues and parses the enclosing file once. The
member locations are kept on `DagFileInfo.definition_locs`, and
`deactivate_deleted_dags` and `clear_orphaned_import_errors` match against
them. Before, a separate zip-only pass reopened every archive to collect member
paths. The `deactivate_deleted_dags(bundle_name, present)` signature is
unchanged.
Other fixes for nested definitions:
- If listing a bundle fails, the error is logged and the bundle is skipped.
It used to be treated as an empty bundle, which deactivated all of its Dags.
- Stale-Dag detection now matches a Dag nested at any depth in a file that
is still present. Before, it only checked one parent level.
- `DagBag.get_dag` / `process_file` now reprocess a nested definition from
its own location. Before, the path had to be mapped back to a `.zip`
(`correct_maybe_zipped`).
- `list_py_file_paths` and `find_dag_file_paths` are removed from
`airflow.utils.file`. They have no remaining callers in this repo.
------------
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Antigravity following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
