dilnazanlid opened a new pull request, #73853:
URL: https://github.com/apache/airflow/pull/73853
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
* **Project Board:** [AIP-85 Dag Importer (Project
#672)](https://github.com/orgs/apache/projects/672)
* 3rd PR for rewiring the Dag processing process with `sdk/importers`:
replace the `dag_processing.importers` by `sdk/importers` and move to Dag
definitions.
------------
DagBag now discovers and imports Dag definitions through the bundle-scoped
Task SDK
importer registry instead of the core `airflow.dag_processing.importers`
package, which is removed.
- Importers can be configured globally with `[dag_processor]
dag_importer_configs`, or
per bundle through an `importers` key in `dag_bundle_config_list`.
- DagBag sets `fileloc`, `relative_fileloc` and `bundle_name` on every
imported Dag, so
importers that build Dags themselves (non-Python formats) are bound to
their definition.
- The source code and language reported by the importer are stored in
`DagCode`. The
source is read once per file in the Dag processor instead of once per Dag
in the manager.
- Importer-namespaced warnings (e.g. `yaml:deprecated_field`) become Dag
warnings and are
cleared when no longer reported. Python warnings keep going to
`captured_warnings`.
- Import errors are cleared per parsed definition, e.g. per zip member.
- `DagFileParsingResult` gains `parsed_definitions` and `dag_source_codes`,
with a
supervisor schema version change.
Behaviour changes in internal APIs:
- `airflow.dag_processing.importers` is removed. It had no configuration or
docs, and its
`import_file` API differs from the SDK one.
- `DagBag.file_last_changed` values are freshness tokens (`str`) instead of
`datetime`,
and zip entries are keyed per member.
- `DagBag.has_logged` is removed. Files skipped by safe mode are now dropped
during
discovery and logged at debug level.
The Dag processor manager still discovers only `.py` and zip files.
Scheduled parsing of
other file types will be done in the follow-up PR that moves manager
discovery to the registry.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Antigravity following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
