Yanislava commented on issue #51321:
URL: https://github.com/apache/airflow/issues/51321#issuecomment-6342277369
Hi! I'm interested in contributing to this issue, and I've been looking into
how Airflow currently handles DAG source information.
Before starting an implementation, I'd like to share a few observations and
ask for feedback on the intended approach.
### Concerns about manually maintained metadata
My main concern with introducing `SourceCodeLocation` (or a generic
`metadata` field) directly into the DAG definition is that source information
can easily become outdated.
For example:
- A DAG can be moved to another repository without updating its metadata.
- The configured branch or ref can change.
- Every new deployment may use a different commit SHA.
The commit SHA is particularly problematic. If we embed it directly in a
serialized DAG field, the DAG's serialized hash may change on every deployment,
even when the DAG definition itself hasn't changed. This could cause Airflow to
create unnecessary `DagVersion` records.
It seems preferable to derive source information from the actual deployed
code whenever Airflow already has access to it, rather than requiring
developers to maintain it manually.
### What Airflow already provides
From looking at the existing DAG Bundle implementation, `GitDagBundle`
appears to expose most of the information needed for
`SourceCodeLocationJobFacet`:
- Repository URL: `repo_url`
- Git ref: `tracking_ref` (which may be a branch, tag, or commit)
- Source path: `subdir` combined with `dag.relative_fileloc`
- Revision: the bundle version associated with the task execution
This suggests that, at least for `GitDagBundle`, the facet could potentially
be generated automatically without introducing additional fields into the DAG
API.
### Runtime experiment with S3DagBundle
I also tested which fields are accessible from a running task in our Airflow
3.2.2 test environment, which uses `S3DagBundle`.
Using `get_current_context()` and inspecting the DAG, RuntimeTaskInstance,
DagRun, and bundle instance, I got the following results:
```text
dag.fileloc:
/tmp/airflow/dag_bundles/ds-dags/smoke/check_metadata.py
dag.relative_fileloc:
smoke/check_metadata.py
dag.bundle_name:
None
ti.bundle_instance:
S3DagBundle
ti.bundle_instance.name:
ds-dags
ti.bundle_instance.repo_url:
None
ti.bundle_instance.tracking_ref:
None
ti.bundle_instance.version:
None
ti.bundle_instance.supports_versioning:
False
dag_run.bundle_version:
None
```
This is expected for a non-versioned S3 bundle, but it confirms that the
runtime exposes the bundle instance and the DAG's relative file path. Git
source information is not available through these fields in this configuration.
### Proposed direction / question
I'm considering whether source provenance should be derived automatically
when the DAG Bundle can provide reliable information, rather than introducing
manually maintained source annotations as the primary mechanism.
For `GitDagBundle`, we could potentially build `SourceCodeLocationJobFacet`
from the existing bundle and runtime data.
For other bundle types, such as `LocalDagBundle` or `S3DagBundle`, I see two
possible approaches:
1. **Omit the facet** when Airflow cannot reliably determine the original
repository, ref, and revision.
2. **Allow optional explicit source metadata** as a fallback, accepting that
its accuracy and maintenance would be the user's responsibility.
Which approach would you prefer for this issue?
Also, would resolving source provenance directly in the OpenLineage provider
be appropriate, or should Airflow Core expose a common interface for retrieving
it from different DAG Bundle implementations?
I'd be happy to work on a smaller initial PR once the intended direction is
clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
