kacpermuda commented on code in PR #70131:
URL: https://github.com/apache/airflow/pull/70131#discussion_r3622969609
##########
providers/dbt/cloud/src/airflow/providers/dbt/cloud/utils/openlineage.py:
##########
@@ -97,6 +99,62 @@ def _get_parent_run_metadata(task_instance):
)
+def _get_openlineage_parent_facet_dict(task_instance) -> dict:
+ """Build the OpenLineage ``ParentRunFacet`` payload (parent and root) for
a task instance."""
+ metadata = _get_parent_run_metadata(task_instance)
+ return {
+ "parent": {
+ "run": {"runId": metadata.run_id},
+ "job": {"namespace": metadata.job_namespace, "name":
metadata.job_name},
+ },
+ "root": {
+ "run": {"runId": metadata.root_parent_run_id},
+ "job": {
+ "namespace": metadata.root_parent_job_namespace,
+ "name": metadata.root_parent_job_name,
+ },
+ },
+ }
+
+
+def inject_parent_job_information_into_dbt_cloud_cause(cause: str | None,
task_instance) -> str | None:
+ """
+ Replace the dbt Cloud run ``cause`` with OpenLineage parent job
information as JSON.
+
+ This serializes the Airflow task's OpenLineage parent (and root) run
identifiers into the
+ triggered run's ``cause`` field. A consumer that reads dbt Cloud runs can
parse this JSON and
+ attach a ``ParentRunFacet``, linking the dbt Cloud run back to the Airflow
task that triggered it.
+
+ The dbt Cloud ``cause`` is limited to ``DBT_CAUSE_MAX_LENGTH`` characters.
The essential ``parent``
+ link is always kept; the ``root`` block is dropped if the two together do
not fit. If even the
+ parent-only payload does not fit, the original ``cause`` is returned
unchanged.
+ """
+ try:
+ facet = _get_openlineage_parent_facet_dict(task_instance)
+ except ImportError:
+ log.warning(
+ "Could not import OpenLineage provider. Skipping the injection of
OpenLineage "
+ "parent job information into the dbt Cloud run cause."
+ )
+ return cause
+
+ candidates = [
+ {"parent": facet["parent"], "root": facet["root"]},
+ {"parent": facet["parent"]},
+ ]
Review Comment:
I know that every character counts since the limit is low, but maybe we
should add encapsulating `{"openlineage": {"parent"...}}` dict? Without the
word `openlineage`, parent and root can be confusing for reader. But, if
somebody sets it to True, they usually know what they're doing. Something to
consider.
##########
providers/dbt/cloud/src/airflow/providers/dbt/cloud/operators/dbt.py:
##########
@@ -219,6 +233,12 @@ def execute(self, context: Context):
f"Triggered via Apache Airflow by task {self.task_id!r} in the
{self.dag.dag_id} DAG."
)
+ if self.openlineage_inject_parent_job_info:
+ self.log.info("Injecting OpenLineage parent job information into
dbt Cloud run cause.")
Review Comment:
```suggestion
self.log.info("Replacing dbt Cloud run cause with OpenLineage
parent job information.")
```
Let's be explicit here, since we replace and not inject (because the char
limit on cause is so low)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
