kacpermuda commented on code in PR #57809:
URL: https://github.com/apache/airflow/pull/57809#discussion_r2494267505
##########
providers/openlineage/docs/guides/user.rst:
##########
@@ -478,6 +478,51 @@ You can enable this automation by setting
``spark_inject_transport_info`` option
AIRFLOW__OPENLINEAGE__SPARK_INJECT_TRANSPORT_INFO=true
+Passing parent information to Airflow DAG
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To enable full OpenLineage lineage tracking across dependent DAGs, you can
pass parent and root job information
+through the DAG's ``dag_run.conf``. When a DAG run configuration includes an
``_openlineage`` section with valid metadata,
+this information is automatically parsed and converted into DAG's
**ParentRunFacet**, from which the root information
+is also propagated to all task runs. If no DAG run configuration is provided,
the DAG run itself is considered the
+lineage root for its tasks.
+
+The ``_openlineage`` dict in conf can contain the following keys:
+
+*(all three values must be included to create a parent reference)*
+
+- **parentRunId** — the unique run ID (uuid) of the direct parent job
+- **parentJobName** — the name of the parent job
+- **parentJobNamespace** — the namespace of the parent job
+
+*(all three values must be included to create a root reference)*
+
+- **rootParentRunId** — the run ID (uuid) of the top-level (root) job
+- **rootParentJobName** — the name of the top-level (root) job
+- **rootParentJobNamespace** — the namespace of the top-level (root) job
+
+
+Example:
+
+.. code-block:: python
+
+ conf = {
+ "_openlineage": {
+ "parentRunId": "3bb703d1-09c1-4a42-8da5-35a0b3216072",
+ "parentJobNamespace": "prod_biz",
+ "parentJobName": "get_files",
+ "rootParentRunId": "9d3b14f7-de91-40b6-aeef-e887e2c7673e",
+ "rootParentJobNamespace": "prod_analytics",
+ "rootParentJobName": "generate_report_sales_e2e",
+ }
+ }
+ trigger_dagrun = TriggerDagRunOperator(
+ task_id="trigger_dagrun",
+ trigger_dag_id="child_dag",
+ conf=conf,
+ )
Review Comment:
Yes, that will be included in a separate PR. I already have it working, but
wanted to split it into smaller chunks. We'll automatically inject this for
users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
