anandhimurali commented on code in PR #38982: URL: https://github.com/apache/airflow/pull/38982#discussion_r1666300729
########## docs/apache-airflow-providers-openlineage/guides/developer.rst: ########## @@ -446,15 +446,96 @@ Conversion from Airflow Table entity to OpenLineage Dataset is made in the follo .. _custom_facets:openlineage: -Custom facets +Custom Facets ============= To learn more about facets in OpenLineage, please refer to `facet documentation <https://openlineage.io/docs/spec/facets/>`_. -Also check out `available Facets <https://github.com/OpenLineage/OpenLineage/blob/main/client/python/openlineage/client/facet.py>`_ +Also check out `available facets <https://github.com/OpenLineage/OpenLineage/blob/main/client/python/openlineage/client/facet.py>`_ The OpenLineage spec might not contain all the facets you need to write your extractor, in which case you will have to make your own `custom facets <https://openlineage.io/docs/spec/facets/custom-facets>`_. More on creating custom facets can be found `here <https://openlineage.io/blog/extending-with-facets/>`_. +Custom Run Facets +----------------- + +You can inject your own custom facets in the lineage event's run facet using the ``custom_run_facets`` Airflow configuration. + +Steps to be taken, + +1. Write a function that returns the custom facet. You can write as many custom facet functions as needed. +2. Register the functions using the ``custom_run_facets`` Airflow configuration. + +Once done, Airflow OpenLineage listener will automatically execute these functions during the lineage event generation +and append their return values to the run facet in the lineage event. + +Writing a custom facet function +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +- **Input arguments:** The function should accept the ``TaskInstance`` as an input argument. +- **Function body:** Perform the logic needed to generate the custom facet. The custom facet should inherit from the ``BaseFacet`` for the ``_producer`` and ``_schemaURL`` to be automatically added for the facet. +- **Return value:** The custom facet to be added to the lineage event. Return type should be ``None`` or ``dict``. You may choose to return ``None`` or ``{}``, if you do not want to add custom facets for certain criteria. + +**Example custom facet function** + +.. code-block:: python + + import attrs + from airflow.models import TaskInstance + from openlineage.client.facet import BaseFacet + + + @attrs.define(slots=False) + class MyCustomRunFacet(BaseFacet): + """Define a custom facet.""" + + name: str + jobState: str + uniqueName: str + displayName: str + dagId: str + taskId: str + cluster: str + + + def get_my_custom_facet(task_instance: TaskInstance): + job_unique_name = f"TEST.{task_instance.dag_id}.{task_instance.task_id}" Review Comment: Addressed in 3455b732e64a70fc3c041390d8f841987c47828f. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
