mobuchowski commented on issue #17984: URL: https://github.com/apache/airflow/issues/17984#issuecomment-926061181
@ashb Sure. I want to implement OpenLineage integration for Airflow 2.1+ using `LineageBackend`. Actually, I've [done MVP of it here](https://github.com/OpenLineage/OpenLineage/blob/airflow/2/integration/airflow/openlineage/airflow/backend.py) :wink: As for failed runs, let's imagine we detected data quality issues in some dataset, which caused our ML models to misbehave. We could look at multiple places to understand what happened, or look at lineage graph for our dataset, and see that Airflow updating it failed. This is not something that is just conceptual, it exists already for Airflow 1.10, Spark, dbt - you can take a look at https://demo.datakin.com or Marquez's Airflow example https://github.com/MarquezProject/marquez/tree/main/examples/airflow -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
