mobuchowski opened a new pull request #18470: URL: https://github.com/apache/airflow/pull/18470
Motivation for this PR is [OpenLineage's](https://openlineage.io/) [Airflow integration.](https://github.com/OpenLineage/OpenLineage/tree/main/integration/airflow) Currently, LineageBackend is notified only when task succesfully completes. This PR implements support for two additional methods for `LineageBackend` - to become notified when task starts and when it fails. Those methods are optional, and any implementations of `LineageBackend` should work exactly the same if they won't implement it. With addition to adding those methods, they are hooked with (AFAIK) appropriate places to do so. Also, changed loading lineage backend to be lazy. Now, I think few things are incomplete. I have no idea what behavior should be for lineage inlets and outlets in context of a failed job - so I just ignored them. Implements https://github.com/apache/airflow/issues/17984 Signed-off-by: Maciej Obuchowski <[email protected]> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
