pgzmnk commented on issue #17984: URL: https://github.com/apache/airflow/issues/17984#issuecomment-924862638
A few thoughts: - This is a great idea. It'd make OpenLineage an easier choice since it'd be easy to implement for multiple DAGs at once. It'd also be easier to maintain as Airflow evolves. - Proposition 2 sounds more aligned with the existing state. Let's ensure it's backward compatible. Projects such as [DataHub subclass LineageBackend](https://github.com/linkedin/datahub/blob/master/metadata-ingestion/src/datahub_provider/lineage/datahub.py#L38). - Otherwise, for proposition 3 it might make sense to decorate `on_failure_callback` (and maybe `on_retry_callback` ?) to keep the method available. On a further downstream note: What differences do you foresee for the lineage 'payload' for successful/started/failed tasks in terms of the OpenLineage schema? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
