potiuk commented on issue #25254: URL: https://github.com/apache/airflow/issues/25254#issuecomment-1196876199
> Another place where it would be useful to surface this information is through the OpenLineage integration. Currently, it sees only state transitions that occurs on the Worker through the TaskInstanceListener. It would be useful to collect more of those through additional listeners in Airflow. The OL integration could send more events besides start and end and capture the additional info as metadata. On that front (might be a good place to discuss)- what do you think @julienledem and @howardyoo about the relation betweeen OTEL traces and OpenLineage ones ? I see those two as pretty orthogonal. OTEL is task/DAG based, and OpenLineage (and lineage in general) are dataset-based. As I see, those are two rather separate and differnet dimentions you can look at when it comes to Airlfow DAGs. They have some things in common, but for anything else than basic HelloWorld, these two will be rather different and will have some common points but the topology of those two will be quite dramatically different. I see the OTEL trace more like "Technical" - more DevOPS thing (where you look at airflwo UI and try to figure if the "system" works as expected, where OpenLineage looks at "data" provenience and lineage (i.e. when you try to see if your data is right). I've recently heard from a few places that the "Observability" term is quite overloaded and it really should be "Data Observability" vs. "Software Observability". Those two overlap of course (and problems in one might even impact the other but they are essentially two rather different dimensions). If my view is correct, Task Instance State is in fact much more of the "Software Observability" than "Data Observability" and as such belongs more to OTEL than Lineage. But I am curious what's your take on it :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
