xBis7 commented on PR #61897: URL: https://github.com/apache/airflow/pull/61897#issuecomment-3947048316
> Changing the identity of the parent span after spans in different processes have already emitted is essentially impossible unless you don't emit any telemetry at all until the whole thing has finished. (It sounds like that was the previous approach?) Not exactly what I meant. For example, you have `task1_span1` as the parent. Everything that has been emitted until `task1_span1` endĪ, is going to be a child of that span. Once it ends, then we start `task1_span2` and we make that span the new parent so that everything that is going to be exported from that time on, is going to be a child of `task1_span2`. To clarify, the original approach was creating the `trace_id` and the `span_id` deterministically using values such as the `dag_id`, the `run_id` and the `task_id`. At the end of the run it was creating all the spans, for the dag_run and the tasks and setting the timings based on what was recorded in the db. If you wanted to create task sub-spans, you had to use the `dag_id`, the `run_id` and the `task_id` to recreate the same `trace_id` and `span_id` so that all spans would be eventually associated. I changed it so that the spans would stay active as long as the process executing the dag_run was also active. I needed to make it more compliant with the general OTel behavior and allow context propagation. That's why I came up with the concept that each scheduler keeps track of its own spans. > I think the thing that will give users what they expect without requiring all kinds of technical gymnastics on our part is to synthesise a dag_run span when the dag run completes, in whichever scheduler is handling completion. Although, I'm not entirely sure about it, the more I think of it, I reach the conclusion that the only thing that matters is the user experience. People familiar with OTel should find the Airflow behavior consistent with what they are used to in any other project. A dag_run or task that hasn't been paused, interrupted or stopped, it should be represented by a continuing span. OTel should be configured the standard way, using the OTel environment variables. They should be able to get the task span context and use it to generate sub-spans. Or start a current span that will be associated with the task span. The last part means that there is an active parent task span. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
