Hi I am looking at our OTEL stuff and I have reached the conclusion that we should rework it so we don't jump through hoops to keep alive very long spans for dag run and task.
We should still have the spans, we just shouldn't jump through hoops to ensure that their start and end times match those in the metastore. Indeed, that's information that's always already available! I do not claim to be an OTEL expert. But intuitively we can see that the current approach is very complicated and confusing, and therein probably less reliable and certainly less maintainable. You can see what I've done so far here: https://github.com/apache/airflow/pull/61897 Sycophantic though it may be, chat gpt seems to agree with me: https://chatgpt.com/share/698fea02-fa18-8013-93f7-1e26215bc3f6 And it makes sense -- just make spans that will automatically be closed when your specific action is over. WDYT? Thanks
