xBis7 commented on PR #62154: URL: https://github.com/apache/airflow/pull/62154#issuecomment-3926113449
Hi @nickstenning, > A desire to provide a from airflow.* import that provides a shared abstraction over the tracing implementation rather than having from opentelemetry.* imports throughout the codebase. We can't use opentelemetry.* imports throughout the codebase and let me explain why. Airflow is a complicated project composed of a bunch of independent processes and subprocesses. You don't have a central in-memory constructor that initializes a bunch of objects and shares them across all processes. When I'm saying processes, I'm referring to a scheduler, a worker, a dag processor, etc. Each one of these, needs to configure the SDK and get its own instance of a tracer. That's why we need a common file with the SDK initialization logic that can be reused. Addiitionally, the current implementation with the metaclass makes sure that the SDK is initialized only once per process. Apart from the initialization, airflow is very hard to auto-instrument. Many operations are depending on db updates. E.g. once that is changed in the db, then take this action. For that reason, we have to handle the span context change manually and start-end spans manually. > A desire to have traces and spans which are emitted conditionally, in order to increase or decrease the level of detail. > > > On (2) I think there's a more fundamental issue, which is that if the desire is to provide configurable verbosity/detail levels for the traces, it doesn't make sense to try and achieve that at the level of a trace. The goal is for users to have continuous visibility under tasks. To be able to figure out, at which step of the execution the task is and then also understand how long each took. Apart from the dag_run traces that also allow the creation of task sub-spans, IMO everything else should be removed. The debug spans are providing details about internal operations and they are only useful to developers who have better tools at their disposal like the IDE debugger or a profiler with flamegraphs. @nickstenning If I understand correctly, you are also agreeing on this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
