howardyoo commented on issue #42424: URL: https://github.com/apache/airflow/issues/42424#issuecomment-2374438262
Auto-instrumentation is a tricky(?) area, especially trying to instrument a complex system like Airflow (It does work extremely well with smaller applications like microservices, where that was the area the auto-instrumentation usually focuses on). Technically, I would say the instrumenting the whole Airflow using otel auto-instrumentation would work, but you may have to do it on your own risk. I would be a little worried if we did that because that would introduce a huge amount of telemetry data (I know how much Airflow calls database queries just to keep it idly running), and also potential impact on its performance has not been greatly studied. So, when I was implementing the AIP-49 (the otel traces for Airflow), I purposely scoped out the auto-instrumentation aspects. However, if there's a good reason / value / need to provide certain level of enabling auto-instrumentation in terms of running operators (e.g. python operators), that may be a good discussion to start with.. So, generally, the Airflow community welcomes contributions of any types as long as those contributions have enough support (things are voted and approved), and the discussion has been made enough. If something sounds like a good idea (or you have found something) - please share with the community and then it can happen as an implementation work! Regards, Howard On Wed, Sep 25, 2024 at 8:33 AM mxmrlt ***@***.***> wrote: > Glad to read it. > > I see you're the creator of airflow_otel_provider. Do you know if there is > a way to auto-instrument our dags so that spans are created and exported > for any custom code like requests calls or kafka publishing etc...? > OpenTelemetry should permit that if I'm right ( > https://opentelemetry.io/docs/zero-code/python/) but perhaps I'm not. > > This would save us from having to declare manually what you show in the > README : > > RequestsInstrumentor().instrument(tracer_provider=otel_hook.tracer_provider) > > If you have any other advice on this topic please tell me. > > Thank you > > — > Reply to this email directly, view it on GitHub > <https://github.com/apache/airflow/issues/42424#issuecomment-2374103765>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AHZNLLQJY55TV6IDKRJRMD3ZYK3UJAVCNFSM6AAAAABOW5EZVOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNZUGEYDGNZWGU> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> > -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
