Hi Kaxil, The OTEL provider would provide hooks for the developers to be able to create spans. Also, if the provider is running on top of Airflow that does not have the OTEL feature enabled/available, it will emit traces on the DAG run level, meaning task instance runs associated with the running DAG.
I guess the motivation for providing OTEL provider is to make the 'custom instrumentation' (like adding your own span, or adding your own span events, attributes, etc.) easier so that developers do not need to directly work with much of the intricacies on OTEL side, but simply use provider to do so. On Wed, Jul 24, 2024 at 2:51 PM Kaxil Naik <kaxiln...@gmail.com> wrote: > @howard: What sort of Operators or Hooks are you planning for the OTEL > provider? > > I am favour of deeper integration for OTEL and Airflow but I don't know > what Operators, Hooks or other things will be part of the provider. > > Regards, > Kaxil > > On Wed, 24 Jul 2024 at 01:00, Jarek Potiuk <ja...@potiuk.com> wrote: > > > +1. I like the idea of how it will add a possibility to customize OTEL > > metrics and spans possibly. With Airflow 2.10 I would also love to see > some > > guidelines and description and maybe some kind of simple How-TO on how > you > > can make "more" use of OTEL - for example users could use > > auto-instrumentation for sqlalchemy, flask and other libraries we are > > using, monitoring memory, cpu, processes etc. (this is all out-of-the-box > > available in OTEL) - and if such documentation describing a number of > > options and what the users can do about it would be great - and provider > > seems to be a good place maybe even to have some ways to enable those > > things more easily. > > > > Maybe just loosely related - but one thing that I am particularly looking > > forward to - is the ability for our users to be able to make a snapshot > of > > a problem they see (with traces) and send it to us. I know Jaegger has > such > > an option, and I saw what you can do, especially if you capture a lot of > > information, this would be the way we always tell all our users "But I > have > > no way to inspect your system - so I can't tell you what is wrong" - > having > > such a snapshot that you can load locally especially with a lot of auto > > instrumentation enabled might be fantastic way to help our users - but in > > order to do that - we need to give them some easy-to-follow-instructions. > > > > If that would be part of the work then I am even +10 on that. > > > > J. > > > > On Tue, Jul 23, 2024 at 2:16 AM Howard Yoo <howard...@gmail.com> wrote: > > > > > Hi Apache Airflow Community, > > > > > > I hope this message finds you well. > > > > > > I am writing to propose the addition of a new provider to Apache > Airflow > > > for OpenTelemetry (https://opentelemetry.io). OpenTelemetry is an > > emerging > > > standard for instrumentation of services and applications, and recently > > has > > > matured to gain huge popularity. > > > > > > Recently, there has been AIP (Airflow Improvement Proposal) no. 49 to > > > implement OpenTelemetry support for Apache Airflow, which will enable > > > Airflow to be able to emit metrics, traces, and task logs in > > OpenTelemetry > > > (PRs: https://github.com/apache/airflow/pull/37948, > > > https://github.com/apache/airflow/pull/40802) > > > > > > Since this feature is to be released soon to the future Airflow, having > > > this provider will further allow users to have more means to instrument > > > their DAGs. This OTEL provider can work independently from Airflow's > OTEL > > > implementation, as well as in conjunction if the feature is available > and > > > enabled. Any DAGs instrumented with OTEL provider will work with > Airflow > > > versions that may not have OTEL support, but also seamlessly with > Airflow > > > that supports OTEL, providing OTEL for everybody. > > > > > > I am willing to contribute to the development and integration effort to > > > ensure a smooth and effective implementation. Please let me know if > there > > > are any specific guidelines or processes that I should follow to > initiate > > > this proposal. > > > > > > Thanks and regards, > > > Howard Yoo > > > > > >
