@howard: What sort of Operators or Hooks are you planning for the OTEL provider?
I am favour of deeper integration for OTEL and Airflow but I don't know what Operators, Hooks or other things will be part of the provider. Regards, Kaxil On Wed, 24 Jul 2024 at 01:00, Jarek Potiuk <ja...@potiuk.com> wrote: > +1. I like the idea of how it will add a possibility to customize OTEL > metrics and spans possibly. With Airflow 2.10 I would also love to see some > guidelines and description and maybe some kind of simple How-TO on how you > can make "more" use of OTEL - for example users could use > auto-instrumentation for sqlalchemy, flask and other libraries we are > using, monitoring memory, cpu, processes etc. (this is all out-of-the-box > available in OTEL) - and if such documentation describing a number of > options and what the users can do about it would be great - and provider > seems to be a good place maybe even to have some ways to enable those > things more easily. > > Maybe just loosely related - but one thing that I am particularly looking > forward to - is the ability for our users to be able to make a snapshot of > a problem they see (with traces) and send it to us. I know Jaegger has such > an option, and I saw what you can do, especially if you capture a lot of > information, this would be the way we always tell all our users "But I have > no way to inspect your system - so I can't tell you what is wrong" - having > such a snapshot that you can load locally especially with a lot of auto > instrumentation enabled might be fantastic way to help our users - but in > order to do that - we need to give them some easy-to-follow-instructions. > > If that would be part of the work then I am even +10 on that. > > J. > > On Tue, Jul 23, 2024 at 2:16 AM Howard Yoo <howard...@gmail.com> wrote: > > > Hi Apache Airflow Community, > > > > I hope this message finds you well. > > > > I am writing to propose the addition of a new provider to Apache Airflow > > for OpenTelemetry (https://opentelemetry.io). OpenTelemetry is an > emerging > > standard for instrumentation of services and applications, and recently > has > > matured to gain huge popularity. > > > > Recently, there has been AIP (Airflow Improvement Proposal) no. 49 to > > implement OpenTelemetry support for Apache Airflow, which will enable > > Airflow to be able to emit metrics, traces, and task logs in > OpenTelemetry > > (PRs: https://github.com/apache/airflow/pull/37948, > > https://github.com/apache/airflow/pull/40802) > > > > Since this feature is to be released soon to the future Airflow, having > > this provider will further allow users to have more means to instrument > > their DAGs. This OTEL provider can work independently from Airflow's OTEL > > implementation, as well as in conjunction if the feature is available and > > enabled. Any DAGs instrumented with OTEL provider will work with Airflow > > versions that may not have OTEL support, but also seamlessly with Airflow > > that supports OTEL, providing OTEL for everybody. > > > > I am willing to contribute to the development and integration effort to > > ensure a smooth and effective implementation. Please let me know if there > > are any specific guidelines or processes that I should follow to initiate > > this proposal. > > > > Thanks and regards, > > Howard Yoo > > >
