+1. I like the idea of how it will add a possibility to customize OTEL metrics and spans possibly. With Airflow 2.10 I would also love to see some guidelines and description and maybe some kind of simple How-TO on how you can make "more" use of OTEL - for example users could use auto-instrumentation for sqlalchemy, flask and other libraries we are using, monitoring memory, cpu, processes etc. (this is all out-of-the-box available in OTEL) - and if such documentation describing a number of options and what the users can do about it would be great - and provider seems to be a good place maybe even to have some ways to enable those things more easily.
Maybe just loosely related - but one thing that I am particularly looking forward to - is the ability for our users to be able to make a snapshot of a problem they see (with traces) and send it to us. I know Jaegger has such an option, and I saw what you can do, especially if you capture a lot of information, this would be the way we always tell all our users "But I have no way to inspect your system - so I can't tell you what is wrong" - having such a snapshot that you can load locally especially with a lot of auto instrumentation enabled might be fantastic way to help our users - but in order to do that - we need to give them some easy-to-follow-instructions. If that would be part of the work then I am even +10 on that. J. On Tue, Jul 23, 2024 at 2:16 AM Howard Yoo <howard...@gmail.com> wrote: > Hi Apache Airflow Community, > > I hope this message finds you well. > > I am writing to propose the addition of a new provider to Apache Airflow > for OpenTelemetry (https://opentelemetry.io). OpenTelemetry is an emerging > standard for instrumentation of services and applications, and recently has > matured to gain huge popularity. > > Recently, there has been AIP (Airflow Improvement Proposal) no. 49 to > implement OpenTelemetry support for Apache Airflow, which will enable > Airflow to be able to emit metrics, traces, and task logs in OpenTelemetry > (PRs: https://github.com/apache/airflow/pull/37948, > https://github.com/apache/airflow/pull/40802) > > Since this feature is to be released soon to the future Airflow, having > this provider will further allow users to have more means to instrument > their DAGs. This OTEL provider can work independently from Airflow's OTEL > implementation, as well as in conjunction if the feature is available and > enabled. Any DAGs instrumented with OTEL provider will work with Airflow > versions that may not have OTEL support, but also seamlessly with Airflow > that supports OTEL, providing OTEL for everybody. > > I am willing to contribute to the development and integration effort to > ensure a smooth and effective implementation. Please let me know if there > are any specific guidelines or processes that I should follow to initiate > this proposal. > > Thanks and regards, > Howard Yoo >
