Howard, I am intrigued by this, but unclear on what this would actually look like and what benefits it would add.
Specifically, I believe that AIP-49 adds support for OTEL emission of metrics and traces, but NOT task logs from Airflow. I am probably being dense here, but I don't quite understand what the OTEL instrumented DAG would look like, and how/when the instrumentation would be utilized. Can you please elaborate on this? Best regards, Vikram On Wed, Jul 24, 2024 at 12:51 PM Kaxil Naik <kaxiln...@gmail.com> wrote: > @howard: What sort of Operators or Hooks are you planning for the OTEL > provider? > > I am favour of deeper integration for OTEL and Airflow but I don't know > what Operators, Hooks or other things will be part of the provider. > > Regards, > Kaxil > > On Wed, 24 Jul 2024 at 01:00, Jarek Potiuk <ja...@potiuk.com> wrote: > > > +1. I like the idea of how it will add a possibility to customize OTEL > > metrics and spans possibly. With Airflow 2.10 I would also love to see > some > > guidelines and description and maybe some kind of simple How-TO on how > you > > can make "more" use of OTEL - for example users could use > > auto-instrumentation for sqlalchemy, flask and other libraries we are > > using, monitoring memory, cpu, processes etc. (this is all out-of-the-box > > available in OTEL) - and if such documentation describing a number of > > options and what the users can do about it would be great - and provider > > seems to be a good place maybe even to have some ways to enable those > > things more easily. > > > > Maybe just loosely related - but one thing that I am particularly looking > > forward to - is the ability for our users to be able to make a snapshot > of > > a problem they see (with traces) and send it to us. I know Jaegger has > such > > an option, and I saw what you can do, especially if you capture a lot of > > information, this would be the way we always tell all our users "But I > have > > no way to inspect your system - so I can't tell you what is wrong" - > having > > such a snapshot that you can load locally especially with a lot of auto > > instrumentation enabled might be fantastic way to help our users - but in > > order to do that - we need to give them some easy-to-follow-instructions. > > > > If that would be part of the work then I am even +10 on that. > > > > J. > > > > On Tue, Jul 23, 2024 at 2:16 AM Howard Yoo <howard...@gmail.com> wrote: > > > > > Hi Apache Airflow Community, > > > > > > I hope this message finds you well. > > > > > > I am writing to propose the addition of a new provider to Apache > Airflow > > > for OpenTelemetry (https://opentelemetry.io). OpenTelemetry is an > > emerging > > > standard for instrumentation of services and applications, and recently > > has > > > matured to gain huge popularity. > > > > > > Recently, there has been AIP (Airflow Improvement Proposal) no. 49 to > > > implement OpenTelemetry support for Apache Airflow, which will enable > > > Airflow to be able to emit metrics, traces, and task logs in > > OpenTelemetry > > > (PRs: https://github.com/apache/airflow/pull/37948, > > > https://github.com/apache/airflow/pull/40802) > > > > > > Since this feature is to be released soon to the future Airflow, having > > > this provider will further allow users to have more means to instrument > > > their DAGs. This OTEL provider can work independently from Airflow's > OTEL > > > implementation, as well as in conjunction if the feature is available > and > > > enabled. Any DAGs instrumented with OTEL provider will work with > Airflow > > > versions that may not have OTEL support, but also seamlessly with > Airflow > > > that supports OTEL, providing OTEL for everybody. > > > > > > I am willing to contribute to the development and integration effort to > > > ensure a smooth and effective implementation. Please let me know if > there > > > are any specific guidelines or processes that I should follow to > initiate > > > this proposal. > > > > > > Thanks and regards, > > > Howard Yoo > > > > > >
