As part of AIP-49, tasks logs are actually included, as span events. It can be turned on/off but when it's enabled, the content of the task log will be included as span events and emitted. Since task logs can be quite a large size, depending on the OTEL backend, the log contents can be truncated as necessary.
OTEL instrumented DAG would create a trace flow of DAG run, based on when and what task instances got invoked and executed, so much similar to the flow chart of Airflow, you can monitor the DAG's execution via any OpenTelemetry compatible backend. Aside from the DAG run, there will be a span-link that would connect a particular task instance run with the instance (or loop) of scheduler, such that you would also know the relationship between what was happening on the scheduler's side and the DAG run itself - connected together. Instrumentation would also allow users to attach or make custom attributes or spans as part of this DAG run graph (which can be achieved easily using OTEL provider), such that not only they can see the OOTB dag run graph, but have their own spans and instrumentation be part of it, to make the monitoring more enriched. Utilizing these data, it is hoped that with larger Airflow environment, it may be easier to monitor any failures, or incidents, or any performance issues much better since the trace, metrics, and logs, will be able to be collected into the single place, and easily correlated. On Thu, Jul 25, 2024 at 5:45 PM Vikram Koka <vik...@astronomer.io.invalid> wrote: > Howard, > > I am intrigued by this, but unclear on what this would actually look like > and what benefits it would add. > > Specifically, I believe that AIP-49 adds support for OTEL emission of > metrics and traces, but NOT task logs from Airflow. > > I am probably being dense here, but I don't quite understand what the OTEL > instrumented DAG would look like, and how/when the instrumentation would be > utilized. Can you please elaborate on this? > > Best regards, > Vikram > > > On Wed, Jul 24, 2024 at 12:51 PM Kaxil Naik <kaxiln...@gmail.com> wrote: > > > @howard: What sort of Operators or Hooks are you planning for the OTEL > > provider? > > > > I am favour of deeper integration for OTEL and Airflow but I don't know > > what Operators, Hooks or other things will be part of the provider. > > > > Regards, > > Kaxil > > > > On Wed, 24 Jul 2024 at 01:00, Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > +1. I like the idea of how it will add a possibility to customize OTEL > > > metrics and spans possibly. With Airflow 2.10 I would also love to see > > some > > > guidelines and description and maybe some kind of simple How-TO on how > > you > > > can make "more" use of OTEL - for example users could use > > > auto-instrumentation for sqlalchemy, flask and other libraries we are > > > using, monitoring memory, cpu, processes etc. (this is all > out-of-the-box > > > available in OTEL) - and if such documentation describing a number of > > > options and what the users can do about it would be great - and > provider > > > seems to be a good place maybe even to have some ways to enable those > > > things more easily. > > > > > > Maybe just loosely related - but one thing that I am particularly > looking > > > forward to - is the ability for our users to be able to make a snapshot > > of > > > a problem they see (with traces) and send it to us. I know Jaegger has > > such > > > an option, and I saw what you can do, especially if you capture a lot > of > > > information, this would be the way we always tell all our users "But I > > have > > > no way to inspect your system - so I can't tell you what is wrong" - > > having > > > such a snapshot that you can load locally especially with a lot of auto > > > instrumentation enabled might be fantastic way to help our users - but > in > > > order to do that - we need to give them some > easy-to-follow-instructions. > > > > > > If that would be part of the work then I am even +10 on that. > > > > > > J. > > > > > > On Tue, Jul 23, 2024 at 2:16 AM Howard Yoo <howard...@gmail.com> > wrote: > > > > > > > Hi Apache Airflow Community, > > > > > > > > I hope this message finds you well. > > > > > > > > I am writing to propose the addition of a new provider to Apache > > Airflow > > > > for OpenTelemetry (https://opentelemetry.io). OpenTelemetry is an > > > emerging > > > > standard for instrumentation of services and applications, and > recently > > > has > > > > matured to gain huge popularity. > > > > > > > > Recently, there has been AIP (Airflow Improvement Proposal) no. 49 to > > > > implement OpenTelemetry support for Apache Airflow, which will enable > > > > Airflow to be able to emit metrics, traces, and task logs in > > > OpenTelemetry > > > > (PRs: https://github.com/apache/airflow/pull/37948, > > > > https://github.com/apache/airflow/pull/40802) > > > > > > > > Since this feature is to be released soon to the future Airflow, > having > > > > this provider will further allow users to have more means to > instrument > > > > their DAGs. This OTEL provider can work independently from Airflow's > > OTEL > > > > implementation, as well as in conjunction if the feature is available > > and > > > > enabled. Any DAGs instrumented with OTEL provider will work with > > Airflow > > > > versions that may not have OTEL support, but also seamlessly with > > Airflow > > > > that supports OTEL, providing OTEL for everybody. > > > > > > > > I am willing to contribute to the development and integration effort > to > > > > ensure a smooth and effective implementation. Please let me know if > > there > > > > are any specific guidelines or processes that I should follow to > > initiate > > > > this proposal. > > > > > > > > Thanks and regards, > > > > Howard Yoo > > > > > > > > > >
