Hey, absolutely! Given the needed focus and attention on Airflow 3.0, I am totally okay with revisiting this topic in May or June! I was just wondering if this was something that we want to eventually do, when the Airflow enters into the world of 3.0.
Regards, Howard On Tue, Apr 15, 2025 at 12:14 PM Ferruzzi, Dennis <ferru...@amazon.com.invalid> wrote: > +1 to what Ash, Jarek, Kaxil, and Vikram said above. > > > - ferruzzi > > > ________________________________ > From: Vikram Koka <vik...@astronomer.io.INVALID> > Sent: Tuesday, April 15, 2025 6:57 AM > To: dev@airflow.apache.org > Subject: RE: [EXT] [DISCUSSION] Opentelemetry support for LOGS in Apache > Airflow > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que > le contenu ne présente aucun risque. > > > > +1 to what Ash, Jarek, and Kaxil said above. > > Let's chat in late May or June on this topic. > > On Tue, Apr 15, 2025 at 7:05 AM Kaxil Naik <kaxiln...@gmail.com> wrote: > > > Yeah, no strength or bandwidth right now for a discussion. And would very > > much prefer we focus on getting 3.0.0 out and any bug fixes after that > > > > On Tue, 15 Apr 2025 at 14:08, Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > Very much agree with Ash here. > > > > > > On Tue, Apr 15, 2025 at 10:30 AM Ash Berlin-Taylor <a...@apache.org> > > wrote: > > > > > > > Hi Howard, > > > > > > > > Your timing is…. Interesting given we are focusing on getting 3.0 > > shipped > > > > and are in the final throws of that. > > > > > > > > Let's revisit this in May or June once the dust has settled from 3.0, > > any > > > > immediate bug fixes we might need, and give people a little bit of > time > > > to > > > > unwind after the crunch. > > > > > > > > -ash > > > > > > > > > On 14 Apr 2025, at 16:22, Howard Yoo <howard...@gmail.com> wrote: > > > > > > > > > > Hi Airflow community members, > > > > > > > > > > Apache Airflow has been moving steadily from conventional statsd > > based > > > > > telemetry into OpenTelemetry (opentelemetry.io), which has gained > > > > > popularity and support for the last few years. There has been AIP > > > > proposals > > > > > which had a 'phased' approach to incorporate > > > > > > > > > > 1. Metrics > > > > > 2. Traces > > > > > 3. Logs > > > > > > > > > > Based on the current availability and maturity of Otel's support. > > Since > > > > the > > > > > metrics emission based on statsd were the fastest low-hanging fruit > > > that > > > > we > > > > > could implement OTEL metrics support relatively easy, which we did > as > > > our > > > > > first try. > > > > > > > > > > Second was the Traces. OTEL support on traces for Python was very > > > mature > > > > > and well established. Even though Airflow did not have any prior > > > > > implementation on emitting Traces, we were able to instrument it > and > > > > > produce traces on both Airflow specific processes (jobs) (e.g. > > > > schedulers, > > > > > triggers, executors), as well as task runs. > > > > > > > > > > However, the area for Logs were not done yet, due at that time a > > > > relatively > > > > > early and immature state of logs support in Otel. > > > > > > > > > > Now, it seems like Logging support on Python has become stable and > > > > reliable > > > > > that we can use it to emit log messages in Otel format. So, the > > timing > > > > > seems to be quite perfect for us to start up a discussion on it. > > > > > > > > > > Logging support in Airflow is basically provided via 'provider' > model > > > > where > > > > > Airflow itself does not have a specific implementation of how > logging > > > is > > > > > done, but requests contributors to provide provider that can > plug-in > > > > their > > > > > logging support. > > > > > > > > > > Would we want to move to that direction, of having 'opentelemetry' > > > > provider > > > > > ? > > > > > Or do we want to have logging support as more of a native support > > way? > > > > > > > > > > The reason why I would like to start this discussion thread was to > > gain > > > > > opinions from the community on whether Airflow has the interest to > > > > support > > > > > OTEL logging first, and then any opinions on specifics of what > would > > be > > > > the > > > > > best way to support OTEL logging. > > > > > > > > > > Please feel free to reply and add any comments to the thread to > voice > > > > your > > > > > opinions. If there isn't any substantial interest in OTEL logging, > > > well, > > > > we > > > > > can live without it. > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > > > > > > >
