Lazy consensus passed. I've opened a PR for the changes https://github.com/apache/airflow/pull/63932.
If anyone is interested, please take a look. Currently, it's a draft because I still have to work out some details related to providers back compat. Christos On Wed, Mar 18, 2026 at 7:31 PM Christos Bisias <[email protected]> wrote: > The behavior should stay the same and we can document how people can > create custom metrics, just like with traces > > > https://github.com/apache/airflow/blob/main/airflow-core/docs/administration-and-deployment/logging-monitoring/traces.rst#adding-custom-spans-in-tasks > > With moving `Stats` under `_shared` and then the addition of the > `DualStatsManager` for handling the legacy metrics, things got too > complicated. I think that the best approach is to remove the > `DualStatsManager` entirely and move the logic under `Stats` but with the > metaclass, it will be tricky and eventually hard to maintain. Things need > to get simplified to accomodate for all the new changes. But the behavior > should stay the same. > > On Wed, Mar 18, 2026 at 7:27 PM Jarek Potiuk <[email protected]> wrote: > >> Also - from me - no `+1` is needed, but I strongly support this direction >> with strong arguments. >> >> I personally introduced the "Public Interface of Airflow" (I had to >> convince many maintainers that it was necessary). In airflow 2.6.0: >> >> https://airflow.apache.org/docs/apache-airflow/2.6.0/public-airflow-interface.html >> . >> The discussion about introducing it (November 2022) is here: >> https://lists.apache.org/thread/1by8ko8jrrp1xwxt5781bwn2tokxjodl (very >> long, but you can take a look what were the arguments and reasoning >> there). >> >> I started the discussion by explaining the basis of Hyrum's Law and SemVer >> there. And explained that in a sufficiently complex system, everything is >> breaking someone's workflow (Hyrum's law). Also, when you look at SemVer - >> this is an intention (even if intentions are not well documented), not a >> fact-based approach. >> >> I think what really matters are our intentions—whether something is a >> public interface—not whether technically we are removing class or method >> that has no "_" in front. While "_" is a clear sign that something is not >> intended to be public, lack of such underscore does not automatically mean >> something is public. Additionally - as you can read in that discussion - >> when evaluating whether a change is breaking or not we should assess the >> risk whether a change will break someone's workflow or not. >> >> Adding the page was a very deliberate move on my part. This was exactly >> what I did to find it useful today. All 3.5 years of having and updating >> the page, were **precisely** for this LAZY CONSENSUS thread (and similar >> that follow). >> >> And I plan to conduct a detailed review and update here at the devlist >> before the 3.2 release. One reason for introducing that page was to allow >> us, several years later, to see whether we communicated our intention to >> make something public or not. This basically follows this FAQ from SemVer >> https://semver.org/#documenting-the-entire-public-api-is-too-much-work >> >> > Documenting the entire public API is too much work! >> > >> > It is your responsibility as a professional developer to properly >> document software that is intended for use by others. Managing software >> complexity is a hugely important part of keeping a project efficient, and >> that’s hard to do if nobody knows how to use your software, > or what >> methods are safe to call. In the long run, Semantic Versioning, and the >> insistence on a well defined public API can keep everyone and everything >> running smoothly. >> > >> >> It turns out it's not "too much work." We did it. We communicated during >> the last 3.5 years. Our users had a chance to adapt and learn from our >> communications. If they didn't, this is entirely their problem. >> >> We did *NOT* communicate our intention to make Stats public. You can take >> a >> look at that page and it's history. I looked at all "public interface" >> versions, and Stats are not mentioned. So we are clear to remove it >> according to SemVer and it's NOT a breaking change according to SemVer. >> >> The risk of removing it is very small - especially if we explain what to >> do >> if someone used them in release notes as a "significant change" >> description. And we are planning to as far as I know. So risk is not an >> issue IMHO. >> >> I am 100% for removal. >> >> J. >> >> >> >> On Wed, Mar 18, 2026 at 5:39 PM Ferruzzi, Dennis <[email protected]> >> wrote: >> >> > I know we don't have to plus-one a lazy consensus, but maybe I can >> provide >> > a little context since I did the bulk of the work on the OTel metrics >> and >> > creating that particular mess. >> > >> > FWIW, I've definitely had people ask if it's possible to ad custom >> metrics >> > in their Dags. If you had asked me ten minutes ago if the process to >> do it >> > was in our official docs, I'd have said yes... but I just looked and >> > couldn't find it. So I guess I lean toward saying it isn't documented, >> > therefore it isn't public interface. >> > >> > I would support this with the caveat that it's still possible for the >> user >> > to add custom metrics to their Dags should they choose to do so. >> > ________________________________ >> > From: Daniel Standish via dev <[email protected]> >> > Sent: Wednesday, March 18, 2026 7:05 AM >> > To: [email protected] <[email protected]> >> > Cc: Daniel Standish <[email protected]> >> > Subject: RE: [EXT] [LAZY CONSENSUS] Remove the airflow.metrics and >> > airflow.stats modules >> > >> > CAUTION: This email originated from outside of the organization. Do not >> > click links or open attachments unless you can confirm the sender and >> know >> > the content is safe. >> > >> > >> > >> > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur >> externe. >> > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne >> pouvez >> > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain >> que >> > le contenu ne présente aucun risque. >> > >> > >> > >> > To try to emphasize some points: >> > >> > * airflow.stats and airflow.metrics have been in there since 2.7 >> > * as of now they are removed in main, and these removals will be >> reflected >> > in 3.2 >> > * this is ok if we agree that these were not user-public interface (i.e. >> > people are not using them in their dags) >> > * a (silent) vote of +1 in this lazy consensus means, yes we can remove >> (do >> > nothing) >> > * a vote of -1 means, we can't just remove we have to keep the >> interfaces >> > around where they were, so we have to add backcompat shims >> > >> > This is the same vote we had for traces (added 3.0), but now for >> > airflow.stats and airflow.metrics added (2.7) >> > >> > >> > On Wed, Mar 18, 2026 at 3:23 AM Christos Bisias <[email protected]> >> > wrote: >> > >> > > Hello everyone, >> > > >> > > For accessing metrics and traces, there were metaclass interfaces, >> called >> > > `Stats` and `Trace` respectively. Each interface was reading the >> config >> > > and then initializing the instance of the appropriate implementation. >> > Each >> > > metaclass was acting like a singleton cache and in all consecutive >> calls >> > > after the first one, it would always access the same existing >> instance. >> > > >> > > Daniel Standish has been leading an effort to simplify the OTel traces >> > > code. He was able to remove the `Trace` interface and add a single >> > > configuration when initializing all the airflow settings in core. >> > > >> > > Traces were simpler because they have a single supported >> implementation >> > > while metrics have multiple. But after discussions, we think that we >> can >> > > follow a similar approach and remove the stats interface. The reason >> is >> > > that there is too much complexity that increases with every new >> addition >> > or >> > > refactoring. It's going to be easier to maintain if simplified. >> > > >> > > The question is whether `airflow.metrics` and `airflow.stats` are >> public >> > > interfaces. They have been around since airflow 2.7 but as far as I >> know, >> > > no one from the community has talked about adding user-defined metrics >> > from >> > > under dags. Additionally, there are no docs saying how users can do >> it. >> > The >> > > only public info is for enabling the project's internal metrics. >> > > >> > > I'm considering these modules as private and I'm assuming that users >> > aren't >> > > aware of them or using them. I think they are safe to remove. >> > > >> > > I would like to call for a lazy consensus vote which will run for 72 >> > hours. >> > > If no one votes -1 during the voting period, the vote will be >> considered >> > > passed. >> > > >> > > Thank you! >> > > >> > > Regards, >> > > Christos >> > > >> > >> >
