> To Niko: I was not aware that there are special cases for hybrid executors in > the core. I'd really propose to clean them up for Airflow 3. Because there is > an alternative. And I assume the gap in Helm chart is almost closed, just > missing a release.
Thanks Jens, glad to have raised awareness! I'll put together the PR that removes the special handling and we can decide when it is okay to have it merged. > In an ideal world, Niko - I would not want you to implement anything in the chart, but I would like, YOU when advocating for making a feature like that stable to say: > It's very easy to say "it's not my issue" (or not my garden) -> but I think in our case - everything we release as community is "ours" and even if it's not necessarily "my garden" - reaching out to community to fix the part of garden that we see as a little abandoned, when we see it impacts "our garden" is what I am advocating for. Thanks for the point of view Jarek :) I'm by no means saying that "it's not my issue" or that we shouldn't cultivate our garden and I'm happy to advocate for work being done in the community or for changes to be made (that's why I started this thread in the first place!). So my apologies if I gave that impression. We own the Helm Chart and it deserves it's due love and attention, agreed. I just simply disagree that a feature is "unstable" unless Helm Chart supports it. To me the stability of a feature is different from whether or not it can be used natively in the Helm Chart. But, I also see your point of view and I'm happy to commit to that one in this case; and I'll see how I can help that work along or get a release of the Helm Chart cut :) Cheers, Niko ________________________________ From: Jarek Potiuk <ja...@potiuk.com> Sent: Saturday, February 15, 2025 5:27:01 AM To: dev@airflow.apache.org Subject: RE: [EXT] [DISCUSS] Mark Multi Exec Config as stable and remove old hardcoded hybrid execs CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque. > Coupling a feature being stable to it's ability to be deployed in one specific circumstance seems like the wrong gating to me As Jens mentioned - yeah, looks like multi-executors are coming in the new release of Helm chart, and that would change my `-0` to '+1'. And it's not really "gatiing". It's really all about what we answer to our users who want to use the 'stable' feature of ours but they can't because they use the Helm Chart of ours. No matter if we like or use Helm Chat - and it was not really requested of you to add the feature :). It was more a bit of a call to our "community" effort here on the chart. For me - when we decided to release Helm Chart as "community" effort after we adopted the Astronomer chart, we agreed (as community) to maintain it and release it with new features. It's a bit of ugly child of ours now., I think and we put very, very little effort into maintaining and releasing the chart (and again it's not a complaint to anyone in particular - it's more "we should do better as a community". Jed who usually releases it has more than full hands with Airflow 3 now, so it's understandable that we have not released a recent version of chart - and there are a number of opened or even merged but not released features there. I'd say for me this case with Multi Executors is not "gating" the stable status of a feature in Airflow but to raise awareness that Helm is somewhat lagging behind. In an ideal world, Niko - I would not want you to implement anything in the chart, but I would like, YOU when advocating for making a feature like that stable to say: "I would love to make Multi-Executor as stable, but I see that the Helm chart of ours does not support it yet, who from the community can help with making it happen so that we can make it "truly stable feature". It's very easy to say "it's not my issue" (or not my garden) -> but I think in our case - everything we release as community is "ours" and even if it's not necessarily "my garden" - reaching out to community to fix the part of garden that we see as a little abandoned, when we see it impacts "our garden" is what I am advocating for. J. On Sat, Feb 15, 2025 at 1:17 PM Jens Scheffler <j_scheff...@gmx.de.invalid> wrote: > To Jarek's comment - I understood that the next release of the Helm > Chart (1.16) would include support for multiple executors. I don't know > to which level and it might be "extendable" but actually we deploy with > the old version and just some "ENV to hack the missing option". I also > hope it will get first-class-citizen, maybe all efforts towards Airflow > 3 are dragging focus off that gap. > > To Niko: I was not aware that there are special cases for hybrid > executors in the core. I'd really propose to clean them up for Airflow > 3. Because there is an alternative. And I assume the gap in Helm chart > is almost closed, just missing a release. Else complexity is low to > close the gap - which I think is less a problem that moving on with > workarounds in core with the major release coming. > > On 15.02.25 00:16, Oliveira, Niko wrote: > > Thanks for the feedback Jens, glad to hear you're using multiple > executors smoothly :) > > > >> In regards of Deprecation: All hybrid executors are in cncf-k8s > provider package. We can deprecate independent of Airflow 3 and clear the > documentation. > > I'm less interested in marking them deprecated in their respective > provider package, what I really want to deprecated is support for them in > Airflow core (i.e. remove all the special case handling we have for them). > So essentially the executor loader would warn if it was instructed to load > one of those executors during the deprecation. > > > >> we could build a "balcony" == check into these executors that they are > raising an Exception when somebody attempts to > >> configure/start them in Airflow 3 (and leave them supported until > support for 2.x ends in the provider package) > > Yes, I agree we should continue to support these executors in Airflow 2 > for as long as we support 2.X versions for that provider. > > > >> For me, it's -0. Yes. Likely some people used it, but we made it > terribly > >> difficult for our Helm chart users to use it. We have no support for > >> multi-executors in the Helm Chart we released. I would only be for > making > >> it stable if we release Helm Chart with multi-executors as a first-class > >> citizen. Even a doc on how to use it with the current Helm chart would > be > >> enough. > > Coupling a feature being stable to it's ability to be deployed in one > specific circumstance seems like the wrong gating to me. The feature itself > is stable, it's interface is unlikely to change, and it will remain > supported. I think that's the usual measurements for stability. I'm not > saying we shouldn't do some work to support it in Helm, for those that use > Helm, but I'm not sure that affects the stability of the multi-exec feature. > > > > I personally do not use Helm or k8s based deployments in my work or > projects, so I wont be terribly much help here. But I'm more than willing > to engage and help disambiguate things from the executor perspective. > > > > Cheers, > > Niko > > > > ________________________________ > > From: Jarek Potiuk <ja...@potiuk.com> > > Sent: Friday, February 14, 2025 12:35:10 PM > > To: dev@airflow.apache.org > > Subject: RE: [EXT] [DISCUSS] Mark Multi Exec Config as stable and remove > old hardcoded hybrid execs > > > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous > ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > certain que le contenu ne présente aucun risque. > > > > > > > > For me, it's -0. Yes. Likely some people used it, but we made it terribly > > difficult for our Helm chart users to use it. We have no support for > > multi-executors in the Helm Chart we released. I would only be for making > > it stable if we release Helm Chart with multi-executors as a first-class > > citizen. Even a doc on how to use it with the current Helm chart would be > > enough. > > > > On Fri, Feb 14, 2025 at 9:15 PM Jens Scheffler > <j_scheff...@gmx.de.invalid> > > wrote: > > > >> +1 marking it stable - actually I am a bit surprised - we are using this > >> since months in production :-D So we can confirm it is rock solid, not > >> only stable. > >> > >> I am not sure whether we really come to a 2.11... > >> > >> In regards of Deprecation: All hybrid executors are in cncf-k8s provider > >> package. We can deprecate independent of Airflow 3 and clear the > >> documentation. Or also we could build a "balcony" == check into these > >> executors that they are raising an Exception when somebody attempts to > >> configure/start them in Airflow 3 (and leave them supported until > >> support for 2.x ends in the provider package) > >> > >> On 14.02.25 19:17, Oliveira, Niko wrote: > >>> Thanks for the reply and the support Constance! > >>> > >>> > >>> Fair call out on moving multi-exec to stable in 2.11.0 release. We > could > >> just deprecate the hardcoded hybrid executors and leave it at that for > >> 2.11, however to me it feels like there should be something the > community > >> sees that is stable for them to migrate to if we're telling them to move > >> away from the old hybrid executors. > >>> Calling it stable just means the public interface won't change in a > >> breaking way and that they can depend on this feature existing into the > >> future instead of being removed at a short notice. We can still collect > >> usage data and fix issues if we find them like we do any other stable > >> feature in Airflow. > >>> Thanks again for weighing in, and I'm curious to hear what others think > >> as well! > >>> Cheers, > >>> > >>> Niko > >>> > >>> > >>> > >>> ________________________________ > >>> From: Constance Martineau <consta...@astronomer.io.INVALID> > >>> Sent: Friday, February 14, 2025 9:27:47 AM > >>> To: dev@airflow.apache.org > >>> Subject: RE: [EXT] [DISCUSS] Mark Multi Exec Config as stable and > remove > >> old hardcoded hybrid execs > >>> CAUTION: This email originated from outside of the organization. Do not > >> click links or open attachments unless you can confirm the sender and > know > >> the content is safe. > >>> > >>> > >>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > >> externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si > vous > >> ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > >> certain que le contenu ne présente aucun risque. > >>> > >>> > >>> Hey Niko, > >>> > >>> Thanks for kicking off this discussion! > >>> > >>> I’m fully on board with removing support for hardcoded hybrid executors > >> in > >>> favor of multi-executor support for 3.0—it’s a great step forward in > >>> simplifying execution models. Adding deprecation warnings in 2.11 also > >>> makes sense to give users a smooth transition period. > >>> > >>> Regarding marking multi-executor support stable in 2.11: The only thing > >> I’d > >>> be mindful of is ensuring we get enough real-world testing across > >> different > >>> multi-executor combinations before marking it fully stable. We haven’t > >> seen > >>> a ton of feedback on that front yet, and I’d rather avoid unexpected > >>> surprises post-release. > >>> > >>> Also, since we’re focused on wrapping up 3.0, I’d prefer to keep > changes > >> in > >>> 2.11 minimal where possible. Deprecation warnings are a great addition, > >> but > >>> I’d want to avoid creating any expectations that we’ll be patching 2.11 > >>> specifically for this feature now that it’s considered stable. > >>> > >>> Curious to hear what others think! > >>> > >>> Cheers, > >>> Constance > >>> > >>> On Thu, Feb 13, 2025 at 8:16 PM Oliveira, Niko > >> <oniko...@amazon.com.invalid> > >>> wrote: > >>> > >>>> Hello folks! > >>>> > >>>> Multiple Executor Configuration (aka hybrid executors, AIP-61) has > been > >>>> out a while now and we haven't had many issues with it. I'm proposing > we > >>>> mark it as stable. > >>>> > >>>> Relatedly I would also like to discuss removing support for the older > >>>> hardcoded hybrid executors (e.g. CeleryKubernetesExecutor, etc) in > >> Airflow > >>>> 3. We still have many lines of code in core Airflow that are coupled > >>>> directly to these executors since they need special handling. They > also > >>>> don't track changes to the base executor very well and so they tend to > >>>> break every time that interface changes [2]. I figure Airflow 3.0 > might > >> be > >>>> the best opportunity we'll have in a while to remove these executors. > >>>> > >>>> We of course don't have to remove them from the provider packages they > >>>> live in, so new versions of providers will still be compatible with > pre > >> 3.0 > >>>> Airflow to use those executors. > >>>> > >>>> I propose that we mark Multiple Executor Configuration stable in 2.11 > >> and > >>>> clearly mark the old hardcoded hybrid executors as deprecated also in > >> 2.11. > >>>> This gives time for folks to migrate if they'd like. Then in 3.0 > remove > >> all > >>>> our special handling and testing from core Airflow relating to these > >>>> hardcoded hybrid executors. > >>>> > >>>> What do y'all think? > >>>> > >>>> Cheers, > >>>> Niko > >>>> > >>>> [1] - https://github.com/apache/airflow/pull/46742 > >>>> [2] - https://github.com/apache/airflow/pull/41602 > >>>> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > >> For additional commands, e-mail: dev-h...@airflow.apache.org > >> > >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
