To Jarek's comment - I understood that the next release of the Helm
Chart (1.16) would include support for multiple executors. I don't know
to which level and it might be "extendable" but actually we deploy with
the old version and just some "ENV to hack the missing option". I also
hope it will get first-class-citizen, maybe all efforts towards Airflow
3 are dragging focus off that gap.
To Niko: I was not aware that there are special cases for hybrid
executors in the core. I'd really propose to clean them up for Airflow
3. Because there is an alternative. And I assume the gap in Helm chart
is almost closed, just missing a release. Else complexity is low to
close the gap - which I think is less a problem that moving on with
workarounds in core with the major release coming.
On 15.02.25 00:16, Oliveira, Niko wrote:
Thanks for the feedback Jens, glad to hear you're using multiple executors
smoothly :)
In regards of Deprecation: All hybrid executors are in cncf-k8s provider
package. We can deprecate independent of Airflow 3 and clear the documentation.
I'm less interested in marking them deprecated in their respective provider
package, what I really want to deprecated is support for them in Airflow core
(i.e. remove all the special case handling we have for them). So essentially
the executor loader would warn if it was instructed to load one of those
executors during the deprecation.
we could build a "balcony" == check into these executors that they are raising
an Exception when somebody attempts to
configure/start them in Airflow 3 (and leave them supported until support for
2.x ends in the provider package)
Yes, I agree we should continue to support these executors in Airflow 2 for as
long as we support 2.X versions for that provider.
For me, it's -0. Yes. Likely some people used it, but we made it terribly
difficult for our Helm chart users to use it. We have no support for
multi-executors in the Helm Chart we released. I would only be for making
it stable if we release Helm Chart with multi-executors as a first-class
citizen. Even a doc on how to use it with the current Helm chart would be
enough.
Coupling a feature being stable to it's ability to be deployed in one specific
circumstance seems like the wrong gating to me. The feature itself is stable,
it's interface is unlikely to change, and it will remain supported. I think
that's the usual measurements for stability. I'm not saying we shouldn't do
some work to support it in Helm, for those that use Helm, but I'm not sure that
affects the stability of the multi-exec feature.
I personally do not use Helm or k8s based deployments in my work or projects,
so I wont be terribly much help here. But I'm more than willing to engage and
help disambiguate things from the executor perspective.
Cheers,
Niko
________________________________
From: Jarek Potiuk <ja...@potiuk.com>
Sent: Friday, February 14, 2025 12:35:10 PM
To: dev@airflow.apache.org
Subject: RE: [EXT] [DISCUSS] Mark Multi Exec Config as stable and remove old
hardcoded hybrid execs
CAUTION: This email originated from outside of the organization. Do not click
links or open attachments unless you can confirm the sender and know the
content is safe.
AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne
cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas
confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le
contenu ne présente aucun risque.
For me, it's -0. Yes. Likely some people used it, but we made it terribly
difficult for our Helm chart users to use it. We have no support for
multi-executors in the Helm Chart we released. I would only be for making
it stable if we release Helm Chart with multi-executors as a first-class
citizen. Even a doc on how to use it with the current Helm chart would be
enough.
On Fri, Feb 14, 2025 at 9:15 PM Jens Scheffler <j_scheff...@gmx.de.invalid>
wrote:
+1 marking it stable - actually I am a bit surprised - we are using this
since months in production :-D So we can confirm it is rock solid, not
only stable.
I am not sure whether we really come to a 2.11...
In regards of Deprecation: All hybrid executors are in cncf-k8s provider
package. We can deprecate independent of Airflow 3 and clear the
documentation. Or also we could build a "balcony" == check into these
executors that they are raising an Exception when somebody attempts to
configure/start them in Airflow 3 (and leave them supported until
support for 2.x ends in the provider package)
On 14.02.25 19:17, Oliveira, Niko wrote:
Thanks for the reply and the support Constance!
Fair call out on moving multi-exec to stable in 2.11.0 release. We could
just deprecate the hardcoded hybrid executors and leave it at that for
2.11, however to me it feels like there should be something the community
sees that is stable for them to migrate to if we're telling them to move
away from the old hybrid executors.
Calling it stable just means the public interface won't change in a
breaking way and that they can depend on this feature existing into the
future instead of being removed at a short notice. We can still collect
usage data and fix issues if we find them like we do any other stable
feature in Airflow.
Thanks again for weighing in, and I'm curious to hear what others think
as well!
Cheers,
Niko
________________________________
From: Constance Martineau <consta...@astronomer.io.INVALID>
Sent: Friday, February 14, 2025 9:27:47 AM
To: dev@airflow.apache.org
Subject: RE: [EXT] [DISCUSS] Mark Multi Exec Config as stable and remove
old hardcoded hybrid execs
CAUTION: This email originated from outside of the organization. Do not
click links or open attachments unless you can confirm the sender and know
the content is safe.
AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous
ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
certain que le contenu ne présente aucun risque.
Hey Niko,
Thanks for kicking off this discussion!
I’m fully on board with removing support for hardcoded hybrid executors
in
favor of multi-executor support for 3.0—it’s a great step forward in
simplifying execution models. Adding deprecation warnings in 2.11 also
makes sense to give users a smooth transition period.
Regarding marking multi-executor support stable in 2.11: The only thing
I’d
be mindful of is ensuring we get enough real-world testing across
different
multi-executor combinations before marking it fully stable. We haven’t
seen
a ton of feedback on that front yet, and I’d rather avoid unexpected
surprises post-release.
Also, since we’re focused on wrapping up 3.0, I’d prefer to keep changes
in
2.11 minimal where possible. Deprecation warnings are a great addition,
but
I’d want to avoid creating any expectations that we’ll be patching 2.11
specifically for this feature now that it’s considered stable.
Curious to hear what others think!
Cheers,
Constance
On Thu, Feb 13, 2025 at 8:16 PM Oliveira, Niko
<oniko...@amazon.com.invalid>
wrote:
Hello folks!
Multiple Executor Configuration (aka hybrid executors, AIP-61) has been
out a while now and we haven't had many issues with it. I'm proposing we
mark it as stable.
Relatedly I would also like to discuss removing support for the older
hardcoded hybrid executors (e.g. CeleryKubernetesExecutor, etc) in
Airflow
3. We still have many lines of code in core Airflow that are coupled
directly to these executors since they need special handling. They also
don't track changes to the base executor very well and so they tend to
break every time that interface changes [2]. I figure Airflow 3.0 might
be
the best opportunity we'll have in a while to remove these executors.
We of course don't have to remove them from the provider packages they
live in, so new versions of providers will still be compatible with pre
3.0
Airflow to use those executors.
I propose that we mark Multiple Executor Configuration stable in 2.11
and
clearly mark the old hardcoded hybrid executors as deprecated also in
2.11.
This gives time for folks to migrate if they'd like. Then in 3.0 remove
all
our special handling and testing from core Airflow relating to these
hardcoded hybrid executors.
What do y'all think?
Cheers,
Niko
[1] - https://github.com/apache/airflow/pull/46742
[2] - https://github.com/apache/airflow/pull/41602
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org
