Thanks for the reply and the support Constance!
Fair call out on moving multi-exec to stable in 2.11.0 release. We could just deprecate the hardcoded hybrid executors and leave it at that for 2.11, however to me it feels like there should be something the community sees that is stable for them to migrate to if we're telling them to move away from the old hybrid executors. Calling it stable just means the public interface won't change in a breaking way and that they can depend on this feature existing into the future instead of being removed at a short notice. We can still collect usage data and fix issues if we find them like we do any other stable feature in Airflow. Thanks again for weighing in, and I'm curious to hear what others think as well! Cheers, Niko ________________________________ From: Constance Martineau <consta...@astronomer.io.INVALID> Sent: Friday, February 14, 2025 9:27:47 AM To: dev@airflow.apache.org Subject: RE: [EXT] [DISCUSS] Mark Multi Exec Config as stable and remove old hardcoded hybrid execs CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque. Hey Niko, Thanks for kicking off this discussion! I’m fully on board with removing support for hardcoded hybrid executors in favor of multi-executor support for 3.0—it’s a great step forward in simplifying execution models. Adding deprecation warnings in 2.11 also makes sense to give users a smooth transition period. Regarding marking multi-executor support stable in 2.11: The only thing I’d be mindful of is ensuring we get enough real-world testing across different multi-executor combinations before marking it fully stable. We haven’t seen a ton of feedback on that front yet, and I’d rather avoid unexpected surprises post-release. Also, since we’re focused on wrapping up 3.0, I’d prefer to keep changes in 2.11 minimal where possible. Deprecation warnings are a great addition, but I’d want to avoid creating any expectations that we’ll be patching 2.11 specifically for this feature now that it’s considered stable. Curious to hear what others think! Cheers, Constance On Thu, Feb 13, 2025 at 8:16 PM Oliveira, Niko <oniko...@amazon.com.invalid> wrote: > Hello folks! > > Multiple Executor Configuration (aka hybrid executors, AIP-61) has been > out a while now and we haven't had many issues with it. I'm proposing we > mark it as stable. > > Relatedly I would also like to discuss removing support for the older > hardcoded hybrid executors (e.g. CeleryKubernetesExecutor, etc) in Airflow > 3. We still have many lines of code in core Airflow that are coupled > directly to these executors since they need special handling. They also > don't track changes to the base executor very well and so they tend to > break every time that interface changes [2]. I figure Airflow 3.0 might be > the best opportunity we'll have in a while to remove these executors. > > We of course don't have to remove them from the provider packages they > live in, so new versions of providers will still be compatible with pre 3.0 > Airflow to use those executors. > > I propose that we mark Multiple Executor Configuration stable in 2.11 and > clearly mark the old hardcoded hybrid executors as deprecated also in 2.11. > This gives time for folks to migrate if they'd like. Then in 3.0 remove all > our special handling and testing from core Airflow relating to these > hardcoded hybrid executors. > > What do y'all think? > > Cheers, > Niko > > [1] - https://github.com/apache/airflow/pull/46742 > [2] - https://github.com/apache/airflow/pull/41602 >
