+1 (binding) On Tue, Feb 21, 2023 at 8:33 PM Josh Fell <josh.d.f...@astronomer.io.invalid> wrote:
> +1 binding > > On Tue, Feb 21, 2023 at 11:21 AM Jarek Potiuk <ja...@potiuk.com> wrote: > >> And just to add as I missed that in the original mail - this is "code >> modification" vote - so all committers have a binding vote. >> https://www.apache.org/foundation/voting.html#votes-on-code-modification >> Also I have not mentioned the time: I think we can keep it open for 72 >> hours from now - which means that it will end on February 24th, 2023, >> 5PM CET. >> >> On Tue, Feb 21, 2023 at 4:26 PM Ash Berlin-Taylor <a...@apache.org> wrote: >> > >> > +1 binding >> > >> > On Feb 21 2023, at 9:21 am, Hussein Awala <huss...@awala.fr> wrote: >> > >> > >> > +1 non-binding.I'm a little concerned that this coupling will reduce >> the fast evolution of providers, but given the benefits on the executor >> side, I vote for it. >> > From: Jarek Potiuk <ja...@potiuk.com> >> > Sent: Tuesday, February 21, 2023 8:45:15 AM >> > To: dev@airflow.apache.org >> > Subject: [VOTE] Move K8S / Celery (and related) executors to respective >> providers >> > >> > Hello everyone, >> > >> > This is a call for the vote to make an internal change to move the code >> of K8S, Celery and related (LocalKubernetes., CeleryKubernetes etc. ) to >> respective providers. >> > >> > Consider it +1 (binding) from my side. >> > >> > This has been discussed in >> https://lists.apache.org/thread/kwwhz62lddygodpgr3fk4b9tthtld9do and let >> me summarize it below: >> > >> > # Why? >> > >> > Multiple reasons: >> > >> > * It will make it easier to manage consistency between K8S Pod Manager >> and K8S executor. In the past there were non-trivial dependencies between >> those that resulted in k8s provider being limited to latest airflow versions >> > * It's non-obvious that the code used in K8S executor uses two >> different artifacts (airflow and cncnf.k8s provider) and it limits our >> abilities to refactor/modify/improve this code as it has to work with >> various combinations of airflow + cncf.kubernete versions >> > * provider's releases (major/minor versions) have much faster release >> cycle and we can both - fix and provide new features to those executors >> > * users who have good reasons to not to upgrade to latest airflow >> releases will be able to use latest k8s/celery executors by updating >> providers only >> > * if there are regressions with executors in newer airflow versions, >> users will be able to downgrade providers - without downgrading the whole >> airflow (downgrading the DB etc.) >> > * this follows the philosophy of Airflow-as-a-platform, where anyone >> can extend Airflow by adding new plugins/providers and moving the executor >> to providers proves the point that anyone can do their own executor and >> that they will have the same capabilities as the ones that are built-in >> > >> > # Why now? >> > >> > We are in the process of finishing AIP-51 with executor decoupling and >> where we got rid of the hard-coded behaviour of Airflow depending on what >> executor was used. It was simply impossible before to move the executors to >> providers, because the hard-coded behaviours had to maintain the knowledge >> about which executor is used. Executor's API was incomplete and some >> behaviours of the executors were hard-coded. With AIP-51 completed executor >> implementation can simply rely on the complete executor's API - including >> exposing properties of the executor that can change airflow core behaviour >> appropriately by inspecting the properties. >> > >> > # Backwards compatibility >> > >> > I believe we will be able to make it fully backwards compatible with >> the usage of PEP 562 and deprecation notices (same as we did with contrib >> packages). Also we seem to be converging on the backwards-compatibility >> approach, specifically excluding the implementation of executors from our >> "Public API list" >> https://lists.apache.org/thread/d90b1yvsbwzy5flnd3vslfjs38x76kyj >> > >> > We will turn "cncf.kubernetes" and "celery" providers into >> "pre-installed" providers, which means that one will be able to use all the >> built-in executors with just "pip install airflow" (interestingly enough >> before that one had to install the k8s provider to make the K8s executor >> work even if they were part of the core which was sub-optimal). >> > >> > Also, resulting from the discussion we will keep documentation for >> available executors in Airflow (so they will still be considered as THE >> executors available and will be discoverable in the same way as today). >> > >> > # Potential problems >> > >> > Seems there are no known problems it can cause. There is the question >> "where to put CeleryKubernetesExecutor?" and the proposal is to put it in >> "cncf.k8s" and treat celery as an optional dependency ("celery" extra) of >> "cncf.k8s" provider. Since both providers will be pre-installed, this is >> not a problem or concern for any use case. >> > >> > J. >> >
