> Should we also create apache-airflow-providers-dask for DaskExecutor? Absolutely.
On Mon, Jan 30, 2023 at 8:14 PM Andrey Anshin <[email protected]> wrote: > > Should we also create apache-airflow-providers-dask for DaskExecutor? > > ---- > Best Wishes > Andrey Anshin > > > > On Sun, 29 Jan 2023 at 13:21, Jarek Potiuk <[email protected]> wrote: >> >> Hello Everyone, >> >> As a follow-up to AIP-51 - when it is completed (with few more quirks >> like the one described by Andrey in the "Rendered Task Instance >> Fields" discussion) - it should now be entirely possible to move >> Kubernetes Executor, Celery Executor and related "joined" executors to >> respective providers. >> >> There is of course question where to move CeleryKubernetesExecutor, >> but we have prior art with Transfer Operators and we might choose to >> add it to "cncf.kubernetes" and make the celery provider an optional >> dependency of K8S provider. >> >> This has multiple advantages, and one of the biggest one I can think >> of is that we could evolve K8S executor faster than Airflow itself and >> we would avoid quite a lot of complexities involved in parallel >> modifications of K8S code in provider and executor (it happened in the >> past that we had do to add very high min-airflow version for >> "cncf.kubernetes" provider in order to make this happens, and we could >> likely avoid similar problems by making sure K8S executor is used as >> "yet another executor" - compliant with the AIP-51 interface and we >> could evolve it much faster. >> >> That would also give celery provider some "real" meaning - so far >> celery provider was merely exposing celery queue sensor, but when we >> move CeleryExecutor to the provider, the provider would finally be a >> useful one. >> >> Doing this requires a few small things: >> * we likely need to add dynamic import in the old "executors" package >> following PEP 562 (and the way we've done operators) and deprecation >> notice in case someone uses them from there. >> * we likely need to add "cncf.kubernetes" and "celery" packages as >> pre-installed providers - alongside http, fttp, common.sql, sqlite, >> imap. >> >> I **think**, after AIP-51 gets implemented, this would be a fully >> backwards-compatible change - it's just a matter of proper management >> of the dependencies. We could add min-cncf.kubernetes/celery versions >> in Airflow 2.6, so that we are sure 2.6+ airflow uses only newer >> providers and kubernetes/celery code and this would be fully >> transparent for the users, I believe. >> >> J.
