We all agree on moving the 5 remote executors to their respective providers. The main question now is whether we should pre-install all the providers or not.
IMHO we should pre-install Celery, Kubernetes, and Dask providers until Airflow 3.0. Not doing so would be a breaking change for some users, and it would go against the Airflow deprecation policy. Once Airflow 3.0 is released, we should remove the pre-installed providers and instead prompt the user to install the provider specific to its executor. On Fri 14 Jul 2023 at 14:54, utkarsh sharma <utkarshar...@gmail.com> wrote: > I'm too in favor of moving Dask Executor into its own provider, which will > make the airflow's codebase more pluggable and orthogonal. Big +1 :) > > Thanks, > Utkarsh Sharma > > On Fri, Jul 14, 2023 at 2:03 AM Ferruzzi, Dennis > <ferru...@amazon.com.invalid> wrote: > > > I'm not sure how much of a train wreck it would turn into, but moving > Dask > > Executor to a provider seems logical to me. > > > > Maybe in the far-flung future of Airflow 3.0 we could move celery and k8s > > into their own as well and make it truly pluggable and executor-agnostic, > > but I agree that at this time they are too deeply integrated. > > > > > > - ferruzzi > > > > > > ________________________________ > > From: Oliveira, Niko <oniko...@amazon.com.INVALID> > > Sent: Wednesday, July 12, 2023 10:40 AM > > To: dev@airflow.apache.org > > Subject: RE: [EXTERNAL][DISCUSS] Moving Dask Executor to a separate > > (optional?) dask provider > > > > CAUTION: This email originated from outside of the organization. Do not > > click links or open attachments unless you can confirm the sender and > know > > the content is safe. > > > > > > > > I think in a perfect world we'd only have the completely vendor neutral > > executors pre-installed (Local, Sequential, Debug) and anything else > would > > need to be specifically installed by admins/users. I think if we were > > starting from scratch this would make the most sense, but clearly > > Kubernetes and Celery executors are so ubiquitous that it'd cause too > much > > wreckage to not install them, but I'd like to push for Dask to _not_ be > > installed by default. If this causes too much wreckage then perhaps we > > should deprecate that (though I'm not sure exactly what that would look > > like in this context), but it's difficult to measure how many folks are > > using the Dask executor. Perhaps we have data from the yearly > > questionnaire/survey we send? > > > > ________________________________ > > From: Jarek Potiuk <ja...@potiuk.com> > > Sent: Wednesday, July 12, 2023 8:05:54 AM > > To: dev@airflow.apache.org > > Subject: [EXTERNAL] [DISCUSS] Moving Dask Executor to a separate > > (optional?) dask provider > > > > CAUTION: This email originated from outside of the organization. Do not > > click links or open attachments unless you can confirm the sender and > know > > the content is safe. > > > > > > > > Hello Everyone, > > > > A small follow up after K8S/Celery executors being moved: > > https://lists.apache.org/thread/7gyw7ty9vm0pokjxq7y3b1zw6mrlxfm8 > > > > We are in the process of moving Celery / Kubernetes executor (Celery > almost > > complete and I am working on K8S next + some common discovery and config > > moving) > > > > But there is one more "questionable" executor - i.e. Dask executor, still > > living in Airflow Core. > > > > When it comes to Celery/Kubernetes, we decided to make the two providers > > preinstalled, because it makes most sense - we are also going to get the > > basic documentation in the "core" airflow documentation so that it is > > easier discoverable and prominently visible - also because of the > > vendor-neutrality. > > > > However when it comes to Dask I am not sure about its status and whether > we > > should make it preinstalled ? > > > > I guess there is no doubt to move it to a provider - this has only the > > benefits same as Celery/K8S move. But whether it should be preinstalled > > with Airflow - I am not sure. I do not know how frequently Dask executor > > (and Dask) is used by people using Airflow, but I personally do not think > > it should be as "closely" connected with Airflow as Celery/Kubernetes > ones. > > > > If we do not make it preinstalled, it is somewhat (but not too much, > > really) breaking change. We still might choose to install dask provider > in > > the PROD reference image, so it will continue to work if you use the > image, > > and when you are installing airflow in venv you will only have to specify > > `pip install apache-airflow[dask]` or manually install > > `apache-airflow-providers-daskexecutor` (for now at least this is the > name > > I could reserve in PyPI). So this is not really breaking, it just > requires > > another dependency to be installed. But some pipelines of installing > > Airflow might get broken because it won't be pre-installed - so this is a > > borderline breaking. > > > > WDYT? Should we make the dask executor pre-installed or not? > > > > J. > > >
