I'm too in favor of moving Dask Executor into its own provider, which will make the airflow's codebase more pluggable and orthogonal. Big +1 :)
Thanks, Utkarsh Sharma On Fri, Jul 14, 2023 at 2:03 AM Ferruzzi, Dennis <ferru...@amazon.com.invalid> wrote: > I'm not sure how much of a train wreck it would turn into, but moving Dask > Executor to a provider seems logical to me. > > Maybe in the far-flung future of Airflow 3.0 we could move celery and k8s > into their own as well and make it truly pluggable and executor-agnostic, > but I agree that at this time they are too deeply integrated. > > > - ferruzzi > > > ________________________________ > From: Oliveira, Niko <oniko...@amazon.com.INVALID> > Sent: Wednesday, July 12, 2023 10:40 AM > To: dev@airflow.apache.org > Subject: RE: [EXTERNAL][DISCUSS] Moving Dask Executor to a separate > (optional?) dask provider > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > I think in a perfect world we'd only have the completely vendor neutral > executors pre-installed (Local, Sequential, Debug) and anything else would > need to be specifically installed by admins/users. I think if we were > starting from scratch this would make the most sense, but clearly > Kubernetes and Celery executors are so ubiquitous that it'd cause too much > wreckage to not install them, but I'd like to push for Dask to _not_ be > installed by default. If this causes too much wreckage then perhaps we > should deprecate that (though I'm not sure exactly what that would look > like in this context), but it's difficult to measure how many folks are > using the Dask executor. Perhaps we have data from the yearly > questionnaire/survey we send? > > ________________________________ > From: Jarek Potiuk <ja...@potiuk.com> > Sent: Wednesday, July 12, 2023 8:05:54 AM > To: dev@airflow.apache.org > Subject: [EXTERNAL] [DISCUSS] Moving Dask Executor to a separate > (optional?) dask provider > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > Hello Everyone, > > A small follow up after K8S/Celery executors being moved: > https://lists.apache.org/thread/7gyw7ty9vm0pokjxq7y3b1zw6mrlxfm8 > > We are in the process of moving Celery / Kubernetes executor (Celery almost > complete and I am working on K8S next + some common discovery and config > moving) > > But there is one more "questionable" executor - i.e. Dask executor, still > living in Airflow Core. > > When it comes to Celery/Kubernetes, we decided to make the two providers > preinstalled, because it makes most sense - we are also going to get the > basic documentation in the "core" airflow documentation so that it is > easier discoverable and prominently visible - also because of the > vendor-neutrality. > > However when it comes to Dask I am not sure about its status and whether we > should make it preinstalled ? > > I guess there is no doubt to move it to a provider - this has only the > benefits same as Celery/K8S move. But whether it should be preinstalled > with Airflow - I am not sure. I do not know how frequently Dask executor > (and Dask) is used by people using Airflow, but I personally do not think > it should be as "closely" connected with Airflow as Celery/Kubernetes ones. > > If we do not make it preinstalled, it is somewhat (but not too much, > really) breaking change. We still might choose to install dask provider in > the PROD reference image, so it will continue to work if you use the image, > and when you are installing airflow in venv you will only have to specify > `pip install apache-airflow[dask]` or manually install > `apache-airflow-providers-daskexecutor` (for now at least this is the name > I could reserve in PyPI). So this is not really breaking, it just requires > another dependency to be installed. But some pipelines of installing > Airflow might get broken because it won't be pre-installed - so this is a > borderline breaking. > > WDYT? Should we make the dask executor pre-installed or not? > > J. >
