Yes. I think this makes much more sense. On Fri, Jul 21, 2023 at 6:19 PM Jed Cunningham <jedcunning...@apache.org> wrote: > > Hello everyone, > > After thinking through whether the dask provider should be pre-installed or > not, it got me thinking about whether we should even be pre-installing the > celery and k8s providers. Core Airflow so far hasn't had the dependencies > required to make those executors functional either - users either had to > use the extra or install the provider directly. So that doesn't really > change if we choose not to preinstall the providers. > > Why would it be beneficial to not pre-install them? The most obvious > benefit is you don't get a handful of new required deps you aren't using > (like celery/flower/billiard/etc if you don't use CeleryExecutor). However > another benefit is we wouldn't have to set a minimum version for these 2 > providers to ensure we have the version with the executors, allowing an > "old" k8s provider (KPO itself is pretty common after all) to be used if > you aren't using KubernetesExecutor. > > The one downside here is how do we make sure users have a new enough > provider, one that has their executor of choice in it. After chatting with > Jarek, he brought up that we could also make the error messaging more > helpful in this situation. This messaging would basically solve the problem > without needing minimum version pinning in core. > > All that said, I think we should strongly consider not pre-installing the > celery and k8s providers as well. > > Thoughts? > > Jed
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
