Yeah. I thought a bit more about that and the risks of forcing Airflow 2.7 users to use the latest k8s provider even if they do not use K8S executor is much bigger (and much more difficult to handle) than risks of not installing the provider in the right version.
Also I think there is one more scenario. This might completely break some deployments and block our users from upgrading to 2.7+ - even for a long time. Imagine you have an environment where user relies on dependency that conflicts with one of the celery/k8s deps. This will effectively block the user from upgrading airflow - even if they do not use K8S executor. This is far more dangerous than any scenario that I can imagine with not preinstalling the providers. @Shubham > In addition to this, we should consider a UI status that clearly outlines the > need to upgrade the provider to use the configured executor. Good idea. @Niko > What do folks think about doing this in stages? I don't see a reason why we > need to make all the changes at once. We could release the executors in > providers and have them come pre-installed for some number of releases. This > would allow the majority of folks to slowly migrate to newer versions of both > core and providers which would then minimize the blast radius when we > eventually turn off the pre-installation of those providers. I think - the opposite. If we find that it causes problems, we can think of pre-installing :D. Eventually I think this all boils down to "how easy" it is to upgrade. Installation is really a deliberate action - and if we have the messaging right - if someone upgrades and has an older version of the provider, we can very easily turn "import error" into "Please upgrade cncf.kubernetes provider to version x.y.z+". I think I will cancel the vote for Dask and restart it for All Executors. I think we should indeed treat them together. J. On Fri, Jul 21, 2023 at 8:14 PM Mehta, Shubham <shu...@amazon.com.invalid> wrote: > > I second Jed's proposal and believe that the implications and benefits, as > laid out, make a lot of sense. > > > we wouldn't have to set a minimum version for these 2 providers to ensure > > we have the version with the executors, allowing an "old" k8s provider (KPO > > itself is pretty common after all) to be used if you aren't using > > KubernetesExecutor. > This is great point. There are indeed many users who fall into this category, > and not compelling them to upgrade their k8s provider is a more user-friendly > approach. > > To mitigate issue of users not having a new enough provider, one that has > their executor of choice, I agree that error messaging would be helpful. In > addition to this, we should consider a UI status that clearly outlines the > need to upgrade the provider to use the configured executor. > > Overall, the proposal helps the Airflow community adhere to the previous > policies without significantly degrading the user experience. > > Shubham > > On 2023-07-21, 9:19 AM, "Jed Cunningham" <jedcunning...@apache.org > <mailto:jedcunning...@apache.org>> wrote: > > > CAUTION: This email originated from outside of the organization. Do not click > links or open attachments unless you can confirm the sender and know the > content is safe. > > > > > > > Hello everyone, > > > After thinking through whether the dask provider should be pre-installed or > not, it got me thinking about whether we should even be pre-installing the > celery and k8s providers. Core Airflow so far hasn't had the dependencies > required to make those executors functional either - users either had to > use the extra or install the provider directly. So that doesn't really > change if we choose not to preinstall the providers. > > > Why would it be beneficial to not pre-install them? The most obvious > benefit is you don't get a handful of new required deps you aren't using > (like celery/flower/billiard/etc if you don't use CeleryExecutor). However > another benefit is we wouldn't have to set a minimum version for these 2 > providers to ensure we have the version with the executors, allowing an > "old" k8s provider (KPO itself is pretty common after all) to be used if > you aren't using KubernetesExecutor. > > > The one downside here is how do we make sure users have a new enough > provider, one that has their executor of choice in it. After chatting with > Jarek, he brought up that we could also make the error messaging more > helpful in this situation. This messaging would basically solve the problem > without needing minimum version pinning in core. > > > All that said, I think we should strongly consider not pre-installing the > celery and k8s providers as well. > > > Thoughts? > > > Jed > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
