I just (when reviewing PRs) found a good example to illustrate my concerns:
https://github.com/apache/airflow/pull/22253 I commented there why I think we need to have a good plan on what to do with the kubernetes provider before we decide what to do with the settings: https://github.com/apache/airflow/pull/22253#issuecomment-1096643179 Otherwise we might need to revert the decision if it turns out that complete decoupling is not possible/feasible/we do not want it and revert to pulling the cncf.kubernetes code back to the core as an "easier" solution. I think coming up with a plan of what we should do with `cncf.kubernetes` will give us clarity on what we should do with the settings. On Tue, Apr 12, 2022 at 10:58 AM Jarek Potiuk <[email protected]> wrote: > > What I am saying is that we should deprecate those settings in core - > but only if it is part of decoupling Kubrenetes Executor from the core > - otherwise this change gives us pretty much no improvement. > > In order to do that - we need to at least know what changes will be > needed in the core - standardizing Executor's behaviours to make them > pluggable - and see if any of the changes will be breaking > compatibility and need Airflow 3 (possibly we can do it in the way > that it won't be needed). > > This change might come in stages - for example deprecating k8s core > settings might be the first change (or maybe not - but we need to have > a plan). > > I think just deprecating settings gives very little if it is not > accompanied with at least a plan on how to decouple the > cncf.kubernetes provider (and especially Kubernetes Executor) from the > core. > > So what I **think** should happen now: > > 1) We should have a proposal (AIP ? ) on how to decouple Executors > (including K8S Executor) from the core (which should contain some kind > of inventory of what should happen to decouple it). > 2) We should propose a staged approach on how to do it and what we can > do without actually breaking compatibility (maybe after the analysis > we will find out that we can do it without breaking the > compatibility). > 3) We should see how deprecation of core settings fits into this plan. > > I think otherwise we might simply move the problem elsewhere rather > than plan to solve it. **Just** moving settings out simply masks the > problem. > > The problem is that no matter where the settings are, the core K8S > executor has a very close binding with the provider and one cannot > leave without the other in the right versions and changing one impacts > the other in unforeseen ways (as we experienced recently when we had > to yank few versions of the cncf.kubernetes provider) > > This is the really problem to solve, not where the settings are. > > J. > > On Mon, Apr 4, 2022 at 6:38 PM Daniel Standish > <[email protected]> wrote: > > > > Thanks Jarek > > > > I register a bit of a mixed message. > > > > Re this > > > >> Maybe I am wrong but I think the separation should be solved together, > >> there is a good reason why some k8s settings are in core. > > > > > > Are you saying I should not proceed with this plan (add warnings in KPO > > that push users to start using K8s Hook connection for config instead of > > "core" k8s config settings before 3.0) until we figure out the other things > > you referenced, or that you are in favor of doing this deprecation but you > > also think that we should resolve those other concerns too by the time we > > actually remove support? > > > > Thanks > >
