The way I perceived it so far, semver and upgrades are tightly coupled and there might be users like me who perceive that way.
In the particular cases of moving a piece of code as an optional dependency, not keeping it part of the core or having it pre-installed, I agree we could have a breaking upgrade in a minor release as long as users don't have to change any DAG code but they only need to install additional dependencies and having well-written communication in the release notes would help reduce the change being interpreted as breaking. In particular, if we agree with *`that breaking upgrade flow is occasionally OK andrelease notes should explain it* ` would it make sense to reword slightly or remove the `strict` word here that mentions we follow strict semver here: https://github.com/apache/airflow/blob/49921763eb15f68f91da826a86690ba4c4155c35/README.md?plain=1#L243 and give examples to reduce ambiguities where breaking upgrades are possible? Regards, Pankaj Koti *Senior Software Engineer, *OSS Engineering Team. Location: Pune, India Timezone: Indian Standard Time (IST) Email: pankaj.k...@astronomer.io Mobile: +91 9730079985 On Fri, Jul 21, 2023 at 2:48 PM Jarek Potiuk <ja...@potiuk.com> wrote: > Hello everyone, > > I wanted to separate one topic into a separate side discussion. Maybe > not useful with Dask discussion but might be something that we might > use for the future. > > > This means that this change might potentially break existing > > setups for some users when they upgrade Airflow version, which > contradicts > > the Airflow deprecation policy > > also there was another commend from Daniel: > > > As a user, your concerns and expectations regarding semver are about > > essentially how the code works, e.g. are you going to have to refactor > all > > of your 500 dags. In other words the API.. But to me this is a lower > > evel thing that doesn't seem should strictly be forbidden. It's like... > > when we change the dependencies in airflow, inevitably that will break > > someone's cluster if they blindly deploy to prod. But this is not a > > prudent thing to do. So I think it's not an unreasonable expectation > that a > > user verify that their cluster will spin up. It's something you can > verify > > without running any dags at all (i.e. if you set dask executor and you > > start airflow, i think airflow will not start). And then users can rely > on > > semver about changing behavior, which is not something you necessarily > can > > tell without running dags. > > I tend to agree with Daniel that technically speaking SemVer does not > tell anything about breaking upgrades. It does not really say "you > might continue upgrades the same way as before and things will work" > but "when you upgrade <CORRECTLY> things will work the way they > worked". This is the promise of https://semver.org/ . > > I re-read it again and could not find any promise that "your process > of upgrade is covered by SemVer promises". Interestingly I also found > out (I did not know) that our rule "breaking changes in our > dependencies is not breaking our compatibility" is explained there in > FAQ: > > > That would be considered compatible since it does not affect > > the public API. Software that explicitly depends on the same > > dependencies as your package should have their own > > dependency specifications and the author will > > notice any conflicts. Determining whether the change > > is a patch level or minor level modification depends on > > whether you updated your dependencies in order to fix a > > bug or introduce new functionality. We would usually expect > > additional code for the latter instance, in which case > > it’s obviously a minor level increment. > > > We do not make a promise that our upgrade process should look the same > between versions - it largely is the same, but IMHO technically we can > throw-in a new step or add a note in the upgrade process telling the > user to do things differently to upgrade. > > IMHO it is perfectly fine if in the release notes/significant section > we tell the user "If you used XX " you need to make sure you install > airflow with [xx] extra. Yes. It might break some installation scripts > and CI workflows, but upgrading to new version of Airflow should > ALWAYS be a deliberate action - with migration of the DB and reading > release notes before doing the upgrade - and failing such automated > workflow of upgrade is at most a signal for whoever maintains it to > make some fixes to their process. > > I also think that guaranteed upgrades work is rather difficult - ys we > have our "reproducible" installation with constraints (which is > guaranteed to work), but there are many other ways and tools people > might install airflow - for example we have poetry or bazel > installation - and for example we already broke basel installation by > adding back limits on pre-installed providers, because basel expects > dependency graph to be DAG and for us (and pip/PyPI) having circles in > dependency graph are quite legitimate use case - see > https://github.com/bazelbuild/rules_python/pull/1166 - so technically > speaking we are not really ablet to guarantee much when it comes > "upgradeability". > > But I would love to hear what others think about it. I would love it > if we get to a consensus on that and write it down in the "Public > Airflow Interface/SamVer" promise of ours so that we can communicate > it better rather than allow different people to have different > assumptions here. > > I think we have two choices if we were to "disambiguate what Semver + > upgrades mean": > > * either we agree that breaking upgrade flow is occasionally OK and > release notes should explain it > > * or we document that the users can 100% rely on it working always > seamlessly when they upgrade - but I think this one is going to be > difficult to describe in the way that will include all the exclusions > such as the "basel" case. > > J. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
