I'm glad to see us start having this discussion. Thank you, Ash. 1) +1 to using SemVer.
2) I'm a strong Yes to Hooks and Operators, moderate on log output, and otherwise pretty weak in terms of the guarantees we should provide. That said, I think some of this is made more problematic for us by Airflow being a single monolith; it makes it hard for us to change some things but not others. If the core engine/models, webserver, and Hooks/Operators were three different projects, we would have a lot more flexibility in how we manage versioning. We could establish a guarantee that everything would remain compatible for one subsequent engine version. For example, engine 2.1 would run DAGs written using Hooks/Operators 2.0, 2.1. People could upgrade their engine and take advantage of new features while having a reasonable migration path to upgrade their DAGs to the new Hooks/Operators. --George On Mon, Jan 8, 2018 at 9:57 AM Chris Riccomini <criccom...@apache.org> wrote: > @ash, I think your proposal is actually pretty reasonable. Do you want to > document it on the wiki, and we can discuss/take a vote? > > On Tue, Dec 19, 2017 at 10:45 AM, Ash Berlin-Taylor < > ash_airflowl...@firemirror.com> wrote: > > > Hi, > > > > A question came up on a github issue about what exactly we meant about > > backwards compatibility, and I figured we as a project should work out > what > > we mean when we say we want to maintain compat. And most importantly > > document it (don't worry, I'm volunteering to do bit, so long as we reach > > consensus). > > > > So the issue that spawned this was https://github.com/apache/ > > incubator-airflow/pull/2806 <https://github.com/apache/ > > incubator-airflow/pull/2806> which changes some config setting names. > > > > In the example of this particular PR it's not a big deal, and supporting > > both names is not a difficult change, but I felt this was a discussion > > worth having. > > > > My view is somewhat influenced by Django which has a view that if you > have > > no deprecation warnings now (say on 1.8) then if you upgrade to 1.9 > things > > will still work, but now you might get deprecation warnings that will > need > > to be fixed before upgrading to 1.10. (Version numbers just for example.) > > > > Bolke's view is: "backwards compatible is more along the lines of > FreeBSD. > > API/ABI compatible, but config changes can happen and are in UPDATING" > > > > Now, Airflow is still a relatively young project so perhaps we want to be > > a little more relaxed about this for the time being? Do we want to just > say > > backwards compatibility is required for operators and hooks -- i.e. > things > > that most users are going to interact with, but other bits of internals > are > > "fair game to change", but perhaps with a best-effort goal. > > > > So some questions: > > > > 1) Do we want to follow something like SemVer (semver.org < > > http://semver.org/>) where backwards-incompatible changes need a major > > version bump, which would be to Airflow 2.0 in this case. I think this is > > what we do, but I don't believe is written down anywhere. > > > > 2) Do we want to limit or exclude the back-compart _guarantees_ to any > > areas? ie. just include Hooks and Operators? What about data model/DB > > tables? Log formats? Log file paths? > > > > Does anyone have any other strong opinions about areas I haven't > mentioned? > > > > My answers are: > > 1) Yes to SemVer > > 2) Strong Yes to Hooks and Operators and anything used directly in DAGs, > > with a weak yes to including config and other python classes too. > > > > My aim here is to start some discussion. If we get to any consensus then > > after the holidays I'll open a PR to update the docs. > > > > Cheers, > > -ash > > > > > > >