1) users "peace of mind" as top priority: clarity of what they can expect from Airflow, and avoiding surprises when upgrading 2) targeting minimal disruption to user's workflows (though we might never reach absolute 100%) 3) making it easy for contributors and maintainers to decide on breaking/non-breaking behaviours
Yupp, I agree, this is an accurate encapsulation of the issues at hand. My proposal to work on documenting our approach for our users (and for maintainers) in a single page: "What is Airflow Public API?" and what users can expect. I think this is actually a very important piece we've been missing. From the SemVer RFC itself it says: "For this system to work, you first need to declare a public API. This may consist of documentation or be enforced by the code itself. Regardless, it is important that this API be clear and precise. Once you identify your public API, you communicate changes to it with specific increments to your version number." So as difficult as I think it will be to accurately describe and automate what the Airflow public API is, I think it's a very useful project to undertake. Perhaps even codifying it in an AIP. At the moment we consider even the deepest/smallest "private" helper function within util provider code to be public. This level of public API makes iterating and maintaining the code very laborious. So I definitely think this is worth the effort. I'll need to have a closer look at that PR, but the exact technical details can certainly be hammered out later. Cheers, Niko ________________________________ From: Jarek Potiuk <ja...@potiuk.com> Sent: Saturday, December 3, 2022 1:25 AM To: dev@airflow.apache.org Subject: RE: [EXTERNAL][DISCUSSION] Assessing what is a breaking change for Airflow (SemVer context) CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. Sorry for not following up on this for a bit - it's been hectic these days for me. I think valid points were said, and from the tone of those I feel that we all who participated have the same sense of what is important: 1) users "peace of mind" as top priority: clarity of what they can expect from Airflow, and avoiding surprises when upgrading 2) targeting minimal disruption to user's workflows (though we might never reach absolute 100%) 3) making it easy for contributors and maintainers to decide on breaking/non-breaking behaviours I think there is a main blocker to all of those (also mentioned in the discussion above): We are extremely cautious about any change because there is a lack of agreement/expectations with our users on what is supposed to be the "public API" . # Proposal My proposal to work on documenting our approach for our users (and for maintainers) in a single page: "What is Airflow Public API?" and what users can expect. There are certain areas where we can define rules and either automate or document (or both) our statement about what is the "public" API and (more importantly) what is clearly NOT on a single page document. Also it should also be accompanied (where possible) with some automation and tooling that would help us to express it in detail (and help our users to validate if they are conforming to the "public API"). We won't solve it very quickly, but once we start doing it, it might turn out that it's not that long of a process in fact. And if we start it now - in a few months we might be in a different place. # Some concrete actions we might take 1) On the 'Code" level - we can start to define the API that is considered as "public" and add verification of those for our users. We could implement a similar solution to what I proposed to common.sql https://github.com/apache/airflow/pull/27962 (where I followed Ash's idea to use MyPy stubgen and pre-commits to flag changes to it, and where we harness MyPy capabilities to control how the API is used). I believe that we could apply a similar solution to all providers and eventually even all parts of core, to make it very clear which part of the Airflow API is public and which is not. I think MyPy and strong-ish typing is taking the Python world by a storm, and we could use it as a standard way of communicating to those who use Airflow as a library, which parts are "public". Having .pyi files as part of our packages with "hidden" parts that are not supported to be exposed, seems to be not only a nice communication tool but also has support for all the kind of tooling from day 0 for our users (IDE integrations, automations to check if the right API is used etc.). We could even easily provide guidelines for the users "Here is how you can check if you are using Airflow code properly". Not 100% foolproof but much better than anything else I can imagine. Also having it in place will allow the providers to be finally separated to separate repositories - and we could use MyPy checks rather than running the full test suite with the Providers to verify if changes in Airflow do not break Providers. That would finally make it possible to loosen the coupling we have between Providers and Airflow (currently we basically run whole suite of tests to be certain things are working - but we could simply run providers with MyPy checks if we have proper .pyi files (not the same confidence but very, very close). 2) On the DB level - we already have "AIP-44" as the foundation of telling the users - those are the "Airflow" you can do "this" when you write your DAGs. Direct DB access will be forbidden and we can specifically communicate to the users "do not use DB any more" and we can even work out warnings when our users do. We could even make it a default behaviour later to block direct access by default (but that is likely only in Airflow 3). 3) On the UI level - we could simply explain that UI changes are exempt from the "no removal" policy. We might simply treat all the UI changes as non-breaking by default and loosen our strictness there. This would be very close to the Chrome/Firefox example by Bolke - I think UI changes are not breaking in the sense that you have to fix your code that uses it, it requires simply changing user's habits. We've already done this, That would be simply acknowledging the approach we already used when TreeView was replaced by GridView. 4) Airflow also has also a few non-code interfaces that are considered as part of the platform: statsd metrics is one of them. I can't think of any more but maybe there are more. We could simply make an inventory and discuss our approach on those ONCE and document it. This will avoid discussions, discussions, discussions, and let our users have some clear expectations and maintainers making quick decisions when approving (or not) PRs. # Question Does it sound like a good plan? Is it worth making such an effort ? Or maybe what we have as status-quo is "good enough" and that would be a waste of effort? WDYT? J. J
