Hello everyone, As part of preparation for the Airflow 3 move and (possible) provider separation (I have some ideas how to do it but that should be a separate discussion) I took on the task of improving our compatibility tests for Providers. I discussed it briefly with Kaxil and Ash and decided to give it a go and see what it takes.
The PR here: https://github.com/apache/airflow/pull/39513 I extended our "import" checks with checks that also run all provider unit tests for specified airflow versions (for now 2.9.1 - but once we get it merged/approved we can make sure the tests are working for 2.7 and 2.8). We will also be able to run "future" compatibility tests in case we decide to leave providers aside from Airflow 3 and will be able to run the tests for both`main` and `pypi`-released versions of airflow. A number of our tests rely on some internals of Airflow and they implicitly rely on the fact that they are run directly in airflow source tree - but there are not many of those - after some initial compatibility fixes I got 50 or so tests failing for 2.9.1 (probably there will be more for 2.8.0 and 2.7.0, but I want to make 2.9.1 works first). I almost got it working (few tests are still failing) with compatibility for 2.9.1 but I will need some help from a few people - around openlineage and serialization but also around recently improved try_number :). I will reach out to the relevant people individually if we see that as a good idea. It requires some care when writing tests to make sure the tests can be run against installed airflow and not from sources. So in the future anyone contributing provider changes will have to make sure the tests pass also for past airflow versions (there are simple instructions explaining how to do it with breeze). But once we merge it, this will be caught on PR level and should be easy to fix any of those problems. The benefit of having the tests is that we not only do simple import tests but actually run provider tests, the drawback is that sometimes tests will have to be adapted to make sure they work also for installed older airflow versions (which is not always straightforward or easy and will need some compatibility code in tests - for example after recent rename of airflow.models.ImportError to ParsingImportError we had to add compat.py to test_utils and import ParsingImportError from there rather than from Airflow directly in tests. I don't think it's too controversial - being able to run unit tests for providers for old (and future) versions of Airflow is generally quite an improvement in stability, but this adds a bit overhead on contributions, so I am letting everyone here know it's coming, so that it's not a surprise to contributors. J.
