Hello here, *TL;DR; We want to have better organized, and much improved new `e2e` test type, a common thing in Airflow - one that others will be able to add tests to and add their own similar tests following the blueprint.*
Over the last few weeks quite a few of us were working on something we have not fully realise we will end up and it was mostly bottom -up grass root ideas that simply "clicked" together and we would like to propose adding (or more formalising) a new type of tests in airflow - e2e tests. The people who worked on it were mostly Amogh, Zhe You lIu, Bugra, myself. Wa already have a few attempts to do it (in various stages - as top-level folders: *docker-tests, k8s-tests, airflow-e2e-tests, airflow-ctl-tests, task-sdk-integration-tests* - yes the lack of consistency in names is very confusing) but with recent improvements in task-sdk-integration-tests - we think we have a good proposal on how we can implement those tests and standardise it across airflow distributions. Current (best) approach is https://github.com/apache/airflow/tree/main/task-sdk-integration-tests And it has the following properties: 1) uses uv pytest (standard pytest tests), docker-compose and python-on-whales together, also has nice breeze wrapper for CI 2) automatically sets-up (and keeps running during iteration) docker-compose based airflow deployment - with components (any - api-server, dag processor, scheduler, triggerer) necessary during the tests and allows to interact with it, including triggering local dags and the like 3) pytest tests are run in a local venv controlled by uv sync 4) it allows for extremely fast (almost like pytest-native in venv) iteration speed with tests - it uses Zhe's hot-reload functionality added a week ago in order to reload components inside the docker-containers, and local sources are mounted to those - which means that just saving a file in your IDE or local env will make automated (sub-second, really) reload of the components you interact with. Basically at the time you press Shift-F10 (or whatever shortcut you have) to re-run your tests, your just auto-saved modified Airflow code already runs in those docker-compose components (!) - *this is the most important feature of it.* 5) All those tests are automatically run in CI - in relevant PRs (driven by selective checks) - but also they are very easily runnable locally in local venv (no breeze CI image needed). Basically: cd task-sdk-integration-tests uv run pytest 6) Last minute addition - we've been literally asked by Kacper an hour ago about having "something like that" for open lineage - every current distribution is able to have their own set of tests like that if the stewards of that part want it Enough bragging.... *Our proposal:* * we consistently rename those tests as `*e2e*` tests * we adapt all of them (including *k8s* tests eventually - where we have to hack kind a bit) - to follow the same patterns and principles * we extract common code for that to `*devel-common*` and reuse across those tests * we get rid of all the top-level `*SMTH-test*` folders we have now and move those different kinds of tests to `*tests/e2e*` (next to `unit`, `system`, `integration` we already have) of distributions that the tests are relevant to For example - instead of: task-sdk src tests task_sdk some_unit_test.py task-sdk-integration-tests dags <- here e2e test dags are stored logs <- here logs from all components are kept (.gitignored) tests task_sdk_tests some_end_2_end_test.py We propose: task-sdk src tests unit task_sdk some_unit_test.py e2e dags logs task_sdk some_end_2_end_test.py Of course - this is a proposal - and if there are other ideas on how to restructure the test folders, that might be a good idea to do it now. *Only serious and complete offers are going to be considered :)*, so I propose constructive, complete proposals rather than criticising the current one. *Let us know what you think,* Jarek, Zhe, Bugra, Amogh
