Hi Jarek, Thanks for proposing the new e2e test structure, the task-sdk e2e setup looks great to me! I have one question about the k8s tests in the future: should they live under airflow-core, under task-sdk, or in a separate dedicated location?
Best regards, Jason On Mon, Nov 17, 2025 at 11:56 PM Jarek Potiuk <[email protected]> wrote: > Hello here, > > *TL;DR; We want to have better organized, and much improved new `e2e` test > type, a common thing in Airflow - one that others will be able to add tests > to and add their own similar tests following the blueprint.* > > Over the last few weeks quite a few of us were working on something we have > not fully realise we will end up and it was mostly bottom -up grass root > ideas that simply "clicked" together and we would like to propose adding > (or more formalising) a new type of tests in airflow - e2e tests. The > people who worked on it were mostly Amogh, Zhe You lIu, Bugra, myself. > > Wa already have a few attempts to do it (in various stages - as top-level > folders: *docker-tests, k8s-tests, airflow-e2e-tests, airflow-ctl-tests, > task-sdk-integration-tests* - yes the lack of consistency in names is very > confusing) but with recent improvements in task-sdk-integration-tests - we > think we have a good proposal on how we can implement those tests and > standardise it across airflow distributions. > > Current (best) approach is > https://github.com/apache/airflow/tree/main/task-sdk-integration-tests > > And it has the following properties: > > 1) uses uv pytest (standard pytest tests), docker-compose and > python-on-whales together, also has nice breeze wrapper for CI > > 2) automatically sets-up (and keeps running during iteration) > docker-compose based airflow deployment - with components (any - > api-server, dag processor, scheduler, triggerer) necessary during the tests > and allows to interact with it, including triggering local dags and the > like > > 3) pytest tests are run in a local venv controlled by uv sync > > 4) it allows for extremely fast (almost like pytest-native in venv) > iteration speed with tests - it uses Zhe's hot-reload functionality added a > week ago in order to reload components inside the docker-containers, and > local sources are mounted to those - which means that just saving a file in > your IDE or local env will make automated (sub-second, really) reload of > the components you interact with. Basically at the time you press Shift-F10 > (or whatever shortcut you have) to re-run your tests, your just auto-saved > modified Airflow code already runs in those docker-compose components (!) > - *this is the most important feature of it.* > > 5) All those tests are automatically run in CI - in relevant PRs (driven by > selective checks) - but also they are very easily runnable locally in local > venv (no breeze CI image needed). Basically: > > cd task-sdk-integration-tests > uv run pytest > > 6) Last minute addition - we've been literally asked by Kacper an hour ago > about having "something like that" for open lineage - every current > distribution is able to have their own set of tests like that if the > stewards of that part want it > > Enough bragging.... > > *Our proposal:* > > * we consistently rename those tests as `*e2e*` tests > > * we adapt all of them (including *k8s* tests eventually - where we have to > hack kind a bit) - to follow the same patterns and principles > > * we extract common code for that to `*devel-common*` and reuse across > those tests > > * we get rid of all the top-level `*SMTH-test*` folders we have now and > move those different kinds of tests to `*tests/e2e*` (next to `unit`, > `system`, `integration` we already have) of distributions that the tests > are relevant to > > For example - instead of: > > task-sdk > src > tests > task_sdk > some_unit_test.py > task-sdk-integration-tests > dags <- here e2e test dags are stored > logs <- here logs from all components are kept (.gitignored) > tests > task_sdk_tests > some_end_2_end_test.py > > We propose: > > > task-sdk > src > tests > unit > task_sdk > some_unit_test.py > e2e > dags > logs > task_sdk > some_end_2_end_test.py > > Of course - this is a proposal - and if there are other ideas on how to > restructure the test folders, that might be a good idea to do it now. *Only > serious and complete offers are going to be considered :)*, so I propose > constructive, complete proposals rather than criticising the current one. > > *Let us know what you think,* > > Jarek, Zhe, Bugra, Amogh >
