On Wed, Feb 9, 2022 at 10:35 PM Giorgio Zoppi <[email protected]> wrote:
> Hello, > we had some experiments last year about using Airflow to run tests > directly from the CI, it was good but at the end we had no way to complete > the solution, also > it was a bit too much. > Yeah. Having "full" airflow to run the tests is a bit too brittle I think and quite an overhead. The experience with running tests in CI is that this **might** be made to work, but then any case where there are problems is super-hard to examine. I think a way to just "fire" a test or a group of those via command line and especially Pytest (the latter gives a lot more flexibility and management of "HOW' to run the multiple tests is really great when you need to solve problems. I think "local reproducibility" of such tests is crucial when it comes to debugging, and the requirement to have "the" version of Airflow with "the" configuration with "the" authentication configuration already makes it next to impossible for "a committer" to reproduce the failure. I think this also falls a bit into the 'environment" discussion part with Niko. I think this is a MUST to have a way to reproduce the environment to run the tests locally - for each system (GCP/AWS etc.). This might be a short documentation (configure Service account X and project Y this way and you can run the tests) and eventually maybe automated terraform/other scripts if more setup is needed but this should be extremely clear and simple how to reproduce a failing test. It might require your own resources (for example your own project in GCP) - but it should be "easy" and "money-safe" (in the sense that you should be able to easily tear down any resources you created by the test). J. > > > Il giorno mar 25 gen 2022 alle ore 17:58 Jarek Potiuk <[email protected]> > ha scritto: > >> Just let me add a bit more context as I see it. >> >> The AIP-47 design (following the AIP-4 idea) is targeted to handle those >> things: >> >> * streamline the execution of the System Tests (which are basically >> executable "example_dags" of ours which we already have in our >> documentation for providers) >> * make them integrated and automatically execute in a "cloud" environment >> - with real cloud services behind >> * make use of the 100s of System Tests already written in Google Provider >> (and many others which we have in in other providers - for example AWS >> should be close to follow) >> * test automation of those using Google Cloud Platform donated credits >> (from what I understand Google is willing to sponsor credits to run those >> tests regularly). >> * I am quite sure when it works also AWS will donate credits for our CI >> that we could use for that. >> * and we should be able to extend it to other providers as well once the >> "CI environment" is in place. >> >> When this one is implemented, we should be finally able not only to run >> "unit" tests with all PRs for a lot of provider code, but also we will be >> able to regularly (not with all PR but likely daily) run the example dags >> with the real services and make sure that at least big part of our >> Providers code (Hooks/Operators) are still "working as expected" - at least >> that they pass "smoke tests". >> >> Currently we rely mostly on the manual tests done when we release >> providers (and our community members are actually great here - as every >> month we have a great deal of help from within the community), but when we >> have system tests automation, the confidence of releasing new versions of >> providers will go up significantly. >> >> I am really looking forward to discussion on this one and implementation >> and deployment for a number of providers. >> >> j. >> >> >> On Tue, Jan 25, 2022 at 2:13 PM Mateusz Nojek <[email protected]> wrote: >> >>> Hello everyone, >>> >>> I created a new AIP draft titled AIP-47 New design of Airflow System >>> Tests >>> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-47+New+design+of+Airflow+System+Tests> >>> . >>> It is related to the new design of system tests that improves how they >>> are written and run which should contribute to higher quality of tests and >>> thus to lower quantity of bugs related to the operators. The AIP-47 is >>> located under AIP-4 >>> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-4+Support+for+Automation+of+System+Tests+for+external+systems> >>> as its direct successor (it more deeply describes some of the concepts). >>> The basic target was to simplify tests execution and enable running them >>> more regularly. >>> >>> I was working on it with Eugene Kostieiev and Bartłomiej Hirsz for some >>> time and soon we are also going to publish a PR with changes based on the >>> content of this AIP. >>> >>> For now, we would like to start a discussion with the developers and >>> contributors of the community of Apache Airflow. The idea was already >>> discussed with Jarek Potiuk, the original author of AIP-4, who cooperated >>> with us on some of the design details. >>> >>> Please, tell me what you think about it and share your ideas, concerns >>> and comments. >>> >>> Kind regards, >>> Mateusz Nojek >>> >> > > -- > Life is a chess game - Anonymous. >
