potiuk commented on PR #43965: URL: https://github.com/apache/airflow/pull/43965#issuecomment-2473784549
Also cc: @ahidalgob @kosteev @pankajkoti @fdemiane @sc250072 except the reviewers that I pinged. This is something I worked on for last few days as first stage of #42632 - it will be quite big eventually and it is still work in progress (I am still working on fixing all the tests in breeze selective checks and need to apply changes to github workflows to run the tests but wanted to give you a heads up of what I am proposing as a change. So do not (yet) comment on subtle details - that will come, but I'd love more to get a heads-up on the whole concept that I am planning to implement for our tests, some simplifications it will bring and impact on the system tests execution (because that is something not yet in our CI and something that should be applied as a change when you will run system tests of Amazon, Google, Astronomer and Teradata people. Few general things: 1) The set of testing commands we will be able to use is this:  This means that we can separate out providers/task_sdk, core tests as completely different jobs in CI (also it will make it easier to run locally. 2) I introduced the concept of "Group Of Tests" - previously the test types (`All/Core/Other/...`) in the core were mixed with `Helm` or `System` test type and it was very confusing. Now we have test group ('Core, Providers, Integration-Core, Integration-Providers, System-Core, System-Providers, Helm) - and each group has it's own "test types" defined (so "Other/Always etc. " are valid for "Core Test Group", but not valid for "Providers Test Group" - also you cannot specify "Providers*" test type for anything else than one of Providers test group. 3) The "options" for each of the test commands are now much more consistent and definition of groups of those options is now much more composable and logical. Example "providers-tests" set of options that are grouped accorting to test type (db/non-db, providers/core etc). For example DB operations are available for the "general" and "db" test scopes and not available for the "non-db" scope.  4) I am still working on cleaning up and restructuring the code a bit - but generally code for different test groups to produce the right pytest command will be separate "per group" when I finish. 5) the overall idea for the change is to be able to keep the same "selective check" optimizations that allow to run only subset of changes for each change and to allo to run tests in parallel on bigger machines (once we have ARC running cc: @hussein-awala ). 6) Running system tests after this is merged should look this way: ``` breeze testing system-providers-tests ./providers/tests/system/amazon/aws/tests/test_aws_auth_manager.py ``` (plus the usual parameters - env vars etc. configured). Let me know what you think - high level - I will work on fixing the tests and adding missing changes and updating docs accordingly in the meantime. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
