The GitHub Actions job "Tests" on airflow.git has succeeded. Run started by GitHub user potiuk (triggered by potiuk).
Head commit for run: 18b6e3cff4e9218cf71fca9ba8aff98cb9ee8c22 / Jarek Potiuk <[email protected]> Parallelise provider tests execution in order to speed them up Previously all tests for providers were executed as single test type, even if only subset of provider tests were executed. However when tests for all providers are executed (for example when main tests are run or when you change some important files and add new dependencies) "Providers" test run all the tests for all providers sequentially which makes them longest running test type by far. This has the side effect that often for those cases Providers test type is running alone where all the other test types had already completed. This means that parallelism of the CI instances is not used to a full extent. This change leverages the current "parallel" framework of breeze that runs tests and rather than running single "Providers[...]" test type, it breakes the Provider test type into separate test type for each provider (if all providers are tested, this means that there are 70+ provider tests/docker composes running - not all in parallel, but up to CPU number ones are run in parallel via pooling mechanism). This required a number of small changes, that resulted not only faster runs in CI and better use of multi-CPU instances, but also it allows much easier to utilise parallelism of local machines of develiopers to run complete set of Airflow tests using as many CPUS as they have allocated to their docker engine. The changes include: * splitting the Provider type into as many providers are being tested ("Providers[nnn]") * escaping test type to avoid rich treating the provider names as not-defined markup and allows to display status in coloured form * pruning of docker-compose networks before, after and during the tests in order to avoid network exhaustion - each test type has its own docker-compose created network * avoiding unnecessary environment checks before each test in case of parallel test execution (produced unnecessary output) * amazon provider, general providers and WWW are always put at the beginning of the list if they are in the list - they are by far the longest so by starting them early we give them a chance to not be the last running test type, using parallelism / multiple CPUS more efficiently. Report URL: https://github.com/apache/airflow/actions/runs/4525445347 With regards, GitHub Actions via GitBox --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
