potiuk opened a new pull request, #30305:
URL: https://github.com/apache/airflow/pull/30305
Previously all tests for providers were executed as single test type, even
if only subset of provider tests were executed. However when tests for all
providers are executed (for example when main tests are run or when you change
some important files and add new dependencies) "Providers" test run all the
tests for all providers sequentially which makes them longest running test type
by far.
This has the side effect that often for those cases Providers test type is
running alone where all the other test types had already completed. This means
that parallelism of the CI instances is not used to a full extent.
This change leverages the current "parallel" framework of breeze that runs
tests and rather than running single "Providers[...]" test type, it breakes the
Provider test type into separate test type for each provider (if all providers
are tested, this means that there are 70+ provider tests/docker composes
running - not all in parallel, but up to CPU number ones are run in parallel
via pooling mechanism).
This required a number of small changes, that resulted not only faster runs
in CI and better use of multi-CPU instances, but also it allows much easier to
utilise parallelism of local machines of develiopers to run complete set of
Airflow tests using as many CPUS as they have allocated to their docker engine.
The changes include:
* splitting the Provider type into as many providers are being tested
("Providers[nnn]")
* escaping test type to avoid rich treating the provider names as
not-defined markup and allows to display status in coloured form
* pruning of docker-compose networks before, after and during the tests in
order to avoid network exhaustion - each test type has its own docker-compose
created network
* avoiding unnecessary environment checks before each test in case of
parallel test execution (produced unnecessary output)
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
