potiuk opened a new pull request #14531: URL: https://github.com/apache/airflow/pull/14531
This is by far the biggest improvements of the test execution time we can get now when we are using self-hosted runners. This change drives down the time of executing all tests on self-hosted runners from >60 minutes to <15 minutes due to heavy parallelisation we can implement for different test types and the fact that our machines for self-hosted runners are far more capable - they have more CPU, more memory and the fact that we are using tmpfs for everything. This change will also drive the cost of our self-hosted runners down. Since we have auto-scaling infrastructure we will simply need the machines to run tests for far shorter time. Since the number of test jobs we run on those self hosted runners is substantial (10 jobs), we are going to save ~ 6 build hours per one PR/merged commit! This also allows the developers to use the power of their development machines - if they have bigger number of CPUs, they can also run full test suite of Airflow in a few minutes by setting `RUN_TESTS_IN_PARALLEL` variable to `true`. On one personal PC (64GB RAM, 8 CPUS/16 cores, fast SSD) the full test suite execution went down from 45 minutes to 8 minutes. Another benefit of this change is that output from tests is only shown in full in CI, when there is an error. There is a continuous progress information printed every 10 seconds when either parallel or sequential tests are run, but the full output is only shown for test types which failed - at the end of the output, clearly marked and groupped in "Red" groups so that it will be much easier to see what are the root causes of the test failures. This works by default for both - sequential and parallell test execution, but for sequential execution, it can be chnged to the previous mode, where output is always fully printed during the execution by setting the variable `TEST_SHOW_ALL_OUTPUT_LINES_AS_THEY_APPEAR` to `true` <!-- Thank you for contributing! Please make sure that your code changes are covered with tests. And in case of new features or big changes remember to adjust the documentation. Feel free to ping committers for the review! In case of existing issue, reference it using one of the following: closes: #ISSUE related: #ISSUE How to write a good git commit message: http://chris.beams.io/posts/git-commit/ --> --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information. In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
