Hello here, Another day - another performance improvement in tests.
I am helping Shahar to test and optimise the self-hosted runner experience to make us independent of the public GitHub Runner queue. Since test parallelism of our tests is **crucial** for faster and cheaper - I finally tackled the problem of "standard" provider DB tests taking by far longest "serialized" time (combination of the tests being "DB tests" and creating/removing a lot of venvs in the process. https://github.com/apache/airflow/pull/68533 - looks small but I **hope** it should solve the problem permanently and improve test speed even without self-hosted runners. It changes all PythonVenv/ExternalEnv tests to be "non-DB tests," meaning they no longer have to run serially. Consequently, we avoid artificially splitting the "standard" provider into smaller "sub-tests," as all these tests will now run in parallel using Xdist. This means parallelism scales automatically and naturally with size of the runner we will run it on. J. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
