potiuk edited a comment on pull request #14531: URL: https://github.com/apache/airflow/pull/14531#issuecomment-795322986
> Before I merged this one I added one more change. This one should allow to dynamically choose (using gnu parallel --semaphore) how many tests we run in parallel depending on the number of CPUs (overrideable with `MAX_PARALLEL_TEST_JOBS` . This should allow to get ~20 minutes even with smaller machines I believe. Also it is very likely (testing it in https://github.com/potiuk/airflow/actions/runs/637728789 that even with default 2 CPUs for public GitHub Runners we should be able to achieve ~2x speedup. That would be awesome. Just update on this one: * indeed the parallel semaphore to limit the number of parallel test types works pretty nicely. With the 4CPU, 32GB machines we get ~ 30 minutes tests which is pretty much what I would expect. * The nice thing is that the tests will automatically adjust parallelism level depending on number of CPUs/Cores available which is also great for developers running the tests on they development machines * I almost got it running with ~ 45 minutes tests in the Github Public runners. There are some hangs which I have to look at - likely due to memory usage. But maybe we can limit that further and get it a bit faster * I want to have predictable sequence of the test types. Right now the execution length of the tests varies a lot because parallel semaphore released the jobs in random order and we might have long time where only last, longest job is run. I would rather get the longest jobs start first followed by the shortest ones as this will give bettter parallelism use and predictability. will have to take a look into that. * there are still few tests failing (but there is a different reason for that I will talk to @kaxil about it ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
