potiuk commented on PR #29223: URL: https://github.com/apache/airflow/pull/29223#issuecomment-1510431945
This one speeds up one of the longest CI jobs we have on `main` / `self-hosted-runners` from 26 minutes to 6 minutes (whoping 5x faster). The `sdist' packages were only instaleld (just for regression) already - we did not run import verification on them (we run import verification for .whl packages only). Installing sdist packages has been always super-slow (and recent `pip` changes made it even slower - because now they always go `sdist -> whl - > install` and the installation cannot be parallelised to use multiple CPUS. However - since we only want to see if provider `sdist` packages can be installed, we can parallelise the installation ourselves - by splitting the list of providers into chunks, and running separate docker container in parallel for each chunk. This way we can install all sdist packages 5 times faster on seff-hosted machine with 8 CPUS (it's not 8x faster because of some common overhead repeated for each on - we install airflow from sdist package in each of the containers. This job is only running for self-hosted runners, so it will have no impact on non-committer PRS (but when we will speed up the remaning Helm Unit test that is very slow as well, this will give faster feedback for commiter PRs and faster builds for main "canary" builds (and some 20 minutes build-time cost for those. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
