potiuk commented on PR #48696: URL: https://github.com/apache/airflow/pull/48696#issuecomment-2786887966
> 1. Instead of adding the complexity to split the list and run more in parallel, have we checked if we can just declare a larger machine which is able to run more in parallel? * This depends on the #44512 - we would need to finally complete our K8S based setup based on ARC - and @hussein-awala still need to complete it :) > 2/ Also if UV needs more time, have we tested if this is really CPU bound? Or wcould we just increase (and over-utilize) CPU and make more speed from the same machine? Are CPU's 100% utilized today with -n 4? The answer to that is really nuanced. Generally speaking. with 4 parallel runs we are pretty much always utilising all CPUs - because we only have 4 CPUS - this is true for tests which are running generally in a single interpreter and generally take 100% CPU. Unless of course there is some I/O contention or active waiting. But from the observation, periods where CPU usage in our tests drops down from 100% is rather rare. It's fully utilized as far as I checked the last time. It's likely a combination of CPU and I/O. Also UV itself is absolutely saturating CPUs and networking when running (big time) - so the parallelisation currently only really speeds up the "test" part - not the UV part - because 4 UVs running in parallell are competing with each other for resources - so increasing parallelism here is not going to help at all - it can even slow down things if the uv's run in parallel will compete even more between themselves. But yes - good idea to check it - and anyone can do it and draw some conclusions -> https://github.com/apache/airflow/pull/48952 - just a matter at looking at the output here https://github.com/apache/airflow/pull/48952 (added `debug ci resources` label. From what I saw, it's just the resolution that takes a lot of time, the "--lowest" check is just What did help - a bit - was the -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
