areusch opened a new pull request #8576: URL: https://github.com/apache/tvm/pull/8576
Enable pytest-xdist to paralellize CI jobs on worker nodes. The thought here is: right now our concurrency model in CI is to keep with 1 CPU to reduce debugging headache and run $(nproc) workers per CI CPU node. Meanwhile our CI is heterogenous so a long ci_cpu-bound step will negatively affect the overall runtime of all PRs. Going the other way, the idea is to run each CI job as fast as possible and let jobs pile into queues where we can see more clearly the bottlenecks. This means that when the queues are drained, devs get faster response times from the CI, and the CPU nodes should still be used optimally (or perhaps get a slight boost since caching may work better with fewer workloads). Restricting to 2 CPUs since this is a test; in the future, CI_PYTEST_NUM_CPUS should be used to actually control from Jenkinsfile. cc @tqchen @jroesch @Lunderberg -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
