On Sun, May 16, 2021 at 3:38 PM Ryan Schmidt <[email protected]> wrote:
> > > On May 16, 2021, at 09:48, Christopher Nielsen wrote: > >> Upgrading them to six-core Xeons would absolutely help, for sure. But I’m >> quite certain that we could also improve the situation, by reducing the >> level of CPU overcommitment. And reducing the vCPUs per VM would help, as >> we simply don’t have the physical CPU power to support eight/VM. >> > > When you say reduce CPU overcommitment, by what means are you referring > to, other than reducing CPUs per VM? Regarding CPU overcommitment: Are the virtual hosts doing any sort of CPU pinning? Many virtualization products have the ability to specify which of the pCPU cores a guest is allowed to use. As far as I can remember, products like KVM and ESXi can do CPU pinning, while VirtualBox cannot. -- Jason Liu On Sun, May 16, 2021 at 3:38 PM Ryan Schmidt <[email protected]> wrote: > > > On May 16, 2021, at 09:48, Christopher Nielsen wrote: > > > In terms of the ratio of vCPUs to GB of RAM, 1:1 isn’t totally > unreasonable. However, we should also reserve 2 GB of RAM for the OS, > including the disk cache. So perhaps 6 vCPUs would be a better choice. > > MacPorts base hasn't ever considered reserving any RAM for the OS. I > cannot confirm that any changes are needed to the formula of how many jobs > we start based on how much RAM is available, but if there are, then they > should be agreed upon and made in base first; then the buildbot setup can > be adjusted accordingly. > > > > As for the total physical CPUs available on our Xserves, here’s the rub: > While hyperthreading does provide some benefit, best-case it generally only > provides 50% more headroom. And sometimes it’s as low as 25%. > > I'm aware. > > > > So if we assume best-case, our Xserve’s only provide the processing > power of 12 CPU cores, when accounting for hyperthreading. So even if only > two builders are active, we’re already well overcommitted on CPU. And with > three or more going, I’d bet the hypervisor is spending more time on > scheduling and pre-emption, than actual processing time. > > I cannot confirm or deny your claims about the hypervisor. > > > > By way of comparison, I’m running on a modest 2008-era MacPro, with only > eight physical CPU cores… and no hyper threading. Plus the Xeons in my > MacPro are one major generation behind the Nehalem-based CPUs on our > Xserves. Yet, my port build times are anywhere from 2x to 10x faster than > we’re seeing on our builders. (And no, that’s not an exaggeration.) > > Are you looking at the time to build just the port, or are you including > the time to install all the dependencies? Because on your system, you > already have the dependencies installed, or if not, they just need to be > downloaded and installed once. On the buildbot, on the other hand, > dependencies are deactivated between builds, and are sometimes activated > and deactivated many times before the main port is built. This can result > in a significant and in my opinion an unnecessarily large amount of time > being taken for dealing with dependencies first. I believe this can be > improved upon. For example, instead of deactivating *all* active ports > between each build and between each dependency, we could develop a way to > deactive only the active ports that are not needed by the next build or > dependency. This would have the added benefit of reducing disk wear, which > has already been a problem before. See > https://trac.macports.org/ticket/62621 > > It is definitely expected that any individual VM will take longer to build > if other VMs on the same host are also busy. Reducing the number of CPUs > each VM has will just ensure that the builds take longer, even if other VMs > on the same host are not busy. > > > > So we need to do something, as the buildbots simply can’t keep up. > > I think they've been keeping up fine, except when a bunch of huge things > need to be built, which I find understandable. > > > > Upgrading them to six-core Xeons would absolutely help, for sure. But > I’m quite certain that we could also improve the situation, by reducing the > level of CPU overcommitment. And reducing the vCPUs per VM would help, as > we simply don’t have the physical CPU power to support eight/VM. > > When you say reduce CPU overcommitment, by what means are you referring > to, other than reducing CPUs per VM? > >
