On 12 April 2018 at 04:23, Ryan Schmidt wrote: > On Apr 11, 2018, at 21:22, Joshua Root wrote: >> On 2018-4-12 11:49 , Ryan Schmidt wrote: >>> >>> These performance problems predate the existence of our >>> getNextBuildOnPortBuilder function. >>> >>> It smells to me like a missing database index problem. I don't know how to >>> determine that though. >> >> I'm guessing we probably run into trouble when the number of pending >> builds exceeds our cachedbuildrequests number. Is that at the default >> 1000 as set in master.cfg? > > Aha! Sure, I don't know why its value would be anything other than what we've > told it to be in master.cfg. So we should increase that to a much higher > number.
Joshua, thank you very much for figuring out what the problem was and saving our buildbot. > Along with maybe some of the other values. You might want to increase the number of builds retained in history. I suspect that currently the builds get deleted after roughly 10.000 new builds come in. I owe a deep apology. I did commit perl updates (with more than 1k ports) a couple of times without ever experience a problem as big as this one. This list seemed super short in comparison, but I didn't anticipate that so many perl subports would be broken and that the list would end up containing more than 1k ports per builder. The queue for 10.13 has finally been digested, so I'll go over the list of failed ports and probably open a ticket to track progress on fixing the issues. Mojca