On Wed, Nov 27 2019 16:14:44 +0100, Marc Espie wrote: > On Fri, Nov 15, 2019 at 03:29:29PM +0200, Lauri Tirkkonen wrote: > > On Fri, Nov 15 2019 15:24:57 +0200, Lauri Tirkkonen wrote: > > > Your points are valid and I agree with them completely. There are > > > clearly problems with lock contention, > > > > and I should mention here that I would probably not have observed the > > magnitude of these problems had I not made make start more jobs at once > > in the first place. So while the make/mk diffs need more work, we've > > already identified more areas of improvement as a result of trying them > > out. But one thing at a time. :) > > Seriously, as much as what you're doing is interesting, you've not identified > anything new in THAT specific area. We've run enough build configurations > to know about those scalability issues for years.
Fair enough, I was being unfair in assuming otherwise. I guess I wanted to offer some validation for my efforts or something ;) > It's cool that you found new races that are hidden by expensive jobs, though, > that much I fully agree with. I'll give you some more background that I maybe should have given earlier already: in my hobby OS Unleashed, we use bmake and earlier did some (slightly hacky) modifications to subdir.mk to enable paralellizing jobs in multiple subdirs at the same time. I don't remember the exact numbers since it's been quite some time, but I was able to cut the base build time by *a lot* with just that change -- during the build there were previously a lot of idle CPUs. Those results served as motivation for me to start looking into doing similar things in OpenBSD, and having a jobserver is sort of the first step there. -- Lauri Tirkkonen | lotheac @ IRCnet
