Thanks for the responses, Dan and Rakesh. You're both absolutely right that a test case is required to demonstrate the problem; I'll need to see if I can put a working example together at a reasonable scale, and I think it's likely that scale itself is part of the problem. Our system uses includes to incorporate around 1500 makefiles (mostly generated from a template) and run them in a single non-recursive make that compiles around 31,000 objects for multiple targets. There are few if any direct dependencies between
In general terms, what I observe is that when we have individual tasks taking more than a second or two Make has no problem keeping up and will maintain around 1000-2000 active jobs, but when jobs are under a second it's only able to keep 10-20 jobs running in parallel, resulting in idle workers. I'll start with a simple test case of defining a makefile with 30,000 null jobs and see how long it takes make to get them all queued up, then start layering in more complexity and see what I can come up with. Thanks also to those who contacted me directly with some alternate sites to post bounties or look for candidates; Here's the list of suggestions in case anyone stumbles across this thread with a similar requirement later: https://freedomsponsors.org/ https://www.bountysource.com/ https://www.fsf.org/resources/jobs On Thu, Oct 2, 2014 at 4:21 AM, Rakesh Sharma <[email protected]> wrote: > From your description what seems to be ailing your build system > is the architecture of your makefile itself. > > Without any clues as to what ails your build (since you didnt provide the > makefile), you need to make your makefile non-recursive in order to reap > the speed dividends. The changes that you talk about (simply-expanded > variables, no shell commands, etc.) go only so far in gaining speed. > > For more help you can look up the Miller paper on non-recursive makefiles > which > explains the reasons for slow/inaccurate builds & also how to overcome > them. > > HTH > > -Rakesh > > > Date: Wed, 1 Oct 2014 13:44:02 -0400 > > Subject: Looking for help improving performance (advice or potential > contract position) > > From: [email protected] > > To: [email protected] > > > > > Greetings all, > > We've been making an effort to optimize our builds and we seem to be > > hitting the limit of what we can accomplish with makefile cleanup. I've > > turned around some significant improvements by using techniques like > > simplifying makefiles where possible, converting recursively-expanded > > variables to simple variables, eliminating shell commands, etc, but we're > > still seeing slow performance from Make itself. > > > > Our build is extremely parallelizable and we run with -j (unlimited) to > run > > as many things as possible in parallel. We use a home-grown job > > distribution tool to send individual tasks to clients on other machines > for > > execution, so there's very little load on the machine running Make > itself. > > What we see is that if we have more than 30 or so clients connected to > the > > build Make is often unable to generate tasks fast enough to keep them > busy. > > > > We brought in Electric Cloud to do an evaluation of their make-compatible > > accelerator tool and found that it was able to instantly saturate as many > > workers as we could throw at it and keep them busy for the duration of > the > > build, cutting the overall time nearly in half. Unfortunately, their > > approach introduces a number of changes that would require significant > > reworking of our build system, and more importantly they want quite a lot > > of money for it (enough to hire additional senior developers for our > team). > > > > Our management doesn't want us taking time to learn the internals of make > > and see if we can improve it ourselves, but they are willing to throw > money > > at the problem, either by offering some sort of donation/bounty for > > performance improvements or directly hiring an expert on a contract > basis; > > any improvements achieved would of course be submitted back to the > > community as patches. > > > > Is there an established place to post this sort of offer? > fossfactory.org > > seems like what I'm looking for, but it doesn't look like it's terribly > > active. > > _______________________________________________ > > Help-make mailing list > > [email protected] > > https://lists.gnu.org/mailman/listinfo/help-make > _______________________________________________ Help-make mailing list [email protected] https://lists.gnu.org/mailman/listinfo/help-make
