On Jun 9, 2005, at 8:01 PM, John Graham-Cumming wrote:
On Thu, 2005-06-09 at 19:11 -0400, Christopher Sean Morrison wrote:
There is already support for processing multiple targets
simultaneously
albeit through some form of locking to ensure that each
dependency/target is being handled by only one of the -j workers (in
that process). That is the simple nature of a -j parallel build. All
I'm suggesting is a means to extend that same locking mechanism on
dependancies being processed by child, sub-process, or otherwise
concurrent makes also.
[snip]
What you are suggesting is, no doubt, possible but you are asking for
locking on a target across independent make processes. Clearly, one
could implement locking of that form (although doing it on every
platform that GNU Make supports might be tricky).
That is indeed what I'd be interested in having. Another alternative
might be to consider the sub-process makes as not being independent,
sharing the exec knowledge of their parent via some mechanism
(environment vars, process id arguments, etc). It would seem more
simple to just handle the more general independent process case, but
considering only sub-process makes and making them dependent or
otherwise state sharing would even be beneficial.
However, I think it's the wrong solution to your problem. It appears
that what you want to do is speed up your build, and you've chosen to
do
that by running a parallel build. If you want to do that then there
are a couple of necessities: you need to make sure that there are no
missing dependencies and that nothing is built twice.
In the cases I have been involved with, those two concerns are
generally not a problem to say the least. Most (including the one in
question) are 'gnu build system' projects where the dependencies are
already in place being described via automake and libtool declarations.
With those declarations in place, the build proceeds just fine with
proper dependency ordering and nothing builds twice. The
double-compile problem that I was working through was clearly
undesirable of course and I'd probably even go so far as to say
building twice is simply 'wrong'. Mind you, I'm referring to the
actual compilation of the source to an object file, not reaching the
Makefile rule and realizing there's nothing to do (e.g. making target
binary foo depend on binary bar's objects and bar itself -- when
processing foo's dependencies, bar's objects are listed explicitly and
indirectly through bar's rule).
The 'problem' is perhaps just a limitation of the make logic being
exported by automake and presumptions on possible target dependancies
(e.g. SUBDIRS, bin_PROGRAMS, lib_LTPROGRAMS, etc are always traversed
in order one at a time). Since it processes each program/library in
order, if there are a lot of 'em (hundreds as there are in my case),
parallel build performance is crippled. It ends up compiling an
object, linking a binary, compiling an object, linking a binary over
and over effectively serializing the build process. All of the objects
are actually independent (and specified as such via the automake rules)
so they can go in parallel, as are all the binaries so they could all
link in parallel so long as the corresponding objects are done.
Minimally, I now know that it 'can' be done, since I just worked in my
own make traversal logic that does just that -- it now compiles all
object files in parallel and then links all binaries together in
parallel.
That said, I was only able to do that by avoiding that separate make
process that automake or autoconf injected on the 'all' target. While
I do heavily concede that it's probably more a design limitation of the
automake/autoconf-generated Makefiles since they could just as easily
provide a similar 'fast' make path like I wrote, I believe that the
cross-process dependency locking would allow much greater potential for
more effective parallel builds without requiring too much hoop-jumping
and customization on the Makefile writer. It would also produce the
exact same end result with no detriment to other parallel or
non-parallel builds (that come to mind).
It should indeed only 'build things once'. I'm suggesting extending
the same logic that exists with the -j option across make instances.
So if make -j2 works, so would running make& twice and it should be
possible to make it execute with practically identical behavior.
Sean
_______________________________________________
Help-make mailing list
[email protected]
http://lists.gnu.org/mailman/listinfo/help-make
