On Thu, 2007-03-29 at 10:42 -0700, Mike Pogue wrote:
> > ...
> > this is a an artifact of the extensive use
> > of wildcards in uts/common/Makefile.rules -- there's one wildcard build
> > rule for each source directory. This means that kernel module makefiles
> > take time proportional to the square of the total number of source files
> > to process.
>
> Very interesting! Bill, would it be possible for you to do a quick
> before-and-after test run on a single directory (and post the
> results), so we could see the potential effect of changing this?
so, a full fix here would require a substantial rewrite of the kernel
makefiles (not very amenable to a quick before-and-after test!)
But an idea of the level of payoff can be measured by nuking the rules
that aren't needed for a small module.
In the following, I'm measuring how long it takes dmake to realize it
doesn't actually have to do anything in a "hot cache" situation (where
the incremental build was run recently).
On zhadum.east.sun.com, a 1.2ghz V880, a null build in uts/sparc/ipsec
for a zfs-based workspace takes just under 2s:
% time dmake all; time dmake all; time dmake all; time dmake all
dmake all 1.23s user 0.30s system 86% cpu 1.770 total
dmake all 1.25s user 0.30s system 86% cpu 1.784 total
dmake all 1.22s user 0.30s system 86% cpu 1.766 total
dmake all 1.22s user 0.29s system 85% cpu 1.761 total
(I'm running this on a busy build machine; look at the user+system time
numbers and ignore the wall-clock figures)
All of the *.c files built into uts/sparc/ipsec live in
uts/common/inet/ip; if I blow away all of the *.c->*.o rules in
uts/common/Makefile.rules save the one for
$(UTSBASE)/common/inet/ip/%.c, I get:
% time dmake all; time dmake all; time dmake all; time dmake all
dmake all 0.77s user 0.27s system 70% cpu 1.468 total
dmake all 0.78s user 0.26s system 76% cpu 1.357 total
dmake all 0.78s user 0.26s system 73% cpu 1.423 total
dmake all 0.78s user 0.26s system 66% cpu 1.560 total
But it turns out that we're reading in a total of five other
uts/.../Makefile.rules files:
% truss dmake all |& egrep Makefile.rules | egrep open | sort -u
open64("../../common/Makefile.rules", O_RDONLY) = 7
open64("../../sparc/Makefile.rules", O_RDONLY) = 7
open64("../../sparc/v9/Makefile.rules", O_RDONLY) = 7
open64("../../sun/Makefile.rules", O_RDONLY) = 7
open64("../../sun4/Makefile.rules", O_RDONLY) = 7
open64("../../sun4u/Makefile.rules", O_RDONLY) = 7
applying the same treatment to these as well and we shave off some more
time:
% time dmake all; time dmake all; time dmake all; time dmake all
dmake all 0.66s user 0.26s system 34% cpu 2.647 total
dmake all 0.69s user 0.29s system 66% cpu 1.476 total
dmake all 0.66s user 0.24s system 68% cpu 1.322 total
dmake all 0.68s user 0.24s system 47% cpu 1.922 total
so we go from ~1.52s to ~0.92 in cpu time by nuking those wildcard
rules.
More interestingly, we cut the number of mis-aimed file lookups from
1065 to 25:
before:
% truss -f dmake all |& egrep ENOENT | wc -l
1065
After:
% truss -f dmake all |& egrep ENOENT | wc -l
25
The exact impact varies depending on the filesystem - 1.5x speedup with
local zfs, and 1.8x speedup over NFS. I suspect the impact in a "cold
cache" situation will be substantially greater but that would require a
more carefully constructed test.
I think this has a lot of promise...
So, what could we do instead of these wildcards?
Before we used the wildcards, each source file had its own .c -> .o rule
in the various Makefile.rules files; this was a pain to maintain, so the
wildcard scheme was introduced by
4421489 use implicit rules rather than explicit ones
during development of SunOS 5.9
I don't think we want to go back there. But I have a few ideas..
Because kernel module builds are so stylized, the one I think has the
most promise in the long run is to come up with a higher-level way to
describe which source files go into each of the many kernel modules, and
then translate that into actual makefiles. This would resemble some of
the mechanics of the traditional *BSD "config" (but wouldn't be used for
building monolithic kernels, of course!). This would also let us move
to a "recursive make considered harmful"-style build environment which
will allow for better management of build parallelism and further speed
the make..
- Bill
_______________________________________________
tools-discuss mailing list
tools-discuss@opensolaris.org
