As we are trying to increase build parallelism, an important question to answer is whether there are any races to build the same directory from two different places. We thought that it would be nice to see what are all such potential races.
So here is the problem - every time the same directory is entered twice during we would like to see what was the "backtrace" and make targets that caused that. The solution came as a combination of DTrace and Perl. I wrote a little DTrace script: http://www.opensolaris.org/os/project/onnv/onnv_build/faster_builds/tools/dtrace/dirtrace.d and a postprocessing script http://www.opensolaris.org/os/project/onnv/onnv_build/faster_builds/tools/dtrace/dirdump.pl This script was used with sparc ans x86 builds and produced quite interesting results: sparc: http://www.opensolaris.org/os/project/onnv/onnv_build/faster_builds/Observe/build_76_sparc.txt x86: http://www.opensolaris.org/os/project/onnv/onnv_build/faster_builds/Observe/build_76_i386.txt Here is a snippet from the sparc file: -------------------------------------------------------------------------------- Directory Time Spent Target -------------------------------------------------------------------------------- usr/src/uts/sparc/ip: 51 usr/src ksh usr/src/tools/scripts/nightly.sh /export/onnv-76//etc/env usr/src 51s 1h37m30s -e install usr/src/uts 2m15s 1h23m43s install usr/src/uts/sun4v 2m18s 16m16s install usr/src/uts/sun4v/unix 2m24s 5m6s install usr/src/uts/sun4v/unix 2m25s 5m4s install.targ usr/src/uts/sun4v/genunix 3m20s 3m51s all.targ usr/src/uts/sparc/ip 3m21s 1m25s ipctf.debug64 usr/src/uts/sparc/ip 3m22s 1m24s debug64/ipctf.a usr/src/uts/sun4v/genunix 7m30s 1m35s install usr/src/uts/sun4v/genunix 7m31s 1m33s install.targ usr/src/uts/sparc/ip 7m32s 11s ipctf.debug64 usr/src/uts/sparc/ip 7m33s 10s debug64/ipctf.a usr/src/uts/sun4v/generic 10m17s 1m55s install usr/src/uts/sun4v/generic 10m18s 1m55s install.targ usr/src/uts/sun4v/generic 10m18s 1m54s def.targ usr/src/uts/sun4v/unix 10m22s 1m50s symcheck usr/src/uts/sun4v/genunix 10m39s 1m31s all.targ usr/src/uts/sparc/ip 10m40s 10s ipctf.debug64 usr/src/uts/sparc/ip 10m41s 10s debug64/ipctf.a usr/src/uts/sun4v/niagara 12m13s 1m56s install usr/src/uts/sun4v/niagara 12m13s 1m56s install.targ usr/src/uts/sun4v/niagara 12m13s 1m55s def.targ usr/src/uts/sun4v/unix 12m19s 1m49s symcheck usr/src/uts/sun4v/genunix 12m36s 1m31s all.targ usr/src/uts/sparc/ip 12m37s 10s ipctf.debug64 usr/src/uts/sparc/ip 12m37s 10s debug64/ipctf.a ... It shows that we entered uts/sparc/ip 51(!) times and shows how it actually happened. It also shows how much wall time was spent in each directory. For comparison, here is the data with our parallel changes: sparc: http://www.opensolaris.org/os/project/onnv/onnv_build/faster_builds/Observe/build_nitro_sparc.txt x86: http://www.opensolaris.org/os/project/onnv/onnv_build/faster_builds/Observe/build.nitro.i386.txt Following is some analysis of these logs. ** Sparc *** sparc/ip conflicts usr/src/uts/sparc/ip is entered 51 times: sun4v/genunix dependency on ipctf comes from the following rules in sun4v/genunix/Makefile: $(IPCTF_TARGET) ipctf_target: FRC @cd $(IPDRV_DIR); pwd; $(MAKE) ipctf.$(OBJS_DIR) @pwd $(GENUNIX): $(IPCTF_TARGET) $(OBJECTS) $(LD) -r $(LDFLAGS) -o $@ $(OBJECTS) $(CTFMERGE_GENUNIX_MERGE) $(POST_PROCESS) usr/src/uts/sun4v install usr/src/uts/sun4v/unix install usr/src/uts/sun4v/genunix all.targ usr/src/uts/sparc/ip ipctf.debug64 usr/src/uts/sun4v/genunix install usr/src/uts/sparc/ip ipctf.debug64 After that the CPU_KMODS (generic niagara niagara2 vfalls) are built which do symcheck in sun4v/genunix which causes traces like this for each of CPU_KMODS (see Makefile.sun4v/shared). Note that CPU_KMODS are built serially explicitely with .NO_PARALLEL directive. usr/src/uts/sun4v/generic install usr/src/uts/sun4v/generic def.targ usr/src/uts/sun4v/unix symcheck usr/src/uts/sun4v/genunix all.targ usr/src/uts/sparc/ip ipctf.debug64 ... Now the path through sun4u -> sun4u/genunix which have similar rules in sun4u/genunix/Makefile: $(IPCTF_TARGET) ipctf_target: FRC @cd $(IPDRV_DIR); pwd; $(MAKE) ipctf.$(OBJS_DIR) @pwd $(GENUNIX): $(IPCTF_TARGET) $(OBJECTS) $(LD) -r $(LDFLAGS) -o $@ $(OBJECTS) $(CTFMERGE_GENUNIX_MERGE) $(POST_PROCESS) usr/src/uts/sun4u install usr/src/uts/sun4u/unix install usr/src/uts/sun4u/genunix all.targ usr/src/uts/sparc/ip ipctf.debug64 sun4u/unix also wants to build sun4u/genunix (why?) and goes through usr/src/uts/sun4u/genunix install usr/src/uts/sun4u/genunix install.targ usr/src/uts/sparc/ip ipctf.debug64 usr/src/uts/sparc/ip debug64/ipctf.a Again, CPU_KMODS do symcheck in usr/src/uts/sun4u/unix. CPU kmods are cheetah cheetahplus jalapeno serrano spitfire hummingbird. They all go through usr/src/uts/sun4u/cheetah install usr/src/uts/sun4u/cheetah install.targ usr/src/uts/sun4u/cheetah def.targ usr/src/uts/sun4u/unix symcheck usr/src/uts/sun4u/genunix all.targ usr/src/uts/sparc/ip ipctf.debug64 Note that CPU_KMODS are built serially explicitely with .NO_PARALLEL directive, so they do not represent a potential race between them. Now, several sun4u platforms have rules in their sun4u subdirs that always build GENLIB: $(GENLIB): FRC @(cd $(GENLIB_DIR); pwd; $(MAKE) all.targ) @pwd It is rebuilt as a dependency on GENLIB in LIBS. The platforms that do this are opl, serengeti, starcat and starfire. Note that they go to sun4u/genunix twice - once for the install.targ and another time for symcheck: Starfire: usr/src/uts/sun4u install usr/src/uts/sun4u/starfire install usr/src/uts/sun4u/starfire/unix install usr/src/uts/sun4u/starfire/unix install.targ usr/src/uts/sun4u/genunix all.targ usr/src/uts/sparc/ip ipctf.debug64 usr/src/uts/sun4u/starfire/spitfire install usr/src/uts/sun4u/starfire/spitfire install.targ usr/src/uts/sun4u/starfire/spitfire def.targ usr/src/uts/sun4u/starfire/unix symcheck usr/src/uts/sun4u/genunix all.targ usr/src/uts/sparc/ip ipctf.debug64 usr/src/uts/sparc/ip debug64/ipctf.a Serengeti: Serengeti visits sun4u/genunix three times because of cheetah and cheetahplus: usr/src/uts/sun4u install usr/src/uts/sun4u/serengeti install usr/src/uts/sun4u/serengeti/unix install usr/src/uts/sun4u/serengeti/unix install.targ usr/src/uts/sun4u/genunix all.targ usr/src/uts/sparc/ip ipctf.debug64 usr/src/uts/sun4u/serengeti/cheetah install usr/src/uts/sun4u/serengeti/cheetah install.targ usr/src/uts/sun4u/serengeti/cheetah def.targ usr/src/uts/sun4u/serengeti/unix symcheck usr/src/uts/sun4u/serengeti/unix symcheck.targ usr/src/uts/sun4u/genunix all.targ usr/src/uts/sparc/ip ipctf.debug64 usr/src/uts/sun4u/serengeti/cheetahplus install usr/src/uts/sun4u/serengeti/cheetahplus install.targ usr/src/uts/sun4u/serengeti/cheetahplus def.targ usr/src/uts/sun4u/serengeti/unix symcheck usr/src/uts/sun4u/serengeti/unix symcheck.targ usr/src/uts/sun4u/genunix all.targ usr/src/uts/sparc/ip ipctf.debug64 starcat: starcat is similar to serengeti: usr/src/uts/sun4u install usr/src/uts/sun4u/starcat install usr/src/uts/sun4u/starcat/unix install usr/src/uts/sun4u/starcat/unix install.targ usr/src/uts/sun4u/genunix all.targ usr/src/uts/sparc/ip ipctf.debug64 usr/src/uts/sun4u/starcat/cheetah install usr/src/uts/sun4u/starcat/cheetah install.targ usr/src/uts/sun4u/starcat/cheetah def.targ usr/src/uts/sun4u/starcat/unix symcheck usr/src/uts/sun4u/starcat/unix symcheck.targ usr/src/uts/sun4u/genunix all.targ usr/src/uts/sparc/ip ipctf.debug64 usr/src/uts/sun4u/starcat/cheetahplus install usr/src/uts/sun4u/starcat/cheetahplus install.targ usr/src/uts/sun4u/starcat/cheetahplus def.targ usr/src/uts/sun4u/starcat/unix symcheck usr/src/uts/sun4u/starcat/unix symcheck.targ usr/src/uts/sun4u/genunix all.targ usr/src/uts/sparc/ip ipctf.debug64 opl: usr/src/uts/sun4u install usr/src/uts/sun4u/opl install usr/src/uts/sun4u/opl/unix install usr/src/uts/sun4u/opl/unix install.targ usr/src/uts/sun4u/genunix all.targ usr/src/uts/sparc/ip ipctf.debug64 usr/src/uts/sun4u/opl/olympus_c install usr/src/uts/sun4u/opl/olympus_c install.targ usr/src/uts/sun4u/opl/olympus_c def.targ usr/src/uts/sun4u/opl/unix symcheck usr/src/uts/sun4u/opl/unix symcheck.targ usr/src/uts/sun4u/genunix all.targ usr/src/uts/sparc/ip ipctf.debug64 Finally we build uts/sparc which tries to build sparc/ip as well: usr/src/uts/sparc install usr/src/uts/sparc/ip install usr/src/uts/sparc/ip install.targ So we have the following potential race: uts subdirs: sun4v sun4u sparc sun4u platforms: unix opl serengeti starcat starfire unix and CPU_KMODS = { cheetah cheetahplus jalapeno serrano spitfire hummingbird } If CPU_KMODS are built in parallel this creates a race with CPU_KMODS *** sun4u/genassym conflicts sun4u install target has a depoendency on genassym which causes usr/src/uts 2m15s 1h23m43s install usr/src/uts/sun4u 18m32s 53m18s install usr/src/uts/sun4u/genassym 18m33s 7s install usr/src/uts/sun4u/genassym 18m33s 7s def.targ Several sun4u subdirss also have a dependency on genassym via DSF_DIR macro defined in Makefile.sun4u.shared: unix lw2plus lw8 opl serengeti starcat starfire zuluvm usr/src/uts/sun4u/unix 18m40s 3m46s install usr/src/uts/sun4u/unix 18m40s 3m45s install.targ usr/src/uts/sun4u/genassym 18m41s 1s all.targ CPU_KMODS do symcheck in sun4u/unix which goes to genassym: usr/src/uts/sun4u/cheetah 25m54s 2m3s install usr/src/uts/sun4u/cheetah 25m55s 2m3s install.targ usr/src/uts/sun4u/cheetah 25m55s 2m2s def.targ usr/src/uts/sun4u/unix 26m7s 1m50s symcheck usr/src/uts/sun4u/genassym 26m8s 1s all.targ ... Also sparc/dtrace has a dependency on genassym: # # Rule to generate assym.h # $(ASSYM_H): FRC x @cd $(DSF_DIR); $(MAKE) all.targ usr/src/uts/sparc 1h11m49s install usr/src/uts/sparc/dtrace 1h11m50s 4m10s install usr/src/uts/sparc/dtrace 1h11m50s 4m10s install.targ usr/src/uts/sun4u/genassym 1h11m51s 2s all.targ So, unix/genassym races are unix CPU_KMODS and uts/sparc/dtrace *** sun4v/genunix conflicts Similar, for sun4v there is a potential conflict between sun4v/unix and CPU_KMODS. *** starcat/genassym conflicts There is a potential starcat/* conflict over starcat/genassym Similarly there is a potential serengeti/* conflict over serengeti/genassym Same for starfire/* and opl/* There is a conflict over sun4u/darwin platmod from darwin and from quasar since quasar depends on darwin: usr/src/uts/sun4u install usr/src/uts/sun4u/darwin install usr/src/uts/sun4u/darwin/platmod install usr/src/uts/sun4u install usr/src/uts/sun4u/quasar install usr/src/uts/sun4u/darwin install usr/src/uts/sun4u/darwin/platmod install Note that there is no conflict shown for lw8 and serengeti! ** x86 *** intel/ip The intel/ip directory is entered when building i86pc, i86xpv and intel to satisfy dependency on IPCTF target. The first time is when building i86pc: usr/src 2s 13m44s -e install usr/src/uts 17s 9m59s install usr/src/uts/i86pc 17s 3m21s install usr/src/uts/i86pc/unix 20s 2m51s install usr/src/uts/i86pc/unix 21s 1m30s install.targ usr/src/uts/intel/genunix 31s 1m10s all.targ usr/src/uts/intel/ip 32s 22s ipctf.debug64 Second time is the same but for 32-bit i86xpv: usr/src/uts/i86xpv 3m38s 1m28s install usr/src/uts/i86xpv/unix 3m41s 1m15s install usr/src/uts/i86xpv/unix 3m42s 37s install.targ usr/src/uts/intel/genunix 3m54s 13s all.targ usr/src/uts/intel/ip 3m55s 3s ipctf.debug64 intel: usr/src/uts/intel 5m6s 5m9s install usr/src/uts/intel/genunix 5m7s 28s install usr/src/uts/intel/genunix 5m8s 13s install.targ usr/src/uts/intel/ip 5m9s 3s ipctf.debug64 And then, again, when intel directory is built: usr/src/uts/intel 5m36s install usr/src/uts/intel/ip 5m36s 3m55s install usr/src/uts/intel/ip 5m38s 58s install.targ usr/src/uts/intel/ip 6m36s 1m18s install.targ usr/src/uts/intel/ip 7m54s 52s def.targ usr/src/uts/intel/ip 8m45s 43s def.targ So there is a race between i86pc/i86xpv and intel directories when these are compiled in parallel. *** i86pc/genassym usr/src 2s 13m44s -e install usr/src/uts 17s 9m59s install usr/src/uts/i86pc 17s 3m21s install usr/src/uts/i86pc/genassym 17s 3s install usr/src/uts/i86pc/genassym 17s 2s def.targ usr/src/uts/i86pc/genassym 19s 1s def.targ usr/src/uts/i86pc/unix 20s 2m51s install usr/src/uts/i86pc/unix 21s 1m30s install.targ usr/src/uts/i86pc/genassym 21s all.targ usr/src/uts/i86pc/genassym 30s all.targ usr/src/uts/i86pc/unix 1m51s 1m19s install.targ usr/src/uts/i86pc/genassym 1m52s all.targ usr/src/uts/i86pc/genassym 2m all.targ usr/src/uts/intel 6m9s install usr/src/uts/intel/dtrace 6m9s 3m27s install usr/src/uts/intel/dtrace 6m10s 1m43s install.targ usr/src/uts/i86pc/genassym 6m11s 2s all.targ usr/src/uts/intel/dtrace 7m53s 1m44s install.targ usr/src/uts/i86pc/genassym 7m54s 2s all.targ Note that building genassym involves building i86pc/unix. Also there is a race between building i86pc and intel/dtrace *** i86xpv/genassym usr/src ksh usr/src/tools/scripts/nightly.sh /export2/builds/on usr/src 2s 13m44s -e install usr/src/uts 17s 9m59s install usr/src/uts/i86xpv 3m38s 1m28s install usr/src/uts/i86xpv/genassym 3m38s 3s install usr/src/uts/i86xpv/genassym 3m38s 2s def.targ usr/src/uts/i86xpv/genassym 3m40s 2s def.targ usr/src/uts/i86xpv/unix 3m41s 1m15s install usr/src/uts/i86xpv/unix 3m42s 37s install.targ usr/src/uts/i86xpv/genassym 3m42s all.targ usr/src/uts/i86xpv/genassym 3m53s all.targ usr/src/uts/i86xpv/unix 4m20s 36s install.targ usr/src/uts/i86xpv/genassym 4m20s all.targ usr/src/uts/i86xpv/genassym 4m30s all.targ Note that building it tries to rebuild i86xpv/unix. ** Sparc results for Nitro With Nitro changes we can see the following: usr/src/uts 6m6s install usr/src/uts/sun4u 6m6s 32m49s install usr/src/uts/sun4u/unix 6m8s 52s install usr/src/uts/sun4u/unix 6m15s 45s install.targ usr/src/uts/sun4u/cheetah 23m15s 39s install usr/src/uts/sun4u/cheetah 23m16s 38s install.targ usr/src/uts/sun4u/cheetah 23m16s 37s def.targ usr/src/uts/sun4u/unix 23m31s 22s symcheck usr/src/uts/sun4u/cheetahplus 23m54s 45s install usr/src/uts/sun4u/cheetahplus 23m55s 45s install.targ usr/src/uts/sun4u/cheetahplus 23m55s 43s def.targ usr/src/uts/sun4u/unix 24m14s 25s symcheck usr/src/uts/sun4u/jalapeno 24m39s 32s install usr/src/uts/sun4u/jalapeno 24m40s 31s install.targ usr/src/uts/sun4u/jalapeno 24m40s 30s def.targ usr/src/uts/sun4u/unix 24m53s 18s symcheck usr/src/uts/sun4u/serrano 25m11s 32s install usr/src/uts/sun4u/serrano 25m11s 31s install.targ usr/src/uts/sun4u/serrano 25m12s 30s def.targ usr/src/uts/sun4u/unix 25m25s 18s symcheck usr/src/uts/sun4u/spitfire 25m43s 28s install usr/src/uts/sun4u/spitfire 25m43s 28s install.targ usr/src/uts/sun4u/spitfire 25m43s 27s def.targ usr/src/uts/sun4u/unix 25m53s 18s symcheck usr/src/uts/sun4u/hummingbird 26m11s 29s install usr/src/uts/sun4u/hummingbird 26m11s 28s install.targ usr/src/uts/sun4u/hummingbird 26m12s 27s def.targ usr/src/uts/sun4u/unix 26m21s 18s symcheck So cheetah cheetahplus jalapeno serrano spitfire and hummingbird call symcheck target on sun4u/unix. These are CPU_KMODS and are executed in serial, so this is just a waste of time. Same thing for sun4v/unix which causes extra entries into sun4v/genunix: usr/src/uts 6m6s install usr/src/uts/sun4v 6m6s 36m33s install usr/src/uts/sun4v/unix 7m3s 18m27s install usr/src/uts/sun4v/unix 7m17s 18m12s install.targ usr/src/uts/sun4v/genunix 19m24s 5m46s all.targ usr/src/uts/sun4v/genunix 25m29s 2m8s install usr/src/uts/sun4v/genunix 25m31s 2m6s install.targ usr/src/uts/sun4v/generic 34m7s 2m31s install usr/src/uts/sun4v/generic 34m9s 2m29s install.targ usr/src/uts/sun4v/generic 34m11s 2m26s def.targ usr/src/uts/sun4v/unix 34m30s 2m6s symcheck usr/src/uts/sun4v/genunix 34m55s 1m40s all.targ usr/src/uts/sun4v/niagara 36m37s 1m53s install usr/src/uts/sun4v/niagara 36m38s 1m53s install.targ usr/src/uts/sun4v/niagara 36m38s 1m52s def.targ usr/src/uts/sun4v/unix 36m43s 1m47s symcheck usr/src/uts/sun4v/genunix 37m2s 1m27s all.targ usr/src/uts/sun4v/niagara2 38m31s 1m47s install usr/src/uts/sun4v/niagara2 38m31s 1m47s install.targ usr/src/uts/sun4v/niagara2 38m31s 1m46s def.targ usr/src/uts/sun4v/unix 38m37s 1m40s symcheck usr/src/uts/sun4v/genunix 38m54s 1m22s all.targ usr/src/uts/sun4v/vfalls 40m18s 1m47s install usr/src/uts/sun4v/vfalls 40m18s 1m46s install.targ usr/src/uts/sun4v/vfalls 40m18s 1m46s def.targ usr/src/uts/sun4v/unix 40m24s 1m40s symcheck usr/src/uts/sun4v/genunix 40m41s 1m22s all.targ And it also causing extra entries into sun4v.genassym: usr/src/uts 6m6s install usr/src/uts/sun4v 6m6s 36m33s install usr/src/uts/sun4v/genassym 6m6s 57s install usr/src/uts/sun4v/genassym 6m7s 56s def.targ usr/src/uts/sun4v/unix 7m3s 18m27s install usr/src/uts/sun4v/unix 7m17s 18m12s install.targ usr/src/uts/sun4v/genassym 7m33s 16s all.targ usr/src/uts/sun4v/generic 34m7s 2m31s install usr/src/uts/sun4v/generic 34m9s 2m29s install.targ usr/src/uts/sun4v/generic 34m11s 2m26s def.targ usr/src/uts/sun4v/unix 34m30s 2m6s symcheck usr/src/uts/sun4v/genassym 34m33s 1s all.targ usr/src/uts/sun4v/niagara 36m37s 1m53s install usr/src/uts/sun4v/niagara 36m38s 1m53s install.targ usr/src/uts/sun4v/niagara 36m38s 1m52s def.targ usr/src/uts/sun4v/unix 36m43s 1m47s symcheck usr/src/uts/sun4v/genassym 36m44s 1s all.targ usr/src/uts/sun4v/niagara2 38m31s 1m47s install usr/src/uts/sun4v/niagara2 38m31s 1m47s install.targ usr/src/uts/sun4v/niagara2 38m31s 1m46s def.targ usr/src/uts/sun4v/unix 38m37s 1m40s symcheck usr/src/uts/sun4v/genassym 38m38s 1s all.targ usr/src/uts/sun4v/vfalls 40m18s 1m47s install usr/src/uts/sun4v/vfalls 40m18s 1m46s install.targ usr/src/uts/sun4v/vfalls 40m18s 1m46s def.targ usr/src/uts/sun4v/unix 40m24s 1m40s symcheck usr/src/uts/sun4v/genassym 40m25s 1s all.targ *** sparc/ip Sparc/ip is entered three times - during prerequisites build, during sparc build and during sun4u/genunix build: usr/src 18s 54m52s -e install usr/src/uts 52s 41m47s install usr/src/uts/sun4 54s 5m11s install.prereq usr/src/uts/sun4u 54s 5m11s install.prereq usr/src/uts/sun4u/genunix 1m2s 3m54s unknown usr/src/uts/sun4u/genunix 1m4s 3m51s def.targ usr/src/uts/sparc/ip 1m5s 1m24s ipctf.debug64 usr/src/uts/sparc 6m6s install usr/src/uts/sparc/ip 6m6s 4m25s install usr/src/uts/sparc/ip 6m8s 1m58s install.targ usr/src/uts/sparc/ip 8m7s 2m10s def.targ usr/src/uts 6m6s install usr/src/uts/sun4u 6m6s 32m49s install usr/src/uts/sun4u/genunix 7m 13m17s install usr/src/uts/sun4u/genunix 7m6s 13m6s install.targ usr/src/uts/sparc/ip 7m25s 2m42s ipctf.debug64 Note that it causes about 7 minutes of wall time. *** sun4u/darwin/platmod usr/src/uts/sun4u/darwin/platmod: 4 usr/src/uts/sun4u 26m40s install usr/src/uts/sun4u/darwin 26m40s 43s install usr/src/uts/sun4u/darwin/platmod 26m41s 42s install usr/src/uts/sun4u/darwin/platmod 26m42s 41s install.targ usr/src/uts/sun4u 27m23s install usr/src/uts/sun4u/quasar 27m31s 2m59s install usr/src/uts/sun4u/darwin 27m33s 2m52s install usr/src/uts/sun4u/darwin/platmod 27m35s 2m49s install usr/src/uts/sun4u/darwin/platmod 30m16s 9s install.targ shows that sun4u/darwin/platmod is entered when building darwin and quasar platforms. Is it the only dependency of quasar on darwin? - akolb _______________________________________________ tools-discuss mailing list [email protected]
