On 8/2/2013 4:13 PM, Gregory Szorc wrote:
# Header dependency hell
We have hundreds of header files that are included in hundreds or even
thousands of other C++ files. Any time one of these widely-used
headers changes, the object files get invalidated by the build system
dependencies and we have to re-invoke the compiler. This also likely
invalidates ccache, so it's just like a clobber build.
No matter what we do to the build backend to make clobber builds
faster, header dependency hell will continue to undermine this
progress for dependency builds.
I don't believe the build config group is in a position to tackle
header dependency hell at this time. We are receptive to good ideas
and will work with people to land patches. Perhaps an ad-hoc group of
Platform developers can band together to address this?
As a simple hacking project, I put together a simple python script that
analyzed all the #include's of everything and then sorted headers by the
number of files [anything that's not a .h file] that include them
transitively, which is a rough estimate of how many files would be
recompiled if that header changed. The full list is uploaded to bug
901132. A brief highlight of results that may surprise you:
The most included header file is prcpucfg.h, followed closely by prtypes.h.
The most included non-NSPR file is mfbt/Compiler.h at #11, followed by 7
headers terminating in Atomics.h.
#18 is prlog.h
#19 is a three-way tie between mozalloc.h, xpcom-config.h and fallible.h
#23 is xpcom's nsError.h and the other related error headers [tie]
#27 is the venerable nscore.h
#40 is nsrootidl.idl/nsISupports.idl
The first JS header is #86
#115 is jsapi.h
#120 is nsIURI.idl, the first header not coming from something you'd
expect to be included in most places
#121 is nsUnicharUtils.h
#164 is nsWrapperCache.h
#165 is a tie between PSpdyPush3.h and nsILoadGroup.idl
... and at this point the list starts including lots of things that's
not from XPCOM, MFBT, JS, mozalloc, or NSPR.
There are 463 headers included by more than 1000 files, 344 by more than
2000, 140 by more than 3000, 80 by more than 4000, 21 by more than 5000,
and 4 by more than 6000. There are also 765 headers included by more
than 500 files, and 1,563 by more than 100. Note that there are 11,432
files overall by this count, so 13% of our "header" files (there are .c
files in that mix, so I wouldn't call them all header files) would
require rebuilding at least 100 files.
Also, note that this is generated from a debug comm-central build. Since
nsDebug.h [#29] includes prprf.h only in debug builds, and comm-central
includes the LDAP C-SDKs which also interact with NSS and NSPR, the NSPR
numbers in particular are slightly inflated from an opt mozilla-central
build.
[Note: these stats differ slightly from the list I posted, since I
realized that my script was including .idl files in the proxy for
terminal files. There was surprisingly little churn among the top
several files when I fixed that.]
Take aways:
1. Ehsan's dieprtypesdie bug needs to aggressively prune NSPR includes
from XPCOM and JS headers to have any hope of working.
2. IDL files should ideally only need to include a single file: the one
that the interface is a base of. Everything else should be a forward
declaration.
3. There is probably not much gains in trying to trim includes in XPCOM
or MFBT: many of them will get included anyways by virtues of being the
core datatype.
4. Public header files that get really included need to trim down their
includes to only the ones they absolutely need *in the header*. Some of
the heavy-included stuff seems to be coming from an unexpectedly-common
header file including other stuff:
DOMJSProxyHandler.h [2614, #197] includes xpcpublic.h [2645, #188]
includes nsIURI [3295, #121]. DOMJSProxyHandler appears to include
xpcpublic.h solely for IsDOMProxy; xpcpublic.h appears to include nsIURI
because it uses it as an nsCOMPtr in the CompartmentStatsExtras class it
defines. The end result is that touching nsIURI will require us to
rebuild all of the DOM bindings.
--
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
