Did you happen to look at any sort of distributed build system? We use 
distributed builds here for WebKit and a full rebuild is pretty reasonable. 
That would negate the need to change how everything is done to get a unity 
build going.

From: webkit-dev 
[mailto:webkit-dev-bounces+don.olmstead=am.sony....@lists.webkit.org] On Behalf 
Of Keith Miller
Sent: Monday, August 28, 2017 5:37 PM
To: WebKit Development Mailing List <webkit-dev@lists.webkit.org>
Subject: [webkit-dev] Growing tired of long build times? Check out this awesome 
new way to speed up your build... soon (HINT: It's not buying a new computer)

Hello fellow Webkittens,

Are you growing tired of long cat naps while waiting 45 minutes for WebKit to 
build? (I’ve definitely never done this 🤐!)
Do you want WebKit to build up to 3-4x faster on a clean build?*
Does seeing your incremental build… stay the same sound good?

You’ll be purring with excitement after you switch to unified source builds!**

* Building JSC with CMake, including CMake configuration time, went from ~8m15s 
to ~2m30s on my MBP using unified sources only on cpp files in JavaScriptCore. 
Final results on WebKit may vary. Using unified sources may cause sudden 
feelings of euphoria and productivity (or grumpiness from lack of naps…). Not 
responsible for any bugs in extra patches produced by using unified source 
builds (excluding build system related bugs) except where required by law 

** Soon. Unified source builds haven’t been landed yet.

How are things going to change? Great question! Let’s find out!

When you fire off the build script it will automagically paw your source files 
into bundles. The size of each bundle is speculatively going to be 8 .cpp files 
but that’s up for experimentation. From my testing so far, the compile time for 
a bundle is at most 20% (~6s vs ~7s for the Air directory) longer than the 
longest file it includes. Given that most of the build time in incremental 
builds is scanning dependencies, this change is probably only barely noticeable.

Bundling happens as part of the CMake configuration process, in the CMake 
build. In the Xcode build it’s more complicated since we cannot dynamically 
choose the files we are going to compile. The only solution I know of is to 
pre-list a bunch of files for the build to dump files into. Occasionally, we’ll 
need to add more files to the build list.

When you add or remove a file in the project, the cost is higher. The current 
plan to bundle source files is by directory. This means that when a .cpp file 
is added or removed you will rebalance the bundles in its directory, and you 
will have to rebuild up to the full directory that file is in.

My goal is to make all of this invisible to developers as much as possible. I 
believe, for the most part, things should operate mostly as they have before. 
As the details on unified source builds get finalized, I’m sending out a 
separate email documenting any workflow changes. I’ll also do my best to answer 
any questions or concerns.

“But Keith, am I going to have to make major changes to the way I develop 
WebKit?” Yes and, hopefully, no. There is one major common workflow change that 
comes with unified sources, the naming of static variables and functions. Since 
C++ does not have a way to only declare a static function or variable in some 
restricted top level scope we are going to need to create some work around. The 
choice I’m thinking we’ll go with is to have an auto-generated macro value 
FILENAME, which is similar to __FILE__ except that it doesn’t have file 
extension (We can’t use FILE since that’s actually a type in C 🙁). This macro 
would be redefined in the unified source directly above the file that is about 
to be included. Then in each cpp file that has static functions/variables 
developers would have:

namespace FILENAME {
            static int myVariable { 0 };
            static int myFunction() { return myVariable; }

The advantage of using a macro over textually writing the filename is that it 
makes code more portable between files. Also, once people get used to the 
concept of FILENAME in the code base it will, hopefully become obvious what it 
means. The main downside to this is using FILENAME in a header would give you 
whatever cpp file first included you (if only we got the compiler magic of 
__FILE__...). We would probably need to prohibit using FILENAME in .h files. 
Unfortunately, you cannot "using FILENAME;” because it will export all the 
values in each subsequent namespace. e.g.

namespace JSC {
    namespace Foo {
            static int myVar { 0 };
    using Foo;

namespace JSC {
    namespace Bar {
            static int myVar { 0 };
    using Bar;

does not compile.

Does anyone have other name recommendations for the macro name other than 
FILENAME? I went with file name since it describes what the substituted value 
will be the best, which is helpful for debugging. The other options I 
considered were FILE_LOCAL and PER_FILE.

Are there other issues that people think might occur with unified sources? 
Note, It’s also likely that there are some source files that need to be 
excluded from the unified source builds for various reasons.

“Is there anything else I need to do with unified sources?”, Yes! Enjoy the 
faster builds!

To help visualize how much of a difference unified sources can make for you, 
I’ve illustrated this helpful graph on the WebKit developer experience:


* Due to size constraints this graph is not to scale.



For those interested there was a long process in deciding how to improve builds:

The first thing I looked at was using C++ modules.

There were a couple of problems with C++ modules. First, different compliers 
support modules in different ways. This was not a major concern since we 
probably would have gone with Clang’s version of modules. As far as I know, 
almost all developers either use clang or could likely switch to clang for 
development. The bigger problem with Clang modules is that it seemed like we 
would have a rough early adopter problem. Additionally, since a lot of Apple’s 
internal SDK’s don’t seem to support modules it was likely to be a major slog 
to get everything working correctly. In a few years I think it will probably be 
worth looking at C++ modules again as, in theory, C++ modules should be better 
than unified sources.

Since we decided against C++ modules that really only leaves us with some form 
of unified sources. Obviously, doing all in one builds would be out of the 
question. While all in ones would have been great for clean builds, it also 
means incremental builds are essentially the same as clean builds. So, I 
concluded that we should go with some kind of automated unified build file 
solution. An automated unified build file solution means that most of the time 
the build system is opaque to developers (well as much as is possible with a 
unified source build).

Since the static variable naming problem was likely to be the biggest issue 
with unified sources, it was important to create a system that made this a pain 
free as possible. There were other options we could go with. We could have a 
script that parses .cpp files and looks for static variables with the same name 
and yells at you. This is pretty annoying and wouldn’t help actual development 
when people hit an error. For example, I know I never run the 
check-webkit-style script until I’m putting a patch up for review.
webkit-dev mailing list

Reply via email to