Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Xabier Rodríguez Calvar]

Hi,

O Lun, 04-09-2017 ás 10:13 +0200, Xabier Rodríguez Calvar escribiu:
> I have another one. You can deploy IceCC distributed compilation and
> CCache. This will work only for people in offices as they those are
> the
> ones who can share the build time. If any of you comes from Cupertino
> to the Web Engines Hackfest, we can help with the set up here as we
> use
> it (at least me and some other folks at Igalia HQ) extensively.

Just to give you some more context, I just did a clean build of WPE
after having done one of WebKitGTK+ (which explains the preprocessed
cached of 5%) and the numbers are:

 Time: 19m:02s.
 CCache stats:
13 direct hits (0%), 264 preprocessed hits (5%),
5047 misses (95%), 5324 total.

This was a clean build without changing anything in the code:

 Time: 04m:16s.
 CCache stats:
5269 direct hits (99%), 55 preprocessed hits (1%),
0 misses (0%), 5324 total.

I can tell you that more than 50% for those four minutes were devoted
to CMake configuration and IDL generation. CCache + CMake + Ninja =
profit! and if you add IceCC to the equation for non cached builds in
an office environment, even better.

Best regards.

PS: I had even wrote a patch for build-webkit to show CCache stats if
present long time ago but it was rejected so I wrap my builds with a
run.pl script doing that.


signature.asc
Description: This is a digitally signed message part
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Xabier Rodríguez Calvar]

Hi,

Gosh, I go on vacation for a week and I find this "controversial"
thread about builds and another even more controversial about RefPtr
:))

Generally speaking, I have the not formed impression that this will
unleash hell and fire and will affect code readability.

O Lun, 28-08-2017 ás 22:10 -0500, Michael Catanzaro escribiu:
> What, with CMake? That's true of the make backend, but definitely
> not 
> the ninja backend. You should pass -GNinja to get the ninja
> generator. 
> I thought the build-webkit script already did this by default.

> Some alternative strawman proposals:

I have another one. You can deploy IceCC distributed compilation and
CCache. This will work only for people in offices as they those are the
ones who can share the build time. If any of you comes from Cupertino
to the Web Engines Hackfest, we can help with the set up here as we use
it (at least me and some other folks at Igalia HQ) extensively.

And answering Chris about incremental builds, yes, depending on how the
bundling is done (dinamically or staticly per build session) it can
give make incremental builds much slower.

Best regards.

-- 
Xabier Rodríguez Calvar
Software Engineer
http://www.igalia.com


signature.asc
Description: This is a digitally signed message part
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Xabier Rodríguez Calvar]

Hi,

O Mar, 29-08-2017 ás 12:27 -0700, Sam Weinig escribiu:
> This isn’t the scenario I find myself in most often.  A much more
> common scenario is working on a change; touch one or two files, and
> then compile and test/debug.  Rinse and repeat.

When I have to change tasks, sometimes I do a clean build which takes
more or less a couple of minutes if you use CCache (and have everything
cached already which will be true most of the times during regular
work).

Best regards.

-- 
Xabier Rodríguez Calvar
Software Engineer
http://www.igalia.com


signature.asc
Description: This is a digitally signed message part
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Geoffrey Garen]

> - have code completion and symbol navigation work very nicely in Xcode

Unified builds will likely make this work better.

Fun fact about Xcode: The way it does symbol indexing for symbols in a header 
is that it compiles the header with a random .cpp and indexes the result. The 
more stuff in your .cpp file, the more likely that the indexed result will be 
correct! :P

Geoff
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Dan Bernstein]

> On Aug 30, 2017, at 7:33 PM, Dean Jackson  wrote:
> 
> Dan, how did you gather the data below (i.e. how did you count the number of 
> files compiled)?

Number of Compile* instances in the build log.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Dean Jackson]

> On 30 Aug 2017, at 8:16 am, Geoffrey Garen  wrote:
> 
> Interesting.
> 
> The majority cases here are 7 or fewer files. I don’t see much difference 
> between these cases and our existing benchmark for one file, where Keith 
> described the build time delta as "barely noticeable".

Was this a .cpp file or a .h file?

I assume that under a unified system, changing a single .cpp file means more 
total code compiled but the same number of files. Changing a single .h file 
would hopefully mean fewer files compiled, but potentially a lot more total 
code.

Does the "you'll hardly notice the difference" rule apply in both cases?

Dan, how did you gather the data below (i.e. how did you count the number of 
files compiled)?

Dean


> 
> For the minority cases that are 23 - 75 files, these challenge Keith’s 
> description that "most of the build time in incremental builds is scanning 
> dependencies” — assuming that you get unlucky enough for none of the files to 
> bundle together.
> 
> If possible, it would be helpful to know if these files were in the same 
> folders or not.
> 
> Alternatively, we can approximate the answer by benchmarking svn up for 
> individual revisions.
> 
> Geoff
> 
>> On Aug 29, 2017, at 2:21 PM, Dan Bernstein > > wrote:
>> 
>> 
>> 
>>> On Aug 29, 2017, at 1:39 PM, Geoffrey Garen >> > wrote:
>>> 
 I see. The right question to ask would have been how much change occurs in 
 their working copy between consecutive incremental builds.
>>> 
>>> If you want to help make our benchmark righter, please do share any data 
>>> you have about the average content of an incremental build that is distinct 
>>> from a daily svn up.
>> 
>> Here is the data from three WebKit contributors surveyed today. For each 
>> contributor, each line corresponds to a single consecutive incremental build 
>> they’ve performed today, and the number shown is the number of files that 
>> were compiled in that build:
>> 
>> A
>> B
>> C
>> 41
>> 4
>> 1
>> 2
>> 1
>> 1
>> 
>> 1
>> 1
>> 
>> 4
>> 7
>> 
>> 4
>> 58
>> 
>> 5
>> 27
>> 
>> 3
>> 23
>> 
>> 4
>> 61
>> 
>> 5
>> 3
>> 
>> 7
>> 75
>> 
>> 1
>> 2
>> 
>> 6
>> 1
>> 
>> 4
>> 2
>> 
>> 4
>> 1
>> 
>> 4
>> 47
>> 
>> 5
>> 
>> 
>> 3
>> 
>> 
>> I hope this helps. It certainly gives me an idea of what a righter benchmark 
>> would be.
> 
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Keith Miller]

> On Aug 30, 2017, at 7:25 PM, Dean Jackson  wrote:
> 
> 
>> On 29 Aug 2017, at 10:37 am, Keith Miller > > wrote:
>> 
>> Are you growing tired of long cat naps while waiting 45 minutes for WebKit 
>> to build? (I’ve definitely never done this 蘿!)
> 
> Never.

Too Bad!

> 
>> Do you want WebKit to build up to 3-4x faster on a clean build?*
> 
> Of course.
> 
>> Does seeing your incremental build… stay the same sound good?
> 
> Yes!
> 
> I don't have any knowledge of build systems or how to make things faster, but 
> I do want to mention things that are important to my workflow that I hope 
> this change won't make worse. I'm a little worried that this appears to be 
> moving away from Xcode in a manner that will make development more difficult. 
> Maybe I've misunderstood some of the suggestions.
> 
> I want to:
> 
> - be able to do most (all) of my editing in Xcode

Yeah, but Eww.

> - have code completion and symbol navigation work very nicely in Xcode

Yep.

> - build and run from Xcode

Yep.

> - build and run from Xcode much faster than I can today

That’s the hope, there is some concern around incremental builds but I believe 
this should work.

> - use Xcode's debugger UI

Yep, modulo maybe the Radars Simon linked.

> - sometimes have just a single project open in Xcode (e.g. WebCore) and build 
> only that part (so I can skip it even having to think about other parts)

I think we will need to support this for Internal Build reasons.

> 
> Will the new magic build system offer these? Will it break any of them?
> 
> If calling build can mean a new Xcode project is created, will I be able to 
> open it and have a nice project structure, with the original files I want to 
> edit, not the unified sources.

That’s not the plan of this project. I’m not sure how unified sources will 
interact Xcode when the CMake only build system is in place. We should make 
sure this is still sane whenever we do the CMake only build.

Cheers,
Keith

> 
> Dean
> 

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Dean Jackson]

> On 29 Aug 2017, at 10:37 am, Keith Miller  wrote:
> 
> Are you growing tired of long cat naps while waiting 45 minutes for WebKit to 
> build? (I’ve definitely never done this 蘿!)

Never.

> Do you want WebKit to build up to 3-4x faster on a clean build?*

Of course.

> Does seeing your incremental build… stay the same sound good?

Yes!

I don't have any knowledge of build systems or how to make things faster, but I 
do want to mention things that are important to my workflow that I hope this 
change won't make worse. I'm a little worried that this appears to be moving 
away from Xcode in a manner that will make development more difficult. Maybe 
I've misunderstood some of the suggestions.

I want to:

- be able to do most (all) of my editing in Xcode
- have code completion and symbol navigation work very nicely in Xcode
- build and run from Xcode
- build and run from Xcode much faster than I can today
- use Xcode's debugger UI
- sometimes have just a single project open in Xcode (e.g. WebCore) and build 
only that part (so I can skip it even having to think about other parts)

Will the new magic build system offer these? Will it break any of them?

If calling build can mean a new Xcode project is created, will I be able to 
open it and have a nice project structure, with the original files I want to 
edit, not the unified sources.

Dean




___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Keith Miller]

> On Aug 29, 2017, at 5:54 PM, Sam Weinig  wrote:
> 
> 
> 
>> On Aug 29, 2017, at 12:46 PM, Geoffrey Garen > > wrote:
>> 
>>> This isn’t the scenario I find myself in most often.  A much more common 
>>> scenario is working on a change; touch one or two files, and then compile 
>>> and test/debug.  Rinse and repeat.
>> 
>> We’ve already tested this case. The worst case slowdown, if you touch a 
>> small file that's in the same bundle as the biggest .cpp file in the 
>> project, is 6s => 7s (20%).
>> 
>> Geoff
> 
> I see larger than ~6 second build times with this scenario, not measure 
> scientifically, but I would approximate it more around 20 - 30 seconds. Do 
> you expect the 20% to scale linearly?

It depends on how things are bundled. Operating under the assumption, which 
needs to be verified, that all the .cpp files in a bundle include *roughly* the 
same, I would expect this to scale sub-linearly. In WebCore+ I expect headers 
make up an even larger percentage of a translation unit so if I had to guess it 
would be an even smaller difference. This is something that I’m planning on 
testing as I proceed with rolling out unified source builds.

Cheers,
Keith

> 
> 
> In a completely other direction, what does this mean for use of Xcode? Can we 
> still build from Xcode? Debug?
> 
> - Sam
> 
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Keith Miller]

> On Aug 30, 2017, at 1:01 PM, Alexey Proskuryakov  wrote:
> 
> 
> 
>> 30 авг. 2017 г., в 11:55, Andy Estes > > написал(а):
>> 
> In a completely other direction, what does this mean for use of Xcode? 
> Can we still build from Xcode? Debug?
 
 CMake can generate Xcode files, so you can still develop and debug in 
 Xcode.
>>> 
>>> This annihilates speed up possibility, as xcodebuild will be used instead 
>>> of ninja
>> 
>> I don’t think so. I believe the generated Xcode project uses CMake when 
>> building.
> 
> 
> Per Keith's original e-mail, the plan is to add native support for unified 
> build in WebKit Xcode projects, not relying on CMake in any way.
> 
> I have a separate question about the unified build plan though. Are we clear 
> on how it will work with using declarations and directives in the global 
> scope of .cpp and .mm files? We use these a lot, all over the place.

One proposal is we have the requirement that all the source files in a 
directory must top level ‘using' the same things. I suppose this could be per 
file extension too since those won’t be bundled together. I don’t know of 
another solution other than excluding specific files from the unified build. I 
guess we could require that they are the last file in any bundle they get 
placed in... D:

It’s pretty sad that C++’s ability to do ‘using’ in a specific scope is so weak.

> 
> - Alexey
> 
> 
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Alexey Proskuryakov]

> 30 авг. 2017 г., в 11:55, Andy Estes  написал(а):
> 
 In a completely other direction, what does this mean for use of Xcode? Can 
 we still build from Xcode? Debug?
>>> 
>>> CMake can generate Xcode files, so you can still develop and debug in Xcode.
>> 
>> This annihilates speed up possibility, as xcodebuild will be used instead of 
>> ninja
> 
> I don’t think so. I believe the generated Xcode project uses CMake when 
> building.


Per Keith's original e-mail, the plan is to add native support for unified 
build in WebKit Xcode projects, not relying on CMake in any way.

I have a separate question about the unified build plan though. Are we clear on 
how it will work with using declarations and directives in the global scope of 
.cpp and .mm files? We use these a lot, all over the place.

- Alexey


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Andy Estes]

> On Aug 30, 2017, at 11:54 AM, Konstantin Tokarev  wrote:
> 
> 
> 
> 30.08.2017, 21:05, "Alex Christensen" :
>>>  On Aug 29, 2017, at 5:54 PM, Sam Weinig  wrote:
>>>  In a completely other direction, what does this mean for use of Xcode? Can 
>>> we still build from Xcode? Debug?
>> 
>> CMake can generate Xcode files, so you can still develop and debug in Xcode.
> 
> This annihilates speed up possibility, as xcodebuild will be used instead of 
> ninja

I don’t think so. I believe the generated Xcode project uses CMake when 
building.

Andy

> 
>> 
>>>  On Aug 29, 2017, at 5:37 PM, Carlos Alberto Lopez Perez 
>>>  wrote:
>>>  Does this means that Apple ports are going to switch to CMake?
>> 
>> We have not decided anything officially yet, and if we were to decide to do 
>> this then it would take quite a while to make the official switch.
>> 
>> ___
>> webkit-dev mailing list
>> webkit-dev@lists.webkit.org
>> https://lists.webkit.org/mailman/listinfo/webkit-dev
> 
> -- 
> Regards,
> Konstantin
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Konstantin Tokarev]

30.08.2017, 21:05, "Alex Christensen" :
>>  On Aug 29, 2017, at 5:54 PM, Sam Weinig  wrote:
>>  In a completely other direction, what does this mean for use of Xcode? Can 
>> we still build from Xcode? Debug?
>
> CMake can generate Xcode files, so you can still develop and debug in Xcode.

This annihilates speed up possibility, as xcodebuild will be used instead of 
ninja

>
>>  On Aug 29, 2017, at 5:37 PM, Carlos Alberto Lopez Perez  
>> wrote:
>>  Does this means that Apple ports are going to switch to CMake?
>
> We have not decided anything officially yet, and if we were to decide to do 
> this then it would take quite a while to make the official switch.
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

-- 
Regards,
Konstantin
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Alex Christensen]

> On Aug 29, 2017, at 5:54 PM, Sam Weinig  wrote:
> In a completely other direction, what does this mean for use of Xcode? Can we 
> still build from Xcode? Debug?
CMake can generate Xcode files, so you can still develop and debug in Xcode.

> On Aug 29, 2017, at 5:37 PM, Carlos Alberto Lopez Perez  
> wrote:
> Does this means that Apple ports are going to switch to CMake?
We have not decided anything officially yet, and if we were to decide to do 
this then it would take quite a while to make the official switch.

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Maciej Stachowiak]

> On Aug 30, 2017, at 2:15 AM, Konstantin Tokarev  wrote:
> 
> 
> 
> 30.08.2017, 03:01, "Michael Catanzaro" :
>> On Tue, Aug 29, 2017 at 6:20 PM, Alicia Boya García 
>> wrote:
>>>  As long as the sets are generated consistently that approach should be
>>>  equally fine and indeed it would reduce the burden for contributors.
>>>  You
>>>  may need at least a file to write exceptions (files not to be included
>>>  in any bundle) though.
>>> 
>>>  -- Alicia.
>> 
>> I agree, with the style checker to check for naming conflicts within
>> the same directory, and with some help from EWS, automatic generation
>> of bundles should work out fine.
> 
> Note that EWS and webkit-patch run check-webkit-style over *diff* only,
> so it won't be possible to detect such naming conflcts outside of patch
> boundaries with check-webkit-style
> 
> 

To be fully effective, we'd have to have a checker script that looks at the 
source tree too, at least files in the same directory as any touched by the 
patch.

 - Maciej

> 
>> 
>> I'm glad we've discarded the original namespace proposal.
>> 
>> Michael
>> 
>> ___
>> webkit-dev mailing list
>> webkit-dev@lists.webkit.org
>> https://lists.webkit.org/mailman/listinfo/webkit-dev
> 
> -- 
> Regards,
> Konstantin
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Konstantin Tokarev]

30.08.2017, 03:01, "Michael Catanzaro" :
> On Tue, Aug 29, 2017 at 6:20 PM, Alicia Boya García 
> wrote:
>>  As long as the sets are generated consistently that approach should be
>>  equally fine and indeed it would reduce the burden for contributors.
>>  You
>>  may need at least a file to write exceptions (files not to be included
>>  in any bundle) though.
>>
>>  -- Alicia.
>
> I agree, with the style checker to check for naming conflicts within
> the same directory, and with some help from EWS, automatic generation
> of bundles should work out fine.

Note that EWS and webkit-patch run check-webkit-style over *diff* only,
so it won't be possible to detect such naming conflcts outside of patch
boundaries with check-webkit-style

>
> I'm glad we've discarded the original namespace proposal.
>
> Michael
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

-- 
Regards,
Konstantin
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Maciej Stachowiak]

> On Aug 29, 2017, at 6:40 PM, Sam Weinig  wrote:
> 
> 
> 
>> On Aug 29, 2017, at 6:36 PM, Maciej Stachowiak > > wrote:
>> 
>> 
>> 
>>> On Aug 29, 2017, at 5:54 PM, Sam Weinig >> > wrote:
>>> 
>>> 
>>> 
 On Aug 29, 2017, at 12:46 PM, Geoffrey Garen > wrote:
 
> This isn’t the scenario I find myself in most often.  A much more common 
> scenario is working on a change; touch one or two files, and then compile 
> and test/debug.  Rinse and repeat.
 
 We’ve already tested this case. The worst case slowdown, if you touch a 
 small file that's in the same bundle as the biggest .cpp file in the 
 project, is 6s => 7s (20%).
 
 Geoff
>>> 
>>> I see larger than ~6 second build times with this scenario, not measure 
>>> scientifically, but I would approximate it more around 20 - 30 seconds. Do 
>>> you expect the 20% to scale linearly?
>> 
>> Is that just compile time or total build time including build system 
>> overhead? I found that for single-file builds the compile step is less than 
>> half the total time.
> 
> 
> I’m not entirely sure. I usually don’t pay attention to what is happening 
> during the compile.  From doing one right now, it seems to be about 50-50 for 
> one small file.

I've tried to reverse engineer this by comparing compile time after touching 
one file to time after touching two and it seems to support ~50%. Given that, 
I'd estimate the impact to a 20-30 second total time at around 1-2 seconds. 
With more files, your probability of a net win increases. If two files in the 
same AllInOne get changed, you save enough to make up for four singletons 
slowing down.

I agree that the single-file incremental time is already not great but I think 
adopting cmake+ninja as the normal way to build is the only easy way to resolve 
that. Maybe we could even consider making the Xcode project invoke the 
cmake/ninja build instead of doing the Xcode thing at some point, to maximally 
help people who build in the GUI.

 - Maciej

> 
> - Sam
> 
>> 
>> 
>>> 
>>> 
>>> In a completely other direction, what does this mean for use of Xcode? Can 
>>> we still build from Xcode? Debug?
>>> 
>>> - Sam
>>> 
>>> ___
>>> webkit-dev mailing list
>>> webkit-dev@lists.webkit.org 
>>> https://lists.webkit.org/mailman/listinfo/webkit-dev 
>>> 
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Sam Weinig]

> On Aug 29, 2017, at 6:36 PM, Maciej Stachowiak  wrote:
> 
> 
> 
>> On Aug 29, 2017, at 5:54 PM, Sam Weinig > > wrote:
>> 
>> 
>> 
>>> On Aug 29, 2017, at 12:46 PM, Geoffrey Garen >> > wrote:
>>> 
 This isn’t the scenario I find myself in most often.  A much more common 
 scenario is working on a change; touch one or two files, and then compile 
 and test/debug.  Rinse and repeat.
>>> 
>>> We’ve already tested this case. The worst case slowdown, if you touch a 
>>> small file that's in the same bundle as the biggest .cpp file in the 
>>> project, is 6s => 7s (20%).
>>> 
>>> Geoff
>> 
>> I see larger than ~6 second build times with this scenario, not measure 
>> scientifically, but I would approximate it more around 20 - 30 seconds. Do 
>> you expect the 20% to scale linearly?
> 
> Is that just compile time or total build time including build system 
> overhead? I found that for single-file builds the compile step is less than 
> half the total time.


I’m not entirely sure. I usually don’t pay attention to what is happening 
during the compile.  From doing one right now, it seems to be about 50-50 for 
one small file.

- Sam

> 
> 
>> 
>> 
>> In a completely other direction, what does this mean for use of Xcode? Can 
>> we still build from Xcode? Debug?
>> 
>> - Sam
>> 
>> ___
>> webkit-dev mailing list
>> webkit-dev@lists.webkit.org 
>> https://lists.webkit.org/mailman/listinfo/webkit-dev 
>> 
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Maciej Stachowiak]

> On Aug 29, 2017, at 5:54 PM, Sam Weinig  wrote:
> 
> 
> 
>> On Aug 29, 2017, at 12:46 PM, Geoffrey Garen > > wrote:
>> 
>>> This isn’t the scenario I find myself in most often.  A much more common 
>>> scenario is working on a change; touch one or two files, and then compile 
>>> and test/debug.  Rinse and repeat.
>> 
>> We’ve already tested this case. The worst case slowdown, if you touch a 
>> small file that's in the same bundle as the biggest .cpp file in the 
>> project, is 6s => 7s (20%).
>> 
>> Geoff
> 
> I see larger than ~6 second build times with this scenario, not measure 
> scientifically, but I would approximate it more around 20 - 30 seconds. Do 
> you expect the 20% to scale linearly?

Is that just compile time or total build time including build system overhead? 
I found that for single-file builds the compile step is less than half the 
total time.


> 
> 
> In a completely other direction, what does this mean for use of Xcode? Can we 
> still build from Xcode? Debug?
> 
> - Sam
> 
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Sam Weinig]

> On Aug 29, 2017, at 12:46 PM, Geoffrey Garen  wrote:
> 
>> This isn’t the scenario I find myself in most often.  A much more common 
>> scenario is working on a change; touch one or two files, and then compile 
>> and test/debug.  Rinse and repeat.
> 
> We’ve already tested this case. The worst case slowdown, if you touch a small 
> file that's in the same bundle as the biggest .cpp file in the project, is 6s 
> => 7s (20%).
> 
> Geoff

I see larger than ~6 second build times with this scenario, not measure 
scientifically, but I would approximate it more around 20 - 30 seconds. Do you 
expect the 20% to scale linearly?


In a completely other direction, what does this mean for use of Xcode? Can we 
still build from Xcode? Debug?

- Sam

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Carlos Alberto Lopez Perez]

On 29/08/17 02:37, Keith Miller wrote:
> Bundling happens as part of the CMake configuration process, in the
> CMake build. In the Xcode build it’s more complicated since we cannot
> dynamically choose the files we are going to compile. The onlysolution
> I know of is to pre-list a bunch of files for the build to dump files
> into. Occasionally, we’ll need to add more files to the build list.

If I understood the proposal right, to benefit from this unified source
feature the developers have to use CMake.

Does this means that Apple ports are going to switch to CMake?

If this switch to CMake is going to be optional, then I only foresee
constant breakage and headaches for those that choose to use the
non-default option (the one not tested by the EWS and release bots).



signature.asc
Description: OpenPGP digital signature
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Michael Catanzaro]

On Tue, Aug 29, 2017 at 3:22 PM, Olmstead, Don  
wrote:
Sure but in a corporate environment it can be a really big win. All 
of our developers have their machines hooked up to the distributed 
build system and we also have servers hanging out that can be used in 
the build. This sort of thing doesn't require anyone to change how 
they write their code which is something a unity build ends up 
requiring.


In terms of WebKit development the only places it falls apart are in 
the source code generation and the link.


Really I was just curious if anything like that was discussed because 
all of this requires pretty big changes to source code. Additionally 
there's other fun things that can happen around includes when doing a 
unity build where a file will stop being able to be built in 
isolation. The same thing has happened with WebCorePrefix where a 
number of files can't be built without it.


I think Keith's new plan for unity builds, which doesn't require major 
changes to our source code, should work out fine, and hopefully save us 
from investing in Threadrippers


Now, distributed builds are a great idea. We use them at Igalia. A 
couple years ago, when we had enough laptops in the office at the same 
time, I saw a clean build of WebKit performed in under five minutes! 
The main limitation is they are not useful for individual contributors. 
GNOME developers complain all the time about how long it takes to build 
WebKit. Distributed builds won't help there. They also don't help 
distributed workers. Most of us at Igalia work from home and only 
benefit from distributed builds when visiting our office.


We discussed distributed builds at the Contributor's Meeting two years 
ago. I believe the main limitation at the time was that they were not 
well-supported by XCode, and many Apple developers had only limited 
interest at the time in switching to CMake. Since the unified build 
proposal also requires CMake, it sounds like that situation may have 
changed. I would certainly recommend that Apple consider distributed 
builds *in addition* to switching to unified builds.


And then we can delete those XCode projects and have one build system 
to rule them all. Muahahaha! Evil laughter!


Michael

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Keith Miller]

> On Aug 29, 2017, at 4:20 PM, Alicia Boya García  wrote:
> 
> On 08/30/2017 12:00 AM, Keith Miller wrote:
> 
>> The automated system should handle your concern about bundling similar .cpp 
>> files together. The proposed system bundles by directory and also sorts the 
>> files in those directories when dividing up the files. Is there any reason 
>> to think that we would not do roughly the same thing if we were bundling by 
>> hand?
> 
> As long as the sets are generated consistently that approach should be
> equally fine and indeed it would reduce the burden for contributors. You
> may need at least a file to write exceptions (files not to be included
> in any bundle) though.

Yeah, that’s probably inevitable. We’ll probably have them both for build 
correctness issues and for build time issues (or in the case of 
B3LowerToAir.cpp both).

> 
> -- Alicia.
> 

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Alicia Boya García]

On 08/30/2017 12:00 AM, Keith Miller wrote:

> The automated system should handle your concern about bundling similar .cpp 
> files together. The proposed system bundles by directory and also sorts the 
> files in those directories when dividing up the files. Is there any reason to 
> think that we would not do roughly the same thing if we were bundling by hand?

As long as the sets are generated consistently that approach should be
equally fine and indeed it would reduce the burden for contributors. You
may need at least a file to write exceptions (files not to be included
in any bundle) though.

-- Alicia.

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Dan Bernstein]

> On Aug 29, 2017, at 3:16 PM, Geoffrey Garen  wrote:
> 
> Interesting.
> 
> The majority cases here are 7 or fewer files. I don’t see much difference 
> between these cases and our existing benchmark for one file, where Keith 
> described the build time delta as "barely noticeable".
> 
> For the minority cases that are 23 - 75 files, these challenge Keith’s 
> description that "most of the build time in incremental builds is scanning 
> dependencies” — assuming that you get unlucky enough for none of the files to 
> bundle together.
> 
> If possible, it would be helpful to know if these files were in the same 
> folders or not.
> 
> Alternatively, we can approximate the answer by benchmarking svn up for 
> individual revisions.

All the evidence that we have so far doesn’t support that such a benchmark 
would be a good approximation, so that’s a bad alternative to gathering more 
evidence.

> 
> Geoff
> 
>> On Aug 29, 2017, at 2:21 PM, Dan Bernstein > > wrote:
>> 
>> 
>> 
>>> On Aug 29, 2017, at 1:39 PM, Geoffrey Garen >> > wrote:
>>> 
 I see. The right question to ask would have been how much change occurs in 
 their working copy between consecutive incremental builds.
>>> 
>>> If you want to help make our benchmark righter, please do share any data 
>>> you have about the average content of an incremental build that is distinct 
>>> from a daily svn up.
>> 
>> Here is the data from three WebKit contributors surveyed today. For each 
>> contributor, each line corresponds to a single consecutive incremental build 
>> they’ve performed today, and the number shown is the number of files that 
>> were compiled in that build:
>> 
>> A
>> B
>> C
>> 41
>> 4
>> 1
>> 2
>> 1
>> 1
>> 
>> 1
>> 1
>> 
>> 4
>> 7
>> 
>> 4
>> 58
>> 
>> 5
>> 27
>> 
>> 3
>> 23
>> 
>> 4
>> 61
>> 
>> 5
>> 3
>> 
>> 7
>> 75
>> 
>> 1
>> 2
>> 
>> 6
>> 1
>> 
>> 4
>> 2
>> 
>> 4
>> 1
>> 
>> 4
>> 47
>> 
>> 5
>> 
>> 
>> 3
>> 
>> 
>> I hope this helps. It certainly gives me an idea of what a righter benchmark 
>> would be.
> 

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Geoffrey Garen]

Interesting.

The majority cases here are 7 or fewer files. I don’t see much difference 
between these cases and our existing benchmark for one file, where Keith 
described the build time delta as "barely noticeable".

For the minority cases that are 23 - 75 files, these challenge Keith’s 
description that "most of the build time in incremental builds is scanning 
dependencies” — assuming that you get unlucky enough for none of the files to 
bundle together.

If possible, it would be helpful to know if these files were in the same 
folders or not.

Alternatively, we can approximate the answer by benchmarking svn up for 
individual revisions.

Geoff

> On Aug 29, 2017, at 2:21 PM, Dan Bernstein  wrote:
> 
> 
> 
>> On Aug 29, 2017, at 1:39 PM, Geoffrey Garen > > wrote:
>> 
>>> I see. The right question to ask would have been how much change occurs in 
>>> their working copy between consecutive incremental builds.
>> 
>> If you want to help make our benchmark righter, please do share any data you 
>> have about the average content of an incremental build that is distinct from 
>> a daily svn up.
> 
> Here is the data from three WebKit contributors surveyed today. For each 
> contributor, each line corresponds to a single consecutive incremental build 
> they’ve performed today, and the number shown is the number of files that 
> were compiled in that build:
> 
> A
> B
> C
> 41
> 4
> 1
> 2
> 1
> 1
> 
> 1
> 1
> 
> 4
> 7
> 
> 4
> 58
> 
> 5
> 27
> 
> 3
> 23
> 
> 4
> 61
> 
> 5
> 3
> 
> 7
> 75
> 
> 1
> 2
> 
> 6
> 1
> 
> 4
> 2
> 
> 4
> 1
> 
> 4
> 47
> 
> 5
> 
> 
> 3
> 
> 
> I hope this helps. It certainly gives me an idea of what a righter benchmark 
> would be.

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Keith Miller]

As far as the static renaming problem, Darin convinced me in the other thread 
that we probably don’t need such a heavy handed approach. At least in JSC, 
there are a number of variables that have generic names that should be put into 
a namespace but the majority probably don’t need to be renamed. I’ll gather 
some stats on name collisions and report back soon. Since my proposal will only 
bundle files by directory, we could mitigate global name collisions by having a 
bot build with each file fully unified. As Saam mentioned before, we should 
probably also have a bot that runs without unified source builds so we don’t 
end up relying on includes from other .cpp files.

I’m not convinced statically bundling is the right approach. By automatically, 
choosing the bundles we don’t require contributors to understand the 
intricacies of how we should load balance our unified source build system. New 
files are quite common and ensuring that people bundle their new files in a 
reasonable manner seems like it would need to be a continual effort, which I 
don’t want to volunteer to do. We could make a script that does this but then 
why not just run that script all the time? There are also likely to be a huge 
number of bundles, for JSC there are ~120 so would that mean there are 120 
checked in unified source files? If not, then I assume the list will be parsed 
by a script and files would be emitted it seems like that script might as well 
automatically decide how to bundle those files. As I described above, I think 
the bot solution should prevent people getting build errors that are not 
reproducible when the add a new file. 

The automated system should handle your concern about bundling similar .cpp 
files together. The proposed system bundles by directory and also sorts the 
files in those directories when dividing up the files. Is there any reason to 
think that we would not do roughly the same thing if we were bundling by hand?

Cheers,
Keith

> On Aug 29, 2017, at 1:53 PM, Alicia Boya García  wrote:
> 
> I'm all for stable bundle .cpp files. Automatic generation can be nice
> for the first time... and once in while we can try to modify the bundles
> to try to make the build even faster, but really most of the time bundle
> contents should not change unless new files are added and everybody
> should get the same bundles.
> 
> If we're going to end up with some name collisions, we'd rather do so
> that every build of the same source tree gets collisions or no build
> gets them.
> 
> The description of these .cpp bundles should become a project artifact,
> shared and versioned in the repository and treated carefully as such.
> 
> Also, in my earlier message about unified builds I explained how the
> distribution of the files within the bundles also affects the number of
> bundles that are rebuilt in response to changes in the source code. I
> also proposed that grouping together files implementing the same
> features could be a good start point to minimize the impact of this issue.
> 
> On other note, I don't think namespacing static functions and variables
> as originally proposed is the way to go.
> 
> The refactor would be ridiculously huge. Running `rg -w '^static' -tcpp
> --no-filename --no-heading |wc -l` in my local repository yields 16797
> instances of static functions and variables, only taking into account
> those where the word static is at the beginning of the line without any
> spaces before, which is a common case but not a requirement in C++. Note
> though that this regex excludes static class members as long as code is
> properly formatted (they would be indented); these are not included as
> they would not need any changes.
> 
> static functions and variables in WebKit are used for hidden
> implementation details, which tend to very very specific and therefore
> receive long specific names with relatively little risk of collisions.
> 
> Furthermore, collisions won't go unnoticed as in every case they will
> trigger a compilation error of the likes of 'redefinition of ‘int a’'.
> In this case the compiler will also show the inclusion stack mentioning
> the bundle file and the conflicting definition. The programmer will
> notice that both are static members in .cpp files, so the only way that
> error can arise is because they had a name collision due to unified
> source builds.
> 
> We'd probably be better off handling the few collisions on a one by one
> basis, renaming the symbols or excluding the worst offenders from
> unified builds rather than plain out forbidding static declarations.
> 
> -- Alicia

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Konstantin Tokarev]

29.08.2017, 14:44, "Alicia Boya García" :
> On 08/29/2017 06:20 AM, Daniel Bates wrote:
>
>>  Do we know what is the cause(es) for the slow clean builds? I am assuming 
>> that much of the speed up from the "unified source" comes from clang being 
>> able to use an in-memory #include cache to avoid disk I/O and re-parsing of 
>> a seen header. Have we exhausted all efforts (or have we given up?) removing 
>> extraneous #includes? Do we think this pursuing this effort would be more 
>> time consuming or have results that pale in comparison to the "unified 
>> source" approach?
>
> Whilst having an in-process-memory #include cache is not a bad thing,
> it's not the greatest gain as the operating systems should already cache
> file reads just fine.

From my experience, Windows OS is particularly bad at this, even when given
quite enough memory for caching.

>
> The greatest gain comes from reducing the amount of times C++ headers
> are parsed. If you are building a certain .cpp file and include a .h
> file, the compiler has to parse it, which can take quite a bit because
> C++ is really complex monster, especially when templates are used. Doing
> this more than necessary raises build times really quickly.
>
> Header files almost always are include-guarded (either with #pragma once
> or traditional #ifndef guards), so including the same header twice
> within the same .cpp file (or any of its included files) has no cost. On
> the other hand, if you then start building a different .cpp file that
> also includes the same header, you have to parse it again because so far
> C++ is concerned, every inclusion could add different symbols to the AST
> the compiler is building, so their output can't be reused*. In turn we
> end up parsing most headers many more times than actually needed (i.e.
> for every .cpp file that includes Node.h the compiler has to parse
> Node.h and all its dependencies from source, that's a lot of wasted
> effort!).
>
> *Note that including twice the same .h within the same .cpp file is not
> fast because the output is cached in anyway, but because the entire .h
> file is skipped, adding no additional nodes to the AST.
>
> The goal of C++ modules is to fix this problem from its root cause:
> instead of literally including text header files, .cpp files declare
> dependencies on module files that can be compiled, stored, loaded and
> referenced from .cpp files in any order, so you would only parse the
> Node module source code once for the entire project, whilst the compiler
> could load directly the AST every other time from a cached module object
> file.
>
> Note the great advantage of modules comes from the fact that they can be
> imported in different contexts and their content is still semantically
> equivalent, whereas with plain C/C++ includes every header file may act
> differently depending on the preprocessor variables defined by the
> includer and other previous inclusions. In the worst case, when they are
> not include-guarded (luckily this is not too common, but it still
> happens), even including the same file in the same .cpp twice could add
> different symbols each time!
>
> Unfortunately C++ modules are a work in progress... There are two
> different competing proposals with implementations, one from Clang and
> another one from Microsoft, and the C++ technical specification is in a
> very early stage too:
> http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2017/n4681.pdf
>
> We know for sure modules are very important for the future of C++, but
> maybe it's still a bit too early to bet a big project like WebKit on them.
>
> So how else can we avoid parsing the same header files so many times and
> speed up our builds? Enter unified builds.
>
> A requirement for unified builds to work correctly is that header files
> are coded in such a way they work as independent units, much like C++
> modules, i.e. including headers should work no matter in what order you
> place them, and in each case they must define the same symbols. On July
> 31 I wrote about some issues we currently have because of not doing
> exactly this in WebKit (particularly, our #include "config.h" lines are
> ambiguous). They can be worked around so they will not become blockers
> for unified builds, but I still think we should fix them at some point.
>
> Once you have a set of .cpp files whose includes all (1) are guarded
> (e.g. by #pragma once) and (2) are independent units according to the
> above rule, you can take advantage of unified builds:
>
> Instead of invoking once the compiler for each .cpp file, you create a
> new artificial "unified" or "bundle" .cpp file that concatenates (or
> #include's) a number of different .cpp files. This way, headers included
> within the bundle are parsed only once, even if they are used by
> different individual .cpp files, as long as they are within the same
> bundle. This can often result in a massive build speed gain.
>
> Unfortunately, there are 

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Dan Bernstein]

> On Aug 29, 2017, at 1:39 PM, Geoffrey Garen  wrote:
> 
>> I see. The right question to ask would have been how much change occurs in 
>> their working copy between consecutive incremental builds.
> 
> If you want to help make our benchmark righter, please do share any data you 
> have about the average content of an incremental build that is distinct from 
> a daily svn up.

Here is the data from three WebKit contributors surveyed today. For each 
contributor, each line corresponds to a single consecutive incremental build 
they’ve performed today, and the number shown is the number of files that were 
compiled in that build:

A
B
C
41
4
1
2
1
1

1
1

4
7

4
58

5
27

3
23

4
61

5
3

7
75

1
2

6
1

4
2

4
1

4
47

5


3


I hope this helps. It certainly gives me an idea of what a righter benchmark 
would be.___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Konstantin Tokarev]

29.08.2017, 23:54, "Alicia Boya García" :
> I'm all for stable bundle .cpp files. Automatic generation can be nice
> for the first time... and once in while we can try to modify the bundles
> to try to make the build even faster, but really most of the time bundle
> contents should not change unless new files are added and everybody
> should get the same bundles.
>
> If we're going to end up with some name collisions, we'd rather do so
> that every build of the same source tree gets collisions or no build
> gets them.
>
> The description of these .cpp bundles should become a project artifact,
> shared and versioned in the repository and treated carefully as such.

This is already the case for *AllInOne.cpp files of WebCore, which are enabled
by -DENABLE_ALLINONE_BUILD=ON option (--allinone-build in build-webkit).

>
> Also, in my earlier message about unified builds I explained how the
> distribution of the files within the bundles also affects the number of
> bundles that are rebuilt in response to changes in the source code. I
> also proposed that grouping together files implementing the same
> features could be a good start point to minimize the impact of this issue.
>
> On other note, I don't think namespacing static functions and variables
> as originally proposed is the way to go.
>
> The refactor would be ridiculously huge. Running `rg -w '^static' -tcpp
> --no-filename --no-heading |wc -l` in my local repository yields 16797
> instances of static functions and variables, only taking into account
> those where the word static is at the beginning of the line without any
> spaces before, which is a common case but not a requirement in C++. Note
> though that this regex excludes static class members as long as code is
> properly formatted (they would be indented); these are not included as
> they would not need any changes.
>
> static functions and variables in WebKit are used for hidden
> implementation details, which tend to very very specific and therefore
> receive long specific names with relatively little risk of collisions.
>
> Furthermore, collisions won't go unnoticed as in every case they will
> trigger a compilation error of the likes of 'redefinition of ‘int a’'.
> In this case the compiler will also show the inclusion stack mentioning
> the bundle file and the conflicting definition. The programmer will
> notice that both are static members in .cpp files, so the only way that
> error can arise is because they had a name collision due to unified
> source builds.
>
> We'd probably be better off handling the few collisions on a one by one
> basis, renaming the symbols or excluding the worst offenders from
> unified builds rather than plain out forbidding static declarations.
>
> -- Alicia
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

-- 
Regards,
Konstantin
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Alicia Boya García]

I'm all for stable bundle .cpp files. Automatic generation can be nice
for the first time... and once in while we can try to modify the bundles
to try to make the build even faster, but really most of the time bundle
contents should not change unless new files are added and everybody
should get the same bundles.

If we're going to end up with some name collisions, we'd rather do so
that every build of the same source tree gets collisions or no build
gets them.

The description of these .cpp bundles should become a project artifact,
shared and versioned in the repository and treated carefully as such.

Also, in my earlier message about unified builds I explained how the
distribution of the files within the bundles also affects the number of
bundles that are rebuilt in response to changes in the source code. I
also proposed that grouping together files implementing the same
features could be a good start point to minimize the impact of this issue.

On other note, I don't think namespacing static functions and variables
as originally proposed is the way to go.

The refactor would be ridiculously huge. Running `rg -w '^static' -tcpp
--no-filename --no-heading |wc -l` in my local repository yields 16797
instances of static functions and variables, only taking into account
those where the word static is at the beginning of the line without any
spaces before, which is a common case but not a requirement in C++. Note
though that this regex excludes static class members as long as code is
properly formatted (they would be indented); these are not included as
they would not need any changes.

static functions and variables in WebKit are used for hidden
implementation details, which tend to very very specific and therefore
receive long specific names with relatively little risk of collisions.

Furthermore, collisions won't go unnoticed as in every case they will
trigger a compilation error of the likes of 'redefinition of ‘int a’'.
In this case the compiler will also show the inclusion stack mentioning
the bundle file and the conflicting definition. The programmer will
notice that both are static members in .cpp files, so the only way that
error can arise is because they had a name collision due to unified
source builds.

We'd probably be better off handling the few collisions on a one by one
basis, renaming the symbols or excluding the worst offenders from
unified builds rather than plain out forbidding static declarations.

-- Alicia
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Geoffrey Garen]

> I see. The right question to ask would have been how much change occurs in 
> their working copy between consecutive incremental builds.

If you want to help make our benchmark righter, please do share any data you 
have about the average content of an incremental build that is distinct from a 
daily svn up.

Geoff
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Saam Barati]

> On Aug 29, 2017, at 1:22 PM, Olmstead, Don  wrote:
> 
> Sure but in a corporate environment it can be a really big win. All of our 
> developers have their machines hooked up to the distributed build system and 
> we also have servers hanging out that can be used in the build. This sort of 
> thing doesn't require anyone to change how they write their code which is 
> something a unity build ends up requiring.
> 
> In terms of WebKit development the only places it falls apart are in the 
> source code generation and the link.
> 
> Really I was just curious if anything like that was discussed because all of 
> this requires pretty big changes to source code. Additionally there's other 
> fun things that can happen around includes when doing a unity build where a 
> file will stop being able to be built in isolation. The same thing has 
> happened with WebCorePrefix where a number of files can't be built without it.
Maybe it’d be worth having an EWS bot builds the non-unity build so we don’t 
get into the same situation here.

- Saam
> 
> -Original Message-
> From: Konstantin Tokarev [mailto:annu...@yandex.ru] 
> Sent: Tuesday, August 29, 2017 11:22 AM
> To: Olmstead, Don ; Keith Miller 
> ; WebKit Development Mailing List 
> 
> Subject: Re: [webkit-dev] Growing tired of long build times? Check out this 
> awesome new way to speed up your build... soon (HINT: It's not buying a new 
> computer)
> 
> 
> 
> 29.08.2017, 21:19, "Olmstead, Don" :
>> Did you happen to look at any sort of distributed build system? We use 
>> distributed builds here for WebKit and a full rebuild is pretty reasonable. 
>> That would negate the need to change how everything is done to get a unity 
>> build going.
> 
> This isn't helpful for contributors who have only one machine at their 
> disposal
> 
>> 
>> From: webkit-dev 
>> [mailto:webkit-dev-bounces+don.olmstead=am.sony@lists.webkit.org] On 
>> Behalf Of Keith Miller
>> Sent: Monday, August 28, 2017 5:37 PM
>> To: WebKit Development Mailing List 
>> Subject: [webkit-dev] Growing tired of long build times? Check out this 
>> awesome new way to speed up your build... soon (HINT: It's not buying a new 
>> computer)
>> 
>> Hello fellow Webkittens,
>> 
>> Are you growing tired of long cat naps while waiting 45 minutes for WebKit 
>> to build? (I’ve definitely never done this 蘿!)
>> 
>> Do you want WebKit to build up to 3-4x faster on a clean build?*
>> 
>> Does seeing your incremental build… stay the same sound good?
>> 
>> You’ll be purring with excitement after you switch to unified source 
>> builds!**
>> 
>> * Building JSC with CMake, including CMake configuration time, went from 
>> ~8m15s to ~2m30s on my MBP using unified sources only on cpp files in 
>> JavaScriptCore. Final results on WebKit may vary. Using unified sources may 
>> cause sudden feelings of euphoria and productivity (or grumpiness from lack 
>> of naps…). Not responsible for any bugs in extra patches produced by using 
>> unified source builds (excluding build system related bugs) except where 
>> required by law (pi...@apple.com).
>> 
>> ** Soon. Unified source builds haven’t been landed yet.
>> 
>> How are things going to change? Great question! Let’s find out!
>> 
>> When you fire off the build script it will automagically paw your source 
>> files into bundles. The size of each bundle is speculatively going to be 8 
>> .cpp files but that’s up for experimentation. From my testing so far, the 
>> compile time for a bundle is at most 20% (~6s vs ~7s for the Air directory) 
>> longer than the longest file it includes. Given that most of the build time 
>> in incremental builds is scanning dependencies, this change is probably only 
>> barely noticeable.
>> 
>> Bundling happens as part of the CMake configuration process, in the CMake 
>> build. In the Xcode build it’s more complicated since we cannot dynamically 
>> choose the files we are going to compile. The only solution I know of is to 
>> pre-list a bunch of files for the build to dump files into. Occasionally, 
>> we’ll need to add more files to the build list.
>> 
>> When you add or remove a file in the project, the cost is higher. The 
>> current plan to bundle source files is by directory. This means that when a 
>> .cpp file is added or removed you will rebalance the bundles in its 
>> directory, and you will have to rebuild up to the full directory that file 
>> is in.
>> 
>> My goal is to make all of this invisible to developers as much as possible. 
>> I believe, for the most part, things should operate mostly as they have 
>> before. As the details on unified source builds get finalized, I’m sending 
>> out a separate email documenting any workflow changes. I’ll also do my best 
>> to answer any questions or concerns.
>> 
>> “But Keith, am I going to have to make major changes to 

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Olmstead, Don]

Sure but in a corporate environment it can be a really big win. All of our 
developers have their machines hooked up to the distributed build system and we 
also have servers hanging out that can be used in the build. This sort of thing 
doesn't require anyone to change how they write their code which is something a 
unity build ends up requiring.

In terms of WebKit development the only places it falls apart are in the source 
code generation and the link.

Really I was just curious if anything like that was discussed because all of 
this requires pretty big changes to source code. Additionally there's other fun 
things that can happen around includes when doing a unity build where a file 
will stop being able to be built in isolation. The same thing has happened with 
WebCorePrefix where a number of files can't be built without it.

-Original Message-
From: Konstantin Tokarev [mailto:annu...@yandex.ru] 
Sent: Tuesday, August 29, 2017 11:22 AM
To: Olmstead, Don ; Keith Miller 
; WebKit Development Mailing List 

Subject: Re: [webkit-dev] Growing tired of long build times? Check out this 
awesome new way to speed up your build... soon (HINT: It's not buying a new 
computer)



29.08.2017, 21:19, "Olmstead, Don" :
> Did you happen to look at any sort of distributed build system? We use 
> distributed builds here for WebKit and a full rebuild is pretty reasonable. 
> That would negate the need to change how everything is done to get a unity 
> build going.

This isn't helpful for contributors who have only one machine at their disposal

>
> From: webkit-dev 
> [mailto:webkit-dev-bounces+don.olmstead=am.sony@lists.webkit.org] On 
> Behalf Of Keith Miller
> Sent: Monday, August 28, 2017 5:37 PM
> To: WebKit Development Mailing List 
> Subject: [webkit-dev] Growing tired of long build times? Check out this 
> awesome new way to speed up your build... soon (HINT: It's not buying a new 
> computer)
>
> Hello fellow Webkittens,
>
> Are you growing tired of long cat naps while waiting 45 minutes for WebKit to 
> build? (I’ve definitely never done this 蘿!)
>
> Do you want WebKit to build up to 3-4x faster on a clean build?*
>
> Does seeing your incremental build… stay the same sound good?
>
> You’ll be purring with excitement after you switch to unified source builds!**
>
> * Building JSC with CMake, including CMake configuration time, went from 
> ~8m15s to ~2m30s on my MBP using unified sources only on cpp files in 
> JavaScriptCore. Final results on WebKit may vary. Using unified sources may 
> cause sudden feelings of euphoria and productivity (or grumpiness from lack 
> of naps…). Not responsible for any bugs in extra patches produced by using 
> unified source builds (excluding build system related bugs) except where 
> required by law (pi...@apple.com).
>
> ** Soon. Unified source builds haven’t been landed yet.
>
> How are things going to change? Great question! Let’s find out!
>
> When you fire off the build script it will automagically paw your source 
> files into bundles. The size of each bundle is speculatively going to be 8 
> .cpp files but that’s up for experimentation. From my testing so far, the 
> compile time for a bundle is at most 20% (~6s vs ~7s for the Air directory) 
> longer than the longest file it includes. Given that most of the build time 
> in incremental builds is scanning dependencies, this change is probably only 
> barely noticeable.
>
> Bundling happens as part of the CMake configuration process, in the CMake 
> build. In the Xcode build it’s more complicated since we cannot dynamically 
> choose the files we are going to compile. The only solution I know of is to 
> pre-list a bunch of files for the build to dump files into. Occasionally, 
> we’ll need to add more files to the build list.
>
> When you add or remove a file in the project, the cost is higher. The current 
> plan to bundle source files is by directory. This means that when a .cpp file 
> is added or removed you will rebalance the bundles in its directory, and you 
> will have to rebuild up to the full directory that file is in.
>
> My goal is to make all of this invisible to developers as much as possible. I 
> believe, for the most part, things should operate mostly as they have before. 
> As the details on unified source builds get finalized, I’m sending out a 
> separate email documenting any workflow changes. I’ll also do my best to 
> answer any questions or concerns.
>
> “But Keith, am I going to have to make major changes to the way I develop 
> WebKit?” Yes and, hopefully, no. There is one major common workflow change 
> that comes with unified sources, the naming of static variables and 
> functions. Since C++ does not have a way to only declare a static function or 
> variable in some restricted top level scope we are going to need to create 
> some work around. The 

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Konstantin Tokarev]

29.08.2017, 22:48, "Geoffrey Garen" :
>>> We have some preliminary data that says incremental builds will be OK, but 
>>> not a full benchmark.
>>>
>>> Here’s a full benchmark I propose to test incremental builds:
>>>
>>> Start 7 days ago in SVN history. Do a clean build.
>>>
>>> SVN update forward by 24 hours. Do an incremental build. Time this build.
>>>
>>> Repeat 7 times. Average the results.
>>
>> How did you arrive at a day’s worth of changes being representative of a 
>> WebKit contributor’s incremental build?
>
> I asked some people how often they svn up and they said every day.

Pulling changes from whole day often means (almost) full rebuild

>
> Geoff
> ,
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev


-- 
Regards,
Konstantin
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Dan Bernstein]

> On Aug 29, 2017, at 12:48 PM, Geoffrey Garen  wrote:
> 
>>> We have some preliminary data that says incremental builds will be OK, but 
>>> not a full benchmark.
>>> 
>>> Here’s a full benchmark I propose to test incremental builds:
>>> 
>>> Start 7 days ago in SVN history. Do a clean build.
>>> 
>>> SVN update forward by 24 hours. Do an incremental build. Time this 
>>> build.
>>> 
>>> Repeat 7 times. Average the results.
>> 
>> How did you arrive at a day’s worth of changes being representative of a 
>> WebKit contributor’s incremental build?
> 
> I asked some people how often they svn up and they said every day.

I see. The right question to ask would have been how much change occurs in 
their working copy between consecutive incremental builds.

> 
> Geoff

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Geoffrey Garen]

>> We have some preliminary data that says incremental builds will be OK, but 
>> not a full benchmark.
>> 
>> Here’s a full benchmark I propose to test incremental builds:
>> 
>>  Start 7 days ago in SVN history. Do a clean build.
>> 
>>  SVN update forward by 24 hours. Do an incremental build. Time this 
>> build.
>> 
>>  Repeat 7 times. Average the results.
> 
> How did you arrive at a day’s worth of changes being representative of a 
> WebKit contributor’s incremental build?

I asked some people how often they svn up and they said every day.

Geoff___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Geoffrey Garen]

> This isn’t the scenario I find myself in most often.  A much more common 
> scenario is working on a change; touch one or two files, and then compile and 
> test/debug.  Rinse and repeat.

We’ve already tested this case. The worst case slowdown, if you touch a small 
file that's in the same bundle as the biggest .cpp file in the project, is 6s 
=> 7s (20%).

Geoff___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Sam Weinig]

> On Aug 29, 2017, at 11:03 AM, Geoffrey Garen  wrote:
> 
>> I worry about adopting unity build because while it makes clean builds 
>> faster, it also slows down incremental builds. As a developer, I rarely do 
>> clean builds, I mostly do incremental builds so this would likely make my 
>> experience worse?
> 
> We have some preliminary data that says incremental builds will be OK, but 
> not a full benchmark.
> 
> Here’s a full benchmark I propose to test incremental builds:
> 
>   Start 7 days ago in SVN history. Do a clean build.
> 
>   SVN update forward by 24 hours. Do an incremental build. Time this 
> build.
> 
>   Repeat 7 times. Average the results.
> 
> I think we should run this benchmark per project as we adopt unity builds, 
> starting with WTF.
> 
> In the beginning, when we adopt unity builds for WTF and then JavaScriptCore, 
> the benchmark can build just these projects and stop. Once we get to WebCore 
> and WebKit, the benchmark will take longer. Oh well.
> 
> If we get the bundling right, this benchmark should show obvious speedups. If 
> it shows no speedup or a slowdown, we’ve done something wrong and we need to 
> reconsider.


This isn’t the scenario I find myself in most often.  A much more common 
scenario is working on a change; touch one or two files, and then compile and 
test/debug.  Rinse and repeat.

- Sam

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Michael Catanzaro]

I'd like to propose a variation on Keith's proposal. I think we should 
manually determine which files should be built together, rather than 
writing CMake code to do that automatically.


Let's consider an example of what use of the FILENAME namespace, as 
Keith proposes, might look like. I opened a couple big files in WebCore 
and settled on this representative code snippet from 
HTMLMediaElement.cpp:



#if ENABLE(MEDIA_SESSION)
typedef HashMap IDToElementMap;

static IDToElementMap& elementIDsToElements()
{
static NeverDestroyed map;
return map;
}

HTMLMediaElement* HTMLMediaElement::elementWithID(uint64_t id)
{
if (id == HTMLMediaElementInvalidID)
return nullptr;

return elementIDsToElements().get(id);
}

static uint64_t nextElementID()
{
static uint64_t elementID = 0;
return ++elementID;
}
#endif


In Keith's proposed scheme, the static functions need to go into the 
FILENAME namespace. The typedef probably should as well, since we'd 
want to ensure it doesn't clash with other hypothetical typedefs in 
other files. (The chances of another file having a typedef of the name 
IDToElementMap are probably rather low, but that's also true for the 
function names as well.) And *the uses of the static functions from 
outside the namespace will now need to be qualified as well* (thanks to 
Alicia for pointing that out; seems obvious, but I'd missed it). So the 
result, if we don't reorder anything, would have to look like this:



#if ENABLE(MEDIA_SESSION)
namespace FILENAME {



typedef HashMap IDToElementMap;

static IDToElementMap& elementIDsToElements()
{
static NeverDestroyed map;
return map;
}



} // namespace FILENAME

HTMLMediaElement* HTMLMediaElement::elementWithID(uint64_t id)
{
if (id == HTMLMediaElementInvalidID)
return nullptr;

return FILENAME::elementIDsToElements().get(id);
}



namespace FILENAME {



static uint64_t nextElementID()
{
static uint64_t elementID = 0;
return ++elementID;
}

} // namespace FILENAME

#endif


We could reorder functions and add more forward declarations to reduce 
the number of extra namespace declarations that we'll need. But there's 
nothing we can do to eliminate the mess caused by needing to use the 
extra FILENAME:: scope when using the static variables and functions 
outside the namespace:


   return FILENAME::elementIDsToElements().get(id);

If it's going to improve clean build times by 80%, then we should 
probably put up the this messiness to get the faster build... but 
surely nobody is going to be very fond of making our code messier. 
(Tangent: I expect this might discourage future use of file-static 
functions and encourage the use of private class member functions. 
Ryusuke and Darin have already shown why that's a bad idea.)


We would be better off *not* attempting to avoid naming clashes 
automatically, i.e. not using the FILENAME namespace. If we give up on 
automatically bundling sources together, and instead group them 
together manually or semi-manually in the build system so that adding a 
new file does not unexpectedly result in changing which other files get 
bundled together, then we can get the benefits of unified builds 
without needing to make significant modifications to our source code. 
We would only need to rename static variables and functions in the rare 
cases that there are actual name collisions with other files in the 
same bundle. I expect such cases will be quite rare. The cost is that 
we would have to manually manage which files get bundled together in 
CMake, rather than having this be determined automatically as under 
Keith's plan.


For example, currently in Source/WebCore/CMakeLists.txt we have:


set(WebCore_SOURCES
Modules/airplay/WebKitPlaybackTargetAvailabilityEvent.cpp

Modules/beacon/NavigatorBeacon.cpp

Modules/cache/Cache.cpp
Modules/cache/CacheStorage.cpp
Modules/cache/CacheStorageConnection.cpp
Modules/cache/DOMCache.cpp
Modules/cache/DOMWindowCaches.cpp
Modules/cache/WorkerCacheStorageConnection.cpp
Modules/cache/WorkerGlobalScopeCaches.cpp

# ...
)


We could instead write something like this:


ADD_SOURCES_BUNDLE(WebCore_SOURCES
Modules/airplay/WebKitPlaybackTargetAvailabilityEvent.cpp
Modules/beacon/NavigatorBeacon.cpp
)

ADD_SOURCES_BUNDLE(WebCore_SOURCES
Modules/cache/Cache.cpp
Modules/cache/CacheStorage.cpp
Modules/cache/CacheStorageConnection.cpp
Modules/cache/DOMCache.cpp
Modules/cache/DOMWindowCaches.cpp
Modules/cache/WorkerCacheStorageConnection.cpp
Modules/cache/WorkerGlobalScopeCaches.cpp
)


We'd have to figure out how to implement ADD_SOURCES_BUNDLE.

I expect manually determining which files to bundle together will be 
less optimal that automatically determining which files should be built 
together, since Keith has already figured out the ideal bundle size and 
our bundles are surely going to 

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Konstantin Tokarev]

  29.08.2017, 21:32, "Maciej Stachowiak" :  On Aug 29, 2017, at 9:10 AM, Chris Dumez  wrote: I worry about adopting unity build because while it makes clean builds faster, it also slows down incremental builds. As a developer, I rarely do clean builds, I mostly do incremental builds so this would likely make my experience worse?Unity builds also require a lot of code churn to resolve naming conflicts and the resulting code does not look as nice. Given the stats presented, if you're rebuilding at least two files in the same slice, you get a speedup.But these 2 modified files are likely to be compiled in parallel For rebuilding only exactly a single file, you get a small slowdown. Your worst case is rebuilding many files but all from a separate slice. Not sure how likely this is. For Apple folks, the penalty would be more than reversed by switching to CMake+Ninja as the main building system for engineering builds (which I think is the plan?)  - Maciej  Finally, the statement that the slow down of incremental build is acceptable because we spend most of our time resolving dependencies seems to assume we keep using make. Using Ninja would speed up incremental builds a lot by notre-resolving dependencies so much. -- Chris Dumez  On Aug 29, 2017, at 9:05 AM, Darin Adler  wrote: On Aug 28, 2017, at 8:34 PM, Ryosuke Niwa  wrote: Wherever possible, we are going to want to convert file-static functionsinto private C++ class member functions.I don't think we should make this change. It would mean that wheneverthe function signature changes, we'd have to modify the header file,which in turn triggers rebuild of every cpp file which includes thatheader file. That was my first thought as well. In fact, I typically ask people to do the opposite: Whenever a function does not require access to private members, convert from a member function that has to be in the header file to a function that is private to the source file. More recently I have started doing something different for the many functions that only have to be used on one place; I use lambdas to create nested functions. This has a couple nice properties: The lambda shares access to private members and anything else that the surrounding function has access to. We have the option of capturing rather than passing arguments, which can be clearer in some cases. It’s not quite as good as a separate function for visual separation; the lambda is often mashed together with the rest of the surrounding function.— Darin___webkit-dev mailing listwebkit-dev@lists.webkit.orghttps://lists.webkit.org/mailman/listinfo/webkit-dev___webkit-dev mailing listwebkit-dev@lists.webkit.orghttps://lists.webkit.org/mailman/listinfo/webkit-dev,___webkit-dev mailing listwebkit-dev@lists.webkit.orghttps://lists.webkit.org/mailman/listinfo/webkit-dev  -- Regards,Konstantin ___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Maciej Stachowiak]

> On Aug 29, 2017, at 9:10 AM, Chris Dumez  wrote:
> 
> I worry about adopting unity build because while it makes clean builds 
> faster, it also slows down incremental builds. As a developer, I rarely do 
> clean builds, I mostly do incremental builds so this would likely make my 
> experience worse?
> Unity builds also require a lot of code churn to resolve naming conflicts and 
> the resulting code does not look as nice.

Given the stats presented, if you're rebuilding at least two files in the same 
slice, you get a speedup. For rebuilding only exactly a single file, you get a 
small slowdown. Your worst case is rebuilding many files but all from a 
separate slice. Not sure how likely this is.

For Apple folks, the penalty would be more than reversed by switching to 
CMake+Ninja as the main building system for engineering builds (which I think 
is the plan?)

 - Maciej

> 
> Finally, the statement that the slow down of incremental build is acceptable 
> because we spend most of our time resolving dependencies seems to assume we 
> keep using make. Using Ninja would speed up incremental builds a lot by not
> re-resolving dependencies so much.
> 
> --
>  Chris Dumez
> 
> 
> 
> 
>> On Aug 29, 2017, at 9:05 AM, Darin Adler > > wrote:
>> 
>>> On Aug 28, 2017, at 8:34 PM, Ryosuke Niwa >> > wrote:
>>> 
 Wherever possible, we are going to want to convert file-static functions
 into private C++ class member functions.
>>> 
>>> I don't think we should make this change. It would mean that whenever
>>> the function signature changes, we'd have to modify the header file,
>>> which in turn triggers rebuild of every cpp file which includes that
>>> header file.
>> 
>> 
>> That was my first thought as well. In fact, I typically ask people to do the 
>> opposite: Whenever a function does not require access to private members, 
>> convert from a member function that has to be in the header file to a 
>> function that is private to the source file.
>> 
>> More recently I have started doing something different for the many 
>> functions that only have to be used on one place; I use lambdas to create 
>> nested functions. This has a couple nice properties: The lambda shares 
>> access to private members and anything else that the surrounding function 
>> has access to. We have the option of capturing rather than passing 
>> arguments, which can be clearer in some cases. It’s not quite as good as a 
>> separate function for visual separation; the lambda is often mashed together 
>> with the rest of the surrounding function.
>> 
>> — Darin
>> ___
>> webkit-dev mailing list
>> webkit-dev@lists.webkit.org 
>> https://lists.webkit.org/mailman/listinfo/webkit-dev
> 
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Keith Miller]

> On Aug 29, 2017, at 11:03 AM, Geoffrey Garen  wrote:
> 
>> I worry about adopting unity build because while it makes clean builds 
>> faster, it also slows down incremental builds. As a developer, I rarely do 
>> clean builds, I mostly do incremental builds so this would likely make my 
>> experience worse?
> 
> We have some preliminary data that says incremental builds will be OK, but 
> not a full benchmark.
> 
> Here’s a full benchmark I propose to test incremental builds:
> 
>   Start 7 days ago in SVN history. Do a clean build.
> 
>   SVN update forward by 24 hours. Do an incremental build. Time this 
> build.
> 
>   Repeat 7 times. Average the results.

I’m not sure this will really test incremental builds. WTF header files are 
touched at least once a day, which means these builds will be closer to a clean 
build. If we want to test incremental builds we’d need someway to just touch 
some set of .cpp files scattered throughout webkit and rebuild.

> 
> I think we should run this benchmark per project as we adopt unity builds, 
> starting with WTF.

I agree, we should come up with some benchmark to test the efficacy of unified 
builds. I think a better test would be to pick some subset of .cpp files we 
think are representative of an incremental build and create a patch that sticks 
some newlines at the top of the file. Hopefully, this patch will always apply 
since copyrights don’t change often. We could also do more than one patch to 
test various versions of incremental builds.

> 
> In the beginning, when we adopt unity builds for WTF and then JavaScriptCore, 
> the benchmark can build just these projects and stop. Once we get to WebCore 
> and WebKit, the benchmark will take longer. Oh well.
> 
> If we get the bundling right, this benchmark should show obvious speedups. If 
> it shows no speedup or a slowdown, we’ve done something wrong and we need to 
> reconsider.
> 
> Geoff

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Konstantin Tokarev]

29.08.2017, 21:19, "Olmstead, Don" :
> Did you happen to look at any sort of distributed build system? We use 
> distributed builds here for WebKit and a full rebuild is pretty reasonable. 
> That would negate the need to change how everything is done to get a unity 
> build going.

This isn't helpful for contributors who have only one machine at their disposal

>
> From: webkit-dev 
> [mailto:webkit-dev-bounces+don.olmstead=am.sony@lists.webkit.org] On 
> Behalf Of Keith Miller
> Sent: Monday, August 28, 2017 5:37 PM
> To: WebKit Development Mailing List 
> Subject: [webkit-dev] Growing tired of long build times? Check out this 
> awesome new way to speed up your build... soon (HINT: It's not buying a new 
> computer)
>
> Hello fellow Webkittens,
>
> Are you growing tired of long cat naps while waiting 45 minutes for WebKit to 
> build? (I’ve definitely never done this 蘿!)
>
> Do you want WebKit to build up to 3-4x faster on a clean build?*
>
> Does seeing your incremental build… stay the same sound good?
>
> You’ll be purring with excitement after you switch to unified source builds!**
>
> * Building JSC with CMake, including CMake configuration time, went from 
> ~8m15s to ~2m30s on my MBP using unified sources only on cpp files in 
> JavaScriptCore. Final results on WebKit may vary. Using unified sources may 
> cause sudden feelings of euphoria and productivity (or grumpiness from lack 
> of naps…). Not responsible for any bugs in extra patches produced by using 
> unified source builds (excluding build system related bugs) except where 
> required by law (pi...@apple.com).
>
> ** Soon. Unified source builds haven’t been landed yet.
>
> How are things going to change? Great question! Let’s find out!
>
> When you fire off the build script it will automagically paw your source 
> files into bundles. The size of each bundle is speculatively going to be 8 
> .cpp files but that’s up for experimentation. From my testing so far, the 
> compile time for a bundle is at most 20% (~6s vs ~7s for the Air directory) 
> longer than the longest file it includes. Given that most of the build time 
> in incremental builds is scanning dependencies, this change is probably only 
> barely noticeable.
>
> Bundling happens as part of the CMake configuration process, in the CMake 
> build. In the Xcode build it’s more complicated since we cannot dynamically 
> choose the files we are going to compile. The only solution I know of is to 
> pre-list a bunch of files for the build to dump files into. Occasionally, 
> we’ll need to add more files to the build list.
>
> When you add or remove a file in the project, the cost is higher. The current 
> plan to bundle source files is by directory. This means that when a .cpp file 
> is added or removed you will rebalance the bundles in its directory, and you 
> will have to rebuild up to the full directory that file is in.
>
> My goal is to make all of this invisible to developers as much as possible. I 
> believe, for the most part, things should operate mostly as they have before. 
> As the details on unified source builds get finalized, I’m sending out a 
> separate email documenting any workflow changes. I’ll also do my best to 
> answer any questions or concerns.
>
> “But Keith, am I going to have to make major changes to the way I develop 
> WebKit?” Yes and, hopefully, no. There is one major common workflow change 
> that comes with unified sources, the naming of static variables and 
> functions. Since C++ does not have a way to only declare a static function or 
> variable in some restricted top level scope we are going to need to create 
> some work around. The choice I’m thinking we’ll go with is to have an 
> auto-generated macro value FILENAME, which is similar to __FILE__ except that 
> it doesn’t have file extension (We can’t use FILE since that’s actually a 
> type in C ). This macro would be redefined in the unified source directly 
> above the file that is about to be included. Then in each cpp file that has 
> static functions/variables developers would have:
>
> namespace FILENAME {
>
>     static int myVariable { 0 };
>
>     static int myFunction() { return myVariable; }
>
> }
>
> The advantage of using a macro over textually writing the filename is that it 
> makes code more portable between files. Also, once people get used to the 
> concept of FILENAME in the code base it will, hopefully become obvious what 
> it means. The main downside to this is using FILENAME in a header would give 
> you whatever cpp file first included you (if only we got the compiler magic 
> of __FILE__...). We would probably need to prohibit using FILENAME in .h 
> files. Unfortunately, you cannot "using FILENAME;” because it will export all 
> the values in each subsequent namespace. e.g.
>
> namespace JSC {
>
>     namespace Foo {
>
>     static int myVar { 0 };
>
>     }
>
>     using Foo;
>
> }
>

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Dan Bernstein]

> On Aug 29, 2017, at 11:03 AM, Geoffrey Garen  wrote:
> 
>> I worry about adopting unity build because while it makes clean builds 
>> faster, it also slows down incremental builds. As a developer, I rarely do 
>> clean builds, I mostly do incremental builds so this would likely make my 
>> experience worse?
> 
> We have some preliminary data that says incremental builds will be OK, but 
> not a full benchmark.
> 
> Here’s a full benchmark I propose to test incremental builds:
> 
>   Start 7 days ago in SVN history. Do a clean build.
> 
>   SVN update forward by 24 hours. Do an incremental build. Time this 
> build.
> 
>   Repeat 7 times. Average the results.

How did you arrive at a day’s worth of changes being representative of a WebKit 
contributor’s incremental build?

> 
> I think we should run this benchmark per project as we adopt unity builds, 
> starting with WTF.
> 
> In the beginning, when we adopt unity builds for WTF and then JavaScriptCore, 
> the benchmark can build just these projects and stop. Once we get to WebCore 
> and WebKit, the benchmark will take longer. Oh well.
> 
> If we get the bundling right, this benchmark should show obvious speedups. If 
> it shows no speedup or a slowdown, we’ve done something wrong and we need to 
> reconsider.
> 
> Geoff
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Olmstead, Don]

Did you happen to look at any sort of distributed build system? We use 
distributed builds here for WebKit and a full rebuild is pretty reasonable. 
That would negate the need to change how everything is done to get a unity 
build going.

From: webkit-dev 
[mailto:webkit-dev-bounces+don.olmstead=am.sony@lists.webkit.org] On Behalf 
Of Keith Miller
Sent: Monday, August 28, 2017 5:37 PM
To: WebKit Development Mailing List 
Subject: [webkit-dev] Growing tired of long build times? Check out this awesome 
new way to speed up your build... soon (HINT: It's not buying a new computer)

Hello fellow Webkittens,

Are you growing tired of long cat naps while waiting 45 minutes for WebKit to 
build? (I’ve definitely never done this 蘿!)
Do you want WebKit to build up to 3-4x faster on a clean build?*
Does seeing your incremental build… stay the same sound good?

You’ll be purring with excitement after you switch to unified source builds!**

* Building JSC with CMake, including CMake configuration time, went from ~8m15s 
to ~2m30s on my MBP using unified sources only on cpp files in JavaScriptCore. 
Final results on WebKit may vary. Using unified sources may cause sudden 
feelings of euphoria and productivity (or grumpiness from lack of naps…). Not 
responsible for any bugs in extra patches produced by using unified source 
builds (excluding build system related bugs) except where required by law 
(pi...@apple.com).

** Soon. Unified source builds haven’t been landed yet.

How are things going to change? Great question! Let’s find out!

When you fire off the build script it will automagically paw your source files 
into bundles. The size of each bundle is speculatively going to be 8 .cpp files 
but that’s up for experimentation. From my testing so far, the compile time for 
a bundle is at most 20% (~6s vs ~7s for the Air directory) longer than the 
longest file it includes. Given that most of the build time in incremental 
builds is scanning dependencies, this change is probably only barely noticeable.

Bundling happens as part of the CMake configuration process, in the CMake 
build. In the Xcode build it’s more complicated since we cannot dynamically 
choose the files we are going to compile. The only solution I know of is to 
pre-list a bunch of files for the build to dump files into. Occasionally, we’ll 
need to add more files to the build list.

When you add or remove a file in the project, the cost is higher. The current 
plan to bundle source files is by directory. This means that when a .cpp file 
is added or removed you will rebalance the bundles in its directory, and you 
will have to rebuild up to the full directory that file is in.

My goal is to make all of this invisible to developers as much as possible. I 
believe, for the most part, things should operate mostly as they have before. 
As the details on unified source builds get finalized, I’m sending out a 
separate email documenting any workflow changes. I’ll also do my best to answer 
any questions or concerns.

“But Keith, am I going to have to make major changes to the way I develop 
WebKit?” Yes and, hopefully, no. There is one major common workflow change that 
comes with unified sources, the naming of static variables and functions. Since 
C++ does not have a way to only declare a static function or variable in some 
restricted top level scope we are going to need to create some work around. The 
choice I’m thinking we’ll go with is to have an auto-generated macro value 
FILENAME, which is similar to __FILE__ except that it doesn’t have file 
extension (We can’t use FILE since that’s actually a type in C ). This macro 
would be redefined in the unified source directly above the file that is about 
to be included. Then in each cpp file that has static functions/variables 
developers would have:

namespace FILENAME {
static int myVariable { 0 };
static int myFunction() { return myVariable; }
}

The advantage of using a macro over textually writing the filename is that it 
makes code more portable between files. Also, once people get used to the 
concept of FILENAME in the code base it will, hopefully become obvious what it 
means. The main downside to this is using FILENAME in a header would give you 
whatever cpp file first included you (if only we got the compiler magic of 
__FILE__...). We would probably need to prohibit using FILENAME in .h files. 
Unfortunately, you cannot "using FILENAME;” because it will export all the 
values in each subsequent namespace. e.g.

namespace JSC {
namespace Foo {
static int myVar { 0 };
}
using Foo;
}

namespace JSC {
namespace Bar {
static int myVar { 0 };
}
using Bar;
}

does not compile.

Does anyone have other name recommendations for the macro name other than 
FILENAME? I went with file name since it describes what the substituted value 
will be the best, which is helpful for 

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Geoffrey Garen]

> I worry about adopting unity build because while it makes clean builds 
> faster, it also slows down incremental builds. As a developer, I rarely do 
> clean builds, I mostly do incremental builds so this would likely make my 
> experience worse?

We have some preliminary data that says incremental builds will be OK, but not 
a full benchmark.

Here’s a full benchmark I propose to test incremental builds:

Start 7 days ago in SVN history. Do a clean build.

SVN update forward by 24 hours. Do an incremental build. Time this 
build.

Repeat 7 times. Average the results.

I think we should run this benchmark per project as we adopt unity builds, 
starting with WTF.

In the beginning, when we adopt unity builds for WTF and then JavaScriptCore, 
the benchmark can build just these projects and stop. Once we get to WebCore 
and WebKit, the benchmark will take longer. Oh well.

If we get the bundling right, this benchmark should show obvious speedups. If 
it shows no speedup or a slowdown, we’ve done something wrong and we need to 
reconsider.

Geoff
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Keith Miller]

> On Aug 29, 2017, at 9:10 AM, Chris Dumez  wrote:
> 
> I worry about adopting unity build because while it makes clean builds 
> faster, it also slows down incremental builds. As a developer, I rarely do 
> clean builds, I mostly do incremental builds so this would likely make my 
> experience worse?
> Unity builds also require a lot of code churn to resolve naming conflicts and 
> the resulting code does not look as nice.

My hope is that it won’t make your experience noticeably worse. I’ll keep doing 
experiments to ensure as little impact on incremental builds as possible. 
Unless, you modify just a single cpp file or many cpp files each in their own 
directory unified sources will still likely be a speedup. It’s likely that 
there are some files that we should exclude from unified sources as they take 
disproportionately longer to compile than the surrounding files 
(FTLLowerDFGToB3.cpp I’m looking at you!). 

The reality is that almost all the time spent (90%) compiling a .cpp file in 
JSC is parsing declarations, most of which are in headers. Since I would expect 
WebCore+ to have even more includes than JSC, the impact of unified sources on 
WebCore incremental builds is probably smaller than JSC. Also, keep in mind the 
20% number was from compiling all ~46 Air .cpp files at the same time. Since we 
are only starting with 8 .cpp files per bundle 20% is probably an overestimate.

As a side question, when you do incremental builds are the files changes 
generally all in one directory? Are the files usually headers or cpp files?

> 
> Finally, the statement that the slow down of incremental build is acceptable 
> because we spend most of our time resolving dependencies seems to assume we 
> keep using make. Using Ninja would speed up incremental builds a lot by not
> re-resolving dependencies so much.

I think the solution to incremental builds, at least for Apple developers, is 
to make the CMake build the primary build system. That’s outside the scope of 
what I’m trying to do here but I think it’s also a worthy effort.

> 
> --
>  Chris Dumez
> 
> 
> 
> 
>> On Aug 29, 2017, at 9:05 AM, Darin Adler > > wrote:
>> 
>>> On Aug 28, 2017, at 8:34 PM, Ryosuke Niwa >> > wrote:
>>> 
 Wherever possible, we are going to want to convert file-static functions
 into private C++ class member functions.
>>> 
>>> I don't think we should make this change. It would mean that whenever
>>> the function signature changes, we'd have to modify the header file,
>>> which in turn triggers rebuild of every cpp file which includes that
>>> header file.
>> 
>> 
>> That was my first thought as well. In fact, I typically ask people to do the 
>> opposite: Whenever a function does not require access to private members, 
>> convert from a member function that has to be in the header file to a 
>> function that is private to the source file.
>> 
>> More recently I have started doing something different for the many 
>> functions that only have to be used on one place; I use lambdas to create 
>> nested functions. This has a couple nice properties: The lambda shares 
>> access to private members and anything else that the surrounding function 
>> has access to. We have the option of capturing rather than passing 
>> arguments, which can be clearer in some cases. It’s not quite as good as a 
>> separate function for visual separation; the lambda is often mashed together 
>> with the rest of the surrounding function.
>> 
>> — Darin
>> ___
>> webkit-dev mailing list
>> webkit-dev@lists.webkit.org 
>> https://lists.webkit.org/mailman/listinfo/webkit-dev
> 
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Ben Kelly]

On Tue, Aug 29, 2017 at 12:10 PM, Chris Dumez  wrote:

> I worry about adopting unity build because while it makes clean builds
> faster, it also slows down incremental builds. As a developer, I rarely do
> clean builds, I mostly do incremental builds so this would likely make my
> experience worse?
> Unity builds also require a lot of code churn to resolve naming conflicts
> and the resulting code does not look as nice.
>

Just to add a data point, the full build time is currently a bit of a
barrier to contributing to the project casually.  If I decided to try to
work on webkit one night I often ended up updating my tree and building,
but not having time for much else.  Even for simple patches this became a
problem for me.  If unified builds cut the full build time by 75% that
could be quite helpful for casual contributors.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Chris Dumez]

> On Aug 29, 2017, at 9:24 AM, Keith Miller  wrote:
> 
>> * We could not use any namespaces and just fix name clashes as they arise, 
>> but only if we have some more predictable way to determine which files will 
>> be built together regardless of port or build configuration. Say files in 
>> small directories all get built together, while files in large directories 
>> get built together if their filenames start with the same letter. That might 
>> be too confusing, but a scheme like that would make it possible for us to 
>> ensure we don't have name clashes regardless of port configuration, while 
>> avoiding the need to add namespaces all over the place.
> 
> I had considered something like this. Ultimately, I decided the FILENAME 
> solution was better because it lets us use descriptive names without having 
> to unique the name. When new cpp files are added to the project, if we don’t 
> use a namespace solution new errors in totally unrelated files may appear due 
> to reordering of the unified source lists. For new developers, errors 
> exclusively in .cpp files they didn’t touch would be particularly frustrating 
> and confusing. e.g. They add Foo.cpp which causes Bar.cpp and Baz.cpp to get 
> bundled together causing a redefinition error in Bar/Baz.cpp. While, new 
> developers might also be annoyed with the FILENAME solution, it’s a pretty 
> simple rule to follow and it doesn’t lead to the same surprises. 


Keep in mind that if you do this, you may not get a build error and still run 
into trouble. If several static functions end up having the same name but 
different parameters, your code may build but may end up crashing or with 
different behavior because we end up calling a different overload. This would 
seem very risky for a big project such as WebKit.

--
 Chris Dumez

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Keith Miller]

> On Aug 28, 2017, at 8:10 PM, Michael Catanzaro  wrote:
> 
> On Mon, Aug 28, 2017 at 7:37 PM, Keith Miller  wrote:
>> Given that most of the build time in incremental builds is scanning 
>> dependencies, this change is probably only barely noticeable.
> 
> What, with CMake? That's true of the make backend, but definitely not the 
> ninja backend. You should pass -GNinja to get the ninja generator. I thought 
> the build-webkit script already did this by default.
> 
>> Bundling happens as part of the CMake configuration process, in the CMake 
>> build.
> 
> Do you have a patch or branch somewhere we could look at? This is very 
> exciting!
> 
>> namespace FILENAME {
>>  static int myVariable { 0 };
>>  static int myFunction() { return myVariable; }
>> }
> 
> I'm afraid you might be underestimating how much pain this would be. We use 
> file-static functions all over the place, scattered across source files. We 
> can't reasonably add namespace guards around individual functions scattered 
> all throughout files, so we're going to need to move them all to one place, 
> in the files, which will seriously hurt readability. Regardless, we will need 
> a lot of forward declarations to make this workable. (At least those can all 
> go in one place.)

There are several solutions that we could do here. We can forward declare the 
static functions and have definitions closer to where they are used. I’m also 
not sure that wrapping statics in namespace is all that bad but I do agree that 
it’s quite unattractive. We could also have multiple clusters of namespace 
FILENAMEs as we see fit.

> 
> I'm not saying this is a bad idea. I think we should try it. We should be 
> willing to put up with some annoyance in order to get faster builds. But I 
> suspect converting the WebCore and WebKit layers is going to be a lot harder 
> than JSC.

I’m not sure this is actually true JSC uses a lot of static functions. Also, I 
wrote a script that does much of the conversion automatically and I can 
manually clean up the rest. I’ll put some different versions of files on 
bugzilla so we can all take a look at what we think the best way to use the 
namespace is. Although, I think the right answer is to likely rely on taste.

> 
> Wherever possible, we are going to want to convert file-static functions into 
> private C++ class member functions. That will go a long way to easing the 
> pain in C++ files, and it will probably be a lot of work, but it seems like a 
> no-brainer task if we're going to be using unified builds. GObject 
> implementations are going to be harder though, since that won't be an option. 
> But it would be simple to just exclude those; they're probably a very small 
> component of overall build time. And I have no idea how this will work with 
> Objective C++, but those could be excluded too if need be.
> 
> Some alternative strawman proposals:
> 
> * We could not use any namespaces and just fix name clashes as they arise, 
> but only if we have some more predictable way to determine which files will 
> be built together regardless of port or build configuration. Say files in 
> small directories all get built together, while files in large directories 
> get built together if their filenames start with the same letter. That might 
> be too confusing, but a scheme like that would make it possible for us to 
> ensure we don't have name clashes regardless of port configuration, while 
> avoiding the need to add namespaces all over the place.

I had considered something like this. Ultimately, I decided the FILENAME 
solution was better because it lets us use descriptive names without having to 
unique the name. When new cpp files are added to the project, if we don’t use a 
namespace solution new errors in totally unrelated files may appear due to 
reordering of the unified source lists. For new developers, errors exclusively 
in .cpp files they didn’t touch would be particularly frustrating and 
confusing. e.g. They add Foo.cpp which causes Bar.cpp and Baz.cpp to get 
bundled together causing a redefinition error in Bar/Baz.cpp. While, new 
developers might also be annoyed with the FILENAME solution, it’s a pretty 
simple rule to follow and it doesn’t lead to the same surprises. 

> 
> * We could run all source files through a script that looks for file-static 
> symbols and manually renames them to include the filename. Of course that 
> would  itself slow down the build, but this would have the least overall 
> impact on our development. The script would need to be smart enough not to 
> touch class-static functions in C++ classes that are declared in source files.
> 

Having the build script change source files seems dangerous and prone to 
introducing bugs. If we copied the files elsewhere then the source in the 
debugger wouldn’t match the source in your editor, which seems extra confusing 
for people not in the know. I’m not saying this isn’t a 

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Chris Dumez]

I worry about adopting unity build because while it makes clean builds faster, 
it also slows down incremental builds. As a developer, I rarely do clean 
builds, I mostly do incremental builds so this would likely make my experience 
worse?
Unity builds also require a lot of code churn to resolve naming conflicts and 
the resulting code does not look as nice.

Finally, the statement that the slow down of incremental build is acceptable 
because we spend most of our time resolving dependencies seems to assume we 
keep using make. Using Ninja would speed up incremental builds a lot by not
re-resolving dependencies so much.

--
 Chris Dumez




> On Aug 29, 2017, at 9:05 AM, Darin Adler  wrote:
> 
>> On Aug 28, 2017, at 8:34 PM, Ryosuke Niwa > > wrote:
>> 
>>> Wherever possible, we are going to want to convert file-static functions
>>> into private C++ class member functions.
>> 
>> I don't think we should make this change. It would mean that whenever
>> the function signature changes, we'd have to modify the header file,
>> which in turn triggers rebuild of every cpp file which includes that
>> header file.
> 
> 
> That was my first thought as well. In fact, I typically ask people to do the 
> opposite: Whenever a function does not require access to private members, 
> convert from a member function that has to be in the header file to a 
> function that is private to the source file.
> 
> More recently I have started doing something different for the many functions 
> that only have to be used on one place; I use lambdas to create nested 
> functions. This has a couple nice properties: The lambda shares access to 
> private members and anything else that the surrounding function has access 
> to. We have the option of capturing rather than passing arguments, which can 
> be clearer in some cases. It’s not quite as good as a separate function for 
> visual separation; the lambda is often mashed together with the rest of the 
> surrounding function.
> 
> — Darin
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Geoffrey Garen]

> I wonder whether the unified sources builds would still allow us to
> keep using compiler caches like ccache.

Our plan is to use ccache.

> Second question/caveat is, wouldn't the bundle build penalize too much
> incremental builds? If your data is correct we would get a <20% build
> time increment for a single change in a cpp file which is I think the
> most common scenario for a WebKit developer.

The penalty for a single file compile is roughly 6s => 7s. The speedup for a 
full project build is roughly 40m => 10m. You would need to perform single file 
compiles 1,800 times per full project build before the tradeoff became “too 
much”.

Geoff
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Myles C. Maxfield]

I often use the “Show Preprocessed Source” option in Xcode. Will I be able to 
continue using this?

> On Aug 29, 2017, at 8:10 AM, Sergio Villar Senin  wrote:
> 
> O Lun, 28-08-2017 ás 17:37 -0700, Keith Miller escribiu:
>> Hello fellow Webkittens,
>> 
>> Are you growing tired of long cat naps while waiting 45 minutes for
>> WebKit to build? (I’ve definitely never done this 蘿!)
>> Do you want WebKit to build up to 3-4x faster on a clean build?*
>> Does seeing your incremental build… stay the same sound good?
>> 
>> You’ll be purring with excitement after you switch to unified source
>> builds!**
> 
> [...]
> 
>> Are there other issues that people think might occur with unified
>> sources? Note, It’s also likely that there are some source files that
>> need to be excluded from the unified source builds for various
>> reasons.
> 
> I wonder whether the unified sources builds would still allow us to
> keep using compiler caches like ccache.
> 
> Second question/caveat is, wouldn't the bundle build penalize too much
> incremental builds? If your data is correct we would get a <20% build
> time increment for a single change in a cpp file which is I think the
> most common scenario for a WebKit developer.
> 
> Don't get me wrong, I'm really excited about this change, just wanted
> to make sure that it does not penalize too much other scenarios.
> 
> BR
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Sergio Villar Senin]

O Lun, 28-08-2017 ás 17:37 -0700, Keith Miller escribiu:
> Hello fellow Webkittens,
> 
> Are you growing tired of long cat naps while waiting 45 minutes for
> WebKit to build? (I’ve definitely never done this 蘿!)
> Do you want WebKit to build up to 3-4x faster on a clean build?*
> Does seeing your incremental build… stay the same sound good?
> 
> You’ll be purring with excitement after you switch to unified source
> builds!**

[...]

> Are there other issues that people think might occur with unified
> sources? Note, It’s also likely that there are some source files that
> need to be excluded from the unified source builds for various
> reasons.

I wonder whether the unified sources builds would still allow us to
keep using compiler caches like ccache.

Second question/caveat is, wouldn't the bundle build penalize too much
incremental builds? If your data is correct we would get a <20% build
time increment for a single change in a cpp file which is I think the
most common scenario for a WebKit developer.

Don't get me wrong, I'm really excited about this change, just wanted
to make sure that it does not penalize too much other scenarios.

BR
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Michael Catanzaro]

On Mon, Aug 28, 2017 at 11:46 PM, Geoffrey Garen  
wrote:

Our plan is to use #include. That has worked in the past.

Geoff


Um yes, that would be better. :)


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Alicia Boya García]

On 08/29/2017 06:20 AM, Daniel Bates wrote:

> Do we know what is the cause(es) for the slow clean builds? I am assuming 
> that much of the speed up from the "unified source" comes from clang being 
> able to use an in-memory #include cache to avoid disk I/O and re-parsing of a 
> seen header. Have we exhausted all efforts (or have we given up?) removing 
> extraneous #includes? Do we think this pursuing this effort would be more 
> time consuming or have results that pale in comparison to the "unified 
> source" approach?

Whilst having an in-process-memory #include cache is not a bad thing,
it's not the greatest gain as the operating systems should already cache
file reads just fine.

The greatest gain comes from reducing the amount of times C++ headers
are parsed. If you are building a certain .cpp file and include a .h
file, the compiler has to parse it, which can take quite a bit because
C++ is really complex monster, especially when templates are used. Doing
this more than necessary raises build times really quickly.

Header files almost always are include-guarded (either with #pragma once
or traditional #ifndef guards), so including the same header twice
within the same .cpp file (or any of its included files) has no cost. On
the other hand, if you then start building a different .cpp file that
also includes the same header, you have to parse it again because so far
C++ is concerned, every inclusion could add different symbols to the AST
the compiler is building, so their output can't be reused*. In turn we
end up parsing most headers many more times than actually needed (i.e.
for every .cpp file that includes Node.h the compiler has to parse
Node.h and all its dependencies from source, that's a lot of wasted
effort!).

*Note that including twice the same .h within the same .cpp file is not
fast because the output is cached in anyway, but because the entire .h
file is skipped, adding no additional nodes to the AST.

The goal of C++ modules is to fix this problem from its root cause:
instead of literally including text header files, .cpp files declare
dependencies on module files that can be compiled, stored, loaded and
referenced from .cpp files in any order, so you would only parse the
Node module source code once for the entire project, whilst the compiler
could load directly the AST every other time from a cached module object
file.

Note the great advantage of modules comes from the fact that they can be
imported in different contexts and their content is still semantically
equivalent, whereas with plain C/C++ includes every header file may act
differently depending on the preprocessor variables defined by the
includer and other previous inclusions. In the worst case, when they are
not include-guarded (luckily this is not too common, but it still
happens), even including the same file in the same .cpp twice could add
different symbols each time!

Unfortunately C++ modules are a work in progress... There are two
different competing proposals with implementations, one from Clang and
another one from Microsoft, and the C++ technical specification is in a
very early stage too:
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2017/n4681.pdf

We know for sure modules are very important for the future of C++, but
maybe it's still a bit too early to bet a big project like WebKit on them.

So how else can we avoid parsing the same header files so many times and
speed up our builds? Enter unified builds.

A requirement for unified builds to work correctly is that header files
are coded in such a way they work as independent units, much like C++
modules, i.e. including headers should work no matter in what order you
place them, and in each case they must define the same symbols. On July
31 I wrote about some issues we currently have because of not doing
exactly this in WebKit (particularly, our #include "config.h" lines are
ambiguous). They can be worked around so they will not become blockers
for unified builds, but I still think we should fix them at some point.

Once you have a set of .cpp files whose includes all (1) are guarded
(e.g. by #pragma once) and (2) are independent units according to the
above rule, you can take advantage of unified builds:

Instead of invoking once the compiler for each .cpp file, you create a
new artificial "unified" or "bundle" .cpp file that concatenates (or
#include's) a number of different .cpp files. This way, headers included
within the bundle are parsed only once, even if they are used by
different individual .cpp files, as long as they are within the same
bundle. This can often result in a massive build speed gain.

Unfortunately, there are some pitfalls, as there is a dissonance between
what the programmer thinks the compilation unit is (the individual .cpp
file) and the actual compilation unit used by the compiler (the bundle
.cpp file).

* `static` variables and functions are intended to be scoped to the
individual .cpp file, but the compiler has no way to 

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Geoffrey Garen]

> The line numbers and filenames will be total nonsense if we just concatenate 
> multiple source files together. But that's very easy to fix if the script 
> that concatenates the sources also adds a #line statement between "files" to 
> change the filename and reset the line number to 1. See 
> https://gcc.gnu.org/onlinedocs/cpp/Line-Control.html (I suspect it will work 
> just fine for Clang as well).

Our plan is to use #include. That has worked in the past.

Geoff
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Michael Catanzaro]

On Mon, Aug 28, 2017 at 11:20 PM, Daniel Bates  
wrote:
How does the speed gain with your proposed "unified source" approach 
compare to using CMake + Ninja to build currently (I think builder 
Apple El Capitan CMake Debug does this)?


I don't have any hard numbers, but anecdotal: the speedup from using 
Ninja instead of Make is insignificant for clean builds. Ninja helps a 
lot with incremental builds where you've only touched a couple files, 
because you don't need to wait ages for make to calculate dependencies. 
But Keith's project is to improve clean builds.


On Mon, Aug 28, 2017 at 11:20 PM, Daniel Bates  
wrote:
How does this work if you build from Xcode? I am assuming these 
bundles won't interfere with debugging in Xcode. When debugging in 
Xcode/lldb/gdb will Xcode/lldb/gdb resolve line information with 
respect to individual files or the bundles?


The line numbers and filenames will be total nonsense if we just 
concatenate multiple source files together. But that's very easy to fix 
if the script that concatenates the sources also adds a #line statement 
between "files" to change the filename and reset the line number to 1. 
See https://gcc.gnu.org/onlinedocs/cpp/Line-Control.html (I suspect it 
will work just fine for Clang as well).


Michael

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Maciej Stachowiak]

> On Aug 28, 2017, at 9:20 PM, Daniel Bates  wrote:
> 
> Thank you for looking into speeding up the build.
> 
> How does the speed gain with your proposed "unified source" approach compare 
> to using CMake + Ninja to build currently (I think builder Apple El Capitan 
> CMake Debug does this)?

I think unified sources help in a way that's orthogonal to what CMake+Ninja do, 
so we'd ideally want to apply both to a build of the whole WebKit stack.

Ideally this combination would become the default for engineering builds of the 
macOS and iOS ports.

> Do we know what is the cause(es) for the slow clean builds? I am assuming 
> that much of the speed up from the "unified source" comes from clang being 
> able to use an in-memory #include cache to avoid disk I/O and re-parsing of a 
> seen header. Have we exhausted all efforts (or have we given up?) removing 
> extraneous #includes? Do we think this pursuing this effort would be more 
> time consuming or have results that pale in comparison to the "unified 
> source" approach?

My 2c:

Probably the better long-term solution would be C++ modules once they are ready 
for prime time. In the short term, unified sources seem like a much faster way 
to get a big win than include cleanup. There's also some reason to believe 
unified sources can result in a higher potential win than include pruning, 
since we in fact have many headers that legitimately need to be included in 
lots of files. 


 - Maciej

> 
> 
>> On Aug 28, 2017, at 5:37 PM, Keith Miller  wrote:
>> 
>> Hello fellow Webkittens,
>> 
>> Are you growing tired of long cat naps while waiting 45 minutes for WebKit 
>> to build? (I’ve definitely never done this 蘿!)
>> Do you want WebKit to build up to 3-4x faster on a clean build?*
>> Does seeing your incremental build… stay the same sound good?
>> 
>> You’ll be purring with excitement after you switch to unified source 
>> builds!**
>> 
>> * Building JSC with CMake, including CMake configuration time, went from 
>> ~8m15s to ~2m30s on my MBP using unified sources only on cpp files in 
>> JavaScriptCore. Final results on WebKit may vary. Using unified sources may 
>> cause sudden feelings of euphoria and productivity (or grumpiness from lack 
>> of naps…). Not responsible for any bugs in extra patches produced by using 
>> unified source builds (excluding build system related bugs) except where 
>> required by law (pi...@apple.com).
>> 
>> ** Soon. Unified source builds haven’t been landed yet.
>> 
>> How are things going to change? Great question! Let’s find out!
>> 
>> When you fire off the build script it will automagically paw your source 
>> files into bundles.
> 
> How does this work if you build from Xcode? I am assuming these bundles won't 
> interfere with debugging in Xcode. When debugging in Xcode/lldb/gdb will 
> Xcode/lldb/gdb resolve line information with respect to individual files or 
> the bundles?
> 
>> The size of each bundle is speculatively going to be 8 .cpp files but that’s 
>> up for experimentation. From my testing so far, the compile time for a 
>> bundle is at most 20% (~6s vs ~7s for the Air directory) longer than the 
>> longest file it includes. Given that most of the build time in incremental 
>> builds is scanning dependencies, this change is probably only barely 
>> noticeable.
>> 
>> Bundling happens as part of the CMake configuration process, in the CMake 
>> build. In the Xcode build it’s more complicated since we cannot dynamically 
>> choose the files we are going to compile. The only solution I know of is to 
>> pre-list a bunch of files for the build to dump files into. Occasionally, 
>> we’ll need to add more files to the build list.
>> 
>> When you add or remove a file in the project, the cost is higher. The 
>> current plan to bundle source files is by directory. This means that when a 
>> .cpp file is added or removed you will rebalance the bundles in its 
>> directory, and you will have to rebuild up to the full directory that file 
>> is in. 
>> 
>> My goal is to make all of this invisible to developers as much as possible. 
>> I believe, for the most part, things should operate mostly as they have 
>> before. As the details on unified source builds get finalized, I’m sending 
>> out a separate email documenting any workflow changes. I’ll also do my best 
>> to answer any questions or concerns.
>> 
>> “But Keith, am I going to have to make major changes to the way I develop 
>> WebKit?” Yes and, hopefully, no. There is one major common workflow change 
>> that comes with unified sources, the naming of static variables and 
>> functions.
> 
> I take it you mean any file-level function or definition with internal 
> linkage. That is, this would also affect file-level constants.
> 
>> Since C++ does not have a way to only declare a static function or variable 
>> in some restricted top level scope we are going to need to create some work 
>> around. The choice I’m 

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Daniel Bates]

Thank you for looking into speeding up the build.

How does the speed gain with your proposed "unified source" approach compare to 
using CMake + Ninja to build currently (I think builder Apple El Capitan CMake 
Debug does this)?

Do we know what is the cause(es) for the slow clean builds? I am assuming that 
much of the speed up from the "unified source" comes from clang being able to 
use an in-memory #include cache to avoid disk I/O and re-parsing of a seen 
header. Have we exhausted all efforts (or have we given up?) removing 
extraneous #includes? Do we think this pursuing this effort would be more time 
consuming or have results that pale in comparison to the "unified source" 
approach?


> On Aug 28, 2017, at 5:37 PM, Keith Miller  wrote:
> 
> Hello fellow Webkittens,
> 
> Are you growing tired of long cat naps while waiting 45 minutes for WebKit to 
> build? (I’ve definitely never done this 蘿!)
> Do you want WebKit to build up to 3-4x faster on a clean build?*
> Does seeing your incremental build… stay the same sound good?
> 
> You’ll be purring with excitement after you switch to unified source builds!**
> 
> * Building JSC with CMake, including CMake configuration time, went from 
> ~8m15s to ~2m30s on my MBP using unified sources only on cpp files in 
> JavaScriptCore. Final results on WebKit may vary. Using unified sources may 
> cause sudden feelings of euphoria and productivity (or grumpiness from lack 
> of naps…). Not responsible for any bugs in extra patches produced by using 
> unified source builds (excluding build system related bugs) except where 
> required by law (pi...@apple.com).
> 
> ** Soon. Unified source builds haven’t been landed yet.
> 
> How are things going to change? Great question! Let’s find out!
> 
> When you fire off the build script it will automagically paw your source 
> files into bundles.

How does this work if you build from Xcode? I am assuming these bundles won't 
interfere with debugging in Xcode. When debugging in Xcode/lldb/gdb will 
Xcode/lldb/gdb resolve line information with respect to individual files or the 
bundles?

> The size of each bundle is speculatively going to be 8 .cpp files but that’s 
> up for experimentation. From my testing so far, the compile time for a bundle 
> is at most 20% (~6s vs ~7s for the Air directory) longer than the longest 
> file it includes. Given that most of the build time in incremental builds is 
> scanning dependencies, this change is probably only barely noticeable.
> 
> Bundling happens as part of the CMake configuration process, in the CMake 
> build. In the Xcode build it’s more complicated since we cannot dynamically 
> choose the files we are going to compile. The only solution I know of is to 
> pre-list a bunch of files for the build to dump files into. Occasionally, 
> we’ll need to add more files to the build list.
> 
> When you add or remove a file in the project, the cost is higher. The current 
> plan to bundle source files is by directory. This means that when a .cpp file 
> is added or removed you will rebalance the bundles in its directory, and you 
> will have to rebuild up to the full directory that file is in. 
> 
> My goal is to make all of this invisible to developers as much as possible. I 
> believe, for the most part, things should operate mostly as they have before. 
> As the details on unified source builds get finalized, I’m sending out a 
> separate email documenting any workflow changes. I’ll also do my best to 
> answer any questions or concerns.
> 
> “But Keith, am I going to have to make major changes to the way I develop 
> WebKit?” Yes and, hopefully, no. There is one major common workflow change 
> that comes with unified sources, the naming of static variables and functions.

I take it you mean any file-level function or definition with internal linkage. 
That is, this would also affect file-level constants.

> Since C++ does not have a way to only declare a static function or variable 
> in some restricted top level scope we are going to need to create some work 
> around. The choice I’m thinking we’ll go with is to have an auto-generated 
> macro value FILENAME, which is similar to __FILE__ except that it doesn’t 
> have file extension (We can’t use FILE since that’s actually a type in C ). 
> This macro would be redefined in the unified source directly above the file 
> that is about to be included. Then in each cpp file that has static 
> functions/variables developers would have:
> 
> namespace FILENAME {
>   static int myVariable { 0 };
>   static int myFunction() { return myVariable; }
> }
> 
> The advantage of using a macro over textually writing the filename is that it 
> makes code more portable between files. Also, once people get used to the 
> concept of FILENAME in the code base it will, hopefully become obvious what 
> it means. The main downside to this is using FILENAME in a header would give 
> you whatever cpp file first included 

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Michael Catanzaro]

On Mon, Aug 28, 2017 at 10:34 PM, Ryosuke Niwa  wrote:

I don't think we should make this change. It would mean that whenever
the function signature changes, we'd have to modify the header file,
which in turn triggers rebuild of every cpp file which includes that
header file.


Good point.

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Ryosuke Niwa]

On Mon, Aug 28, 2017 at 8:10 PM, Michael Catanzaro
 wrote:
> On Mon, Aug 28, 2017 at 7:37 PM, Keith Miller 
> wrote:
>>
>> Given that most of the build time in incremental builds is scanning
>> dependencies, this change is probably only barely noticeable.
>
>
> What, with CMake? That's true of the make backend, but definitely not the
> ninja backend. You should pass -GNinja to get the ninja generator. I thought
> the build-webkit script already did this by default.
>
>> Bundling happens as part of the CMake configuration process, in the CMake
>> build.
>
>
> Do you have a patch or branch somewhere we could look at? This is very
> exciting!
>
>> namespace FILENAME {
>> static int myVariable { 0 };
>> static int myFunction() { return myVariable; }
>> }
>
>
> I'm afraid you might be underestimating how much pain this would be. We use
> file-static functions all over the place, scattered across source files. We
> can't reasonably add namespace guards around individual functions scattered
> all throughout files, so we're going to need to move them all to one place,
> in the files, which will seriously hurt readability. Regardless, we will
> need a lot of forward declarations to make this workable. (At least those
> can all go in one place.)
>
> I'm not saying this is a bad idea. I think we should try it. We should be
> willing to put up with some annoyance in order to get faster builds. But I
> suspect converting the WebCore and WebKit layers is going to be a lot harder
> than JSC.
>
> Wherever possible, we are going to want to convert file-static functions
> into private C++ class member functions.

I don't think we should make this change. It would mean that whenever
the function signature changes, we'd have to modify the header file,
which in turn triggers rebuild of every cpp file which includes that
header file.

- R. Niwa
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Michael Catanzaro]

On Mon, Aug 28, 2017 at 7:37 PM, Keith Miller  
wrote:
Given that most of the build time in incremental builds is scanning 
dependencies, this change is probably only barely noticeable.


What, with CMake? That's true of the make backend, but definitely not 
the ninja backend. You should pass -GNinja to get the ninja generator. 
I thought the build-webkit script already did this by default.


Bundling happens as part of the CMake configuration process, in the 
CMake build.


Do you have a patch or branch somewhere we could look at? This is very 
exciting!



namespace FILENAME {
static int myVariable { 0 };
static int myFunction() { return myVariable; }
}


I'm afraid you might be underestimating how much pain this would be. We 
use file-static functions all over the place, scattered across source 
files. We can't reasonably add namespace guards around individual 
functions scattered all throughout files, so we're going to need to 
move them all to one place, in the files, which will seriously hurt 
readability. Regardless, we will need a lot of forward declarations to 
make this workable. (At least those can all go in one place.)


I'm not saying this is a bad idea. I think we should try it. We should 
be willing to put up with some annoyance in order to get faster builds. 
But I suspect converting the WebCore and WebKit layers is going to be a 
lot harder than JSC.


Wherever possible, we are going to want to convert file-static 
functions into private C++ class member functions. That will go a long 
way to easing the pain in C++ files, and it will probably be a lot of 
work, but it seems like a no-brainer task if we're going to be using 
unified builds. GObject implementations are going to be harder though, 
since that won't be an option. But it would be simple to just exclude 
those; they're probably a very small component of overall build time. 
And I have no idea how this will work with Objective C++, but those 
could be excluded too if need be.


Some alternative strawman proposals:

* We could not use any namespaces and just fix name clashes as they 
arise, but only if we have some more predictable way to determine which 
files will be built together regardless of port or build configuration. 
Say files in small directories all get built together, while files in 
large directories get built together if their filenames start with the 
same letter. That might be too confusing, but a scheme like that would 
make it possible for us to ensure we don't have name clashes regardless 
of port configuration, while avoiding the need to add namespaces all 
over the place.


* We could run all source files through a script that looks for 
file-static symbols and manually renames them to include the filename. 
Of course that would  itself slow down the build, but this would have 
the least overall impact on our development. The script would need to 
be smart enough not to touch class-static functions in C++ classes that 
are declared in source files.


Michael

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev