Re: [webkit-dev] Compiling WebKit up to 25% faster in Windows?
Den 2013-03-28 18:53:48 skrev Daniel Bratell brat...@opera.com: On a Xeon W3550 (quad 3.06GHz), with plenty of RAM but a spinning disk and Windows 7: webcore_dom: 58 seconds - 38 seconds (-35%) webcore_rendering: 106 seconds - 73 seconds (-30%) webcore_platform: 59 seconds - 34 seconds (-43%) (Yes, better than the 25% mentioned in the subject but this was on a different computer) I understand that this is not as interesting anymore since gyp is gone and Windows is a smaller platform than it used to be, but for the record, I've kept working on it in between other things and the total gain is about 7 minutes, which is better than the 25% estimate. This is in chromium land with the patch in http://pastebin.com/90vx4sep and the script in http://pastebin.com/WmzGY8zs. Both are pre-blink so they should look familiar, though they apply to gyp files which are gone. Neither have been cleaned up so they are a bit embarrassing but if someone wants to keep working on this in WebKit they might be a good starting point. To do in that case: * Port to the new build system for Windows. * Test compile a lot of platform and verify gains in Windows. * Integrate the idltopath.pm map generator into the build system. * Apply. /Daniel ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Compiling WebKit up to 25% faster in Windows?
Den 2013-03-26 19:21:32 skrev Daniel Bratell brat...@opera.com: As an experiment we took the (chromium) project webcore_dom, that normally compiles in 56 seconds in Windows on a generic computer and fixed it. Removing the many include paths in the build system and instead specifying the path in the include directives changed that to 42 seconds, a 25% reduction. I've looked some more at this today and it still looks promising. On a Xeon W3550 (quad 3.06GHz), with plenty of RAM but a spinning disk and Windows 7: webcore_dom: 58 seconds - 38 seconds (-35%) webcore_rendering: 106 seconds - 73 seconds (-30%) webcore_platform: 59 seconds - 34 seconds (-43%) (Yes, better than the 25% mentioned in the subject but this was on a different computer) Just adding paths to the files cut a few percent of the compilation time. The big gain is from shrinking the list of include paths sent to the compiler. The data points reported here are the best times, but compile times were consistent over a number of attempts so now I trust them 100%. There are some roadblocks though, and I wouldn't mind some pointers here. (This is my first deep dive into how the source generation/build system works in WebKit) A lot of the source code, basically all WebCore projects I didn't list above, use files automatically generated by a number of massive perl scripts. The generated files include header files without any idea where they are. It would be good if they could generate something correct. Any suggestion there? My easter celebrating colleague talked about trying the suggestion to symlink all headers in a single directory. I've not tried that though and I have no comparable data. /Daniel ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Compiling WebKit up to 25% faster in Windows?
Den 2013-03-27 02:11:45 skrev Nico Weber tha...@chromium.org: On Tue, Mar 26, 2013 at 4:51 PM, Daniel Bratell brat...@opera.com wrote: Den 2013-03-26 21:20:10 skrev Dirk Pranke dpra...@chromium.org: If we have consensus that we should just switch to paths relative to Source (or maybe a couple different options), that would be (IMO) a big win. It sounds like Daniel co. have already done the big bang conversion. Not the whole conversion, no. Just enough to confirm the suspicion that it affects compile time in Windows. As other people in the thread have mentioned there are some tricky areas so before investing too much time in this, it seemed to be best to check with people with longer experience of the code if there was something we've missed. Though the conversion is scripted so it can possibly be applied to large portions of the source code and then the tricky areas can be manually fixed. Can you share your scripts, so that we can measure how this changes things on other platforms? I can show it to you but it's not creating a compilable set of sources. It has bugs and you need to go through the source and fix a lot. http://pastebin.com/JSFzFwTt has the last version I have access to. Another guy here have a better version but he's celebrating easter somewhere so I haven't been able to get access to it today. :-p /Daniel ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Compiling WebKit up to 25% faster in Windows?
Den 2013-03-27 20:05:03 skrev Brent Fulgham bfulg...@gmail.com: Hi Daniel, On Tue, Mar 26, 2013 at 11:21 AM, Daniel Bratell brat...@opera.com wrote: As an experiment we took the (chromium) project webcore_dom, that normally compiles in 56 seconds in Windows on a generic computer and fixed it. Removing the many include paths in the build system and instead specifying the path in the include directives changed that to 42 seconds, a 25% reduction. I thought that much of this was supposed to be addressed by the use of precompiled headers. Presumably, if the header files are properly incorporated into the PCH, shouldn't any gains from relative paths be small? Obviously your statistic says otherwise, but I'm not sure that a single test on a single system is definitive proof of anything. Precompiled headers could help some but they also cost some, and will trigger much bigger recompilations when you are changing files. I would welcome any way to make it easier and faster to develop WebKit but I haven't looked at that myself. Maybe someone else at Opera has but I've not heard about it. Did you run the test multiple times to get a feel for how reproducible the improvements was? I know I have fooled myself in the past into thinking I had improved something, only to discover that unrelated computer activity (e.g., backups, virus scans, etc.) were contributing to slow build times. For sure it needs more analysis. For instance, is the webcore_dom savings representative of the whole of webkit? Is it reproducible across every machine or is the slowness magnified by some factor that doesn't apply to everyone? Examples of good questions that I can't answer yet. If the initial analysis is correct, and the fix scales to the whole of WebKit the savings are huge for Windows developers but it's hard to say for sure without actually doing the whole conversion and we (or at least not I) have not done that yet. It's also not done in a few minutes by someone rather unfamiliar with all the different header files that share the same name so it's easy to mess up. The reason I posted here was to get some feedback and avoid spending a lot of precious time on a fool's errand. /Daniel ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Compiling WebKit up to 25% faster in Windows?
Den 2013-03-27 20:40:19 skrev Pau Garcia i Quiles pgqui...@elpauer.org: Hello, This thread already contains about 30 speculative messages. What about providing a patch for the whole WebKit and some benchmarks on the main platforms and compilers? Easier said than done. But for sure we will keep looking at this if nobody else does it. Nothing here has discouraged me, though such a patch, should it work, will be huge (7500 changed files in my tree right now) and tricky to handle. The potential gain is huge too though. /Daniel ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
[webkit-dev] Compiling WebKit up to 25% faster in Windows?
Hi WebKittens, As you might be aware, we at Opera now have Chromium based products which means that we compile WebKit quite a lot. A big issue for us and our automatic systems has been the long time needed to compile WebKit (inside Chromium), especially in Windows. The big compilation time difference between Linux and Windows didn't seem to make sense so we did some analysis of what was going on and we think that we have found a cause. In WebKit include directives are without path, and instead the compiler is given a very long list of directories to search through. That process takes a lot of time in Windows. It must take some time in OSX and in Linux too but probably less. As an experiment we took the (chromium) project webcore_dom, that normally compiles in 56 seconds in Windows on a generic computer and fixed it. Removing the many include paths in the build system and instead specifying the path in the include directives changed that to 42 seconds, a 25% reduction. There is no reason to think the same reduction doesn't apply to all projects and then there are many many minutes to save for developers and build servers here. Caveat: I don't know if the resulting binary is correct. Is this something that has been talked about in the past, and would you be interested in replacing the long list of directories to search for every include with paths (relative some good base) directly in the include directives? /Daniel ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Compiling WebKit up to 25% faster in Windows?
Den 2013-03-26 21:29:32 skrev Benjamin Poulain benja...@webkit.org: On Tue, Mar 26, 2013 at 1:20 PM, Dirk Pranke dpra...@chromium.org wrote If we have consensus that we should just switch to paths relative to Source (or maybe a couple different options), that would be (IMO) a big win. It sounds like Daniel co. have already done the big bang conversion. I think using full path would be a serious hit regarding hackability. I would rather spend some time tweaking my compiler to cache each directory content than waste time finding where is every single header I need to include. I guess you mean that it will be more job moving files around, but that is a rather rare operation, while reading an include directive and wondering what it's part of is rather common (both for compilers, tools and humans). I like the paths as a tool to indicate module dependencies. You can more easily see that a file depends on foo and bar (but not on fie) if you see: #include foo/object.h #include foo/thing.h #include bar/stuff.h That will tell you useful things, and avoid making layering violations by accident. But I realize it's a question of style and as such there is not a right and a wrong, unless there are other factors. And here we have the seemingly heavy compilation time cost for it which I think is a strong argument against delegating the task of finding the header files to the compiler. /Daniel ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Compiling WebKit up to 25% faster in Windows?
Den 2013-03-26 21:20:10 skrev Dirk Pranke dpra...@chromium.org: If we have consensus that we should just switch to paths relative to Source (or maybe a couple different options), that would be (IMO) a big win. It sounds like Daniel co. have already done the big bang conversion. Not the whole conversion, no. Just enough to confirm the suspicion that it affects compile time in Windows. As other people in the thread have mentioned there are some tricky areas so before investing too much time in this, it seemed to be best to check with people with longer experience of the code if there was something we've missed. Though the conversion is scripted so it can possibly be applied to large portions of the source code and then the tricky areas can be manually fixed. /Daniel ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
