Re: Analyze C++/compiler usage and code stats easily
On 11/29/13, 4:00 AM, Ehsan Akhgari wrote: On 11/25/2013, 7:41 PM, Gregory Szorc wrote: The repo was hiding that old changeset because it has been obsoleted (Mercurial changeset evolution magic). I updated the server config to expose hidden changesets, so the link works again. However, the changeset has been obsoleted, so you'll want to fetch the newest one. The best way to fetch it is to pull the bookmark tracking it: $ hg pull -B gps/build/measure-compiler http://hg.gregoryszorc.com/gecko-collab Or, look for that bookmark at http://hg.gregoryszorc.com/gecko-collab/bookmarks and track down the changeset. I'm not sure how to integrate any of this into my workflow... :( Do you plan to land this stuff in the near future? There are no plans to land this. File a bug and we can make it happen. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Analyze C++/compiler usage and code stats easily
On 11/25/2013, 7:41 PM, Gregory Szorc wrote: The repo was hiding that old changeset because it has been obsoleted (Mercurial changeset evolution magic). I updated the server config to expose hidden changesets, so the link works again. However, the changeset has been obsoleted, so you'll want to fetch the newest one. The best way to fetch it is to pull the bookmark tracking it: $ hg pull -B gps/build/measure-compiler http://hg.gregoryszorc.com/gecko-collab Or, look for that bookmark at http://hg.gregoryszorc.com/gecko-collab/bookmarks and track down the changeset. I'm not sure how to integrate any of this into my workflow... :( Do you plan to land this stuff in the near future? Thanks! Ehsan ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Analyze C++/compiler usage and code stats easily
Gregory Szorc wrote: The best way to fetch it is to pull the bookmark tracking it: $ hg pull -B gps/build/measure-compiler http://hg.gregoryszorc.com/gecko-collab Or, look for that bookmark at http://hg.gregoryszorc.com/gecko-collab/bookmarks and track down the changeset. Note that for bookmarks that don't contain /s you can just use (e.g.) http://hg.gregoryszorc.com/gecko-collab/rev/fx to view the changeset. (Also, while looking at the Mercurial source code to see if I could expose hidden changesets through the web UI, I noticed that the official resource/URL for linking to changesets is changeset not rev. So, all our Bugzilla URLs using /rev/node are using an alias. I'm curious if this is intentional or an accident.) Interestingly the bookmarks page itself links to /rev/ too... -- Warning: May contain traces of nuts. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Analyze C++/compiler usage and code stats easily
This patch doesn't seem to exist any more. Do you have another copy of it lying somewhere? Thanks! -- Ehsan http://ehsanakhgari.org/ On Fri, Nov 15, 2013 at 12:43 AM, Gregory Szorc g...@mozilla.com wrote: C++ developers, Over 90% of the CPU time required to build the tree is spent compiling or linking C/C++. So, anything we can do to make that faster will make the overall build faster. I put together a quick patch [1] to make it rather simple to extract compiler resource usage and very basic code metrics during builds. Simply apply that patch and build with `mach build --profile-compiler` and your machine will produce all kinds of potentially interesting measurements. They will be stuffed into objdir/.mozbuild/compilerprof/. If you don't feel like waiting (it will take about 5x longer than a regular build because it performs separate preprocessing, ast, and codegen compiler invocations 3 times each), grab an archive of an OS X build I just performed from [2] and extract it to objdir/.mozbuild/. I put together an extremely simple `mach compiler-analyze` command to sift through the results. e.g. $ mach compiler-analyze preprocessor-relevant-lines $ mach compiler-analyze codegen-sizes $ mach compiler-analyze codegen-total-times Just run `mach help compiler-analyze` to see the full list of what can be printed. Or, write your own code to analyze the produced JSON files. I'm sure people who love getting down and dirty with C++ will be interested in this data. I have no doubt that are compiler time and code size wins waiting to be discovered through this data. We may even uncover a perf issue or two. Who knows. Here are some questions I have after casually looking at the data: * The mean ratio of .o size to lines from preprocessor is 16.35 bytes/line. Why do 38/4916 (0.8%) files have a ratio over 100? Why are a lot of these in js/ and gfx? * What's in the 150 files that have 100k+ lines after preprocessing that makes them so large? * Why does lots of js/'s source code gravitate towards the bad extreme for most of the metrics (code size, compiler time, preprocessor size)? Disclaimer: This patch is currently hacked together. If there is an interest in getting this checked into the tree, I can clean it up and do it. Just file a bug in Core :: Build Config and I'll make it happen when I have time. Or, if an interested party wants to champion getting it landed, I'll happily hand it off :) [1] http://hg.gregoryszorc.com/gecko-collab/rev/741f3074e313 [2] https://people.mozilla.org/~gszorc/compiler_profiles.tar.bz2 ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Analyze C++/compiler usage and code stats easily
The repo was hiding that old changeset because it has been obsoleted (Mercurial changeset evolution magic). I updated the server config to expose hidden changesets, so the link works again. However, the changeset has been obsoleted, so you'll want to fetch the newest one. The best way to fetch it is to pull the bookmark tracking it: $ hg pull -B gps/build/measure-compiler http://hg.gregoryszorc.com/gecko-collab Or, look for that bookmark at http://hg.gregoryszorc.com/gecko-collab/bookmarks and track down the changeset. (Also, while looking at the Mercurial source code to see if I could expose hidden changesets through the web UI, I noticed that the official resource/URL for linking to changesets is changeset not rev. So, all our Bugzilla URLs using /rev/node are using an alias. I'm curious if this is intentional or an accident.) On 11/25/13, 9:35 PM, Ehsan Akhgari wrote: This patch doesn't seem to exist any more. Do you have another copy of it lying somewhere? Thanks! -- Ehsan http://ehsanakhgari.org/ On Fri, Nov 15, 2013 at 12:43 AM, Gregory Szorc g...@mozilla.com mailto:g...@mozilla.com wrote: C++ developers, Over 90% of the CPU time required to build the tree is spent compiling or linking C/C++. So, anything we can do to make that faster will make the overall build faster. I put together a quick patch [1] to make it rather simple to extract compiler resource usage and very basic code metrics during builds. Simply apply that patch and build with `mach build --profile-compiler` and your machine will produce all kinds of potentially interesting measurements. They will be stuffed into objdir/.mozbuild/compilerprof/__. If you don't feel like waiting (it will take about 5x longer than a regular build because it performs separate preprocessing, ast, and codegen compiler invocations 3 times each), grab an archive of an OS X build I just performed from [2] and extract it to objdir/.mozbuild/. I put together an extremely simple `mach compiler-analyze` command to sift through the results. e.g. $ mach compiler-analyze preprocessor-relevant-lines $ mach compiler-analyze codegen-sizes $ mach compiler-analyze codegen-total-times Just run `mach help compiler-analyze` to see the full list of what can be printed. Or, write your own code to analyze the produced JSON files. I'm sure people who love getting down and dirty with C++ will be interested in this data. I have no doubt that are compiler time and code size wins waiting to be discovered through this data. We may even uncover a perf issue or two. Who knows. Here are some questions I have after casually looking at the data: * The mean ratio of .o size to lines from preprocessor is 16.35 bytes/line. Why do 38/4916 (0.8%) files have a ratio over 100? Why are a lot of these in js/ and gfx? * What's in the 150 files that have 100k+ lines after preprocessing that makes them so large? * Why does lots of js/'s source code gravitate towards the bad extreme for most of the metrics (code size, compiler time, preprocessor size)? Disclaimer: This patch is currently hacked together. If there is an interest in getting this checked into the tree, I can clean it up and do it. Just file a bug in Core :: Build Config and I'll make it happen when I have time. Or, if an interested party wants to champion getting it landed, I'll happily hand it off :) [1] http://hg.gregoryszorc.com/__gecko-collab/rev/741f3074e313 http://hg.gregoryszorc.com/gecko-collab/rev/741f3074e313 [2] https://people.mozilla.org/~__gszorc/compiler_profiles.tar.__bz2 https://people.mozilla.org/~gszorc/compiler_profiles.tar.bz2 _ dev-platform mailing list dev-platform@lists.mozilla.org mailto:dev-platform@lists.mozilla.org https://lists.mozilla.org/__listinfo/dev-platform https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Analyze C++/compiler usage and code stats easily
On 11/15/2013 05:37 PM, Gregory Szorc wrote: On 11/15/13, 12:26 PM, Terrence Cole wrote: The problem this mess is solving, at least in the GC, is that gecko needs to be able to inline barriers, UnmarkGray, and misc other stuff. Whenever we accidentally out-of-line anything in these paths we see a tremendous slowdown on dromaeo_dom and dromaeo_css, so we do have strong evidence that this optimization is critical. We hide as many of the internal GC details as possible behind magic number offsets, void*, and shadow structs, but this still -- particularly combined with templates -- leaves us with a tremendous amount of code in headers. I think our best hope for improving compilation time with these limitations is C++ modules. It wouldn't really improve the code layout mess, but would at least mean we only have to compile the massive pile of inlined code once. Could you compile parts of SpiderMonkey in unified mode? Would be very useful for me, but I don't think I'm the common case for SpiderMonkey hackers. I actually had great results playing with clang's pre-compiled headers: 30+% speedup on compilation of jsgc.cpp. Getting it production ready looked like it would be a major headache though, so I didn't push forward with it. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Analyze C++/compiler usage and code stats easily
I updated the patch to be a little bit more robust. New version at http://hg.gregoryszorc.com/gecko-collab/rev/6af8ba812e82 I also put together some lists of compiler CPU time per directory: https://gps.pastebin.mozilla.org/3613688 https://gps.pastebin.mozilla.org/3620053 In terms of percent: https://gps.pastebin.mozilla.org/3620075 https://gps.pastebin.mozilla.org/3620086 On 11/14/13, 11:43 PM, Gregory Szorc wrote: C++ developers, Over 90% of the CPU time required to build the tree is spent compiling or linking C/C++. So, anything we can do to make that faster will make the overall build faster. I put together a quick patch [1] to make it rather simple to extract compiler resource usage and very basic code metrics during builds. Simply apply that patch and build with `mach build --profile-compiler` and your machine will produce all kinds of potentially interesting measurements. They will be stuffed into objdir/.mozbuild/compilerprof/. If you don't feel like waiting (it will take about 5x longer than a regular build because it performs separate preprocessing, ast, and codegen compiler invocations 3 times each), grab an archive of an OS X build I just performed from [2] and extract it to objdir/.mozbuild/. I put together an extremely simple `mach compiler-analyze` command to sift through the results. e.g. $ mach compiler-analyze preprocessor-relevant-lines $ mach compiler-analyze codegen-sizes $ mach compiler-analyze codegen-total-times Just run `mach help compiler-analyze` to see the full list of what can be printed. Or, write your own code to analyze the produced JSON files. I'm sure people who love getting down and dirty with C++ will be interested in this data. I have no doubt that are compiler time and code size wins waiting to be discovered through this data. We may even uncover a perf issue or two. Who knows. Here are some questions I have after casually looking at the data: * The mean ratio of .o size to lines from preprocessor is 16.35 bytes/line. Why do 38/4916 (0.8%) files have a ratio over 100? Why are a lot of these in js/ and gfx? * What's in the 150 files that have 100k+ lines after preprocessing that makes them so large? * Why does lots of js/'s source code gravitate towards the bad extreme for most of the metrics (code size, compiler time, preprocessor size)? Disclaimer: This patch is currently hacked together. If there is an interest in getting this checked into the tree, I can clean it up and do it. Just file a bug in Core :: Build Config and I'll make it happen when I have time. Or, if an interested party wants to champion getting it landed, I'll happily hand it off :) [1] http://hg.gregoryszorc.com/gecko-collab/rev/741f3074e313 [2] https://people.mozilla.org/~gszorc/compiler_profiles.tar.bz2 ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Analyze C++/compiler usage and code stats easily
On 11/15/2013 06:51 AM, Jason Orendorff wrote: On 11/14/13 11:43 PM, Gregory Szorc wrote: * Why does lots of js/'s source code gravitate towards the bad extreme for most of the metrics (code size, compiler time, preprocessor size)? We use templates very heavily and we inline like mad. Templates make C++ compilers go slowly. The GC smart pointer classes are all templates. They are used everywhere. This will not change. I'd go so far as to say it cannot change. If we wanted to retain the same level of type safety without templates we'd need to write (or macro-generate) 10x as much code, getting us right back to where we are now. On to inlines. -inl.h and inlines.h headers in mozilla-central (js/src has more than everything else put together): https://gist.github.com/jorendorff/7484945 Inlining a function in C++ often requires moving something else, like a class definition the inline function requiers, into a header too. So the affordances of C++ are such that unless you pay extremely close attention to code organization when writing inline-heavy code, you make a huge mess. We have made a huge mess. The problem this mess is solving, at least in the GC, is that gecko needs to be able to inline barriers, UnmarkGray, and misc other stuff. Whenever we accidentally out-of-line anything in these paths we see a tremendous slowdown on dromaeo_dom and dromaeo_css, so we do have strong evidence that this optimization is critical. We hide as many of the internal GC details as possible behind magic number offsets, void*, and shadow structs, but this still -- particularly combined with templates -- leaves us with a tremendous amount of code in headers. I think our best hope for improving compilation time with these limitations is C++ modules. It wouldn't really improve the code layout mess, but would at least mean we only have to compile the massive pile of inlined code once. Cheers, Terrence I don't yet know how to tidy up this kind of mess. I'd love to adopt C++ style rules that would help, but I don't know what those would be. Nicholas Nethercote and I have spent time fighting this fire: Bug 872416 -js/src #includes are completely out of control https://bugzilla.mozilla.org/show_bug.cgi?id=872416 Bug 879831 - js/src #includes are still out of control https://bugzilla.mozilla.org/show_bug.cgi?id=879831 Bug 886205 - Slim down inlines.h/-inl.h files https://bugzilla.mozilla.org/show_bug.cgi?id=886205 Bug 888083 - Do not #include inline-headers from non-inline headers https://bugzilla.mozilla.org/show_bug.cgi?id=888083 Bug 880088 - Break the build on #include madness https://bugzilla.mozilla.org/show_bug.cgi?id=880088 This kind of work is high-opportunity-cost. It involves a lot of clobbering and try servering, and it doesn't get any JS engine bugs fixed or features implemented. -j ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Analyze C++/compiler usage and code stats easily
On 11/15/13, 12:26 PM, Terrence Cole wrote: The problem this mess is solving, at least in the GC, is that gecko needs to be able to inline barriers, UnmarkGray, and misc other stuff. Whenever we accidentally out-of-line anything in these paths we see a tremendous slowdown on dromaeo_dom and dromaeo_css, so we do have strong evidence that this optimization is critical. We hide as many of the internal GC details as possible behind magic number offsets, void*, and shadow structs, but this still -- particularly combined with templates -- leaves us with a tremendous amount of code in headers. I think our best hope for improving compilation time with these limitations is C++ modules. It wouldn't really improve the code layout mess, but would at least mean we only have to compile the massive pile of inlined code once. Could you compile parts of SpiderMonkey in unified mode? ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
