Re: Recent build time improvements due to unified sources

2013-11-29 Thread Gregory Szorc
On 11/29/13, 5:12 AM, Ehsan Akhgari wrote: On 11/28/2013, 4:55 PM, Mike Hoye wrote: On 11/28/2013, 4:45 PM, Nicholas Nethercote wrote: I'm pretty sure that gps was saying if you're paid to work by Mozilla, get a faster machine. More generally, we're all in furious agreement: fast builds are

Re: Recent build time improvements due to unified sources

2013-11-28 Thread Gabriele Svelto
On 28/11/2013 10:57, Neil wrote: I often build in a VM. I allocate 2 CPUs and 2GB of RAM to it, which seems to be enough even to link xul.dll with debug symbols, although it takes a few minutes. (Linking it without symbols takes just seconds.) Given the amount of memory needed to link I haven't

Re: Recent build time improvements due to unified sources

2013-11-28 Thread Ehsan Akhgari
On 11/28/2013, 2:29 AM, Gabriele Svelto wrote: On 28/11/2013 08:09, Gregory Szorc wrote: Peak RSS likely has increased significantly (hundreds to gigabytes range). OK, that's what I was curious about. Memory is cheap and getting cheaper. Nobody paid by Mozilla to develop Firefox should have

Re: Recent build time improvements due to unified sources

2013-11-28 Thread Mike Hoye
On 11/28/2013, 12:14 PM, Ehsan Akhgari wrote: Please file a bug (and CC me) with more details about where the build fails, how much memory the machine has, and how much memory each compiler invocation consumes (test with make -j1). We can adjust the number of files we build in one chunk per

Re: Recent build time improvements due to unified sources

2013-11-28 Thread Ehsan Akhgari
On 11/28/2013, 1:21 PM, Mike Hoye wrote: On 11/28/2013, 12:14 PM, Ehsan Akhgari wrote: Please file a bug (and CC me) with more details about where the build fails, how much memory the machine has, and how much memory each compiler invocation consumes (test with make -j1). We can adjust the

Re: Recent build time improvements due to unified sources

2013-11-28 Thread Mike Hoye
On 11/28/2013, 4:45 PM, Nicholas Nethercote wrote: I'm pretty sure that gps was saying if you're paid to work by Mozilla, get a faster machine. More generally, we're all in furious agreement: fast builds are good; achieving them via multiple means is worthwhile; those with the option of

Re: Recent build time improvements due to unified sources

2013-11-28 Thread Ehsan Akhgari
On 11/28/2013, 4:55 PM, Mike Hoye wrote: On 11/28/2013, 4:45 PM, Nicholas Nethercote wrote: I'm pretty sure that gps was saying if you're paid to work by Mozilla, get a faster machine. More generally, we're all in furious agreement: fast builds are good; achieving them via multiple means is

Re: Recent build time improvements due to unified sources

2013-11-28 Thread Mike Hommey
On Thu, Nov 28, 2013 at 05:12:53PM -0500, Ehsan Akhgari wrote: On 11/28/2013, 4:55 PM, Mike Hoye wrote: On 11/28/2013, 4:45 PM, Nicholas Nethercote wrote: I'm pretty sure that gps was saying if you're paid to work by Mozilla, get a faster machine. More generally, we're all in furious

Re: Recent build time improvements due to unified sources

2013-11-27 Thread Gregory Szorc
On 11/21/13, 10:38 PM, Gregory Szorc wrote: On 11/20/13, 9:57 PM, Gregory Szorc wrote: On 11/19/2013 10:08 PM, Gregory Szorc wrote: On 11/18/13, 11:15 PM, Gregory Szorc wrote: Do builds feel like they've gotten faster in the last few weeks^hours? It's because they have. When I did my MacBook

Re: Recent build time improvements due to unified sources

2013-11-27 Thread Gabriele Svelto
On 28/11/2013 06:06, Gregory Szorc wrote: On de5aa094b55f, we're now down to: Wall 7:37 (457) User 45:38 (2738) Sys3:54 (234) Total 49:32 (2972) That's with WebRTC and ICU enabled. Looking at my own stats while building I was wondering if anybody has looked at peak memory

Re: Recent build time improvements due to unified sources

2013-11-27 Thread Gregory Szorc
On 11/28/13, 12:21 PM, Gabriele Svelto wrote: On 28/11/2013 06:06, Gregory Szorc wrote: On de5aa094b55f, we're now down to: Wall 7:37 (457) User 45:38 (2738) Sys3:54 (234) Total 49:32 (2972) That's with WebRTC and ICU enabled. Looking at my own stats while building I was

Re: Recent build time improvements due to unified sources

2013-11-27 Thread Gabriele Svelto
On 28/11/2013 08:09, Gregory Szorc wrote: Peak RSS likely has increased significantly (hundreds to gigabytes range). OK, that's what I was curious about. Memory is cheap and getting cheaper. Nobody paid by Mozilla to develop Firefox should have a machine with less than 16 GB. Adding 25%+ to

Re: Recent build time improvements due to unified sources

2013-11-21 Thread Neil
Zack Weinberg wrote: On 2013-11-20 12:37 PM, Benoit Jacob wrote: Talking about ideas for further extending the impact of UNIFIED_SOURCES, it seems that the biggest limitation at the moment is that sources can't be unified between different moz.build's. Because of that, source directories

Re: Recent build time improvements due to unified sources

2013-11-21 Thread Michael Shal
From: Neil n...@parkwaycc.co.uk There used to be a limitation that source files had to be in the VPATH. This limitation obviously does not apply to unified sources (the compiler will use the -I path to find the source.) so you shouldn't have a problem setting UNIFIED_SOURCES in a parent

Re: Recent build time improvements due to unified sources

2013-11-21 Thread Nicholas Nethercote
On Thu, Nov 21, 2013 at 7:38 AM, Gregory Szorc g...@mozilla.com wrote: You people are sick. I go to bed, wake up and my builds have gotten faster And I've gone from 7.5 minutes to 6.75 minutes in the past day or two. Nick ___ dev-platform mailing

Re: Recent build time improvements due to unified sources

2013-11-21 Thread Gregory Szorc
On 11/20/13, 9:57 PM, Gregory Szorc wrote: On 11/19/2013 10:08 PM, Gregory Szorc wrote: On 11/18/13, 11:15 PM, Gregory Szorc wrote: Do builds feel like they've gotten faster in the last few weeks^hours? It's because they have. When I did my MacBook Pro comparison [1] with a changeset from Oct

Re: Recent build time improvements due to unified sources

2013-11-20 Thread Nicholas Nethercote
On September 12, a debug clobber build on my new Linux desktop took 12.7 minutes. Just then it took 7.5 minutes. Woo! Nick ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform

Re: Recent build time improvements due to unified sources

2013-11-20 Thread Patrick McManus
I was skeptical of this work - so I need to say now that it is paying dividends bigger and faster than I thought it could. very nice! On Wed, Nov 20, 2013 at 3:38 AM, Nicholas Nethercote n.netherc...@gmail.com wrote: On September 12, a debug clobber build on my new Linux desktop took 12.7

Re: Recent build time improvements due to unified sources

2013-11-20 Thread Ehsan Akhgari
On 2013-11-20 12:37 PM, Benoit Jacob wrote: Talking about ideas for further extending the impact of UNIFIED_SOURCES, it seems that the biggest limitation at the moment is that sources can't be unified between different moz.build's. Because of that, source directories that consist of many small

Re: Recent build time improvements due to unified sources

2013-11-20 Thread Ehsan Akhgari
On 2013-11-20 12:09 PM, Chris Peterson wrote: It might be useful to add a files_per_unified_file parameter to mozconfig or mach build. People could benchmark different values of files_per_unified_file (trading off clobber vs incremental build times). The same parameter could also be used to

Re: Recent build time improvements due to unified sources

2013-11-20 Thread Gregory Szorc
On Nov 20, 2013, at 9:37, Benoit Jacob jacob.benoi...@gmail.com wrote: Talking about ideas for further extending the impact of UNIFIED_SOURCES, it seems that the biggest limitation at the moment is that sources can't be unified between different moz.build's. Because of that, source directories

Re: Recent build time improvements due to unified sources

2013-11-20 Thread Ehsan Akhgari
While this analysis is interesting, it doesn't measure the impact of the unified builds project directly, so I decided that now that a good chunk of code is being compiled in unified mode we should get some specific numbers on the improvements. What I did was I took inbound as of

Re: Recent build time improvements due to unified sources

2013-11-20 Thread Robert O'Callahan
On Thu, Nov 21, 2013 at 11:06 AM, Zack Weinberg za...@panix.com wrote: On 2013-11-20 12:37 PM, Benoit Jacob wrote: Talking about ideas for further extending the impact of UNIFIED_SOURCES, it seems that the biggest limitation at the moment is that sources can't be unified between different

Re: Recent build time improvements due to unified sources

2013-11-20 Thread Ehsan Akhgari
On 2013-11-20 5:27 PM, Robert O'Callahan wrote: On Thu, Nov 21, 2013 at 11:06 AM, Zack Weinberg za...@panix.com wrote: On 2013-11-20 12:37 PM, Benoit Jacob wrote: Talking about ideas for further extending the impact of UNIFIED_SOURCES, it seems that the biggest limitation at the moment is

Re: Recent build time improvements due to unified sources

2013-11-20 Thread Benoit Jacob
2013/11/20 Ehsan Akhgari ehsan.akhg...@gmail.com On 2013-11-20 5:27 PM, Robert O'Callahan wrote: On Thu, Nov 21, 2013 at 11:06 AM, Zack Weinberg za...@panix.com wrote: On 2013-11-20 12:37 PM, Benoit Jacob wrote: Talking about ideas for further extending the impact of UNIFIED_SOURCES, it

Re: Recent build time improvements due to unified sources

2013-11-20 Thread Gregory Szorc
On 11/19/2013 10:08 PM, Gregory Szorc wrote: On 11/18/13, 11:15 PM, Gregory Szorc wrote: Do builds feel like they've gotten faster in the last few weeks^hours? It's because they have. When I did my MacBook Pro comparison [1] with a changeset from Oct 28, build times on my 2013 2.6 GHz

Re: Recent build time improvements due to unified sources

2013-11-19 Thread Mike Hommey
On Mon, Nov 18, 2013 at 11:15:16PM -0800, Gregory Szorc wrote: Do builds feel like they've gotten faster in the last few weeks^hours? It's because they have. When I did my MacBook Pro comparison [1] with a changeset from Oct 28, build times on my 2013 2.6 GHz MacBook Pro were as follows:

Re: Recent build time improvements due to unified sources

2013-11-19 Thread Gregory Szorc
On 11/18/13, 11:15 PM, Gregory Szorc wrote: Do builds feel like they've gotten faster in the last few weeks^hours? It's because they have. When I did my MacBook Pro comparison [1] with a changeset from Oct 28, build times on my 2013 2.6 GHz MacBook Pro were as follows: Wall 11:13 (673) User

Recent build time improvements due to unified sources

2013-11-18 Thread Gregory Szorc
Do builds feel like they've gotten faster in the last few weeks^hours? It's because they have. When I did my MacBook Pro comparison [1] with a changeset from Oct 28, build times on my 2013 2.6 GHz MacBook Pro were as follows: Wall 11:13 (673) User 69:55 (4195) Sys6:04 (364) Total 75:59