Re: Decommissioning "dumbmake"

[Nicholas Nethercote]

On Sun, Oct 18, 2015 at 8:37 PM, Boris Zbarsky  wrote:
>>
>> Eventually |mach build| should just do the right
>> thing, no matter what files you've touched...
>
> The problem is that definitions of "right thing" differ depending on the
> goal, right?

I'm using the standard build system sense of "right thing" where all
dependencies are known and correct and the build system conservatively
rebuilds things based on modification timestamps and at file
granularity. Which we currently don't have but which glandium and co.
are steadily working towards.

This is good enough 99% of the time but then there's the 1% cases such
as the one bholley is discussing. And those cases are valid; we just
want them to be as simple as possible while still being useful so they
don't get in the way of other improvements. AIUI glandium started this
thread because dumbmake is exactly a system for those 1% cases that is
*not* all that simple and *is* getting in the way of other
improvements.

Nick
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decommissioning "dumbmake"

[Josh Matthews]

On 2015-10-19 1:17 AM, Bobby Holley wrote:

I've heard two compelling arguments against the current setup:
(1) It's inconsistent.
(2) It causes unnecessary overhead when you do |mach build foo bar| because
we link between foo and bar.


I don't recall reading (2) in this thread, and it certainly wasn't the 
case when dumbmake was created.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decommissioning "dumbmake"

[Steve Fink]

On 10/18/2015 10:17 PM, Bobby Holley wrote:

On Sun, Oct 18, 2015 at 8:37 PM, Boris Zbarsky  wrote:


On 10/18/15 7:14 PM, Nicholas Nethercote wrote:


Eventually |mach build| should just do the right
thing, no matter what files you've touched...


The problem is that definitions of "right thing" differ depending on the
goal, right?


Exactly. It's not clear to me that even the perfect build system would
solve this problem (unless it were perfectly fast, in which case I don't
care about any of this). The C++ compilation model requires that we rebuild
everything that #includes a header file that changed, which can take a long
time, so I use my wetware capabilities to short-circuit that heavy rebuild
when I know it isn't needed.

I know this may void my warranty in general, but (as Jonas points out) the
only thing I really care about is making sure that the stuff I rebuilt gets
linked, which IMO is something the build system should be able to
guarantee. This may be simpler than it used to be now that (most?)
everything is linked into libxul, but I think there are still various
corner cases in the tree (correct me if I'm wrong).


If I were to sketch out a "principled" version of this functionality, I 
think it would look something like having different views of shared 
files when rebuilding. So when you change someCommonHeader.h and request 
a minimal inconsistent rebuild of dom/base, all of the dependency logic 
for files within dom/base would see (the timestamp of) the new file, 
while all source files within dom/base would see the old version. More 
generally, you'd want to define an arbitrary subset of directories that 
see the new file.


You could even do exactly that with filesystem namespaces, completely in 
userspace in linux 3.8+, but if I understand the current build system 
architecture properly, that would mean modifying all the backends to 
understand these contortions.


Would it be possible to make a |./mach frankenbuild dom/base| that 
copied all of the dom/base headers into frankendist/include/, leaving 
the existing headers in dist/include/, and make the dependency files 
within dom/base (and the rest of the please-rebuild subset) refer to 
files from frankendist? And of course, the include path for the actual 
compile commands would have to put frankendist/include before 
dist/include in the include search order (for the subset to rebuild). 
When you did a regular build, it would then have to rebuild everything 
that was using frankendist, of course, but that would happen automatically.


I may just not understand the full scope of the problem, but it seems 
like handing the backend a fully correct description of the dependencies 
you want and letting it Do The Right Thing (as in, rebuild everything 
whose dependencies have changed full stop) has got to be better than 
hardcoding in metarules about specific directories.


Or maybe the problem is that you're touching stuff in dom/base and you 
only want things in dom/bindings to see your changes? So it should be 
more like |./mach frankebuild --modified 'dom/base/**.h' dom/bindings|? 
Same basic solution.


Or maybe I should keep my mouth shut until I know what the actual 
problem is.


___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decommissioning "dumbmake"

[Michael Shal]

>
> Also, .h dependency proliferation is a real problem for build system
> performance, especially with the autogenerated mozilla-include.h (which
> contains things from every AC_DEFINE in configure.in - an add AC_DEFINE,
> invalidate everything). If you build the tips of mozilla-central from 24h
> intervals, you'll find that we pretty much rebuild every Gecko source file
> due to headers changing. I would encourage a Council of Elrond to somehow
> tackle this problem. There is preliminary work on a tool in bug 1108293 to
> help identify dependencies. The last time I showed output to a Gecko
> developer, there were definitely some WTF reactions.
>
>
There were some ideas & half-baked implementations floating around in bug
902825 for splitting up mozilla-config.h. It might be worth revisiting that
as a starting point.

-Mike
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decommissioning "dumbmake"

[Gregory Szorc]

On Mon, Oct 19, 2015 at 11:47 AM, Bobby Holley 
wrote:

> On Mon, Oct 19, 2015 at 8:20 AM, Josh Matthews 
> wrote:
>
> > On 2015-10-19 1:17 AM, Bobby Holley wrote:
> >
> >> I've heard two compelling arguments against the current setup:
> >> (1) It's inconsistent.
> >> (2) It causes unnecessary overhead when you do |mach build foo bar|
> >> because
> >> we link between foo and bar.
> >>
> >
> > I don't recall reading (2) in this thread, and it certainly wasn't the
> > case when dumbmake was created.
> >
>
> I was referring to Benoit's point, which admittedly involved two separate
> invocations of |mach|.


Responding here to a bunch of points because I don't want to write N>1
responses.

* The existing build system is a "frankenbuild" because each Makefile is
its own DAG / world view and thus can lose dependencies. This is a reason
we need to clobber periodically.
* `mach build dom/base toolkit/library` is a "partial build"
* `mach build binaries` is a "full build" (at least for the Gecko/XUL world
view, which is what most readers of dev-platform care about).
* "Partial builds" are "frankenbuilds" by definition because you are
intentionally skipping dependencies.
* "Full builds," while ideal from build system purity perspective, are not
always suitable because if you change a highly-used .h file, you could have
to wait minutes for all dependencies to rebuild. Sometimes you just want a
quick syntax or compilation check in a small set of dependent source files,
targets, libraries, etc.
* "frankenbuilds" and "partial builds" of today require humans to absorb
build system knowledge [in order to work around limitations with "full
builds" workflows]. This is not ideal: we would prefer to only expose
high-level targets like "binaries" or build modes that take changed files
into account to only rebuild a minimal subset (if that's what the developer
insists).

Also, .h dependency proliferation is a real problem for build system
performance, especially with the autogenerated mozilla-include.h (which
contains things from every AC_DEFINE in configure.in - an add AC_DEFINE,
invalidate everything). If you build the tips of mozilla-central from 24h
intervals, you'll find that we pretty much rebuild every Gecko source file
due to headers changing. I would encourage a Council of Elrond to somehow
tackle this problem. There is preliminary work on a tool in bug 1108293 to
help identify dependencies. The last time I showed output to a Gecko
developer, there were definitely some WTF reactions.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decommissioning "dumbmake"

[Bobby Holley]

On Mon, Oct 19, 2015 at 8:20 AM, Josh Matthews 
wrote:

> On 2015-10-19 1:17 AM, Bobby Holley wrote:
>
>> I've heard two compelling arguments against the current setup:
>> (1) It's inconsistent.
>> (2) It causes unnecessary overhead when you do |mach build foo bar|
>> because
>> we link between foo and bar.
>>
>
> I don't recall reading (2) in this thread, and it certainly wasn't the
> case when dumbmake was created.
>

I was referring to Benoit's point, which admittedly involved two separate
invocations of |mach|.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decommissioning "dumbmake"

[Philip Chee]

On 16/10/2015 08:15, Mike Hommey wrote:

> There are however now two build targets that can do the right thing for
> most use cases:
> - mach build binaries, which will build C/C++ related things
>   appropriately
> - mach build faster, which will build JS, XUL, CSS, etc. (iow, non
>   C/C++) (although it skips what doesn't end up in dist/bin)

I tried mach build faster recently and it failed to build comm-central
because it was looking for Firefox (/browser). Any ETA on when MBF will
work on comm-central?

Phil

-- 
Philip Chee , 
http://flashblock.mozdev.org/ http://xsidebar.mozdev.org
Guard us from the she-wolf and the wolf, and guard us from the thief,
oh Night, and so be good for us to pass.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decommissioning "dumbmake"

[Nicholas Alexander]

On Sun, Oct 18, 2015 at 3:12 PM, Eric Rescorla  wrote:

>
>
> On Sun, Oct 18, 2015 at 2:52 PM, Nicholas Alexander <
> nalexan...@mozilla.com> wrote:
>
>>
>>
>> On Thu, Oct 15, 2015 at 5:15 PM, Mike Hommey  wrote:
>>
>>> Hi,
>>>
>>> I started a thread with the same subject almost two years ago. The
>>> motivation hasn't changed, but the context surely has, so it's probably
>>> time to reconsider.
>>>
>>> As a reminder, "dumbmake" is the feature that makes "mach build foo/bar"
>>> sometimes rebuild in some other directories as well. For example, "mach
>>> build gfx" will build gfx, as well as toolkit/library.
>>>
>>> OTOH, it is pretty limited, and, for instance, "mach build gfx/2d" will
>>> only build gfx/2d.
>>>
>>> There are however now two build targets that can do the right thing for
>>> most use cases:
>>> - mach build binaries, which will build C/C++ related things
>>>   appropriately
>>> - mach build faster, which will build JS, XUL, CSS, etc. (iow, non
>>>   C/C++) (although it skips what doesn't end up in dist/bin)
>>>
>>> At this point, I think "dumbmake" is more harmful than helpful, and the
>>> above two targets should be used instead. Removing "dumbmake" would mean
>>> that "mach build foo/bar" would still work, but would stop to "magically"
>>> do something else than what was requested (or fail to do that thing for
>>> all the cases it doesn't know about).
>>>
>>> Are there still objections to go forward, within the new context?
>>>
>>
>> I agree that dumbmake is more harmful than helpful.  I did some of the
>> work to port it forward to the mach world; IIRC, my motivation was to ease
>> the transition to "mach and smarter build targets".  We're now seeing
>> smarter build targets that aren't based on (recursive) Make, and dumbmake
>> hasn't helped anybody transition to anything: there have been a wopping 10
>> commits to the underlying dependency tree [1], some of which are backouts
>> and relandings.
>>
>> If we need smarter build targets, let's make that happen at the mach
>> level, not at the Make level
>>
>
> Slightly off-topic, but I'd like to argue against smarter build targets,
> if by that you mean more stuff like "mach build binaries"
>
> What's needed here is a dependency management system that
> simply builds what's needed regardless of what's changed, not more
> ways for the user to tell the build system "only rebuild some stuff".
>

In general, I agree, but there are legitimate differences in user intention
that do need to be capture somewhere.  I claim that should be at the mach
command level, but I don't really care if those are top-level (mach
rebuild-xul) or subcommand level (mach build xul), and whether they are
internally "smarter build targets".

In Android land, we're using Gradle to task switch like this -- build, yes;
but also lint; and also run tests; etc.

Nick
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decommissioning "dumbmake"

[Eric Rescorla]

On Sun, Oct 18, 2015 at 2:52 PM, Nicholas Alexander 
wrote:

>
>
> On Thu, Oct 15, 2015 at 5:15 PM, Mike Hommey  wrote:
>
>> Hi,
>>
>> I started a thread with the same subject almost two years ago. The
>> motivation hasn't changed, but the context surely has, so it's probably
>> time to reconsider.
>>
>> As a reminder, "dumbmake" is the feature that makes "mach build foo/bar"
>> sometimes rebuild in some other directories as well. For example, "mach
>> build gfx" will build gfx, as well as toolkit/library.
>>
>> OTOH, it is pretty limited, and, for instance, "mach build gfx/2d" will
>> only build gfx/2d.
>>
>> There are however now two build targets that can do the right thing for
>> most use cases:
>> - mach build binaries, which will build C/C++ related things
>>   appropriately
>> - mach build faster, which will build JS, XUL, CSS, etc. (iow, non
>>   C/C++) (although it skips what doesn't end up in dist/bin)
>>
>> At this point, I think "dumbmake" is more harmful than helpful, and the
>> above two targets should be used instead. Removing "dumbmake" would mean
>> that "mach build foo/bar" would still work, but would stop to "magically"
>> do something else than what was requested (or fail to do that thing for
>> all the cases it doesn't know about).
>>
>> Are there still objections to go forward, within the new context?
>>
>
> I agree that dumbmake is more harmful than helpful.  I did some of the
> work to port it forward to the mach world; IIRC, my motivation was to ease
> the transition to "mach and smarter build targets".  We're now seeing
> smarter build targets that aren't based on (recursive) Make, and dumbmake
> hasn't helped anybody transition to anything: there have been a wopping 10
> commits to the underlying dependency tree [1], some of which are backouts
> and relandings.
>
> If we need smarter build targets, let's make that happen at the mach
> level, not at the Make level
>

Slightly off-topic, but I'd like to argue against smarter build targets,
if by that you mean more stuff like "mach build binaries"

What's needed here is a dependency management system that
simply builds what's needed regardless of what's changed, not more
ways for the user to tell the build system "only rebuild some stuff".

-Ekr


> Nick
>
> [1]
> https://hg.mozilla.org/mozilla-central/filelog/e8c7dfe727cd970e2c3294934e2927b14143c205/build/dumbmake-dependencies
>
>
> ___
> firefox-dev mailing list
> firefox-...@mozilla.org
> https://mail.mozilla.org/listinfo/firefox-dev
>
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decommissioning "dumbmake"

[Nicholas Alexander]

On Thu, Oct 15, 2015 at 5:15 PM, Mike Hommey  wrote:

> Hi,
>
> I started a thread with the same subject almost two years ago. The
> motivation hasn't changed, but the context surely has, so it's probably
> time to reconsider.
>
> As a reminder, "dumbmake" is the feature that makes "mach build foo/bar"
> sometimes rebuild in some other directories as well. For example, "mach
> build gfx" will build gfx, as well as toolkit/library.
>
> OTOH, it is pretty limited, and, for instance, "mach build gfx/2d" will
> only build gfx/2d.
>
> There are however now two build targets that can do the right thing for
> most use cases:
> - mach build binaries, which will build C/C++ related things
>   appropriately
> - mach build faster, which will build JS, XUL, CSS, etc. (iow, non
>   C/C++) (although it skips what doesn't end up in dist/bin)
>
> At this point, I think "dumbmake" is more harmful than helpful, and the
> above two targets should be used instead. Removing "dumbmake" would mean
> that "mach build foo/bar" would still work, but would stop to "magically"
> do something else than what was requested (or fail to do that thing for
> all the cases it doesn't know about).
>
> Are there still objections to go forward, within the new context?
>

I agree that dumbmake is more harmful than helpful.  I did some of the work
to port it forward to the mach world; IIRC, my motivation was to ease the
transition to "mach and smarter build targets".  We're now seeing smarter
build targets that aren't based on (recursive) Make, and dumbmake hasn't
helped anybody transition to anything: there have been a wopping 10 commits
to the underlying dependency tree [1], some of which are backouts and
relandings.

If we need smarter build targets, let's make that happen at the mach level,
not at the Make level.

Nick

[1]
https://hg.mozilla.org/mozilla-central/filelog/e8c7dfe727cd970e2c3294934e2927b14143c205/build/dumbmake-dependencies
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decommissioning "dumbmake"

[Nicholas Nethercote]

On Sun, Oct 18, 2015 at 3:12 PM, Eric Rescorla  wrote:
>
> What's needed here is a dependency management system that
> simply builds what's needed regardless of what's changed,

Otherwise known as "a proper build system". glandium and co. have been
working towards that for a long time. It's a big, difficult job. |mach
build binaries| and |mach build faster| are temporary waypoints along
the way that approximate "a proper build system" for a couple of
common workflows. Eventually |mach build| should just do the right
thing, no matter what files you've touched...

> not more ways for the user to tell the build system "only rebuild some stuff".

... except that bholley and ehsan are asking for a way to override the
dependency tracking and just rebuild particular directories. Which is
reasonable, up to a point. I argue that support for this kind of thing
should be very limited and simple, to avoid getting in glandium's way
as he works towards "a proper build system". IMO, if you're elite
enough that you're regularly telling the build system "I know better
than you" then you're elite enough to remember which directories need
to be built in which order, or to encode those ordering into some
aliases.

Nick
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decommissioning "dumbmake"

[Eric Rescorla]

On Sun, Oct 18, 2015 at 3:18 PM, Nicholas Alexander 
wrote:

>
>
> On Sun, Oct 18, 2015 at 3:12 PM, Eric Rescorla  wrote:
>
>>
>>
>> On Sun, Oct 18, 2015 at 2:52 PM, Nicholas Alexander <
>> nalexan...@mozilla.com> wrote:
>>
>>>
>>>
>>> On Thu, Oct 15, 2015 at 5:15 PM, Mike Hommey  wrote:
>>>
 Hi,

 I started a thread with the same subject almost two years ago. The
 motivation hasn't changed, but the context surely has, so it's probably
 time to reconsider.

 As a reminder, "dumbmake" is the feature that makes "mach build foo/bar"
 sometimes rebuild in some other directories as well. For example, "mach
 build gfx" will build gfx, as well as toolkit/library.

 OTOH, it is pretty limited, and, for instance, "mach build gfx/2d" will
 only build gfx/2d.

 There are however now two build targets that can do the right thing for
 most use cases:
 - mach build binaries, which will build C/C++ related things
   appropriately
 - mach build faster, which will build JS, XUL, CSS, etc. (iow, non
   C/C++) (although it skips what doesn't end up in dist/bin)

 At this point, I think "dumbmake" is more harmful than helpful, and the
 above two targets should be used instead. Removing "dumbmake" would mean
 that "mach build foo/bar" would still work, but would stop to
 "magically"
 do something else than what was requested (or fail to do that thing for
 all the cases it doesn't know about).

 Are there still objections to go forward, within the new context?

>>>
>>> I agree that dumbmake is more harmful than helpful.  I did some of the
>>> work to port it forward to the mach world; IIRC, my motivation was to ease
>>> the transition to "mach and smarter build targets".  We're now seeing
>>> smarter build targets that aren't based on (recursive) Make, and dumbmake
>>> hasn't helped anybody transition to anything: there have been a wopping 10
>>> commits to the underlying dependency tree [1], some of which are backouts
>>> and relandings.
>>>
>>> If we need smarter build targets, let's make that happen at the mach
>>> level, not at the Make level
>>>
>>
>> Slightly off-topic, but I'd like to argue against smarter build targets,
>> if by that you mean more stuff like "mach build binaries"
>>
>> What's needed here is a dependency management system that
>> simply builds what's needed regardless of what's changed, not more
>> ways for the user to tell the build system "only rebuild some stuff".
>>
>
> In general, I agree, but there are legitimate differences in user
> intention that do need to be capture somewhere.
>

Can you provide some examples that aren't rebuilds of part of the system?

I agree that lint and tests are relevant, but that's a different category.

-Ekr




> I claim that should be at the mach command level, but I don't really care
> if those are top-level (mach rebuild-xul) or subcommand level (mach build
> xul), and whether they are internally "smarter build targets".
>
> In Android land, we're using Gradle to task switch like this -- build,
> yes; but also lint; and also run tests; etc.
>
> Nick
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decommissioning "dumbmake"

[Lawrence Mandel]

On Sun, Oct 18, 2015 at 7:14 PM, Nicholas Nethercote  wrote:

> On Sun, Oct 18, 2015 at 3:12 PM, Eric Rescorla  wrote:
> >
> > What's needed here is a dependency management system that
> > simply builds what's needed regardless of what's changed,
>
> Otherwise known as "a proper build system". glandium and co. have been
> working towards that for a long time. It's a big, difficult job. |mach
> build binaries| and |mach build faster| are temporary waypoints along
> the way that approximate "a proper build system" for a couple of
> common workflows. Eventually |mach build| should just do the right
> thing, no matter what files you've touched...
>
> > not more ways for the user to tell the build system "only rebuild some
> stuff".
>
> ... except that bholley and ehsan are asking for a way to override the
> dependency tracking and just rebuild particular directories. Which is
> reasonable, up to a point. I argue that support for this kind of thing
> should be very limited and simple, to avoid getting in glandium's way
> as he works towards "a proper build system". IMO, if you're elite
> enough that you're regularly telling the build system "I know better
> than you" then you're elite enough to remember which directories need
> to be built in which order, or to encode those ordering into some
> aliases.
>

A proper build system is something that we needed yesterday. jgriffin and
co are working on a plan to create a proper build system (no doubt with
glandium) starting in Q1. (Actually, ground work has already started.) We
can hack on things until then but we really just need to do the right
things that have been discussed in this and other threads.

Lawrence
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decommissioning "dumbmake"

[Karl Tomlinson]

On Fri, 16 Oct 2015 10:31:43 -0700, Gregory Szorc wrote:

> But as awesome as
> these targets are, they can still build more than is desired (especially in
> the edit .h file case). This slows down iteration cycles and slows down
> developers.
>
> For this reason, I think dumbmake needs to remain or be replaced by
> something that allows Gecko developers to easily perform a partial (and
> probably incorrect end state) build. I /think/ this could be as simple as a
> "xul" target that linked libxul. This way, people could type `mach build
> dom/base xul` and build all binaries (including static libraries) under a
> source directory and link libxul.

Are you suggesting that would do something different to this?
./mach build dom/base toolkit/library
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decommissioning "dumbmake"

[Gerald Squelart]

On Monday, October 19, 2015 at 11:09:46 AM UTC+11, Eric Rescorla wrote:
> On Sun, Oct 18, 2015 at 4:14 PM, Nicholas Nethercote
> wrote:
> 
> > On Sun, Oct 18, 2015 at 3:12 PM, Eric Rescorla wrote:
> > >
> > > What's needed here is a dependency management system that
> > > simply builds what's needed regardless of what's changed,
> >
> > Otherwise known as "a proper build system". glandium and co. have been
> > working towards that for a long time. It's a big, difficult job. |mach
> > build binaries| and |mach build faster| are temporary waypoints along
> > the way that approximate "a proper build system" for a couple of
> > common workflows. Eventually |mach build| should just do the right
> > thing, no matter what files you've touched...
> >
> > > not more ways for the user to tell the build system "only rebuild some
> > stuff".
> >
> > ... except that bholley and ehsan are asking for a way to override the
> > dependency tracking and just rebuild particular directories.
> 
> 
> I don't want to speak for bholley and ehsan, but when *I* run into this
> problem it's because the build system isn't doing the right thing. If it
> were, I wouldn't have to.
> 
> -Ekr

I think the situation they're talking about, and which I've experienced myself, 
is that sneezing at a header file that's included almost everywhere means a 
very long rebuild time -- which is the right thing for the build system to do 
of course!

But if we've only changed a comment, or added a method, or are just trying to 
implement/fix a bug and test a patch quickly in isolation, then we should have 
the ability to locally override the build system decisions.

There are probably ways around that (e.g. experiment in a cpp file before 
moving code to a header), but it's sometimes cumbersome or just not worth the 
effort, compared with rebuilding only parts of the system.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decommissioning "dumbmake"

[Bobby Holley]

On Sun, Oct 18, 2015 at 8:37 PM, Boris Zbarsky  wrote:

> On 10/18/15 7:14 PM, Nicholas Nethercote wrote:
>
>> Eventually |mach build| should just do the right
>> thing, no matter what files you've touched...
>>
>
> The problem is that definitions of "right thing" differ depending on the
> goal, right?
>

Exactly. It's not clear to me that even the perfect build system would
solve this problem (unless it were perfectly fast, in which case I don't
care about any of this). The C++ compilation model requires that we rebuild
everything that #includes a header file that changed, which can take a long
time, so I use my wetware capabilities to short-circuit that heavy rebuild
when I know it isn't needed.

I know this may void my warranty in general, but (as Jonas points out) the
only thing I really care about is making sure that the stuff I rebuilt gets
linked, which IMO is something the build system should be able to
guarantee. This may be simpler than it used to be now that (most?)
everything is linked into libxul, but I think there are still various
corner cases in the tree (correct me if I'm wrong).

I've heard two compelling arguments against the current setup:
(1) It's inconsistent.
(2) It causes unnecessary overhead when you do |mach build foo bar| because
we link between foo and bar.

for (1), I would be satisfied with a stripped down version of dumbmake that
doesn't do any building, but _does_ guarantee to link everything. (2) could
be solved with a flag to |mach build| to indicate whether or not the link
step should be performed (though I think linking should be the default).

bholley
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decommissioning "dumbmake"

[Boris Zbarsky]

On 10/18/15 7:14 PM, Nicholas Nethercote wrote:

Eventually |mach build| should just do the right
thing, no matter what files you've touched...


The problem is that definitions of "right thing" differ depending on the 
goal, right?


-Boris
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decommissioning "dumbmake"

[Jonas Sicking]

On Sun, Oct 18, 2015 at 4:14 PM, Nicholas Nethercote
 wrote:
>
>> not more ways for the user to tell the build system "only rebuild some 
>> stuff".
>
> ... except that bholley and ehsan are asking for a way to override the
> dependency tracking and just rebuild particular directories. Which is
> reasonable, up to a point. I argue that support for this kind of thing
> should be very limited and simple, to avoid getting in glandium's way
> as he works towards "a proper build system".

This makes sense to me.

It occurred to me that maybe this can be solved by having a mach
command similar to "build binaries", but which *only* links libraries
and doesn't compile any cpp files.

That way it's easy enough to do "mach build dom/whatever && mach build
linklibraries". This wouldn't need the smarts to figure out which
libraries needs to be relinked if a given source file has change, it
would just attempt to relink all of them.

I think the big timesave that bholley and ehsan is looking for is
avoiding recompiling a bunch of C++ files, which this would still
accomplish.

/ Jonas
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decommissioning "dumbmake"

[Eric Rescorla]

On Sun, Oct 18, 2015 at 4:14 PM, Nicholas Nethercote  wrote:

> On Sun, Oct 18, 2015 at 3:12 PM, Eric Rescorla  wrote:
> >
> > What's needed here is a dependency management system that
> > simply builds what's needed regardless of what's changed,
>
> Otherwise known as "a proper build system". glandium and co. have been
> working towards that for a long time. It's a big, difficult job. |mach
> build binaries| and |mach build faster| are temporary waypoints along
> the way that approximate "a proper build system" for a couple of
> common workflows. Eventually |mach build| should just do the right
> thing, no matter what files you've touched...
>
> > not more ways for the user to tell the build system "only rebuild some
> stuff".
>
> ... except that bholley and ehsan are asking for a way to override the
> dependency tracking and just rebuild particular directories.


I don't want to speak for bholley and ehsan, but when *I* run into this
problem it's because the build system isn't doing the right thing. If it
were, I wouldn't have to.

-Ekr
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decommissioning "dumbmake"

[Gregory Szorc]

On Thu, Oct 15, 2015 at 6:37 PM, Nicholas Nethercote  wrote:

> On Fri, Oct 16, 2015 at 11:37 AM, Bobby Holley 
> wrote:
> >
> > |mach build binaries| is much slower for me than the present behavior,
> > because I often hack on header files that are included all over the tree,
> > but whose #include-ers generally don't need to be rebuilt all of the
> time.
>
> I used to have a bunch of my own aliases for building particular
> directories (and toolkit/library if necessary) and then relinking.
> (Sounds like maybe I partially reimplemented dumbmake myself? Not
> sure.) I stopped using them once |mach build binaries| came along
> because it was better for my use cases.
>
> I suspect the very small number of people who are in your position can
> do something similar?
>

This is pretty much how everyone did everything 3+ years ago: waiting
45-90s for a `make -f client.mk` to traverse all the tiers and directories
when a single C++/JS file changed was just wasteful. So lots of people
cobbled together shell aliases and scripts for patterns like `make -j8
services && make -j8 toolkit/components && make -j8 browser` or `make -j8
-C dom/base && make -j8 -C toolkit/library`. It was a good hack to gain
build/development speed while sacrificing a little build purity.

dumbmake was a formal establishment of these practices so that people
didn't have to know as many inner workings of the build system and code up
these aliases.

The "binaries" target came along after dumbmake and made the C++ build
procedure infinitely better. Now we have "fastermake," which is the rough
equivalent but for frontend and JavaScript developers. But as awesome as
these targets are, they can still build more than is desired (especially in
the edit .h file case). This slows down iteration cycles and slows down
developers.

For this reason, I think dumbmake needs to remain or be replaced by
something that allows Gecko developers to easily perform a partial (and
probably incorrect end state) build. I /think/ this could be as simple as a
"xul" target that linked libxul. This way, people could type `mach build
dom/base xul` and build all binaries (including static libraries) under a
source directory and link libxul.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decommissioning "dumbmake"

[Mike Hommey]

On Thu, Oct 15, 2015 at 08:43:53PM -0400, Ehsan Akhgari wrote:
> On 2015-10-15 8:37 PM, Bobby Holley wrote:
> >On Thu, Oct 15, 2015 at 5:26 PM, Mike Hommey  >> wrote:
> >
> >On Thu, Oct 15, 2015 at 05:21:44PM -0700, Bobby Holley wrote:
> >> Will building in an arbitrary source directory continue to link 
> > libxul? It
> >> was really great when we stopped needing to build in toolkit/library 
> > all
> >> the time.
> >
> >The point is, that doesn't reliably happen currently. You should
> >just use mach
> >build binaries, which will, reliably, and even better, only if needed.
> >
> >
> >|mach build binaries| is much slower for me than the present behavior,
> >because I often hack on header files that are included all over the
> >tree, but whose #include-ers generally don't need to be rebuilt all of
> >the time.
> 
> I have the exact same use case as Bobby and as such object to removing this
> feature.  Especially since it is unclear what we gain from removing it.

Consistency. Currently, when you do `mach build foo/bar`, you don't know
what's going to happen. Some people will get toolkit/library rebuild,
some won't. Or other directories. Or not.

"dumbmake" also breaks `mach build foo/bar/baz` when the expected outcome
is `make -C objdir/foo bar/baz` instead of `make -C objdir/foo/bar baz`.

Mike
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decommissioning "dumbmake"

[Benoit Girard]

+1

For my use case breaking dumbmake is preferable given that we now have
'build binaries'. When touching commonly included header I often like to
run ./mach build gfx && ./mach build binaries. This effectively let's me
say 'Make sure my gfx changes are good before you recompile the rest of
gecko' giving me control over the recompile order. This will notify me
quickly if there's a compile error without having to rebuild everything
that include the header that I changed giving me compile errors much
faster. With the current behavior './mach build gfx' will try to link when
I haven't yet recompiled all the files that I need.

On Thu, Oct 15, 2015 at 8:15 PM, Mike Hommey  wrote:

> Hi,
>
> I started a thread with the same subject almost two years ago. The
> motivation hasn't changed, but the context surely has, so it's probably
> time to reconsider.
>
> As a reminder, "dumbmake" is the feature that makes "mach build foo/bar"
> sometimes rebuild in some other directories as well. For example, "mach
> build gfx" will build gfx, as well as toolkit/library.
>
> OTOH, it is pretty limited, and, for instance, "mach build gfx/2d" will
> only build gfx/2d.
>
> There are however now two build targets that can do the right thing for
> most use cases:
> - mach build binaries, which will build C/C++ related things
>   appropriately
> - mach build faster, which will build JS, XUL, CSS, etc. (iow, non
>   C/C++) (although it skips what doesn't end up in dist/bin)
>
> At this point, I think "dumbmake" is more harmful than helpful, and the
> above two targets should be used instead. Removing "dumbmake" would mean
> that "mach build foo/bar" would still work, but would stop to "magically"
> do something else than what was requested (or fail to do that thing for
> all the cases it doesn't know about).
>
> Are there still objections to go forward, within the new context?
>
> Cheers,
>
> Mike
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decommissioning "dumbmake"

[Mike Hommey]

On Thu, Oct 15, 2015 at 05:21:44PM -0700, Bobby Holley wrote:
> Will building in an arbitrary source directory continue to link libxul? It
> was really great when we stopped needing to build in toolkit/library all
> the time.

The point is, that doesn't reliably happen currently. You should just use mach
build binaries, which will, reliably, and even better, only if needed.

Mike
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decommissioning "dumbmake"

[Bobby Holley]

Also, interactive rebase often dirties a bunch of files unnecessarily - I
know they haven't changed, but the build system doesn't.

Basically, the best-effort approach of rebuilding directories saves a lot
of time for certain Gecko hackers like myself, and it would be really nice
if the libxul linking was preserved, if nothing else.

On Thu, Oct 15, 2015 at 5:37 PM, Bobby Holley  wrote:

> On Thu, Oct 15, 2015 at 5:26 PM, Mike Hommey  wrote:
>
>> On Thu, Oct 15, 2015 at 05:21:44PM -0700, Bobby Holley wrote:
>> > Will building in an arbitrary source directory continue to link libxul?
>> It
>> > was really great when we stopped needing to build in toolkit/library all
>> > the time.
>>
>> The point is, that doesn't reliably happen currently. You should just use
>> mach
>> build binaries, which will, reliably, and even better, only if needed.
>>
>
> |mach build binaries| is much slower for me than the present behavior,
> because I often hack on header files that are included all over the tree,
> but whose #include-ers generally don't need to be rebuilt all of the time.
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decommissioning "dumbmake"

[Ehsan Akhgari]

On 2015-10-15 8:37 PM, Bobby Holley wrote:

On Thu, Oct 15, 2015 at 5:26 PM, Mike Hommey > wrote:

On Thu, Oct 15, 2015 at 05:21:44PM -0700, Bobby Holley wrote:
> Will building in an arbitrary source directory continue to link libxul? It
> was really great when we stopped needing to build in toolkit/library all
> the time.

The point is, that doesn't reliably happen currently. You should
just use mach
build binaries, which will, reliably, and even better, only if needed.


|mach build binaries| is much slower for me than the present behavior,
because I often hack on header files that are included all over the
tree, but whose #include-ers generally don't need to be rebuilt all of
the time.


I have the exact same use case as Bobby and as such object to removing 
this feature.  Especially since it is unclear what we gain from removing it.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decommissioning "dumbmake"

[Nicholas Nethercote]

On Fri, Oct 16, 2015 at 11:37 AM, Bobby Holley  wrote:
>
> |mach build binaries| is much slower for me than the present behavior,
> because I often hack on header files that are included all over the tree,
> but whose #include-ers generally don't need to be rebuilt all of the time.

I used to have a bunch of my own aliases for building particular
directories (and toolkit/library if necessary) and then relinking.
(Sounds like maybe I partially reimplemented dumbmake myself? Not
sure.) I stopped using them once |mach build binaries| came along
because it was better for my use cases.

I suspect the very small number of people who are in your position can
do something similar?

Nick
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decommissioning "dumbmake"

[Bobby Holley]

On Thu, Oct 15, 2015 at 5:26 PM, Mike Hommey  wrote:

> On Thu, Oct 15, 2015 at 05:21:44PM -0700, Bobby Holley wrote:
> > Will building in an arbitrary source directory continue to link libxul?
> It
> > was really great when we stopped needing to build in toolkit/library all
> > the time.
>
> The point is, that doesn't reliably happen currently. You should just use
> mach
> build binaries, which will, reliably, and even better, only if needed.
>

|mach build binaries| is much slower for me than the present behavior,
because I often hack on header files that are included all over the tree,
but whose #include-ers generally don't need to be rebuilt all of the time.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decommissioning "dumbmake"

[Bobby Holley]

Will building in an arbitrary source directory continue to link libxul? It
was really great when we stopped needing to build in toolkit/library all
the time.

On Thu, Oct 15, 2015 at 5:15 PM, Mike Hommey  wrote:

> Hi,
>
> I started a thread with the same subject almost two years ago. The
> motivation hasn't changed, but the context surely has, so it's probably
> time to reconsider.
>
> As a reminder, "dumbmake" is the feature that makes "mach build foo/bar"
> sometimes rebuild in some other directories as well. For example, "mach
> build gfx" will build gfx, as well as toolkit/library.
>
> OTOH, it is pretty limited, and, for instance, "mach build gfx/2d" will
> only build gfx/2d.
>
> There are however now two build targets that can do the right thing for
> most use cases:
> - mach build binaries, which will build C/C++ related things
>   appropriately
> - mach build faster, which will build JS, XUL, CSS, etc. (iow, non
>   C/C++) (although it skips what doesn't end up in dist/bin)
>
> At this point, I think "dumbmake" is more harmful than helpful, and the
> above two targets should be used instead. Removing "dumbmake" would mean
> that "mach build foo/bar" would still work, but would stop to "magically"
> do something else than what was requested (or fail to do that thing for
> all the cases it doesn't know about).
>
> Are there still objections to go forward, within the new context?
>
> Cheers,
>
> Mike
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decommissioning "dumbmake"

[Jonas Sicking]

> |mach build binaries| is much slower for me than the present behavior,
> because I often hack on header files that are included all over the tree,
> but whose #include-ers generally don't need to be rebuilt all of the time.

Before we had mach, I had an alias for linking libxul. If there is a
simple mach command for linking libxul you might be able to just build
your source dir and then linking libxul?

/ Jonas
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decommissioning "dumbmake"

[Mike Hommey]

On Thu, Oct 15, 2015 at 09:15:01PM -0700, Jonas Sicking wrote:
> > |mach build binaries| is much slower for me than the present behavior,
> > because I often hack on header files that are included all over the tree,
> > but whose #include-ers generally don't need to be rebuilt all of the time.
> 
> Before we had mach, I had an alias for linking libxul. If there is a
> simple mach command for linking libxul you might be able to just build
> your source dir and then linking libxul?

Actually, you can pass multiple directories to mach, so you can do

   mach build foo toolkit/library

in which case mach will do

   make -C objdir/foo
   make -C objdir/toolkit/library
 
And will only do the latter if the former succeeds.

Mike
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decommissioning dumbmake

[Philip Chee]

On 15/01/2014 02:37, Matt Brubeck wrote:
 On 1/13/2014 8:05 PM, Mike Hommey wrote:
 I hear frontend developers are using it for frontend code, I would like
 to know if you would be bothered if it were to disappear.
 
 As a front-end developer, I think dumbmake is harmful almost as often as 
 it is helpful, and should be replaced with something better.  In 
 addition to cases where it builds too little, there are also common 
 cases where it builds too much.  For example, when I run mach build 
 toolkit/content, it wastes time relinking libxul even if I changed only 
 JavaScript code.
 
 I agree that that the best solution would be a command similar in spirit 
 to mach build binaries but for non-C/C++ changes, to provide fast 
 correct builds for front-end-only changes without requiring developers 
 to memorize dependencies between directories. But I'm not sure how one 
 might implement this.

I miss the old build tier_app target. Can whatever replaces dumbmake do
something like that?

Phil

-- 
Philip Chee phi...@aleytys.pc.my, philip.c...@gmail.com
http://flashblock.mozdev.org/ http://xsidebar.mozdev.org
Guard us from the she-wolf and the wolf, and guard us from the thief,
oh Night, and so be good for us to pass.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decommissioning dumbmake

[Till Schneidereit]

(Some messages in this thread seem to be taking an absurdly long time to
arrive, so I apologize if what I'm talking about has long been addressed.)


On Thu, Jan 16, 2014 at 2:00 PM, Philip Chee philip.c...@gmail.com wrote:

 On 15/01/2014 02:37, Matt Brubeck wrote:
  On 1/13/2014 8:05 PM, Mike Hommey wrote:
  I hear frontend developers are using it for frontend code, I would like
  to know if you would be bothered if it were to disappear.


I'd be extremely unhappy if this disappeared without any replacement.
Changing a cpp file under js/src and running `mach` takes about 2 minutes
for me, whereas running `mach js/src` takes about 18 seconds.

As one of the people behind the potentially-over-ambitiously-named
smartmake that became mach dumbmake, I know that it is far from perfect.
For some parts of gecko it's very valuable, however.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decommissioning dumbmake

[Ted Mielczarek]

On 1/16/2014 11:26 AM, Till Schneidereit wrote:

 I'd be extremely unhappy if this disappeared without any replacement.
 Changing a cpp file under js/src and running `mach` takes about 2 minutes
 for me, whereas running `mach js/src` takes about 18 seconds.

 As one of the people behind the potentially-over-ambitiously-named
 smartmake that became mach dumbmake, I know that it is far from perfect.
 For some parts of gecko it's very valuable, however.

Have you tried `mach build binaries`? That's the proposed replacement
for your usecase.

-Ted

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decommissioning dumbmake

[Boris Zbarsky]

On 1/16/14 12:05 PM, Ted Mielczarek wrote:

Have you tried `mach build binaries`?


That doesn't do the same thing as mach build dirname, though. 
Certainly not for dom/bindings...


-Boris
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decommissioning dumbmake

[Till Schneidereit]

On Thu, Jan 16, 2014 at 6:05 PM, Ted Mielczarek t...@mielczarek.org wrote:

 On 1/16/2014 11:26 AM, Till Schneidereit wrote:
 
  I'd be extremely unhappy if this disappeared without any replacement.
  Changing a cpp file under js/src and running `mach` takes about 2 minutes
  for me, whereas running `mach js/src` takes about 18 seconds.
 
  As one of the people behind the potentially-over-ambitiously-named
  smartmake that became mach dumbmake, I know that it is far from perfect.
  For some parts of gecko it's very valuable, however.
 
 Have you tried `mach build binaries`? That's the proposed replacement
 for your usecase.


That works exactly as well as `mach build js/src`! \o/
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform