Re: Must we rebuild all our rust code constantly?

[Chris M.]

On Mon, Aug 19, 2019 at 10:32 PM Kris Maglione 
wrote:

> On Tue, Aug 20, 2019 at 02:23:06PM +0900, ISHIKAWA,chiaki wrote:
> >On 2019/08/20 9:11, Dave Townsend wrote:
> >>Thanks to a tip I've tracked this down. This seems to only be the case
> when
> >>I have sccache enabled. Disabling it gives me nice quick incremental
> builds
> >>again. Of course that isn't an ideal solution but it will do for now.
> >>
> >>On Mon, Aug 19, 2019 at 1:55 PM Dave Townsend 
> wrote:
> >>
> >>>For a couple of weeks now I've seen that any attempt to build Firefox,
> >>>even incremental builds seem to rebuild an awful lot of rust code. I
> found
> >>>this in the source which seems to suggest why:
> >>>
> https://searchfox.org/mozilla-central/source/config/makefiles/rust.mk#238.
> >>>But, this means that now an incremental build with a couple of code
> change
> >>>that have no impact on rust is taking upwards of 4 minutes to complete
> in
> >>>comparison to around 40 seconds, and the log file is full of cargo
> output.
> >>>I've heard similar comments from other developers.
> >>>
> >>>This is a pretty big increase in the time to compile and test and is
> >>>really slowing down my work. Is there any way we can avoid this?
> >>>
> >>
> >
> >
> >I am using linux for local development and noticed something similar.
> >
> >So I should disable sccache (!?).
>
> For what it's worth, Rust is now configured to use incremental
> compilation, which has its own cache which isn't cleared after
> clobber, so sccache isn't actually needed anymore. Ordinary
> ccache should be sufficient.
>
sccache will still cache dependencies from crates.io. From a quick check
building the rust in the tree on my laptop, a full cache hit for rust with
sccache produces a significant speedup (2x) versus incremental alone.

> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Must we rebuild all our rust code constantly?

[Chris M.]

On Tue, Aug 20, 2019 at 12:33 PM Markus Stange  wrote:

> On 2019-08-19 8:11 p.m., Dave Townsend wrote:
> > Thanks to a tip I've tracked this down. This seems to only be the case
> when
> > I have sccache enabled. Disabling it gives me nice quick incremental
> builds
> > again.
>
> What's your sccache version? I think you may be hitting the following
> sccache bug which has been fixed already:
> https://github.com/mozilla/sccache/issues/436
>
> I'm using sccache 0.2.9 and I don't see rust code rebuilding constantly.
>
I've never been able to reproduce this issue, and that would explain why.

Can someone who's experienced this confirm upgrading sccache fixes it?

>
> Markus
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To what extent is sccache's distributed compilation usable?

[Chris M.]

I've been working through issues with distributed sccache, but I've
recently sent instructions to people currently running icecream in the SF
office on how to switch to running sccache-dist with IT managed hardware.
We're going to use that as a test bed before rolling out more widely, and
if all goes well there this should happen shortly (likely shortly after
returning from Whistler).

Regarding that bug, that's tracking IT work for the most part so probably
needs to stay restricted. Most of the work I've been doing is tracked in
sccache's github repo and can be followed there, or tracked by
https://bugzilla.mozilla.org/show_bug.cgi?id=1499145.

Chris

On Wed, Jun 5, 2019 at 1:52 AM Miko Mynttinen 
wrote:

> Has there been any progress with this? Bug 1524533 is still restricted.
>
> Miko
>
> > On Apr 2, 2019, at 10:39 PM, Jeff Muizelaar 
> wrote:
> >
> > That bug is restricted. Does it need to be?
> >
> > -Jeff
> >
> > On Tue, Apr 2, 2019 at 4:22 PM Kim Moir  wrote:
> >>
> >> The tracking bug for sccache being deployed to offices is here
> >>
> >> https://bugzilla.mozilla.org/show_bug.cgi?id=1524533
> >>
> >> Kim
> >>
> >> On Tue, Apr 2, 2019 at 3:22 AM Frederik Braun 
> wrote:
> >>
> >>> Am 01.04.19 um 22:16 schrieb Chris M.:
> >>>> Hi Emilio, if you're interested you're encouraged to try it out,
> however
> >>>> we're shipping machines to offices now to run the schedulers, the
> first
> >>> of
> >>>> which I'll be testing in the SF office this week, so we're planning an
> >>>> officially supported setup in the near future.
> >>>>
> >>>
> >>> That's great news. Can we follow progress or detailed plans somewhere?
> >>> ___
> >>> dev-platform mailing list
> >>> dev-platform@lists.mozilla.org
> >>> https://lists.mozilla.org/listinfo/dev-platform
> >>>
> >> ___
> >> dev-platform mailing list
> >> dev-platform@lists.mozilla.org
> >> https://lists.mozilla.org/listinfo/dev-platform
> > ___
> > dev-platform mailing list
> > dev-platform@lists.mozilla.org
> > https://lists.mozilla.org/listinfo/dev-platform
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To what extent is sccache's distributed compilation usable?

[Chris M.]

On Mon, Apr 1, 2019 at 12:23 PM Emilio Cobos Álvarez 
wrote:

> Hi,
>
> Over in #layout people were ranting about how much of a pain it was to
> set up icecc on MacOS. I pointed out that sccache in theory does support
> distributed compilation[1] and in theory also supports MacOS clients.
>
> Does anybody know to what extent is it in a usable state? Can it replace
> icecc? I'm a Linux user, and icecc works pretty well, but if sccache
> supports Rust as well that's a huge plus.
>
> Are people encouraged to try it out? Or only experimentally? Or should
> we wait?
>
Hi Emilio, if you're interested you're encouraged to try it out, however
we're shipping machines to offices now to run the schedulers, the first of
which I'll be testing in the SF office this week, so we're planning an
officially supported setup in the near future.

>
> I know Ted encouraged me to try it in Berlin, and I still have to do
> that (d'oh :/). So given that I suspect the answer is "it's worth trying
> out and report bugs". But worth knowing if it's more generally usable or
> mostly experimental.
>
> Thanks,
>
>  -- Emilio
>
> [1]:
>
> https://github.com/mozilla/sccache/blob/master/docs/DistributedQuickstart.md
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Proposal to adjust testing to run on PGO builds only and not test on OPT builds

[Chris M.]

Earlier today I landed a fix for bug 1517532 that will mean that an
artifact build with MOZ_PGO set will pull artifacts from an automation pgo
build. As a result artifact pgo builds as trigger by a "-p all
--artifact..." will succeed now as well (and consume pgo'd artifacts).

If we end up wanting to turn off opt builds in automation after all we may
be able to pull artifacts from pgo builds for local artifact builds by
default. The behavior of the compiled code shouldn't be different -- this
probably wouldn't matter to people developing front end code locally.

Chris

On Fri, Jan 4, 2019 at 12:25 PM jmaher  wrote:

> thanks everyone for your comments on this.  It sounds like from a
> practical standpoint until we can get the runtimes of PGO builds on try and
> in integration to be less than debug build times this is not a desirable
> change.
>
> A few common responses:
> * artifact opt builds on try are fast for quick iterations, a must have
> * can we do artifact builds for PGO? (thanks :nalexander for bug 1517533
> and bug 1517532)
> * what about talos?  we need to investigate this more, I have always
> argued against pgo only for talos, but maybe we can revisit that (bug
> 1514829)
> * do we turn off builds as well?  I had proposed just the tests, if we
> decide to turn off talos it would make sense to turn off builds.
>
> Thanks all for the quick feedback, when the bugs in this thread are
> further along, or if I see another simpler solution for reducing the
> duplication, I will follow up.
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Better incremental builds - Tup backend available on Linux

[Chris M.]

On Mon, Oct 1, 2018 at 8:05 AM Randell Jesup  wrote:

> >Tup is a modern build system that enables fast and correct builds. I'm
> >pleased to announce the availability of Tup for building Firefox on Linux.
> >Compared to Make, building with Tup will result in faster incremental
> build
> >times and reduce the need for clobber builds. See the in-tree docs for
> >details [1].
>
> So this seems to be faster for "./mach build" if you have small changes
> to c++ code . However, "./mach build binaries" doesn't work at all.
> (Perhaps with this it's no longer necessary, but if so a message stating
> that would be good).
>

Thanks for trying it out! I've posted a change to  re-direct mach build
"binaries" and "faster" to simply "./mach build" in bug 1493272
. There are more
details in the bug, but simply running "./mach build" in the tup backend
should be as fast and cover cases we'd want "binaries" or "faster" in a
make based backend.


> >cd ~/.mozbuild && mach artifact toolchain --from-build linux64-tup
>
> This doesn't work... no "mach" in my path.  Going to a tree and typing
> './mach artifact toolchain --from-build linux64-tup' works though.
>

We will update the docs to not  assume "mach" is on PATH in bug 1495593



>
> --
> Randell Jesup, Mozilla Corp
> remove "news" for personal email
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform