Re: To what extent is sccache's distributed compilation usable?

[Nicholas Alexander]

On Thu, Mar 12, 2020 at 5:30 PM Marcos Caceres  wrote:

> On Wednesday, October 30, 2019 at 11:59:45 PM UTC+11, Jeff Muizelaar wrote:
> > On Tue, Oct 29, 2019 at 9:45 PM Marcos Caceres 
> wrote:
> > sccache supports doing cross compilation so you don't actually need to
> > buy a fast Mac to get fast Mac builds. A fast cheap Linux desktop will
> > get you much better bang for your buck.
>
> Now that we are all be stuck at home for 1-2 months, might be fun to
> revisit this :)
>
> If anyone manages to get sccache working across multiple Macs/Linux boxes
> at home, could you post some instructions on how to get it going?


I have achieved this two times with different configurations.  Both times I
followed
https://github.com/mozilla/sccache/blob/master/docs/DistributedQuickstart.md
and asked questions in https://chat.mozilla.org/#/room/#build:mozilla.org.
Each time I was mostly building mobile/android; I had issues targeting
macOS (that I think I resolved, once).

The first time I had three devices: a beefy Linux box, a Linux laptop, and
a macOS laptop.  I ran the scheduler and a builder on the beefy Linux box,
I ran the Linux laptop as a builder, and the macOS laptop was (necessarily)
a client.  I used no authentication.

The second I had two devices: the beefy Linux box and the macOS laptop.  I
used the in-office scheduler and ran the beefy Linux box as a builder.

Much of the expertise here has moved on but hopefully we can get things
working for folks.

Best,
Nick
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To what extent is sccache's distributed compilation usable?

[Marcos Caceres]

On Wednesday, October 30, 2019 at 11:59:45 PM UTC+11, Jeff Muizelaar wrote:
> On Tue, Oct 29, 2019 at 9:45 PM Marcos Caceres  wrote: 
> sccache supports doing cross compilation so you don't actually need to
> buy a fast Mac to get fast Mac builds. A fast cheap Linux desktop will
> get you much better bang for your buck.

Now that we are all be stuck at home for 1-2 months, might be fun to revisit 
this :) 

If anyone manages to get sccache working across multiple Macs/Linux boxes at 
home, could you post some instructions on how to get it going? 
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To what extent is sccache's distributed compilation usable?

[Jeff Muizelaar]

On Tue, Oct 29, 2019 at 9:45 PM Marcos Caceres  wrote:

> Thanks Steve... so it does really sound like perhaps just investing in more 
> individual computing power might be the way to go. I think that's ok... even 
> if it means, in my own case, personally paying the extra "Apple Tax" for a 
> Mac Pro once they are released (I guess I'd be looking at ~US$10,000). The 
> increased productivity might make it worth it tho.
>

sccache supports doing cross compilation so you don't actually need to
buy a fast Mac to get fast Mac builds. A fast cheap Linux desktop will
get you much better bang for your buck.

-Jeff
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To what extent is sccache's distributed compilation usable?

[Gabriele Svelto]

 Hi all,
when dealing with builds on slower machines I usually run a build every
morning (4AM) so that when I wake up I've got it ready (together with a
warm cache) and following builds won't be as bad.

This can be done easily on a Mac by setting up a cronjob that will run a
build on a freshly updated mozilla-central clone (you might want to
stash your changes before updating). I then set a schedule with the
energy saver preferences to wake up my Mac from sleep a few minutes
before the cronjob is supposed to run.

This is not a solution for changes that end up affecting large parts of
the build (such as touching a common header) but usually saves me a lot
of time.

 Gabriele

On 29/10/19 18:53, Steve Fink wrote:
> On 10/28/19 9:17 PM, Marcos Caceres wrote:
>> On Tuesday, October 29, 2019 at 3:27:52 AM UTC+11, smaug wrote:
>>> Quite often one has just a laptop. Not compiling tons of Rust stuff
>>> all the time would be really nice.
>>> (I haven't figured out when stylo decides to recompile itself - it
>>> seems to be somewhat random.)
>> Probably a gross misunderstanding on my part, but the sccache project
>> page states [1]: "It is used as a compiler wrapper and avoids
>> compilation when possible, storing a cache in a remote storage using
>> the Amazon Simple Cloud Storage Service (S3) API, the Google Cloud
>> Storage (GCS) API, or Redis."
>>
>> I'm still (possibly naively) imagining that we will leverage the "the
>> cloud"™️ to speed up compiles? Or am I totally misreading what the
>> above is saying?
>>
>> [1] https://github.com/mozilla/sccache#sccache---shared-compilation-cache
> 
> My experience with other distributed compilation tools (distcc, icecc)
> indicates that cloud resources are going to be of very limited use here.
> Compiles are just way too sensitive to network bandwidth and latency,
> especially when compiling with debuginfo which tends to be extremely
> large. Even if the network transfer takes way less time than the
> compile, the sending/receiving scheduling never seems to work out very
> well and things collapse down to a trickle.
> 
> Also, I've had very limited luck with using slow local machines. A CPU
> is not a CPU  -- even on a local gigabit network, farming off compiles
> to slow machines is more likely to slow things down than speed them up.
> Despite the fancy graphical tools, I was never completely satisfied with
> my understanding of exactly why that is. It could be that a lack of
> parallelism meant that everything ended up repeatedly waiting on the
> slow machine to finish the last file in a directory (or whatever your
> boundary of parallelism is). Or it could be network contention,
> especially when your object files have massive debuginfo portions. (I
> always wanted to have a way to generate split debuginfo, and not block
> on the debuginfo transfers.) The tools tended to show things working
> great for a while, and then slowing down to a snail's pace.
> 
> I've long thought [1] that predictive prefetching would be cool: when
> you do something (eg pull from mozilla-central), a background task
> starts prefetching cached build results that were generated remotely.
> Your local compile would use them if they were available, or generate
> them locally if not. That would at least do no harm (if you don't count
> network bandwidth).
> 
> sccache's usage of S3 makes sense when running from within AWS. I'm
> skeptical of its utility when running remotely. But I haven't tried
> setting up sccache on my local network, and my internet connectivity
> isn't great anyway.
> 
> I really ought to put my decade-old desktop into action again. My last
> attempt was with icecc, and though it worked pretty well when it worked,
> the pain in keeping it alive wasn't worth the benefit.
> 
> 
> [1] Ancient history -
> https://wiki.mozilla.org/Sfink/Thought_Experiment_-_One_Minute_Builds
> 
> 
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform




signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To what extent is sccache's distributed compilation usable?

[Reuben Morais]

> On 29. Oct 2019, at 18:53, Steve Fink  wrote:
> 
> On 10/28/19 9:17 PM, Marcos Caceres wrote:
>> On Tuesday, October 29, 2019 at 3:27:52 AM UTC+11, smaug wrote:
>>> Quite often one has just a laptop. Not compiling tons of Rust stuff all the 
>>> time would be really nice.
>>> (I haven't figured out when stylo decides to recompile itself - it seems to 
>>> be somewhat random.)
>> Probably a gross misunderstanding on my part, but the sccache project page 
>> states [1]: "It is used as a compiler wrapper and avoids compilation when 
>> possible, storing a cache in a remote storage using the Amazon Simple Cloud 
>> Storage Service (S3) API, the Google Cloud Storage (GCS) API, or Redis."
>> 
>> I'm still (possibly naively) imagining that we will leverage the "the 
>> cloud"™️ to speed up compiles? Or am I totally misreading what the above is 
>> saying?
>> 
>> [1] https://github.com/mozilla/sccache#sccache---shared-compilation-cache
> 
> My experience with other distributed compilation tools (distcc, icecc) 
> indicates that cloud resources are going to be of very limited use here. 
> Compiles are just way too sensitive to network bandwidth and latency, 
> especially when compiling with debuginfo which tends to be extremely large. 
> Even if the network transfer takes way less time than the compile, the 
> sending/receiving scheduling never seems to work out very well and things 
> collapse down to a trickle.
> 
> Also, I've had very limited luck with using slow local machines. A CPU is not 
> a CPU  -- even on a local gigabit network, farming off compiles to slow 
> machines is more likely to slow things down than speed them up. Despite the 
> fancy graphical tools, I was never completely satisfied with my understanding 
> of exactly why that is. It could be that a lack of parallelism meant that 
> everything ended up repeatedly waiting on the slow machine to finish the last 
> file in a directory (or whatever your boundary of parallelism is). Or it 
> could be network contention, especially when your object files have massive 
> debuginfo portions. (I always wanted to have a way to generate split 
> debuginfo, and not block on the debuginfo transfers.) The tools tended to 
> show things working great for a while, and then slowing down to a snail's 
> pace.

This research offers some insight: 
https://www.usenix.org/system/files/atc19-fouladi.pdf / 
https://github.com/StanfordSNR/gg

In particular:

> Existing remote compilation systems, including distcc and icecc, send data 
> between a master node and the workers frequently during the build. These 
> systems perform best on a local network, and add substantial latency when 
> building on more remote servers in the cloud. In contrast, gg uploads all the 
> build input once and executes and exchanges data purely within the cloud, 
> reducing the effects of network latency.

Their system using 8000 AWS Lambdas is 2x faster than an icecc cluster when 
building Chromium, although that is still a *remote* icecc cluster. Maybe the 
speedup is less significant with a local cluster like we're running in our 
offices.

-- reuben
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To what extent is sccache's distributed compilation usable?

[Marcos Caceres]

On Wednesday, October 30, 2019 at 4:53:55 AM UTC+11, Steve Fink wrote: 
> I really ought to put my decade-old desktop into action again. My last 
> attempt was with icecc, and though it worked pretty well when it worked, 
> the pain in keeping it alive wasn't worth the benefit.

Thanks Steve... so it does really sound like perhaps just investing in more 
individual computing power might be the way to go. I think that's ok... even if 
it means, in my own case, personally paying the extra "Apple Tax" for a Mac Pro 
once they are released (I guess I'd be looking at ~US$10,000). The increased 
productivity might make it worth it tho.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To what extent is sccache's distributed compilation usable?

[Steve Fink]

On 10/28/19 9:17 PM, Marcos Caceres wrote:

On Tuesday, October 29, 2019 at 3:27:52 AM UTC+11, smaug wrote:

Quite often one has just a laptop. Not compiling tons of Rust stuff all the 
time would be really nice.
(I haven't figured out when stylo decides to recompile itself - it seems to be 
somewhat random.)

Probably a gross misunderstanding on my part, but the sccache project page states [1]: 
"It is used as a compiler wrapper and avoids compilation when possible, storing a 
cache in a remote storage using the Amazon Simple Cloud Storage Service (S3) API, the 
Google Cloud Storage (GCS) API, or Redis."

I'm still (possibly naively) imagining that we will leverage the "the cloud"™️ 
to speed up compiles? Or am I totally misreading what the above is saying?

[1] https://github.com/mozilla/sccache#sccache---shared-compilation-cache


My experience with other distributed compilation tools (distcc, icecc) 
indicates that cloud resources are going to be of very limited use here. 
Compiles are just way too sensitive to network bandwidth and latency, 
especially when compiling with debuginfo which tends to be extremely 
large. Even if the network transfer takes way less time than the 
compile, the sending/receiving scheduling never seems to work out very 
well and things collapse down to a trickle.


Also, I've had very limited luck with using slow local machines. A CPU 
is not a CPU  -- even on a local gigabit network, farming off compiles 
to slow machines is more likely to slow things down than speed them up. 
Despite the fancy graphical tools, I was never completely satisfied with 
my understanding of exactly why that is. It could be that a lack of 
parallelism meant that everything ended up repeatedly waiting on the 
slow machine to finish the last file in a directory (or whatever your 
boundary of parallelism is). Or it could be network contention, 
especially when your object files have massive debuginfo portions. (I 
always wanted to have a way to generate split debuginfo, and not block 
on the debuginfo transfers.) The tools tended to show things working 
great for a while, and then slowing down to a snail's pace.


I've long thought [1] that predictive prefetching would be cool: when 
you do something (eg pull from mozilla-central), a background task 
starts prefetching cached build results that were generated remotely. 
Your local compile would use them if they were available, or generate 
them locally if not. That would at least do no harm (if you don't count 
network bandwidth).


sccache's usage of S3 makes sense when running from within AWS. I'm 
skeptical of its utility when running remotely. But I haven't tried 
setting up sccache on my local network, and my internet connectivity 
isn't great anyway.


I really ought to put my decade-old desktop into action again. My last 
attempt was with icecc, and though it worked pretty well when it worked, 
the pain in keeping it alive wasn't worth the benefit.



[1] Ancient history - 
https://wiki.mozilla.org/Sfink/Thought_Experiment_-_One_Minute_Builds



___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To what extent is sccache's distributed compilation usable?

[Simon Sapin]

On 28/10/2019 17:27, smaug wrote:

Quite often one has just a laptop. Not compiling tons of Rust stuff all the 
time would be really nice.
(I haven't figured out when stylo decides to recompile itself - it seems to be 
somewhat random.)


I’m pretty sure there is no call to a random number generator in Cargo’s 
code for dependency tracking. Spurious recompiles are bugs that are 
worth filing so that they can be fixed.


If you manage to find steps to reproduce, building with a 
CARGO_LOG=cargo::core::compiler::fingerprint environment variable should 
print some details about why Cargo things some crate (and there for 
those that transitively depend on it) needs to be rebuilt.


https://github.com/servo/mozjs/pull/203 is an example of a fix for such 
a bug: a combination of a `build.rs` script being over-eager in using 
`rerun-if-changed` and calling something that ends up writing to the 
source directory.


`rerun-if-changed` is documented at 
https://doc.rust-lang.org/cargo/reference/build-scripts.html



On 29/10/2019 00:54, Gerald Squelart wrote:

It's a bit annoying to see things like "force-cargo-...-build" when
nothing has changed,


That much makes sense to me: running `cargo build` is part of figuring 
out that nothing has changed. Cargo has a lot of dependency tracking 
logic, duplicating it all to avoid calling it would be unproductive and 
fragile. Running `cargo build` with a warm filesystem cache when there’s 
nothing to do should not take more than a couple of seconds. (Looking at 
the mtime of each source file.)




accompanied by lots of "waiting for file lock on package cache".


This line is printed − once − when Cargo is waiting for a filesystem 
lock. It showing up N times is a sign that there are at least N+1 copies 
of Cargo running concurrently.


It would be nice to run Cargo only once and tell it up-front everything 
it needs to do (with `-p foo` multiple times, or `--all`, or 
`default-memebers` 
https://doc.rust-lang.org/cargo/reference/manifest.html#package-selection) 
rather that having multiples copies competing for a shared resource.


--
Simon Sapin
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To what extent is sccache's distributed compilation usable?

[Gerald Squelart]

On Tuesday, October 29, 2019 at 3:27:52 AM UTC+11, smaug wrote:
> On 10/24/19 9:17 AM, Marcos Caceres wrote:
> > On Thursday, October 24, 2019 at 6:46:08 AM UTC+11, Christopher Manchester 
> > wrote:
> >> As announced in this week's project call, sccache-dist schedulers have been
> >> deployed to mozilla offices, and those with hardware available can enroll
> >> servers based on the documentation here
> >> .
> > 
> > Just a reminder to not forget about us remotees! We are dying out here with 
> > 1+ hour compile times so anything would really help (even 20-20% 
> > improvement would go a long way). A lot of us have spare old macs, and if 
> > we can put them to good use with this, that would be amazing.
> > 
> 
> 
> Quite often one has just a laptop. Not compiling tons of Rust stuff all the 
> time would be really nice.
> (I haven't figured out when stylo decides to recompile itself - it seems to 
> be somewhat random.)

It's a bit annoying to see things like "force-cargo-...-build" when nothing has 
changed, accompanied by lots of "waiting for file lock on package cache".
It's adding tens of seconds to every build. sccache doesn't help.
I opened https://bugzilla.mozilla.org/show_bug.cgi?id=1568168 for that a while 
ago.

On the sccache topic, if it supports distributed builds, great! I'll have to 
give it a try...
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To what extent is sccache's distributed compilation usable?

[smaug]

On 10/24/19 9:17 AM, Marcos Caceres wrote:

On Thursday, October 24, 2019 at 6:46:08 AM UTC+11, Christopher Manchester 
wrote:

As announced in this week's project call, sccache-dist schedulers have been
deployed to mozilla offices, and those with hardware available can enroll
servers based on the documentation here
.


Just a reminder to not forget about us remotees! We are dying out here with 1+ 
hour compile times so anything would really help (even 20-20% improvement would 
go a long way). A lot of us have spare old macs, and if we can put them to good 
use with this, that would be amazing.




Quite often one has just a laptop. Not compiling tons of Rust stuff all the 
time would be really nice.
(I haven't figured out when stylo decides to recompile itself - it seems to be 
somewhat random.)
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To what extent is sccache's distributed compilation usable?

[Marcos Caceres]

On Saturday, October 26, 2019 at 11:47:51 PM UTC+11, Emilio Cobos Álvarez 
wrote: 
> So you have two options, I think:
> 
>   * Setup a Linux box as an sccache-dist scheduler and builder. This 
> should require no SDKs or what not, but requires some setup in your mac 
> as described here[1]. Then you'd just do regular sccache-enabled ./mach 
> build from your Mac, which would then distribute work to your Linux box.

I think I'll try option 1... I have an old Surface Pro running Ubuntu which I 
can use as the scheduler. It's probably too underpowered to be helpful in 
compiling... but you never know - every CPU helps, I guess... even at 1GHz. 
Then I've got a couple of old (2014) MacBook Pros to throw into the mix, which, 
together with my 2018 MacBook Pro should get me to around 20+ CPUs. I guess if 
that works, I can look at buying some cheap Linux setup with Ryzen in it to 
feed it more CPUs. 

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To what extent is sccache's distributed compilation usable?

[Emilio Cobos Álvarez]

On 10/25/19 2:54 AM, Marcos Caceres wrote:

Do you know if it's possible to cross compile on one of those for to target 
MacOS? I'd like to build on, and for, MacOS.


Our official mac builds are cross-compiled from Linux, IIRC. So yeah, it 
should be possible to do that.


Cross-compiling to target MacOS is a bit involved, requiring getting the 
relevant SDKs in the Linux box, the relevant mozconfig flags, and what 
not. But probably nothing too bad? I've done it in the past, mostly 
cargo-culting from the official mozconfigs in the tree. Not sure if 
there are official docs somewhere though.


I also know that sccache-dist can be used somewhat painlessly from Mac, 
unlike icecc (requires some setup as described here[1], but doesn't seem 
too bad).


So you have two options, I think:

 * Setup a Linux box as an sccache-dist scheduler and builder. This 
should require no SDKs or what not, but requires some setup in your mac 
as described here[1]. Then you'd just do regular sccache-enabled ./mach 
build from your Mac, which would then distribute work to your Linux box.


 * Setup a Linux box with a full cross-compilation setup, build there, 
and somehow pull the objdir from there back to your mac (not sure what's 
best here for incremental builds, rsync or something?).


If you only have one builder, and your Mac is slow, the second option is 
probably a bit faster to get a clobber build than sccache (as you don't 
need to do network stuff for every compilation unit and such). But 
obviously I haven't measured, so it may be the case that the sccache 
overhead is low and the extra cores from your mac add up for it.


Some caveats about the second setup:

 * I don't know how to best do the objdir / incremental builds with 
such a setup (some people in this list probably know though, and I'd be 
interested in knowing too!).


 * You need to do your changes over ssh, or have a setup where you also 
sync the src directories or something.


The first option should also be relatively straight-forward, hopefully, 
looking at the docs here[2], though I haven't set up an sccache 
scheduler myself.


Again there's people in this list that know more than me, so probably 
they can help more :)


Cheers,

 -- Emilio

[1]: 
https://firefox-source-docs.mozilla.org/build/buildsystem/sccache-dist.html
[2]: 
https://github.com/mozilla/sccache/blob/master/docs/DistributedQuickstart.md#configure-a-scheduler

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To what extent is sccache's distributed compilation usable?

[Jeff Gilbert]

I don't know. If nothing else, it looks like it's possible to run macOS in
a VM in some cases, so maybe that's worth someone exploring. Unfortunately,
the upcoming Mac Pros look to be over four times as expensive as my machine.

On Thu, Oct 24, 2019 at 5:55 PM Marcos Caceres  wrote:

> On Friday, October 25, 2019 at 4:14:05 AM UTC+11, Jeff Gilbert wrote:
> > My home Ryzen 3900x builds windows debug clobbers in under 10 minutes.
>
> Whoa, that's awesome!
>
> > Beefy desktops are relatively cheap, and I believe we continue to
> encourage
> > firefox-builders to request desktop workstations for builds.
>
> Do you know if it's possible to cross compile on one of those for to
> target MacOS? I'd like to build on, and for, MacOS.
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To what extent is sccache's distributed compilation usable?

[Marcos Caceres]

On Friday, October 25, 2019 at 4:14:05 AM UTC+11, Jeff Gilbert wrote:
> My home Ryzen 3900x builds windows debug clobbers in under 10 minutes.

Whoa, that's awesome! 

> Beefy desktops are relatively cheap, and I believe we continue to encourage
> firefox-builders to request desktop workstations for builds.

Do you know if it's possible to cross compile on one of those for to target 
MacOS? I'd like to build on, and for, MacOS. 

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To what extent is sccache's distributed compilation usable?

[Jeff Gilbert]

My home Ryzen 3900x builds windows debug clobbers in under 10 minutes.
Beefy desktops are relatively cheap, and I believe we continue to encourage
firefox-builders to request desktop workstations for builds.

On Wed, Oct 23, 2019 at 11:20 PM Marcos Caceres 
wrote:

> On Thursday, October 24, 2019 at 6:46:08 AM UTC+11, Christopher Manchester
> wrote:
> > As announced in this week's project call, sccache-dist schedulers have
> been
> > deployed to mozilla offices, and those with hardware available can enroll
> > servers based on the documentation here
> > <
> https://firefox-source-docs.mozilla.org/build/buildsystem/sccache-dist.html#steps-for-setting-up-a-server
> >.
>
> Just a reminder to not forget about us remotees! We are dying out here
> with 1+ hour compile times so anything would really help (even 20-20%
> improvement would go a long way). A lot of us have spare old macs, and if
> we can put them to good use with this, that would be amazing.
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To what extent is sccache's distributed compilation usable?

[Marcos Caceres]

On Thursday, October 24, 2019 at 6:46:08 AM UTC+11, Christopher Manchester 
wrote:
> As announced in this week's project call, sccache-dist schedulers have been
> deployed to mozilla offices, and those with hardware available can enroll
> servers based on the documentation here
> .

Just a reminder to not forget about us remotees! We are dying out here with 1+ 
hour compile times so anything would really help (even 20-20% improvement would 
go a long way). A lot of us have spare old macs, and if we can put them to good 
use with this, that would be amazing.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To what extent is sccache's distributed compilation usable?

[Christopher Manchester]

As announced in this week's project call, sccache-dist schedulers have been
deployed to mozilla offices, and those with hardware available can enroll
servers based on the documentation here
.
That document also contains instructions for building as a client after
servers have been enrolled.

We're still actively working on improvements, but early testing has proved
the current setup to be usable and provide compelling build time
improvements.
Visit https://github.com/mozilla/sccache/issues for open issues or to file
as they come up. Issues specific to integration with Mozilla's build system
are tracked by bug 1439584
.

Please join #sccache on Slack for questions or discussion.

Chris



On Fri, Jun 7, 2019 at 10:13 AM Chris M.  wrote:

> I've been working through issues with distributed sccache, but I've
> recently sent instructions to people currently running icecream in the SF
> office on how to switch to running sccache-dist with IT managed hardware.
> We're going to use that as a test bed before rolling out more widely, and
> if all goes well there this should happen shortly (likely shortly after
> returning from Whistler).
>
> Regarding that bug, that's tracking IT work for the most part so probably
> needs to stay restricted. Most of the work I've been doing is tracked in
> sccache's github repo and can be followed there, or tracked by
> https://bugzilla.mozilla.org/show_bug.cgi?id=1499145.
>
> Chris
>
> On Wed, Jun 5, 2019 at 1:52 AM Miko Mynttinen 
> wrote:
>
>> Has there been any progress with this? Bug 1524533 is still restricted.
>>
>> Miko
>>
>> > On Apr 2, 2019, at 10:39 PM, Jeff Muizelaar 
>> wrote:
>> >
>> > That bug is restricted. Does it need to be?
>> >
>> > -Jeff
>> >
>> > On Tue, Apr 2, 2019 at 4:22 PM Kim Moir  wrote:
>> >>
>> >> The tracking bug for sccache being deployed to offices is here
>> >>
>> >> https://bugzilla.mozilla.org/show_bug.cgi?id=1524533
>> >>
>> >> Kim
>> >>
>> >> On Tue, Apr 2, 2019 at 3:22 AM Frederik Braun 
>> wrote:
>> >>
>> >>> Am 01.04.19 um 22:16 schrieb Chris M.:
>>  Hi Emilio, if you're interested you're encouraged to try it out,
>> however
>>  we're shipping machines to offices now to run the schedulers, the
>> first
>> >>> of
>>  which I'll be testing in the SF office this week, so we're planning
>> an
>>  officially supported setup in the near future.
>> 
>> >>>
>> >>> That's great news. Can we follow progress or detailed plans somewhere?
>> >>> ___
>> >>> dev-platform mailing list
>> >>> dev-platform@lists.mozilla.org
>> >>> https://lists.mozilla.org/listinfo/dev-platform
>> >>>
>> >> ___
>> >> dev-platform mailing list
>> >> dev-platform@lists.mozilla.org
>> >> https://lists.mozilla.org/listinfo/dev-platform
>> > ___
>> > dev-platform mailing list
>> > dev-platform@lists.mozilla.org
>> > https://lists.mozilla.org/listinfo/dev-platform
>>
>> ___
>> dev-platform mailing list
>> dev-platform@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-platform
>>
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To what extent is sccache's distributed compilation usable?

[Jonathan Watt]

Just in case you're not aware of it, there are a couple of perf issues that may
make people prefer ccache for now:

https://github.com/mozilla/sccache/issues/219
https://github.com/mozilla/sccache/issues/160

Jonathan

On 07/06/2019 18:13, Chris M. wrote:
> I've been working through issues with distributed sccache, but I've
> recently sent instructions to people currently running icecream in the SF
> office on how to switch to running sccache-dist with IT managed hardware.
> We're going to use that as a test bed before rolling out more widely, and
> if all goes well there this should happen shortly (likely shortly after
> returning from Whistler).
> 
> Regarding that bug, that's tracking IT work for the most part so probably
> needs to stay restricted. Most of the work I've been doing is tracked in
> sccache's github repo and can be followed there, or tracked by
> https://bugzilla.mozilla.org/show_bug.cgi?id=1499145.
> 
> Chris
> 
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To what extent is sccache's distributed compilation usable?

[Chris M.]

I've been working through issues with distributed sccache, but I've
recently sent instructions to people currently running icecream in the SF
office on how to switch to running sccache-dist with IT managed hardware.
We're going to use that as a test bed before rolling out more widely, and
if all goes well there this should happen shortly (likely shortly after
returning from Whistler).

Regarding that bug, that's tracking IT work for the most part so probably
needs to stay restricted. Most of the work I've been doing is tracked in
sccache's github repo and can be followed there, or tracked by
https://bugzilla.mozilla.org/show_bug.cgi?id=1499145.

Chris

On Wed, Jun 5, 2019 at 1:52 AM Miko Mynttinen 
wrote:

> Has there been any progress with this? Bug 1524533 is still restricted.
>
> Miko
>
> > On Apr 2, 2019, at 10:39 PM, Jeff Muizelaar 
> wrote:
> >
> > That bug is restricted. Does it need to be?
> >
> > -Jeff
> >
> > On Tue, Apr 2, 2019 at 4:22 PM Kim Moir  wrote:
> >>
> >> The tracking bug for sccache being deployed to offices is here
> >>
> >> https://bugzilla.mozilla.org/show_bug.cgi?id=1524533
> >>
> >> Kim
> >>
> >> On Tue, Apr 2, 2019 at 3:22 AM Frederik Braun 
> wrote:
> >>
> >>> Am 01.04.19 um 22:16 schrieb Chris M.:
>  Hi Emilio, if you're interested you're encouraged to try it out,
> however
>  we're shipping machines to offices now to run the schedulers, the
> first
> >>> of
>  which I'll be testing in the SF office this week, so we're planning an
>  officially supported setup in the near future.
> 
> >>>
> >>> That's great news. Can we follow progress or detailed plans somewhere?
> >>> ___
> >>> dev-platform mailing list
> >>> dev-platform@lists.mozilla.org
> >>> https://lists.mozilla.org/listinfo/dev-platform
> >>>
> >> ___
> >> dev-platform mailing list
> >> dev-platform@lists.mozilla.org
> >> https://lists.mozilla.org/listinfo/dev-platform
> > ___
> > dev-platform mailing list
> > dev-platform@lists.mozilla.org
> > https://lists.mozilla.org/listinfo/dev-platform
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To what extent is sccache's distributed compilation usable?

[Miko Mynttinen]

Has there been any progress with this? Bug 1524533 is still restricted.

Miko

> On Apr 2, 2019, at 10:39 PM, Jeff Muizelaar  wrote:
> 
> That bug is restricted. Does it need to be?
> 
> -Jeff
> 
> On Tue, Apr 2, 2019 at 4:22 PM Kim Moir  wrote:
>> 
>> The tracking bug for sccache being deployed to offices is here
>> 
>> https://bugzilla.mozilla.org/show_bug.cgi?id=1524533
>> 
>> Kim
>> 
>> On Tue, Apr 2, 2019 at 3:22 AM Frederik Braun  wrote:
>> 
>>> Am 01.04.19 um 22:16 schrieb Chris M.:
 Hi Emilio, if you're interested you're encouraged to try it out, however
 we're shipping machines to offices now to run the schedulers, the first
>>> of
 which I'll be testing in the SF office this week, so we're planning an
 officially supported setup in the near future.
 
>>> 
>>> That's great news. Can we follow progress or detailed plans somewhere?
>>> ___
>>> dev-platform mailing list
>>> dev-platform@lists.mozilla.org
>>> https://lists.mozilla.org/listinfo/dev-platform
>>> 
>> ___
>> dev-platform mailing list
>> dev-platform@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-platform
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To what extent is sccache's distributed compilation usable?

[Jeff Muizelaar]

That bug is restricted. Does it need to be?

-Jeff

On Tue, Apr 2, 2019 at 4:22 PM Kim Moir  wrote:
>
> The tracking bug for sccache being deployed to offices is here
>
> https://bugzilla.mozilla.org/show_bug.cgi?id=1524533
>
> Kim
>
> On Tue, Apr 2, 2019 at 3:22 AM Frederik Braun  wrote:
>
> > Am 01.04.19 um 22:16 schrieb Chris M.:
> > > Hi Emilio, if you're interested you're encouraged to try it out, however
> > > we're shipping machines to offices now to run the schedulers, the first
> > of
> > > which I'll be testing in the SF office this week, so we're planning an
> > > officially supported setup in the near future.
> > >
> >
> > That's great news. Can we follow progress or detailed plans somewhere?
> > ___
> > dev-platform mailing list
> > dev-platform@lists.mozilla.org
> > https://lists.mozilla.org/listinfo/dev-platform
> >
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To what extent is sccache's distributed compilation usable?

[Kim Moir]

The tracking bug for sccache being deployed to offices is here

https://bugzilla.mozilla.org/show_bug.cgi?id=1524533

Kim

On Tue, Apr 2, 2019 at 3:22 AM Frederik Braun  wrote:

> Am 01.04.19 um 22:16 schrieb Chris M.:
> > Hi Emilio, if you're interested you're encouraged to try it out, however
> > we're shipping machines to offices now to run the schedulers, the first
> of
> > which I'll be testing in the SF office this week, so we're planning an
> > officially supported setup in the near future.
> >
>
> That's great news. Can we follow progress or detailed plans somewhere?
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To what extent is sccache's distributed compilation usable?

[Frederik Braun]

Am 01.04.19 um 22:16 schrieb Chris M.:
> Hi Emilio, if you're interested you're encouraged to try it out, however
> we're shipping machines to offices now to run the schedulers, the first of
> which I'll be testing in the SF office this week, so we're planning an
> officially supported setup in the near future.
> 

That's great news. Can we follow progress or detailed plans somewhere?
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To what extent is sccache's distributed compilation usable?

[Nicholas Alexander]

On Mon, Apr 1, 2019 at 1:03 PM Emilio Cobos Álvarez 
wrote:

> On 01/04/2019 21:40, Bobby Holley wrote:
> > My (possibly outdated) understanding is that the sccache distributed
> > compilation stuff is restricted to the build machines we control.
> Naively,
> > I'm not sure there's a secure way to have machines in the wild contribute
> > build artifacts, given the risk of malicious code.
> >
> > It seems plausible that we could at least allow read-only access to the
> > artifacts generated by the build machines, but that would presumably only
> > be useful if the developer's system configuration exactly matched that of
> > the build machine.
>
> Oh, the setup I was thinking of would be similar to how icecc works now,
> on a local network. So, for example, N developers in the Berlin office
> sharing a scheduler and build resources in the same network.
>
> I'm not 100% sure whether the distributed compilation part of sccache is
> only about sharing compilation artifacts, or also compiling stuff in
> remote machines (like icecc does), or both, but afaik even in a local
> network it could still be a pretty nice build-time / productivity
> improvement.
>

It does distribute compilation tasks to remote hosts.  At this time those
hosts must be running Linux (for the `bubblewrap` chroot-like package).
Months ago I configured this, following
https://github.com/mozilla/sccache/blob/master/docs/DistributedQuickstart.md,
for a Linux host and a macOS device.  Setting up the toolchains was
frustrating but do-able.  I did not get this working on the YVR office
build monster, just 'cuz I don't do a lot of native compilation.

Nick
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To what extent is sccache's distributed compilation usable?

[Chris M.]

On Mon, Apr 1, 2019 at 12:23 PM Emilio Cobos Álvarez 
wrote:

> Hi,
>
> Over in #layout people were ranting about how much of a pain it was to
> set up icecc on MacOS. I pointed out that sccache in theory does support
> distributed compilation[1] and in theory also supports MacOS clients.
>
> Does anybody know to what extent is it in a usable state? Can it replace
> icecc? I'm a Linux user, and icecc works pretty well, but if sccache
> supports Rust as well that's a huge plus.
>
> Are people encouraged to try it out? Or only experimentally? Or should
> we wait?
>
Hi Emilio, if you're interested you're encouraged to try it out, however
we're shipping machines to offices now to run the schedulers, the first of
which I'll be testing in the SF office this week, so we're planning an
officially supported setup in the near future.

>
> I know Ted encouraged me to try it in Berlin, and I still have to do
> that (d'oh :/). So given that I suspect the answer is "it's worth trying
> out and report bugs". But worth knowing if it's more generally usable or
> mostly experimental.
>
> Thanks,
>
>  -- Emilio
>
> [1]:
>
> https://github.com/mozilla/sccache/blob/master/docs/DistributedQuickstart.md
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To what extent is sccache's distributed compilation usable?

[Emilio Cobos Álvarez]

On 01/04/2019 21:40, Bobby Holley wrote:
> My (possibly outdated) understanding is that the sccache distributed
> compilation stuff is restricted to the build machines we control. Naively,
> I'm not sure there's a secure way to have machines in the wild contribute
> build artifacts, given the risk of malicious code.
> 
> It seems plausible that we could at least allow read-only access to the
> artifacts generated by the build machines, but that would presumably only
> be useful if the developer's system configuration exactly matched that of
> the build machine.

Oh, the setup I was thinking of would be similar to how icecc works now,
on a local network. So, for example, N developers in the Berlin office
sharing a scheduler and build resources in the same network.

I'm not 100% sure whether the distributed compilation part of sccache is
only about sharing compilation artifacts, or also compiling stuff in
remote machines (like icecc does), or both, but afaik even in a local
network it could still be a pretty nice build-time / productivity
improvement.

 -- Emilio

> On Mon, Apr 1, 2019 at 12:23 PM Emilio Cobos Álvarez 
> wrote:
> 
>> Hi,
>>
>> Over in #layout people were ranting about how much of a pain it was to
>> set up icecc on MacOS. I pointed out that sccache in theory does support
>> distributed compilation[1] and in theory also supports MacOS clients.
>>
>> Does anybody know to what extent is it in a usable state? Can it replace
>> icecc? I'm a Linux user, and icecc works pretty well, but if sccache
>> supports Rust as well that's a huge plus.
>>
>> Are people encouraged to try it out? Or only experimentally? Or should
>> we wait?
>>
>> I know Ted encouraged me to try it in Berlin, and I still have to do
>> that (d'oh :/). So given that I suspect the answer is "it's worth trying
>> out and report bugs". But worth knowing if it's more generally usable or
>> mostly experimental.
>>
>> Thanks,
>>
>>  -- Emilio
>>
>> [1]:
>>
>> https://github.com/mozilla/sccache/blob/master/docs/DistributedQuickstart.md
>> ___
>> dev-platform mailing list
>> dev-platform@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-platform
>>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
> 
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To what extent is sccache's distributed compilation usable?

[Bobby Holley]

My (possibly outdated) understanding is that the sccache distributed
compilation stuff is restricted to the build machines we control. Naively,
I'm not sure there's a secure way to have machines in the wild contribute
build artifacts, given the risk of malicious code.

It seems plausible that we could at least allow read-only access to the
artifacts generated by the build machines, but that would presumably only
be useful if the developer's system configuration exactly matched that of
the build machine.

On Mon, Apr 1, 2019 at 12:23 PM Emilio Cobos Álvarez 
wrote:

> Hi,
>
> Over in #layout people were ranting about how much of a pain it was to
> set up icecc on MacOS. I pointed out that sccache in theory does support
> distributed compilation[1] and in theory also supports MacOS clients.
>
> Does anybody know to what extent is it in a usable state? Can it replace
> icecc? I'm a Linux user, and icecc works pretty well, but if sccache
> supports Rust as well that's a huge plus.
>
> Are people encouraged to try it out? Or only experimentally? Or should
> we wait?
>
> I know Ted encouraged me to try it in Berlin, and I still have to do
> that (d'oh :/). So given that I suspect the answer is "it's worth trying
> out and report bugs". But worth knowing if it's more generally usable or
> mostly experimental.
>
> Thanks,
>
>  -- Emilio
>
> [1]:
>
> https://github.com/mozilla/sccache/blob/master/docs/DistributedQuickstart.md
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

To what extent is sccache's distributed compilation usable?

[Emilio Cobos Álvarez]

Hi,

Over in #layout people were ranting about how much of a pain it was to
set up icecc on MacOS. I pointed out that sccache in theory does support
distributed compilation[1] and in theory also supports MacOS clients.

Does anybody know to what extent is it in a usable state? Can it replace
icecc? I'm a Linux user, and icecc works pretty well, but if sccache
supports Rust as well that's a huge plus.

Are people encouraged to try it out? Or only experimentally? Or should
we wait?

I know Ted encouraged me to try it in Berlin, and I still have to do
that (d'oh :/). So given that I suspect the answer is "it's worth trying
out and report bugs". But worth knowing if it's more generally usable or
mostly experimental.

Thanks,

 -- Emilio

[1]:
https://github.com/mozilla/sccache/blob/master/docs/DistributedQuickstart.md
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform