Re: Must we rebuild all our rust code constantly?

2019-08-28 Thread mwoerister
On Friday, August 23, 2019 at 9:37:12 AM UTC+2, Simon Sapin wrote:
> On 23/08/2019 04:00, Mike Hommey wrote:
> > I had a similar experience on a 36-core machine earlier today. It seems
> > incremental is not working properly for the style crate: incremental is
> > supposed to have the same effect as sccache, but it still takes a long
> > time to compile style. I'm told mw is going to look into this in the
> > coming weeks.
> 
> I don’t know how much it has evolved since, but the initial versions of 
> incremental compilation in rustc only cached code generation in LLVM. 
> Type checking, trait resolution, etc. was still done every time rustc 
> was invoked for a crate.
> 
> -- 
> Simon Sapin


With recent versions of `rustc`, incremental compilation caches pretty much 
everything after parsing and macro expansion. However, the style crate in 
particular has proven to be rather resistant to our incremental compilation 
approach so far. I will indeed spend quite a bit of time investigating this 
over the next few months.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: Must we rebuild all our rust code constantly?

2019-08-24 Thread ISHIKAWA,chiaki

Here is my own answer to clean up the mess of version mismatches, etc.

I did the following.

1. I removed ~/.mozbuild and ~/.cargo directory and
re-ran |make bootstrap|.
(I am not sure if removing ~/.cargo directory was a good idea.)

2. I revered my local changes before I update the M-C and C-C (./comm).

3. Then I noticed mozilla/Cargo.lock has been modified when I tried to 
reapply my local changes.

This file was presumably changed when I did my own sccache installation.

So I reverted the change to Cargo.lock, and now it seems that the 
compilation is proceeding... (Not sure if the result is OK.)



4. When I did this, I noticed there is sccache 0.2.11-Alpha.0 under 
~/.mozbuild/sccache directory.
Obviously, I was using my own installed older version of sccache which I 
installed many months ago.
So I added the path to this sccache in my script for building TB and it 
seems to be compiling TB.


I added the following in my mozconfig for TB, but I am not sure if this 
is the right way to disable incremental compilation.


mk_add_options "export CARGO_INCREMENTAL=0"

TIA

On 2019/08/22 11:22, ISHIKAWA,chiaki wrote:
Well, I have a problem now after trying to update sccache just in case 
I need a new version in the future.


I did the following:

cargo install --force sccache

(I was not so sure of what the proper update procedure of already 
installed package. sccache 2.0.8-alpha-something was already installed.)


Then when I try to compile TB using the command script which worked 
for more than a few years suddenly printed the following and failed. I 
think I have messed up my cargo metadata somehow by the above command.


I have run |mach bootstrap| to see if it fixes the issue. It did not.

ERROR: Couldn't execute `cargo metadata` with manifest 
"/NREF-COMM-CENTRAL/mozilla/toolkit/library/rust/Cargo.toml": 
Metadata(Output { status: ExitStatus(ExitStatus(25856)), stdout: "", 
stderr: "error: failed to select a version for the requirement `libc = 
\"= 0.2.62\"`\n  candidate versions found which didn\'t match: 
0.2.60\n  location searched: directory source 
`/NREF-COMM-CENTRAL/mozilla/third_party/rust` (which is replacing 
registry `https://github.com/rust-lang/crates.io-index`)\nrequired by 
package `mozjs_sys v0.0.0 
(/NREF-COMM-CENTRAL/mozilla/js/src)`\nperhaps a crate was updated and 
forgotten to be re-vendored?\n" })
ERROR: Couldn't generate bindings for 
/NREF-COMM-CENTRAL/mozilla/toolkit/library/rust.

make[4]: *** [backend.mk:72: .deps/ServoStyleConsts.h.stub] Error 1
make[3]: *** [/NREF-COMM-CENTRAL/mozilla/config/recurse.mk:101: 
layout/style/export] Error 2

make[3]: *** Waiting for unfinished jobs
ERROR: Couldn't execute `cargo metadata` with manifest 
"/NREF-COMM-CENTRAL/mozilla/toolkit/library/rust/Cargo.toml": 
Metadata(Output { status: ExitStatus(ExitStatus(25856)), stdout: "", 
stderr: "    Blocking waiting for file lock on package cache 
lock\nerror: failed to select a version for the requirement `libc = 
\"= 0.2.62\"`\n  candidate versions found which didn\'t match: 
0.2.60\n  location searched: directory source 
`/NREF-COMM-CENTRAL/mozilla/third_party/rust` (which is replacing 
registry `https://github.com/rust-lang/crates.io-index`)\nrequired by 
package `mozjs_sys v0.0.0 
(/NREF-COMM-CENTRAL/mozilla/js/src)`\nperhaps a crate was updated and 
forgotten to be re-vendored?\n" })
ERROR: Couldn't generate bindings for 
/NREF-COMM-CENTRAL/mozilla/toolkit/library/rust.
make[4]: *** [backend.mk:11: .deps/webrender_ffi_generated.h.stub] 
Error 1
make[3]: *** [/NREF-COMM-CENTRAL/mozilla/config/recurse.mk:101: 
gfx/webrender_bindings/export] Error 2


Does anyone have a clue how to restore order here?

TIA

Chiaki


On 2019/08/21 4:33, Markus Stange wrote:

On 2019-08-19 8:11 p.m., Dave Townsend wrote:
Thanks to a tip I've tracked this down. This seems to only be the 
case when
I have sccache enabled. Disabling it gives me nice quick incremental 
builds

again.


What's your sccache version? I think you may be hitting the following 
sccache bug which has been fixed already:

https://github.com/mozilla/sccache/issues/436

I'm using sccache 0.2.9 and I don't see rust code rebuilding constantly.

Markus
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform






___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: Must we rebuild all our rust code constantly?

2019-08-23 Thread Eric Rahm
A fix for this (https://bugzilla.mozilla.org/show_bug.cgi?id=1576030)
should be on m-c shortly, no need to tweak mozconfigs.

-e

On Fri, Aug 23, 2019 at 9:45 AM Nicholas Alexander 
wrote:

> On Fri, Aug 23, 2019 at 2:11 AM ishikawa  wrote:
>
> > On 2019/08/23 11:00, Mike Hommey wrote:
> > >
> > > In the meanwhile, as discussed on irc, it seems fair to disable
> > > CARGO_INCREMENTAL when building with sccache.
> >
> > So how can we disable this?
> > Setting environment variable as in the following?
> >
> > export CARGO_INCREMENTAL=
> >
>
> `ac_add_options --disable-cargo-incremental` in your mozconfig should do
> it.  See:
>
>
> https://searchfox.org/mozilla-central/rev/597a69c70a5cce6f42f159eb54ad1ef6745f5432/build/moz.configure/toolchain.configure#1865
>
> Nick
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: Must we rebuild all our rust code constantly?

2019-08-23 Thread Nicholas Alexander
On Fri, Aug 23, 2019 at 2:11 AM ishikawa  wrote:

> On 2019/08/23 11:00, Mike Hommey wrote:
> >
> > In the meanwhile, as discussed on irc, it seems fair to disable
> > CARGO_INCREMENTAL when building with sccache.
>
> So how can we disable this?
> Setting environment variable as in the following?
>
> export CARGO_INCREMENTAL=
>

`ac_add_options --disable-cargo-incremental` in your mozconfig should do
it.  See:

https://searchfox.org/mozilla-central/rev/597a69c70a5cce6f42f159eb54ad1ef6745f5432/build/moz.configure/toolchain.configure#1865

Nick
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: Must we rebuild all our rust code constantly?

2019-08-23 Thread ishikawa
On 2019/08/23 11:00, Mike Hommey wrote:
> 
> In the meanwhile, as discussed on irc, it seems fair to disable
> CARGO_INCREMENTAL when building with sccache.

So how can we disable this?
Setting environment variable as in the following?

export CARGO_INCREMENTAL=

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: Must we rebuild all our rust code constantly?

2019-08-23 Thread Simon Sapin

On 23/08/2019 04:00, Mike Hommey wrote:

I had a similar experience on a 36-core machine earlier today. It seems
incremental is not working properly for the style crate: incremental is
supposed to have the same effect as sccache, but it still takes a long
time to compile style. I'm told mw is going to look into this in the
coming weeks.


I don’t know how much it has evolved since, but the initial versions of 
incremental compilation in rustc only cached code generation in LLVM. 
Type checking, trait resolution, etc. was still done every time rustc 
was invoked for a crate.


--
Simon Sapin
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: Must we rebuild all our rust code constantly?

2019-08-22 Thread Mike Hommey
On Thu, Aug 22, 2019 at 06:48:03PM -0700, Chris M. wrote:
> On Mon, Aug 19, 2019 at 10:32 PM Kris Maglione 
> wrote:
> 
> > On Tue, Aug 20, 2019 at 02:23:06PM +0900, ISHIKAWA,chiaki wrote:
> > >On 2019/08/20 9:11, Dave Townsend wrote:
> > >>Thanks to a tip I've tracked this down. This seems to only be the case
> > when
> > >>I have sccache enabled. Disabling it gives me nice quick incremental
> > builds
> > >>again. Of course that isn't an ideal solution but it will do for now.
> > >>
> > >>On Mon, Aug 19, 2019 at 1:55 PM Dave Townsend 
> > wrote:
> > >>
> > >>>For a couple of weeks now I've seen that any attempt to build Firefox,
> > >>>even incremental builds seem to rebuild an awful lot of rust code. I
> > found
> > >>>this in the source which seems to suggest why:
> > >>>
> > https://searchfox.org/mozilla-central/source/config/makefiles/rust.mk#238.
> > >>>But, this means that now an incremental build with a couple of code
> > change
> > >>>that have no impact on rust is taking upwards of 4 minutes to complete
> > in
> > >>>comparison to around 40 seconds, and the log file is full of cargo
> > output.
> > >>>I've heard similar comments from other developers.
> > >>>
> > >>>This is a pretty big increase in the time to compile and test and is
> > >>>really slowing down my work. Is there any way we can avoid this?
> > >>>
> > >>
> > >
> > >
> > >I am using linux for local development and noticed something similar.
> > >
> > >So I should disable sccache (!?).
> >
> > For what it's worth, Rust is now configured to use incremental
> > compilation, which has its own cache which isn't cleared after
> > clobber, so sccache isn't actually needed anymore. Ordinary
> > ccache should be sufficient.
> >
> sccache will still cache dependencies from crates.io. From a quick check
> building the rust in the tree on my laptop, a full cache hit for rust with
> sccache produces a significant speedup (2x) versus incremental alone.

I had a similar experience on a 36-core machine earlier today. It seems
incremental is not working properly for the style crate: incremental is
supposed to have the same effect as sccache, but it still takes a long
time to compile style. I'm told mw is going to look into this in the
coming weeks.

In the meanwhile, as discussed on irc, it seems fair to disable
CARGO_INCREMENTAL when building with sccache.

Mike
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: Must we rebuild all our rust code constantly?

2019-08-22 Thread Chris M.
On Mon, Aug 19, 2019 at 10:32 PM Kris Maglione 
wrote:

> On Tue, Aug 20, 2019 at 02:23:06PM +0900, ISHIKAWA,chiaki wrote:
> >On 2019/08/20 9:11, Dave Townsend wrote:
> >>Thanks to a tip I've tracked this down. This seems to only be the case
> when
> >>I have sccache enabled. Disabling it gives me nice quick incremental
> builds
> >>again. Of course that isn't an ideal solution but it will do for now.
> >>
> >>On Mon, Aug 19, 2019 at 1:55 PM Dave Townsend 
> wrote:
> >>
> >>>For a couple of weeks now I've seen that any attempt to build Firefox,
> >>>even incremental builds seem to rebuild an awful lot of rust code. I
> found
> >>>this in the source which seems to suggest why:
> >>>
> https://searchfox.org/mozilla-central/source/config/makefiles/rust.mk#238.
> >>>But, this means that now an incremental build with a couple of code
> change
> >>>that have no impact on rust is taking upwards of 4 minutes to complete
> in
> >>>comparison to around 40 seconds, and the log file is full of cargo
> output.
> >>>I've heard similar comments from other developers.
> >>>
> >>>This is a pretty big increase in the time to compile and test and is
> >>>really slowing down my work. Is there any way we can avoid this?
> >>>
> >>
> >
> >
> >I am using linux for local development and noticed something similar.
> >
> >So I should disable sccache (!?).
>
> For what it's worth, Rust is now configured to use incremental
> compilation, which has its own cache which isn't cleared after
> clobber, so sccache isn't actually needed anymore. Ordinary
> ccache should be sufficient.
>
sccache will still cache dependencies from crates.io. From a quick check
building the rust in the tree on my laptop, a full cache hit for rust with
sccache produces a significant speedup (2x) versus incremental alone.

> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: Must we rebuild all our rust code constantly?

2019-08-22 Thread Chris M.
On Tue, Aug 20, 2019 at 12:33 PM Markus Stange  wrote:

> On 2019-08-19 8:11 p.m., Dave Townsend wrote:
> > Thanks to a tip I've tracked this down. This seems to only be the case
> when
> > I have sccache enabled. Disabling it gives me nice quick incremental
> builds
> > again.
>
> What's your sccache version? I think you may be hitting the following
> sccache bug which has been fixed already:
> https://github.com/mozilla/sccache/issues/436
>
> I'm using sccache 0.2.9 and I don't see rust code rebuilding constantly.
>
I've never been able to reproduce this issue, and that would explain why.

Can someone who's experienced this confirm upgrading sccache fixes it?

>
> Markus
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: Must we rebuild all our rust code constantly?

2019-08-22 Thread Simon Sapin

On 22/08/2019 03:42, ISHIKAWA, Chiaki wrote:

At the same time, I have a feeling that the debug symbol that rustc
generates may be a tad bigger than I would like it to be, but I need to
investigate more about this.


Sounds like this known bug of rustc:
https://github.com/rust-lang/rust/issues/56068

--
Simon Sapin
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: Must we rebuild all our rust code constantly?

2019-08-21 Thread ISHIKAWA,chiaki
Well, I have a problem now after trying to update sccache just in case I 
need a new version in the future.


I did the following:

cargo install --force sccache

(I was not so sure of what the proper update procedure of already 
installed package. sccache 2.0.8-alpha-something was already installed.)


Then when I try to compile TB using the command script which worked for 
more than a few years suddenly printed the following and failed. I think 
I have messed up my cargo metadata somehow by the above command.


I have run |mach bootstrap| to see if it fixes the issue. It did not.

ERROR: Couldn't execute `cargo metadata` with manifest 
"/NREF-COMM-CENTRAL/mozilla/toolkit/library/rust/Cargo.toml": 
Metadata(Output { status: ExitStatus(ExitStatus(25856)), stdout: "", 
stderr: "error: failed to select a version for the requirement `libc = 
\"= 0.2.62\"`\n  candidate versions found which didn\'t match: 0.2.60\n  
location searched: directory source 
`/NREF-COMM-CENTRAL/mozilla/third_party/rust` (which is replacing 
registry `https://github.com/rust-lang/crates.io-index`)\nrequired by 
package `mozjs_sys v0.0.0 (/NREF-COMM-CENTRAL/mozilla/js/src)`\nperhaps 
a crate was updated and forgotten to be re-vendored?\n" })
ERROR: Couldn't generate bindings for 
/NREF-COMM-CENTRAL/mozilla/toolkit/library/rust.

make[4]: *** [backend.mk:72: .deps/ServoStyleConsts.h.stub] Error 1
make[3]: *** [/NREF-COMM-CENTRAL/mozilla/config/recurse.mk:101: 
layout/style/export] Error 2

make[3]: *** Waiting for unfinished jobs
ERROR: Couldn't execute `cargo metadata` with manifest 
"/NREF-COMM-CENTRAL/mozilla/toolkit/library/rust/Cargo.toml": 
Metadata(Output { status: ExitStatus(ExitStatus(25856)), stdout: "", 
stderr: "    Blocking waiting for file lock on package cache 
lock\nerror: failed to select a version for the requirement `libc = \"= 
0.2.62\"`\n  candidate versions found which didn\'t match: 0.2.60\n  
location searched: directory source 
`/NREF-COMM-CENTRAL/mozilla/third_party/rust` (which is replacing 
registry `https://github.com/rust-lang/crates.io-index`)\nrequired by 
package `mozjs_sys v0.0.0 (/NREF-COMM-CENTRAL/mozilla/js/src)`\nperhaps 
a crate was updated and forgotten to be re-vendored?\n" })
ERROR: Couldn't generate bindings for 
/NREF-COMM-CENTRAL/mozilla/toolkit/library/rust.

make[4]: *** [backend.mk:11: .deps/webrender_ffi_generated.h.stub] Error 1
make[3]: *** [/NREF-COMM-CENTRAL/mozilla/config/recurse.mk:101: 
gfx/webrender_bindings/export] Error 2


Does anyone have a clue how to restore order here?

TIA

Chiaki


On 2019/08/21 4:33, Markus Stange wrote:

On 2019-08-19 8:11 p.m., Dave Townsend wrote:
Thanks to a tip I've tracked this down. This seems to only be the 
case when
I have sccache enabled. Disabling it gives me nice quick incremental 
builds

again.


What's your sccache version? I think you may be hitting the following 
sccache bug which has been fixed already:

https://github.com/mozilla/sccache/issues/436

I'm using sccache 0.2.9 and I don't see rust code rebuilding constantly.

Markus
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform




___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: Must we rebuild all our rust code constantly?

2019-08-21 Thread ISHIKAWA, Chiaki

On 2019/08/21 3:52, Eric Rahm wrote:

mach clobber --full


Thank you for the tips.

I will try this.

At the same time, I have a feeling that the debug symbol that rustc 
generates may be a tad bigger than I would like it to be, but I need to 
investigate more about this.


Chiaki
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: Must we rebuild all our rust code constantly?

2019-08-20 Thread Markus Stange

On 2019-08-19 8:11 p.m., Dave Townsend wrote:

Thanks to a tip I've tracked this down. This seems to only be the case when
I have sccache enabled. Disabling it gives me nice quick incremental builds
again.


What's your sccache version? I think you may be hitting the following 
sccache bug which has been fixed already:

https://github.com/mozilla/sccache/issues/436

I'm using sccache 0.2.9 and I don't see rust code rebuilding constantly.

Markus
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: Must we rebuild all our rust code constantly?

2019-08-20 Thread Eric Rahm
You can use |mach clobber --full| to remove the incremental cache. It might
be worth filing a bug for the large size, my best guess is that we're
keeping around items from old compiler versions. It's possible we could do
a full clobber of the IC when we detect a new compiler version, but that
might not be a great experience for folks testing on different rust
versions.

-e

On Tue, Aug 20, 2019 at 11:02 AM ISHIKAWA,chiaki 
wrote:

> On 2019/08/20 14:32, Kris Maglione wrote:
> > On Tue, Aug 20, 2019 at 02:23:06PM +0900, ISHIKAWA,chiaki wrote:
> >> On 2019/08/20 9:11, Dave Townsend wrote:
> >>> Thanks to a tip I've tracked this down. This seems to only be the
> >>> case when
> >>> I have sccache enabled. Disabling it gives me nice quick incremental
> >>> builds
> >>> again. Of course that isn't an ideal solution but it will do for now.
> >>>
> >>> On Mon, Aug 19, 2019 at 1:55 PM Dave Townsend
> >>>  wrote:
> >>>
>  For a couple of weeks now I've seen that any attempt to build Firefox,
>  even incremental builds seem to rebuild an awful lot of rust code.
>  I found
>  this in the source which seems to suggest why:
> 
> https://searchfox.org/mozilla-central/source/config/makefiles/rust.mk#238.
>
> 
>  But, this means that now an incremental build with a couple of code
>  change
>  that have no impact on rust is taking upwards of 4 minutes to
>  complete in
>  comparison to around 40 seconds, and the log file is full of cargo
>  output.
>  I've heard similar comments from other developers.
> 
>  This is a pretty big increase in the time to compile and test and is
>  really slowing down my work. Is there any way we can avoid this?
> 
> >>> 
> >>
> >>
> >> I am using linux for local development and noticed something similar.
> >>
> >> So I should disable sccache (!?).
> >
> > For what it's worth, Rust is now configured to use incremental
> > compilation, which has its own cache which isn't cleared after
> > clobber, so sccache isn't actually needed anymore. Ordinary ccache
> > should be sufficient.
> >
> >
>
> I was not sure where this incremental compilation cache is located.
> Sure enough on my computer it seems to live under
> MOZ_OBJ/x86_64-unknown-linux-gnu/debug/incremental
>
> I have a serious question: does this incremental cache have a control
> mechanism to purge old entries?
>
> I have been hit with file system overflow errors quite frequently since
> early summer and I figure
> this is due to the bloated
> MOZ_OBJ/x86_64-unknown-linux-gnu/debug/incremental directory.
>
> It is now 26.6 GB with 67840 items according baobab file system usage
> monitor.
>
> 26.6 GB for this incremental cache alone is way too much number that was
> suggested by mozilla development document before.
> In comparsion, ccache I use is usable with max size of 5GB
> as set by
> CCACHE_SIZE=5.0GB
> in my script.
>
> On my local PC, I see that the file directory size (with the sizes of
> subdirectories added to it) is as follows:
>
> MOZ_OBJ is 45.8 GB large. (as of now: I had to stop TB compilation due
> to the 50 GB smallish partition overflow where this directory is located.)
> MOZ_OBJ/x86_64-unknown-linux-gnu/ is 33.9 GB large .
> MOZ_OBJ/x86_64-unknown-linux-gnu/debug/  is 33.9 GB large.
>  I think /debug/ is because I create full debug version of TB locally.
> MOZ_OBJ/MOZ_OBJ/x86_64-unknown-linux-gnu/debug/incremental is 26.5 GB
> large.
>
> I need a way to shrink this incremental cache reasonably.
> It seems this incremental directory grows without limit. Or, it does not
> trim old unusable cache files automatically.
>
> TIA
>
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: Must we rebuild all our rust code constantly?

2019-08-20 Thread ISHIKAWA,chiaki

On 2019/08/20 14:32, Kris Maglione wrote:

On Tue, Aug 20, 2019 at 02:23:06PM +0900, ISHIKAWA,chiaki wrote:

On 2019/08/20 9:11, Dave Townsend wrote:
Thanks to a tip I've tracked this down. This seems to only be the 
case when
I have sccache enabled. Disabling it gives me nice quick incremental 
builds

again. Of course that isn't an ideal solution but it will do for now.

On Mon, Aug 19, 2019 at 1:55 PM Dave Townsend 
 wrote:



For a couple of weeks now I've seen that any attempt to build Firefox,
even incremental builds seem to rebuild an awful lot of rust code. 
I found

this in the source which seems to suggest why:
https://searchfox.org/mozilla-central/source/config/makefiles/rust.mk#238. 

But, this means that now an incremental build with a couple of code 
change
that have no impact on rust is taking upwards of 4 minutes to 
complete in
comparison to around 40 seconds, and the log file is full of cargo 
output.

I've heard similar comments from other developers.

This is a pretty big increase in the time to compile and test and is
really slowing down my work. Is there any way we can avoid this?






I am using linux for local development and noticed something similar.

So I should disable sccache (!?).


For what it's worth, Rust is now configured to use incremental 
compilation, which has its own cache which isn't cleared after 
clobber, so sccache isn't actually needed anymore. Ordinary ccache 
should be sufficient.





I was not sure where this incremental compilation cache is located.
Sure enough on my computer it seems to live under
MOZ_OBJ/x86_64-unknown-linux-gnu/debug/incremental

I have a serious question: does this incremental cache have a control 
mechanism to purge old entries?


I have been hit with file system overflow errors quite frequently since 
early summer and I figure
this is due to the bloated 
MOZ_OBJ/x86_64-unknown-linux-gnu/debug/incremental directory.


It is now 26.6 GB with 67840 items according baobab file system usage 
monitor.


26.6 GB for this incremental cache alone is way too much number that was 
suggested by mozilla development document before.

In comparsion, ccache I use is usable with max size of 5GB
as set by
CCACHE_SIZE=5.0GB
in my script.

On my local PC, I see that the file directory size (with the sizes of 
subdirectories added to it) is as follows:


MOZ_OBJ is 45.8 GB large. (as of now: I had to stop TB compilation due 
to the 50 GB smallish partition overflow where this directory is located.)

MOZ_OBJ/x86_64-unknown-linux-gnu/ is 33.9 GB large .
MOZ_OBJ/x86_64-unknown-linux-gnu/debug/  is 33.9 GB large.
    I think /debug/ is because I create full debug version of TB locally.
MOZ_OBJ/MOZ_OBJ/x86_64-unknown-linux-gnu/debug/incremental is 26.5 GB large.

I need a way to shrink this incremental cache reasonably.
It seems this incremental directory grows without limit. Or, it does not 
trim old unusable cache files automatically.


TIA


___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: Must we rebuild all our rust code constantly?

2019-08-19 Thread Jörg Knobloch

Hi all,

I'm not sure whether Thunderbirds are different, but the issue is that 
(on Windows) xul.dll is constantly relinked although there haven't been 
any changes to C++ or RS files. Log below.


Oh, you can also see some ugly hunspell warnings that have crept in a 
while ago.


Jörg.

$ mach build
 0:02.41 Clobber not needed.
 0:02.42 Adding make options from c:\mozilla-source\comm-central\.mozconfig
    MOZ_MAKE_FLAGS=-j16
MOZ_OBJDIR=c:/mozilla-source/comm-central/obj-x86_64-pc-mingw32
    OBJDIR=c:/mozilla-source/comm-central/obj-x86_64-pc-mingw32
    FOUND_MOZCONFIG=c:/mozilla-source/comm-central/.mozconfig
    export FOUND_MOZCONFIG
 0:02.45 c:\mozilla-build\bin\mozmake.exe -f client.mk 
MOZ_PARALLEL_BUILD=16 -s
 0:02.88 Elapsed: 0.00s; From dist/public: Kept 0 existing; 
Added/updated 0; Removed 0 files and 0 directories.
 0:02.89 Elapsed: 0.00s; From dist/private: Kept 0 existing; 
Added/updated 0; Removed 0 files and 0 directories.
 0:02.91 Elapsed: 0.03s; From dist/xpi-stage: Kept 58 existing; 
Added/updated 0; Removed 0 files and 0 directories.
 0:03.77 Elapsed: 0.88s; From _tests: Kept 0 existing; Added/updated 
1521; Removed 0 files and 0 directories.
 0:03.99 Elapsed: 1.09s; From dist/bin: Kept 0 existing; Added/updated 
2858; Removed 0 files and 0 directories.
 0:04.94 Elapsed: 2.05s; From dist/include: Kept 0 existing; 
Added/updated 6385; Removed 0 files and 0 directories.

 0:04.99 ./buildid.h.stub
 0:05.19 ./source-repo.h.stub
 0:06.02 build/application.ini.stub
 0:06.32 build/application.ini.h.stub
 0:07.03 build/clang-plugin/tests
 0:07.85 toolkit/library/rust/force-cargo-library-build
 0:07.85 js/src/rust/force-cargo-library-build
 0:07.85 js/src/frontend/binast/force-cargo-host-program-build
 0:07.86 testing/geckodriver/force-cargo-program-build
 0:08.05     Blocking waiting for file lock on 
package cache lock
 0:08.05  BlockingBlocking  
waiting for file lock on package cache lockwaiting for file lock on 
package cache lock
 0:09.83     Blocking waiting for file lock on 
package cache lock
 0:11.11     Blocking waiting for file lock on 
package cache lock
 0:12.26     Blocking waiting for file lock on 
package cache lock

 0:13.18 comm/mail/app
 0:13.23 toolkit/library/buildid.cpp.stub
 0:13.45     Blocking waiting for file lock on 
package cache lock
 0:13.45     Blocking waiting for file lock on 
package cache lock
 0:13.48     Blocking waiting for file lock on 
package cache lock
 0:13.48     Blocking waiting for file lock on 
package cache lock
 0:13.48     Blocking waiting for file lock on 
package cache lock
 0:13.50     Blocking waiting for file lock on 
build directory
 0:13.54     Finished dev [optimized + debuginfo] 
target(s) in 5.65s

 0:13.63 toolkit/library
 0:13.78     Finished dev [optimized + debuginfo] 
target(s) in 5.88s
 0:14.17     Finished dev [optimized + debuginfo] 
target(s) in 6.27s
 0:14.26     Finished dev [optimized + debuginfo] 
target(s) in 6.38s

 0:14.42 toolkit/library/build/xul.dll
 0:15.07 comm/mail/app/module.res
 0:15.09 Creating Resource file: module.res
 0:15.12 comm/mail/app/thunderbird.exe
 0:15.26 Embedding manifest from 
../../../../comm/mail/app/thunderbird.exe.manifest
 0:19.93 lld-link: warning: 
..\..\..\extensions\spellcheck\hunspell\glue\Unified_cpp_hunspell_glue0.obj: 
locally defined symbol imported: "public: __cdecl 
Hunspell::~Hunspell(void)" (??1Hunspell@@QEAA@XZ) (defined in 
..\..\..\extensions\spellcheck\hunspell\src\Unified_cpp_hunspell_src0.obj) 
[LNK4217]
 0:19.93 lld-link: warning: 
..\..\..\extensions\spellcheck\hunspell\glue\Unified_cpp_hunspell_glue0.obj: 
locally defined symbol imported: "public: __cdecl 
Hunspell::Hunspell(char const *, char const *, char const *)" 
(??0Hunspell@@QEAA@PEBD00@Z) (defined in 
..\..\..\extensions\spellcheck\hunspell\src\Unified_cpp_hunspell_src0.obj) 
[LNK4217]
 0:19.93 lld-link: warning: 
..\..\..\extensions\spellcheck\hunspell\glue\Unified_cpp_hunspell_glue0.obj: 
locally defined symbol imported: "public: class std::basic_stringstruct std::char_traits, class std::allocator> const & 
__cdecl Hunspell::get_dict_encoding(void) const" 
(?get_dict_encoding@Hunspell@@QEBAAEBV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@XZ) 
(defined in 
..\..\..\extensions\spellcheck\hunspell\src\Unified_cpp_hunspell_src0.obj) 
[LNK4217]
 0:19.93 lld-link: warning: 
..\..\..\extensions\spellcheck\hunspell\glue\Unified_cpp_hunspell_glue0.obj: 
locally defined symbol imported: "public: bool __cdecl 
Hunspell::spell(class std::basic_stringstd::char_traits, class std::allocator> const &, int *, 
class std::basic_string, class 
std::allocator> *)" 

Re: Must we rebuild all our rust code constantly?

2019-08-19 Thread Kris Maglione

On Tue, Aug 20, 2019 at 02:23:06PM +0900, ISHIKAWA,chiaki wrote:

On 2019/08/20 9:11, Dave Townsend wrote:

Thanks to a tip I've tracked this down. This seems to only be the case when
I have sccache enabled. Disabling it gives me nice quick incremental builds
again. Of course that isn't an ideal solution but it will do for now.

On Mon, Aug 19, 2019 at 1:55 PM Dave Townsend  wrote:


For a couple of weeks now I've seen that any attempt to build Firefox,
even incremental builds seem to rebuild an awful lot of rust code. I found
this in the source which seems to suggest why:
https://searchfox.org/mozilla-central/source/config/makefiles/rust.mk#238.
But, this means that now an incremental build with a couple of code change
that have no impact on rust is taking upwards of 4 minutes to complete in
comparison to around 40 seconds, and the log file is full of cargo output.
I've heard similar comments from other developers.

This is a pretty big increase in the time to compile and test and is
really slowing down my work. Is there any way we can avoid this?






I am using linux for local development and noticed something similar.

So I should disable sccache (!?).


For what it's worth, Rust is now configured to use incremental 
compilation, which has its own cache which isn't cleared after 
clobber, so sccache isn't actually needed anymore. Ordinary 
ccache should be sufficient.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: Must we rebuild all our rust code constantly?

2019-08-19 Thread ISHIKAWA,chiaki

On 2019/08/20 9:11, Dave Townsend wrote:

Thanks to a tip I've tracked this down. This seems to only be the case when
I have sccache enabled. Disabling it gives me nice quick incremental builds
again. Of course that isn't an ideal solution but it will do for now.

On Mon, Aug 19, 2019 at 1:55 PM Dave Townsend  wrote:


For a couple of weeks now I've seen that any attempt to build Firefox,
even incremental builds seem to rebuild an awful lot of rust code. I found
this in the source which seems to suggest why:
https://searchfox.org/mozilla-central/source/config/makefiles/rust.mk#238.
But, this means that now an incremental build with a couple of code change
that have no impact on rust is taking upwards of 4 minutes to complete in
comparison to around 40 seconds, and the log file is full of cargo output.
I've heard similar comments from other developers.

This is a pretty big increase in the time to compile and test and is
really slowing down my work. Is there any way we can avoid this?






I am using linux for local development and noticed something similar.

So I should disable sccache (!?).



___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform




___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: Must we rebuild all our rust code constantly?

2019-08-19 Thread Adam Gashlin
Is this https://bugzilla.mozilla.org/show_bug.cgi?id=1427313 ?

On Mon, Aug 19, 2019 at 5:27 PM Kris Maglione  wrote:

> This is apparently a known bug that no-one seems to be able to
> track down the cause of. It suddenly started happening to me one
> night for every build, even if I changed nothing. Then, just as
> suddenly, stopped happening after a couple of hours.
>
> On Mon, Aug 19, 2019 at 05:11:19PM -0700, Dave Townsend wrote:
> >Thanks to a tip I've tracked this down. This seems to only be the case
> when
> >I have sccache enabled. Disabling it gives me nice quick incremental
> builds
> >again. Of course that isn't an ideal solution but it will do for now.
> >
> >On Mon, Aug 19, 2019 at 1:55 PM Dave Townsend 
> wrote:
> >
> >> For a couple of weeks now I've seen that any attempt to build Firefox,
> >> even incremental builds seem to rebuild an awful lot of rust code. I
> found
> >> this in the source which seems to suggest why:
> >>
> https://searchfox.org/mozilla-central/source/config/makefiles/rust.mk#238.
> >> But, this means that now an incremental build with a couple of code
> change
> >> that have no impact on rust is taking upwards of 4 minutes to complete
> in
> >> comparison to around 40 seconds, and the log file is full of cargo
> output.
> >> I've heard similar comments from other developers.
> >>
> >> This is a pretty big increase in the time to compile and test and is
> >> really slowing down my work. Is there any way we can avoid this?
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: Must we rebuild all our rust code constantly?

2019-08-19 Thread Kris Maglione
This is apparently a known bug that no-one seems to be able to 
track down the cause of. It suddenly started happening to me one 
night for every build, even if I changed nothing. Then, just as 
suddenly, stopped happening after a couple of hours.


On Mon, Aug 19, 2019 at 05:11:19PM -0700, Dave Townsend wrote:

Thanks to a tip I've tracked this down. This seems to only be the case when
I have sccache enabled. Disabling it gives me nice quick incremental builds
again. Of course that isn't an ideal solution but it will do for now.

On Mon, Aug 19, 2019 at 1:55 PM Dave Townsend  wrote:


For a couple of weeks now I've seen that any attempt to build Firefox,
even incremental builds seem to rebuild an awful lot of rust code. I found
this in the source which seems to suggest why:
https://searchfox.org/mozilla-central/source/config/makefiles/rust.mk#238.
But, this means that now an incremental build with a couple of code change
that have no impact on rust is taking upwards of 4 minutes to complete in
comparison to around 40 seconds, and the log file is full of cargo output.
I've heard similar comments from other developers.

This is a pretty big increase in the time to compile and test and is
really slowing down my work. Is there any way we can avoid this?

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: Must we rebuild all our rust code constantly?

2019-08-19 Thread Dave Townsend
Thanks to a tip I've tracked this down. This seems to only be the case when
I have sccache enabled. Disabling it gives me nice quick incremental builds
again. Of course that isn't an ideal solution but it will do for now.

On Mon, Aug 19, 2019 at 1:55 PM Dave Townsend  wrote:

> For a couple of weeks now I've seen that any attempt to build Firefox,
> even incremental builds seem to rebuild an awful lot of rust code. I found
> this in the source which seems to suggest why:
> https://searchfox.org/mozilla-central/source/config/makefiles/rust.mk#238.
> But, this means that now an incremental build with a couple of code change
> that have no impact on rust is taking upwards of 4 minutes to complete in
> comparison to around 40 seconds, and the log file is full of cargo output.
> I've heard similar comments from other developers.
>
> This is a pretty big increase in the time to compile and test and is
> really slowing down my work. Is there any way we can avoid this?
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: Must we rebuild all our rust code constantly?

2019-08-19 Thread Simon Sapin

On 20/08/2019 00:20, Simon Sapin wrote:

It would be good to have steps to reproduce the issue. Then Cargo’s
debug logs should show what it thinks is out of date.


Running with a CARGO_LOG=cargo::core::compiler::fingerprint environment 
variable should show some relevant information.


--
Simon Sapin
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: Must we rebuild all our rust code constantly?

2019-08-19 Thread Simon Sapin

On 19/08/2019 22:55, Dave Townsend wrote:

For a couple of weeks now I've seen that any attempt to build Firefox, even
incremental builds seem to rebuild an awful lot of rust code. I found this
in the source which seems to suggest why:
https://searchfox.org/mozilla-central/source/config/makefiles/rust.mk#238.
But, this means that now an incremental build with a couple of code change
that have no impact on rust is taking upwards of 4 minutes to complete in
comparison to around 40 seconds, and the log file is full of cargo output.
I've heard similar comments from other developers.

This is a pretty big increase in the time to compile and test and is really
slowing down my work. Is there any way we can avoid this?


Running `cargo build` every time makes sense to me, for the reasons 
given in that makefile comment. However when everything is already up to 
date it should finish quickly (single-digit seconds or less) after 
printing something like:



Finished dev [unoptimized + debuginfo] target(s) in 1.24s


… and without printing any line that starts with "Compiling".


It would be good to have steps to reproduce the issue. Then Cargo’s 
debug logs should show what it thinks is out of date. (But I don’t 
remember at hand the exact invocation to obtain them, and they can be 
kinda hard to read.)


--
Simon Sapin
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform