Re: Must we rebuild all our rust code constantly?
On Friday, August 23, 2019 at 9:37:12 AM UTC+2, Simon Sapin wrote: > On 23/08/2019 04:00, Mike Hommey wrote: > > I had a similar experience on a 36-core machine earlier today. It seems > > incremental is not working properly for the style crate: incremental is > > supposed to have the same effect as sccache, but it still takes a long > > time to compile style. I'm told mw is going to look into this in the > > coming weeks. > > I don’t know how much it has evolved since, but the initial versions of > incremental compilation in rustc only cached code generation in LLVM. > Type checking, trait resolution, etc. was still done every time rustc > was invoked for a crate. > > -- > Simon Sapin With recent versions of `rustc`, incremental compilation caches pretty much everything after parsing and macro expansion. However, the style crate in particular has proven to be rather resistant to our incremental compilation approach so far. I will indeed spend quite a bit of time investigating this over the next few months. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Must we rebuild all our rust code constantly?
Here is my own answer to clean up the mess of version mismatches, etc.
I did the following.
1. I removed ~/.mozbuild and ~/.cargo directory and
re-ran |make bootstrap|.
(I am not sure if removing ~/.cargo directory was a good idea.)
2. I revered my local changes before I update the M-C and C-C (./comm).
3. Then I noticed mozilla/Cargo.lock has been modified when I tried to
reapply my local changes.
This file was presumably changed when I did my own sccache installation.
So I reverted the change to Cargo.lock, and now it seems that the
compilation is proceeding... (Not sure if the result is OK.)
4. When I did this, I noticed there is sccache 0.2.11-Alpha.0 under
~/.mozbuild/sccache directory.
Obviously, I was using my own installed older version of sccache which I
installed many months ago.
So I added the path to this sccache in my script for building TB and it
seems to be compiling TB.
I added the following in my mozconfig for TB, but I am not sure if this
is the right way to disable incremental compilation.
mk_add_options "export CARGO_INCREMENTAL=0"
TIA
On 2019/08/22 11:22, ISHIKAWA,chiaki wrote:
Well, I have a problem now after trying to update sccache just in case
I need a new version in the future.
I did the following:
cargo install --force sccache
(I was not so sure of what the proper update procedure of already
installed package. sccache 2.0.8-alpha-something was already installed.)
Then when I try to compile TB using the command script which worked
for more than a few years suddenly printed the following and failed. I
think I have messed up my cargo metadata somehow by the above command.
I have run |mach bootstrap| to see if it fixes the issue. It did not.
ERROR: Couldn't execute `cargo metadata` with manifest
"/NREF-COMM-CENTRAL/mozilla/toolkit/library/rust/Cargo.toml":
Metadata(Output { status: ExitStatus(ExitStatus(25856)), stdout: "",
stderr: "error: failed to select a version for the requirement `libc =
\"= 0.2.62\"`\n candidate versions found which didn\'t match:
0.2.60\n location searched: directory source
`/NREF-COMM-CENTRAL/mozilla/third_party/rust` (which is replacing
registry `https://github.com/rust-lang/crates.io-index`)\nrequired by
package `mozjs_sys v0.0.0
(/NREF-COMM-CENTRAL/mozilla/js/src)`\nperhaps a crate was updated and
forgotten to be re-vendored?\n" })
ERROR: Couldn't generate bindings for
/NREF-COMM-CENTRAL/mozilla/toolkit/library/rust.
make[4]: *** [backend.mk:72: .deps/ServoStyleConsts.h.stub] Error 1
make[3]: *** [/NREF-COMM-CENTRAL/mozilla/config/recurse.mk:101:
layout/style/export] Error 2
make[3]: *** Waiting for unfinished jobs
ERROR: Couldn't execute `cargo metadata` with manifest
"/NREF-COMM-CENTRAL/mozilla/toolkit/library/rust/Cargo.toml":
Metadata(Output { status: ExitStatus(ExitStatus(25856)), stdout: "",
stderr: " Blocking waiting for file lock on package cache
lock\nerror: failed to select a version for the requirement `libc =
\"= 0.2.62\"`\n candidate versions found which didn\'t match:
0.2.60\n location searched: directory source
`/NREF-COMM-CENTRAL/mozilla/third_party/rust` (which is replacing
registry `https://github.com/rust-lang/crates.io-index`)\nrequired by
package `mozjs_sys v0.0.0
(/NREF-COMM-CENTRAL/mozilla/js/src)`\nperhaps a crate was updated and
forgotten to be re-vendored?\n" })
ERROR: Couldn't generate bindings for
/NREF-COMM-CENTRAL/mozilla/toolkit/library/rust.
make[4]: *** [backend.mk:11: .deps/webrender_ffi_generated.h.stub]
Error 1
make[3]: *** [/NREF-COMM-CENTRAL/mozilla/config/recurse.mk:101:
gfx/webrender_bindings/export] Error 2
Does anyone have a clue how to restore order here?
TIA
Chiaki
On 2019/08/21 4:33, Markus Stange wrote:
On 2019-08-19 8:11 p.m., Dave Townsend wrote:
Thanks to a tip I've tracked this down. This seems to only be the
case when
I have sccache enabled. Disabling it gives me nice quick incremental
builds
again.
What's your sccache version? I think you may be hitting the following
sccache bug which has been fixed already:
https://github.com/mozilla/sccache/issues/436
I'm using sccache 0.2.9 and I don't see rust code rebuilding constantly.
Markus
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
Re: Must we rebuild all our rust code constantly?
A fix for this (https://bugzilla.mozilla.org/show_bug.cgi?id=1576030) should be on m-c shortly, no need to tweak mozconfigs. -e On Fri, Aug 23, 2019 at 9:45 AM Nicholas Alexander wrote: > On Fri, Aug 23, 2019 at 2:11 AM ishikawa wrote: > > > On 2019/08/23 11:00, Mike Hommey wrote: > > > > > > In the meanwhile, as discussed on irc, it seems fair to disable > > > CARGO_INCREMENTAL when building with sccache. > > > > So how can we disable this? > > Setting environment variable as in the following? > > > > export CARGO_INCREMENTAL= > > > > `ac_add_options --disable-cargo-incremental` in your mozconfig should do > it. See: > > > https://searchfox.org/mozilla-central/rev/597a69c70a5cce6f42f159eb54ad1ef6745f5432/build/moz.configure/toolchain.configure#1865 > > Nick > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Must we rebuild all our rust code constantly?
On Fri, Aug 23, 2019 at 2:11 AM ishikawa wrote: > On 2019/08/23 11:00, Mike Hommey wrote: > > > > In the meanwhile, as discussed on irc, it seems fair to disable > > CARGO_INCREMENTAL when building with sccache. > > So how can we disable this? > Setting environment variable as in the following? > > export CARGO_INCREMENTAL= > `ac_add_options --disable-cargo-incremental` in your mozconfig should do it. See: https://searchfox.org/mozilla-central/rev/597a69c70a5cce6f42f159eb54ad1ef6745f5432/build/moz.configure/toolchain.configure#1865 Nick ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Must we rebuild all our rust code constantly?
On 2019/08/23 11:00, Mike Hommey wrote: > > In the meanwhile, as discussed on irc, it seems fair to disable > CARGO_INCREMENTAL when building with sccache. So how can we disable this? Setting environment variable as in the following? export CARGO_INCREMENTAL= ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Must we rebuild all our rust code constantly?
On 23/08/2019 04:00, Mike Hommey wrote: I had a similar experience on a 36-core machine earlier today. It seems incremental is not working properly for the style crate: incremental is supposed to have the same effect as sccache, but it still takes a long time to compile style. I'm told mw is going to look into this in the coming weeks. I don’t know how much it has evolved since, but the initial versions of incremental compilation in rustc only cached code generation in LLVM. Type checking, trait resolution, etc. was still done every time rustc was invoked for a crate. -- Simon Sapin ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Must we rebuild all our rust code constantly?
On Thu, Aug 22, 2019 at 06:48:03PM -0700, Chris M. wrote: > On Mon, Aug 19, 2019 at 10:32 PM Kris Maglione > wrote: > > > On Tue, Aug 20, 2019 at 02:23:06PM +0900, ISHIKAWA,chiaki wrote: > > >On 2019/08/20 9:11, Dave Townsend wrote: > > >>Thanks to a tip I've tracked this down. This seems to only be the case > > when > > >>I have sccache enabled. Disabling it gives me nice quick incremental > > builds > > >>again. Of course that isn't an ideal solution but it will do for now. > > >> > > >>On Mon, Aug 19, 2019 at 1:55 PM Dave Townsend > > wrote: > > >> > > >>>For a couple of weeks now I've seen that any attempt to build Firefox, > > >>>even incremental builds seem to rebuild an awful lot of rust code. I > > found > > >>>this in the source which seems to suggest why: > > >>> > > https://searchfox.org/mozilla-central/source/config/makefiles/rust.mk#238. > > >>>But, this means that now an incremental build with a couple of code > > change > > >>>that have no impact on rust is taking upwards of 4 minutes to complete > > in > > >>>comparison to around 40 seconds, and the log file is full of cargo > > output. > > >>>I've heard similar comments from other developers. > > >>> > > >>>This is a pretty big increase in the time to compile and test and is > > >>>really slowing down my work. Is there any way we can avoid this? > > >>> > > >> > > > > > > > > >I am using linux for local development and noticed something similar. > > > > > >So I should disable sccache (!?). > > > > For what it's worth, Rust is now configured to use incremental > > compilation, which has its own cache which isn't cleared after > > clobber, so sccache isn't actually needed anymore. Ordinary > > ccache should be sufficient. > > > sccache will still cache dependencies from crates.io. From a quick check > building the rust in the tree on my laptop, a full cache hit for rust with > sccache produces a significant speedup (2x) versus incremental alone. I had a similar experience on a 36-core machine earlier today. It seems incremental is not working properly for the style crate: incremental is supposed to have the same effect as sccache, but it still takes a long time to compile style. I'm told mw is going to look into this in the coming weeks. In the meanwhile, as discussed on irc, it seems fair to disable CARGO_INCREMENTAL when building with sccache. Mike ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Must we rebuild all our rust code constantly?
On Mon, Aug 19, 2019 at 10:32 PM Kris Maglione wrote: > On Tue, Aug 20, 2019 at 02:23:06PM +0900, ISHIKAWA,chiaki wrote: > >On 2019/08/20 9:11, Dave Townsend wrote: > >>Thanks to a tip I've tracked this down. This seems to only be the case > when > >>I have sccache enabled. Disabling it gives me nice quick incremental > builds > >>again. Of course that isn't an ideal solution but it will do for now. > >> > >>On Mon, Aug 19, 2019 at 1:55 PM Dave Townsend > wrote: > >> > >>>For a couple of weeks now I've seen that any attempt to build Firefox, > >>>even incremental builds seem to rebuild an awful lot of rust code. I > found > >>>this in the source which seems to suggest why: > >>> > https://searchfox.org/mozilla-central/source/config/makefiles/rust.mk#238. > >>>But, this means that now an incremental build with a couple of code > change > >>>that have no impact on rust is taking upwards of 4 minutes to complete > in > >>>comparison to around 40 seconds, and the log file is full of cargo > output. > >>>I've heard similar comments from other developers. > >>> > >>>This is a pretty big increase in the time to compile and test and is > >>>really slowing down my work. Is there any way we can avoid this? > >>> > >> > > > > > >I am using linux for local development and noticed something similar. > > > >So I should disable sccache (!?). > > For what it's worth, Rust is now configured to use incremental > compilation, which has its own cache which isn't cleared after > clobber, so sccache isn't actually needed anymore. Ordinary > ccache should be sufficient. > sccache will still cache dependencies from crates.io. From a quick check building the rust in the tree on my laptop, a full cache hit for rust with sccache produces a significant speedup (2x) versus incremental alone. > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Must we rebuild all our rust code constantly?
On Tue, Aug 20, 2019 at 12:33 PM Markus Stange wrote: > On 2019-08-19 8:11 p.m., Dave Townsend wrote: > > Thanks to a tip I've tracked this down. This seems to only be the case > when > > I have sccache enabled. Disabling it gives me nice quick incremental > builds > > again. > > What's your sccache version? I think you may be hitting the following > sccache bug which has been fixed already: > https://github.com/mozilla/sccache/issues/436 > > I'm using sccache 0.2.9 and I don't see rust code rebuilding constantly. > I've never been able to reproduce this issue, and that would explain why. Can someone who's experienced this confirm upgrading sccache fixes it? > > Markus > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Must we rebuild all our rust code constantly?
On 22/08/2019 03:42, ISHIKAWA, Chiaki wrote: At the same time, I have a feeling that the debug symbol that rustc generates may be a tad bigger than I would like it to be, but I need to investigate more about this. Sounds like this known bug of rustc: https://github.com/rust-lang/rust/issues/56068 -- Simon Sapin ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Must we rebuild all our rust code constantly?
Well, I have a problem now after trying to update sccache just in case I
need a new version in the future.
I did the following:
cargo install --force sccache
(I was not so sure of what the proper update procedure of already
installed package. sccache 2.0.8-alpha-something was already installed.)
Then when I try to compile TB using the command script which worked for
more than a few years suddenly printed the following and failed. I think
I have messed up my cargo metadata somehow by the above command.
I have run |mach bootstrap| to see if it fixes the issue. It did not.
ERROR: Couldn't execute `cargo metadata` with manifest
"/NREF-COMM-CENTRAL/mozilla/toolkit/library/rust/Cargo.toml":
Metadata(Output { status: ExitStatus(ExitStatus(25856)), stdout: "",
stderr: "error: failed to select a version for the requirement `libc =
\"= 0.2.62\"`\n candidate versions found which didn\'t match: 0.2.60\n
location searched: directory source
`/NREF-COMM-CENTRAL/mozilla/third_party/rust` (which is replacing
registry `https://github.com/rust-lang/crates.io-index`)\nrequired by
package `mozjs_sys v0.0.0 (/NREF-COMM-CENTRAL/mozilla/js/src)`\nperhaps
a crate was updated and forgotten to be re-vendored?\n" })
ERROR: Couldn't generate bindings for
/NREF-COMM-CENTRAL/mozilla/toolkit/library/rust.
make[4]: *** [backend.mk:72: .deps/ServoStyleConsts.h.stub] Error 1
make[3]: *** [/NREF-COMM-CENTRAL/mozilla/config/recurse.mk:101:
layout/style/export] Error 2
make[3]: *** Waiting for unfinished jobs
ERROR: Couldn't execute `cargo metadata` with manifest
"/NREF-COMM-CENTRAL/mozilla/toolkit/library/rust/Cargo.toml":
Metadata(Output { status: ExitStatus(ExitStatus(25856)), stdout: "",
stderr: " Blocking waiting for file lock on package cache
lock\nerror: failed to select a version for the requirement `libc = \"=
0.2.62\"`\n candidate versions found which didn\'t match: 0.2.60\n
location searched: directory source
`/NREF-COMM-CENTRAL/mozilla/third_party/rust` (which is replacing
registry `https://github.com/rust-lang/crates.io-index`)\nrequired by
package `mozjs_sys v0.0.0 (/NREF-COMM-CENTRAL/mozilla/js/src)`\nperhaps
a crate was updated and forgotten to be re-vendored?\n" })
ERROR: Couldn't generate bindings for
/NREF-COMM-CENTRAL/mozilla/toolkit/library/rust.
make[4]: *** [backend.mk:11: .deps/webrender_ffi_generated.h.stub] Error 1
make[3]: *** [/NREF-COMM-CENTRAL/mozilla/config/recurse.mk:101:
gfx/webrender_bindings/export] Error 2
Does anyone have a clue how to restore order here?
TIA
Chiaki
On 2019/08/21 4:33, Markus Stange wrote:
On 2019-08-19 8:11 p.m., Dave Townsend wrote:
Thanks to a tip I've tracked this down. This seems to only be the
case when
I have sccache enabled. Disabling it gives me nice quick incremental
builds
again.
What's your sccache version? I think you may be hitting the following
sccache bug which has been fixed already:
https://github.com/mozilla/sccache/issues/436
I'm using sccache 0.2.9 and I don't see rust code rebuilding constantly.
Markus
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
Re: Must we rebuild all our rust code constantly?
On 2019/08/21 3:52, Eric Rahm wrote: mach clobber --full Thank you for the tips. I will try this. At the same time, I have a feeling that the debug symbol that rustc generates may be a tad bigger than I would like it to be, but I need to investigate more about this. Chiaki ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Must we rebuild all our rust code constantly?
On 2019-08-19 8:11 p.m., Dave Townsend wrote: Thanks to a tip I've tracked this down. This seems to only be the case when I have sccache enabled. Disabling it gives me nice quick incremental builds again. What's your sccache version? I think you may be hitting the following sccache bug which has been fixed already: https://github.com/mozilla/sccache/issues/436 I'm using sccache 0.2.9 and I don't see rust code rebuilding constantly. Markus ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Must we rebuild all our rust code constantly?
You can use |mach clobber --full| to remove the incremental cache. It might be worth filing a bug for the large size, my best guess is that we're keeping around items from old compiler versions. It's possible we could do a full clobber of the IC when we detect a new compiler version, but that might not be a great experience for folks testing on different rust versions. -e On Tue, Aug 20, 2019 at 11:02 AM ISHIKAWA,chiaki wrote: > On 2019/08/20 14:32, Kris Maglione wrote: > > On Tue, Aug 20, 2019 at 02:23:06PM +0900, ISHIKAWA,chiaki wrote: > >> On 2019/08/20 9:11, Dave Townsend wrote: > >>> Thanks to a tip I've tracked this down. This seems to only be the > >>> case when > >>> I have sccache enabled. Disabling it gives me nice quick incremental > >>> builds > >>> again. Of course that isn't an ideal solution but it will do for now. > >>> > >>> On Mon, Aug 19, 2019 at 1:55 PM Dave Townsend > >>> wrote: > >>> > For a couple of weeks now I've seen that any attempt to build Firefox, > even incremental builds seem to rebuild an awful lot of rust code. > I found > this in the source which seems to suggest why: > > https://searchfox.org/mozilla-central/source/config/makefiles/rust.mk#238. > > > But, this means that now an incremental build with a couple of code > change > that have no impact on rust is taking upwards of 4 minutes to > complete in > comparison to around 40 seconds, and the log file is full of cargo > output. > I've heard similar comments from other developers. > > This is a pretty big increase in the time to compile and test and is > really slowing down my work. Is there any way we can avoid this? > > >>> > >> > >> > >> I am using linux for local development and noticed something similar. > >> > >> So I should disable sccache (!?). > > > > For what it's worth, Rust is now configured to use incremental > > compilation, which has its own cache which isn't cleared after > > clobber, so sccache isn't actually needed anymore. Ordinary ccache > > should be sufficient. > > > > > > I was not sure where this incremental compilation cache is located. > Sure enough on my computer it seems to live under > MOZ_OBJ/x86_64-unknown-linux-gnu/debug/incremental > > I have a serious question: does this incremental cache have a control > mechanism to purge old entries? > > I have been hit with file system overflow errors quite frequently since > early summer and I figure > this is due to the bloated > MOZ_OBJ/x86_64-unknown-linux-gnu/debug/incremental directory. > > It is now 26.6 GB with 67840 items according baobab file system usage > monitor. > > 26.6 GB for this incremental cache alone is way too much number that was > suggested by mozilla development document before. > In comparsion, ccache I use is usable with max size of 5GB > as set by > CCACHE_SIZE=5.0GB > in my script. > > On my local PC, I see that the file directory size (with the sizes of > subdirectories added to it) is as follows: > > MOZ_OBJ is 45.8 GB large. (as of now: I had to stop TB compilation due > to the 50 GB smallish partition overflow where this directory is located.) > MOZ_OBJ/x86_64-unknown-linux-gnu/ is 33.9 GB large . > MOZ_OBJ/x86_64-unknown-linux-gnu/debug/ is 33.9 GB large. > I think /debug/ is because I create full debug version of TB locally. > MOZ_OBJ/MOZ_OBJ/x86_64-unknown-linux-gnu/debug/incremental is 26.5 GB > large. > > I need a way to shrink this incremental cache reasonably. > It seems this incremental directory grows without limit. Or, it does not > trim old unusable cache files automatically. > > TIA > > > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Must we rebuild all our rust code constantly?
On 2019/08/20 14:32, Kris Maglione wrote: On Tue, Aug 20, 2019 at 02:23:06PM +0900, ISHIKAWA,chiaki wrote: On 2019/08/20 9:11, Dave Townsend wrote: Thanks to a tip I've tracked this down. This seems to only be the case when I have sccache enabled. Disabling it gives me nice quick incremental builds again. Of course that isn't an ideal solution but it will do for now. On Mon, Aug 19, 2019 at 1:55 PM Dave Townsend wrote: For a couple of weeks now I've seen that any attempt to build Firefox, even incremental builds seem to rebuild an awful lot of rust code. I found this in the source which seems to suggest why: https://searchfox.org/mozilla-central/source/config/makefiles/rust.mk#238. But, this means that now an incremental build with a couple of code change that have no impact on rust is taking upwards of 4 minutes to complete in comparison to around 40 seconds, and the log file is full of cargo output. I've heard similar comments from other developers. This is a pretty big increase in the time to compile and test and is really slowing down my work. Is there any way we can avoid this? I am using linux for local development and noticed something similar. So I should disable sccache (!?). For what it's worth, Rust is now configured to use incremental compilation, which has its own cache which isn't cleared after clobber, so sccache isn't actually needed anymore. Ordinary ccache should be sufficient. I was not sure where this incremental compilation cache is located. Sure enough on my computer it seems to live under MOZ_OBJ/x86_64-unknown-linux-gnu/debug/incremental I have a serious question: does this incremental cache have a control mechanism to purge old entries? I have been hit with file system overflow errors quite frequently since early summer and I figure this is due to the bloated MOZ_OBJ/x86_64-unknown-linux-gnu/debug/incremental directory. It is now 26.6 GB with 67840 items according baobab file system usage monitor. 26.6 GB for this incremental cache alone is way too much number that was suggested by mozilla development document before. In comparsion, ccache I use is usable with max size of 5GB as set by CCACHE_SIZE=5.0GB in my script. On my local PC, I see that the file directory size (with the sizes of subdirectories added to it) is as follows: MOZ_OBJ is 45.8 GB large. (as of now: I had to stop TB compilation due to the 50 GB smallish partition overflow where this directory is located.) MOZ_OBJ/x86_64-unknown-linux-gnu/ is 33.9 GB large . MOZ_OBJ/x86_64-unknown-linux-gnu/debug/ is 33.9 GB large. I think /debug/ is because I create full debug version of TB locally. MOZ_OBJ/MOZ_OBJ/x86_64-unknown-linux-gnu/debug/incremental is 26.5 GB large. I need a way to shrink this incremental cache reasonably. It seems this incremental directory grows without limit. Or, it does not trim old unusable cache files automatically. TIA ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Must we rebuild all our rust code constantly?
Hi all, I'm not sure whether Thunderbirds are different, but the issue is that (on Windows) xul.dll is constantly relinked although there haven't been any changes to C++ or RS files. Log below. Oh, you can also see some ugly hunspell warnings that have crept in a while ago. Jörg. $ mach build 0:02.41 Clobber not needed. 0:02.42 Adding make options from c:\mozilla-source\comm-central\.mozconfig MOZ_MAKE_FLAGS=-j16 MOZ_OBJDIR=c:/mozilla-source/comm-central/obj-x86_64-pc-mingw32 OBJDIR=c:/mozilla-source/comm-central/obj-x86_64-pc-mingw32 FOUND_MOZCONFIG=c:/mozilla-source/comm-central/.mozconfig export FOUND_MOZCONFIG 0:02.45 c:\mozilla-build\bin\mozmake.exe -f client.mk MOZ_PARALLEL_BUILD=16 -s 0:02.88 Elapsed: 0.00s; From dist/public: Kept 0 existing; Added/updated 0; Removed 0 files and 0 directories. 0:02.89 Elapsed: 0.00s; From dist/private: Kept 0 existing; Added/updated 0; Removed 0 files and 0 directories. 0:02.91 Elapsed: 0.03s; From dist/xpi-stage: Kept 58 existing; Added/updated 0; Removed 0 files and 0 directories. 0:03.77 Elapsed: 0.88s; From _tests: Kept 0 existing; Added/updated 1521; Removed 0 files and 0 directories. 0:03.99 Elapsed: 1.09s; From dist/bin: Kept 0 existing; Added/updated 2858; Removed 0 files and 0 directories. 0:04.94 Elapsed: 2.05s; From dist/include: Kept 0 existing; Added/updated 6385; Removed 0 files and 0 directories. 0:04.99 ./buildid.h.stub 0:05.19 ./source-repo.h.stub 0:06.02 build/application.ini.stub 0:06.32 build/application.ini.h.stub 0:07.03 build/clang-plugin/tests 0:07.85 toolkit/library/rust/force-cargo-library-build 0:07.85 js/src/rust/force-cargo-library-build 0:07.85 js/src/frontend/binast/force-cargo-host-program-build 0:07.86 testing/geckodriver/force-cargo-program-build 0:08.05 [0m[0m[1m[36m Blocking[0m waiting for file lock on package cache lock 0:08.05 [0m[0m[0m[0m[1m[1m[36m[36m BlockingBlocking[0m[0m waiting for file lock on package cache lockwaiting for file lock on package cache lock 0:09.83 [0m[0m[1m[36m Blocking[0m waiting for file lock on package cache lock 0:11.11 [0m[0m[1m[36m Blocking[0m waiting for file lock on package cache lock 0:12.26 [0m[0m[1m[36m Blocking[0m waiting for file lock on package cache lock 0:13.18 comm/mail/app 0:13.23 toolkit/library/buildid.cpp.stub 0:13.45 [0m[0m[1m[36m Blocking[0m waiting for file lock on package cache lock 0:13.45 [0m[0m[1m[36m Blocking[0m waiting for file lock on package cache lock 0:13.48 [0m[0m[1m[36m Blocking[0m waiting for file lock on package cache lock 0:13.48 [0m[0m[1m[36m Blocking[0m waiting for file lock on package cache lock 0:13.48 [0m[0m[1m[36m Blocking[0m waiting for file lock on package cache lock 0:13.50 [0m[0m[1m[36m Blocking[0m waiting for file lock on build directory 0:13.54 [0m[0m[1m[32m Finished[0m dev [optimized + debuginfo] target(s) in 5.65s 0:13.63 toolkit/library 0:13.78 [0m[0m[1m[32m Finished[0m dev [optimized + debuginfo] target(s) in 5.88s 0:14.17 [0m[0m[1m[32m Finished[0m dev [optimized + debuginfo] target(s) in 6.27s 0:14.26 [0m[0m[1m[32m Finished[0m dev [optimized + debuginfo] target(s) in 6.38s 0:14.42 toolkit/library/build/xul.dll 0:15.07 comm/mail/app/module.res 0:15.09 Creating Resource file: module.res 0:15.12 comm/mail/app/thunderbird.exe 0:15.26 Embedding manifest from ../../../../comm/mail/app/thunderbird.exe.manifest 0:19.93 lld-link: warning: ..\..\..\extensions\spellcheck\hunspell\glue\Unified_cpp_hunspell_glue0.obj: locally defined symbol imported: "public: __cdecl Hunspell::~Hunspell(void)" (??1Hunspell@@QEAA@XZ) (defined in ..\..\..\extensions\spellcheck\hunspell\src\Unified_cpp_hunspell_src0.obj) [LNK4217] 0:19.93 lld-link: warning: ..\..\..\extensions\spellcheck\hunspell\glue\Unified_cpp_hunspell_glue0.obj: locally defined symbol imported: "public: __cdecl Hunspell::Hunspell(char const *, char const *, char const *)" (??0Hunspell@@QEAA@PEBD00@Z) (defined in ..\..\..\extensions\spellcheck\hunspell\src\Unified_cpp_hunspell_src0.obj) [LNK4217] 0:19.93 lld-link: warning: ..\..\..\extensions\spellcheck\hunspell\glue\Unified_cpp_hunspell_glue0.obj: locally defined symbol imported: "public: class std::basic_stringstruct std::char_traits, class std::allocator> const & __cdecl Hunspell::get_dict_encoding(void) const" (?get_dict_encoding@Hunspell@@QEBAAEBV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@XZ) (defined in ..\..\..\extensions\spellcheck\hunspell\src\Unified_cpp_hunspell_src0.obj) [LNK4217] 0:19.93 lld-link: warning: ..\..\..\extensions\spellcheck\hunspell\glue\Unified_cpp_hunspell_glue0.obj: locally defined symbol imported: "public: bool __cdecl Hunspell::spell(class std::basic_stringstd::char_traits, class std::allocator> const &, int *, class std::basic_string, class std::allocator> *)"
Re: Must we rebuild all our rust code constantly?
On Tue, Aug 20, 2019 at 02:23:06PM +0900, ISHIKAWA,chiaki wrote: On 2019/08/20 9:11, Dave Townsend wrote: Thanks to a tip I've tracked this down. This seems to only be the case when I have sccache enabled. Disabling it gives me nice quick incremental builds again. Of course that isn't an ideal solution but it will do for now. On Mon, Aug 19, 2019 at 1:55 PM Dave Townsend wrote: For a couple of weeks now I've seen that any attempt to build Firefox, even incremental builds seem to rebuild an awful lot of rust code. I found this in the source which seems to suggest why: https://searchfox.org/mozilla-central/source/config/makefiles/rust.mk#238. But, this means that now an incremental build with a couple of code change that have no impact on rust is taking upwards of 4 minutes to complete in comparison to around 40 seconds, and the log file is full of cargo output. I've heard similar comments from other developers. This is a pretty big increase in the time to compile and test and is really slowing down my work. Is there any way we can avoid this? I am using linux for local development and noticed something similar. So I should disable sccache (!?). For what it's worth, Rust is now configured to use incremental compilation, which has its own cache which isn't cleared after clobber, so sccache isn't actually needed anymore. Ordinary ccache should be sufficient. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Must we rebuild all our rust code constantly?
On 2019/08/20 9:11, Dave Townsend wrote: Thanks to a tip I've tracked this down. This seems to only be the case when I have sccache enabled. Disabling it gives me nice quick incremental builds again. Of course that isn't an ideal solution but it will do for now. On Mon, Aug 19, 2019 at 1:55 PM Dave Townsend wrote: For a couple of weeks now I've seen that any attempt to build Firefox, even incremental builds seem to rebuild an awful lot of rust code. I found this in the source which seems to suggest why: https://searchfox.org/mozilla-central/source/config/makefiles/rust.mk#238. But, this means that now an incremental build with a couple of code change that have no impact on rust is taking upwards of 4 minutes to complete in comparison to around 40 seconds, and the log file is full of cargo output. I've heard similar comments from other developers. This is a pretty big increase in the time to compile and test and is really slowing down my work. Is there any way we can avoid this? I am using linux for local development and noticed something similar. So I should disable sccache (!?). ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Must we rebuild all our rust code constantly?
Is this https://bugzilla.mozilla.org/show_bug.cgi?id=1427313 ? On Mon, Aug 19, 2019 at 5:27 PM Kris Maglione wrote: > This is apparently a known bug that no-one seems to be able to > track down the cause of. It suddenly started happening to me one > night for every build, even if I changed nothing. Then, just as > suddenly, stopped happening after a couple of hours. > > On Mon, Aug 19, 2019 at 05:11:19PM -0700, Dave Townsend wrote: > >Thanks to a tip I've tracked this down. This seems to only be the case > when > >I have sccache enabled. Disabling it gives me nice quick incremental > builds > >again. Of course that isn't an ideal solution but it will do for now. > > > >On Mon, Aug 19, 2019 at 1:55 PM Dave Townsend > wrote: > > > >> For a couple of weeks now I've seen that any attempt to build Firefox, > >> even incremental builds seem to rebuild an awful lot of rust code. I > found > >> this in the source which seems to suggest why: > >> > https://searchfox.org/mozilla-central/source/config/makefiles/rust.mk#238. > >> But, this means that now an incremental build with a couple of code > change > >> that have no impact on rust is taking upwards of 4 minutes to complete > in > >> comparison to around 40 seconds, and the log file is full of cargo > output. > >> I've heard similar comments from other developers. > >> > >> This is a pretty big increase in the time to compile and test and is > >> really slowing down my work. Is there any way we can avoid this? > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Must we rebuild all our rust code constantly?
This is apparently a known bug that no-one seems to be able to track down the cause of. It suddenly started happening to me one night for every build, even if I changed nothing. Then, just as suddenly, stopped happening after a couple of hours. On Mon, Aug 19, 2019 at 05:11:19PM -0700, Dave Townsend wrote: Thanks to a tip I've tracked this down. This seems to only be the case when I have sccache enabled. Disabling it gives me nice quick incremental builds again. Of course that isn't an ideal solution but it will do for now. On Mon, Aug 19, 2019 at 1:55 PM Dave Townsend wrote: For a couple of weeks now I've seen that any attempt to build Firefox, even incremental builds seem to rebuild an awful lot of rust code. I found this in the source which seems to suggest why: https://searchfox.org/mozilla-central/source/config/makefiles/rust.mk#238. But, this means that now an incremental build with a couple of code change that have no impact on rust is taking upwards of 4 minutes to complete in comparison to around 40 seconds, and the log file is full of cargo output. I've heard similar comments from other developers. This is a pretty big increase in the time to compile and test and is really slowing down my work. Is there any way we can avoid this? ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Must we rebuild all our rust code constantly?
Thanks to a tip I've tracked this down. This seems to only be the case when I have sccache enabled. Disabling it gives me nice quick incremental builds again. Of course that isn't an ideal solution but it will do for now. On Mon, Aug 19, 2019 at 1:55 PM Dave Townsend wrote: > For a couple of weeks now I've seen that any attempt to build Firefox, > even incremental builds seem to rebuild an awful lot of rust code. I found > this in the source which seems to suggest why: > https://searchfox.org/mozilla-central/source/config/makefiles/rust.mk#238. > But, this means that now an incremental build with a couple of code change > that have no impact on rust is taking upwards of 4 minutes to complete in > comparison to around 40 seconds, and the log file is full of cargo output. > I've heard similar comments from other developers. > > This is a pretty big increase in the time to compile and test and is > really slowing down my work. Is there any way we can avoid this? > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
