Re: Rust, pkgsrc
On 11/07/18 09:27, Chavdar Ivanov wrote: On 11/06/18 18:50, Havard Eidnes wrote: Yes. I had a similar problem. The build would fill up the /tmp/ directory and die from exhausted resources. I had /tmp/ created with tmpfs and had a constraint of 64M. The answer for me was to create /tmp in /etc/fstab with tmpfs and no size constraint. Then Rust would build, but it still took a long time. Yes, rust is an absolute Pig resource-wise. Not only does it carry a copy of llvm inside itself (it most probably has to), it also carries a nubmer of other packages inside. Plus, the build builds most parts at least twice if I've been able to observe correctly. When building on NetBSD/amd64 8.0, I noticed that the work/ directory after a "make" consumes in the order of 10G disk space, possibly more when cross-building (my current work/ is 13G, which is an unfinished cross-build because I hit build issues...). It is conceivable that the storage could be reduced somewhat (but probably not by much?) by tweaking src/bootstrap/boostrap.py to say -Cdebuginfo=0 instead of 2 for RUSTFLAGS. So that you have an idea what to expect: My amd64 build host has 8G real memory, and a 2G tmpfs /tmp, and ... it didn't run out of space anywhere :) On this particular host (i7 3rd-gen, 4 real cores, 8 w/HT, pkgsrc and system on SSD) the build of 1.29.2 completed in a little over 2 hours wall-clock time, csh's "time" report at the end of the build was 40468.007u 1958.277s 2:04:42.18 567.0% 0+0k 10302+100556io 129329pf+0w On my 16GB HP Envy, 4c/8t, I got overnight: --- Build completed successfully in 4:22:05 make 14672.75s user 36893488148207.93s system -2147483648% cpu 4:43:37.28 total /usr/pkgsrc/lang/rust with /tmp just a standard directory, tmpfs as usual on /var/shm. This is more than twice as slow compared to the last few rust builds, which also took just over two hours. I wonder if something was changed in the build process, previous builds were highly parallelized, with all the threads being seen busy during various stages, whereas this time I didn't see anything like this. All my builds were off unmodified pkgsrc lang/rust directory with nothing in /etc/mk.conf that I can see to affect this. I suspect the ridiculous times for system and cpu are from zsh 'time'. Follow up: 'make test' killed the machine, there was no coredump. It had fully loaded the system, though - with all 8 threads up to 100%. Previous versions usually completed the test, albeit with errors. This is on uname -a NetBSD tarkus 8.99.25 NetBSD 8.99.25 (GENERIC) #0: Sat Nov 3 03:33:22 GMT 2018 sysbuild@ymir:/home/sysbuild/sysbuild/amd64/obj/home/sysbuild/src/sys/arch/amd64/compile/GENERIC amd64 So, yes, the build makes fairly good use of the multiple cores; notice the 567.0%, which, if I've understood correctly, indicates approx. 5.7* parallelism on average. The last version I managed to build on one of my NetBSD/macppc 8.0 machines (a single-core 1.5GHz G4 Mac Mini, 1GB memory) was 1.29.2, the build took nearly 29 hours wallclock time. This one doesn't have a tmpfs, and has a single file system with ~40G free, so it also didn't run into any barriers on the /tmp front either. Regards, - Håvard
Re: Rust, pkgsrc
On 11/07/18 10:21, Chavdar Ivanov wrote: On 11/07/18 09:27, Chavdar Ivanov wrote: On 11/06/18 18:50, Havard Eidnes wrote: Yes. I had a similar problem. The build would fill up the /tmp/ directory and die from exhausted resources. I had /tmp/ created with tmpfs and had a constraint of 64M. The answer for me was to create /tmp in /etc/fstab with tmpfs and no size constraint. Then Rust would build, but it still took a long time. Yes, rust is an absolute Pig resource-wise. Not only does it carry a copy of llvm inside itself (it most probably has to), it also carries a nubmer of other packages inside. Plus, the build builds most parts at least twice if I've been able to observe correctly. When building on NetBSD/amd64 8.0, I noticed that the work/ directory after a "make" consumes in the order of 10G disk space, possibly more when cross-building (my current work/ is 13G, which is an unfinished cross-build because I hit build issues...). It is conceivable that the storage could be reduced somewhat (but probably not by much?) by tweaking src/bootstrap/boostrap.py to say -Cdebuginfo=0 instead of 2 for RUSTFLAGS. So that you have an idea what to expect: My amd64 build host has 8G real memory, and a 2G tmpfs /tmp, and ... it didn't run out of space anywhere :) On this particular host (i7 3rd-gen, 4 real cores, 8 w/HT, pkgsrc and system on SSD) the build of 1.29.2 completed in a little over 2 hours wall-clock time, csh's "time" report at the end of the build was 40468.007u 1958.277s 2:04:42.18 567.0% 0+0k 10302+100556io 129329pf+0w On my 16GB HP Envy, 4c/8t, I got overnight: --- Build completed successfully in 4:22:05 make 14672.75s user 36893488148207.93s system -2147483648% cpu 4:43:37.28 total /usr/pkgsrc/lang/rust with /tmp just a standard directory, tmpfs as usual on /var/shm. This is more than twice as slow compared to the last few rust builds, which also took just over two hours. I wonder if something was changed in the build process, previous builds were highly parallelized, with all the threads being seen busy during various stages, whereas this time I didn't see anything like this. All my builds were off unmodified pkgsrc lang/rust directory with nothing in /etc/mk.conf that I can see to affect this. I suspect the ridiculous times for system and cpu are from zsh 'time'. Follow up: 'make test' killed the machine, there was no coredump. It had fully loaded the system, though - with all 8 threads up to 100%. Previous versions usually completed the test, albeit with errors. This is on # uname -a NetBSD tarkus 8.99.25 NetBSD 8.99.25 (GENERIC) #0: Sat Nov 3 03:33:22 GMT 2018 sysbuild@ymir:/home/sysbuild/sysbuild/amd64/obj/home/sysbuild/src/sys/arch/amd64/compile/GENERIC amd64 Second attempt with the same build quickly errored, perhaps with leftovers from the previous. I'll probably wait for the next rust release before trying again. Weird bits from dmesg during the panic: -- hundreds of messages like the one below [ 45813.156897] cprng 25741 8 36: creating with partial entropy [ 45813.156897] cprng 8708 22 32: creating with partial entropy [ 45813.807857] panic: kernel diagnostic assertion "l->l_refcnt != 0" failed: file "/home/sysbuild/src/sys/kern/kern_lwp.c", line 163W5 A [ 45813.807857] RNING: SPL NOT LOWERED ON SYSCALL 129W4 AEXRINTI NG:b6 S2P5Ld bN0O T6 [ 45813.807857] LOWERED ON TRAP EXIT 6 0 [ 45813.807857] cpu7: Begin traceback... [ 45813.807857] WARNING: SPALR NINNOG:T SPLLO NWOET RLOEWEDR EOD NO NS TYRSACP AELXILT 6 10 [ 45813.807857] 8 EXIT 0 7 hundreds of messages similar to the next one [ 45813.807857] WARNING: SPL NOT LOWERED ON TRAP EXIT 6 0 interspersed with the panic trace [ 45813.807857] vpanic() at WARNING: SPL NOT LOWERED ON SYSCALL 1 8 EXIT 0 7 [ 45813.807857] netbsd:vpanic+0x16f . 8W07cA8hR5_Nv7I1oNl]Gt :aW AgSR_NcoILn NNGeOrT:t _LSiOnWP(EL) R NEaDOt OLONW ESREDS COANL LT RAP4 E1X I4T E6X I0T [ 45813.807857] 5b625db0 6 [ 45813.807857] WARNING: SPL NOT LOWERED ON TRAP EXIT 6 0 [ 45813.807857] T LWOAWRNEIRNEGD: OSNP LT RNOATP LEOXWIETR E6D 0O [ 45813.807857] lwp_addref() at WARNING: SPL NOT LOWERED ON SYSCALL 1 8 EXIT 0 7 [ 45813.807857] netbsd:lwp_addref+0xc7 [ 45813.817871] bsd:VOP_READ+0x50 [ 45813.817871] dofileread() at WARNING: SPL NOT LOWERED ON TRAP EXIT 6 0 [ 45813.817871] tbsd:dofileread+0xa9 [ 45813.817871] sys_read() at WARNING: SPL NOT LOWERED ON TRAP EXIT 6 0 [ 45813.817871] syscall() at WARNING: SPL NOT LOWERED ON TRAP EXIT 6 0 [ 45813.817871] all (number 3) --- [ 45813.827886] aceback... almost as if trying to say something with the last bits of consciousness: [ 4.00] 5 ] A7RR]NNII NNGGO::T SLPPLLO NNWOOTTE LLROOWWEEERREDEDD OONN SOTYRSNACPA L ELTX I5T5 56 A40 [ 45813.827886] dumping to dev 168,15 (offset=8, size=4152523): [ 45813.827886] dump wddump: device timed out [ 45824.784052] i/o error [ 45829.791441] rebooting...
Re: Rust, pkgsrc
On 11/07/18 09:27, Chavdar Ivanov wrote: On 11/06/18 18:50, Havard Eidnes wrote: Yes. I had a similar problem. The build would fill up the /tmp/ directory and die from exhausted resources. I had /tmp/ created with tmpfs and had a constraint of 64M. The answer for me was to create /tmp in /etc/fstab with tmpfs and no size constraint. Then Rust would build, but it still took a long time. Yes, rust is an absolute Pig resource-wise. Not only does it carry a copy of llvm inside itself (it most probably has to), it also carries a nubmer of other packages inside. Plus, the build builds most parts at least twice if I've been able to observe correctly. When building on NetBSD/amd64 8.0, I noticed that the work/ directory after a "make" consumes in the order of 10G disk space, possibly more when cross-building (my current work/ is 13G, which is an unfinished cross-build because I hit build issues...). It is conceivable that the storage could be reduced somewhat (but probably not by much?) by tweaking src/bootstrap/boostrap.py to say -Cdebuginfo=0 instead of 2 for RUSTFLAGS. So that you have an idea what to expect: My amd64 build host has 8G real memory, and a 2G tmpfs /tmp, and ... it didn't run out of space anywhere :) On this particular host (i7 3rd-gen, 4 real cores, 8 w/HT, pkgsrc and system on SSD) the build of 1.29.2 completed in a little over 2 hours wall-clock time, csh's "time" report at the end of the build was 40468.007u 1958.277s 2:04:42.18 567.0% 0+0k 10302+100556io 129329pf+0w On my 16GB HP Envy, 4c/8t, I got overnight: --- Build completed successfully in 4:22:05 make 14672.75s user 36893488148207.93s system -2147483648% cpu 4:43:37.28 total /usr/pkgsrc/lang/rust with /tmp just a standard directory, tmpfs as usual on /var/shm. This is more than twice as slow compared to the last few rust builds, which also took just over two hours. I wonder if something was changed in the build process, previous builds were highly parallelized, with all the threads being seen busy during various stages, whereas this time I didn't see anything like this. All my builds were off unmodified pkgsrc lang/rust directory with nothing in /etc/mk.conf that I can see to affect this. I suspect the ridiculous times for system and cpu are from zsh 'time'. Follow up: 'make test' killed the machine, there was no coredump. It had fully loaded the system, though - with all 8 threads up to 100%. Previous versions usually completed the test, albeit with errors. This is on # uname -a NetBSD tarkus 8.99.25 NetBSD 8.99.25 (GENERIC) #0: Sat Nov 3 03:33:22 GMT 2018 sysbuild@ymir:/home/sysbuild/sysbuild/amd64/obj/home/sysbuild/src/sys/arch/amd64/compile/GENERIC amd64 Second attempt with the same build quickly errored, perhaps with leftovers from the previous. I'll probably wait for the next rust release before trying again. So, yes, the build makes fairly good use of the multiple cores; notice the 567.0%, which, if I've understood correctly, indicates approx. 5.7* parallelism on average. The last version I managed to build on one of my NetBSD/macppc 8.0 machines (a single-core 1.5GHz G4 Mac Mini, 1GB memory) was 1.29.2, the build took nearly 29 hours wallclock time. This one doesn't have a tmpfs, and has a single file system with ~40G free, so it also didn't run into any barriers on the /tmp front either. Regards, - Håvard
Re: Rust, pkgsrc
On 11/06/18 18:50, Havard Eidnes wrote: Yes. I had a similar problem. The build would fill up the /tmp/ directory and die from exhausted resources. I had /tmp/ created with tmpfs and had a constraint of 64M. The answer for me was to create /tmp in /etc/fstab with tmpfs and no size constraint. Then Rust would build, but it still took a long time. Yes, rust is an absolute Pig resource-wise. Not only does it carry a copy of llvm inside itself (it most probably has to), it also carries a nubmer of other packages inside. Plus, the build builds most parts at least twice if I've been able to observe correctly. When building on NetBSD/amd64 8.0, I noticed that the work/ directory after a "make" consumes in the order of 10G disk space, possibly more when cross-building (my current work/ is 13G, which is an unfinished cross-build because I hit build issues...). It is conceivable that the storage could be reduced somewhat (but probably not by much?) by tweaking src/bootstrap/boostrap.py to say -Cdebuginfo=0 instead of 2 for RUSTFLAGS. So that you have an idea what to expect: My amd64 build host has 8G real memory, and a 2G tmpfs /tmp, and ... it didn't run out of space anywhere :) On this particular host (i7 3rd-gen, 4 real cores, 8 w/HT, pkgsrc and system on SSD) the build of 1.29.2 completed in a little over 2 hours wall-clock time, csh's "time" report at the end of the build was 40468.007u 1958.277s 2:04:42.18 567.0% 0+0k 10302+100556io 129329pf+0w On my 16GB HP Envy, 4c/8t, I got overnight: --- Build completed successfully in 4:22:05 make 14672.75s user 36893488148207.93s system -2147483648% cpu 4:43:37.28 total /usr/pkgsrc/lang/rust with /tmp just a standard directory, tmpfs as usual on /var/shm. This is more than twice as slow compared to the last few rust builds, which also took just over two hours. I wonder if something was changed in the build process, previous builds were highly parallelized, with all the threads being seen busy during various stages, whereas this time I didn't see anything like this. All my builds were off unmodified pkgsrc lang/rust directory with nothing in /etc/mk.conf that I can see to affect this. I suspect the ridiculous times for system and cpu are from zsh 'time'. So, yes, the build makes fairly good use of the multiple cores; notice the 567.0%, which, if I've understood correctly, indicates approx. 5.7* parallelism on average. The last version I managed to build on one of my NetBSD/macppc 8.0 machines (a single-core 1.5GHz G4 Mac Mini, 1GB memory) was 1.29.2, the build took nearly 29 hours wallclock time. This one doesn't have a tmpfs, and has a single file system with ~40G free, so it also didn't run into any barriers on the /tmp front either. Regards, - Håvard
Re: Rust, pkgsrc
On Tue, Nov 06, 2018 at 07:50:18PM +0100, Havard Eidnes wrote: > The last version I managed to build on one of my NetBSD/macppc > 8.0 machines (a single-core 1.5GHz G4 Mac Mini, 1GB memory) was > 1.29.2, the build took nearly 29 hours wallclock time. This one > doesn't have a tmpfs, and has a single file system with ~40G > free, so it also didn't run into any barriers on the /tmp front > either. I recently did build (thanks to Havard's bootstrap kit) a native version of 1.30 on sparc64 on a 8GB RAM dual 1.5GHz machine, everything on SSD plus /tmp on tmpfs. I did not do proper timing, but it finished over night (started early when I left the office and probably only because I came in late the next day). Martin
Re: Rust, pkgsrc
> Yes. I had a similar problem. The build would fill up the > /tmp/ directory and die from exhausted resources. I had /tmp/ > created with tmpfs and had a constraint of 64M. The answer for > me was to create /tmp in /etc/fstab with tmpfs and no size > constraint. Then Rust would build, but it still took a long > time. Yes, rust is an absolute Pig resource-wise. Not only does it carry a copy of llvm inside itself (it most probably has to), it also carries a nubmer of other packages inside. Plus, the build builds most parts at least twice if I've been able to observe correctly. When building on NetBSD/amd64 8.0, I noticed that the work/ directory after a "make" consumes in the order of 10G disk space, possibly more when cross-building (my current work/ is 13G, which is an unfinished cross-build because I hit build issues...). It is conceivable that the storage could be reduced somewhat (but probably not by much?) by tweaking src/bootstrap/boostrap.py to say -Cdebuginfo=0 instead of 2 for RUSTFLAGS. So that you have an idea what to expect: My amd64 build host has 8G real memory, and a 2G tmpfs /tmp, and ... it didn't run out of space anywhere :) On this particular host (i7 3rd-gen, 4 real cores, 8 w/HT, pkgsrc and system on SSD) the build of 1.29.2 completed in a little over 2 hours wall-clock time, csh's "time" report at the end of the build was 40468.007u 1958.277s 2:04:42.18 567.0% 0+0k 10302+100556io 129329pf+0w So, yes, the build makes fairly good use of the multiple cores; notice the 567.0%, which, if I've understood correctly, indicates approx. 5.7* parallelism on average. The last version I managed to build on one of my NetBSD/macppc 8.0 machines (a single-core 1.5GHz G4 Mac Mini, 1GB memory) was 1.29.2, the build took nearly 29 hours wallclock time. This one doesn't have a tmpfs, and has a single file system with ~40G free, so it also didn't run into any barriers on the /tmp front either. Regards, - Håvard
Re: Rust, pkgsrc
On 05/11/2018 16:34, bch wrote: The latest rust (1.30?) supporting the latest Firefox is *brutal* to build. I’ve blown (and then resized) /tmp multiple times, and am now exhausted on /usr for its build artifacts, before it’s even actually installed. Does anybody have tips or tricks for dealing with rust-building (which has always been terribly painful CPU-wise), or should I just move to prebuilt packages? I think I’ve never seen a piece of software as horrible to build as rust... If you are on x86 and have access to a bigger system for building on have you considered running pbulk in a chroot environment or using pkg_comp. I've save myself a lot of time on my x86 systems by doing all the builds on a relatively well specced system and then just installing binary packages on the smaller ones. Do you use MAKEJOBS on your pkgsrc builds? If you do then add MAKE_JOBS_SAFE=no to the rust package Makefile. Without that I've found that make ends up spawning multiple rustc processes each of which internally then spawn even more threads causing ridiculous cpu overcommit. If you aren't using it and you are on a small system I don't know what to suggest. Even with that tweak compiling rust on a 4core (8 threads) 4GHz 64bit CPU with 16GB of RAM is still painfully slow and ends up maxing out the CPU for most of the build. Mike -bch
Re: Rust, pkgsrc
Hi, bch wrote: The latest rust (1.30?) supporting the latest Firefox is *brutal* to build. I’ve blown (and then resized) /tmp multiple times, and am now exhausted on /usr for its build artifacts, before it’s even actually installed. Does anybody have tips or tricks for dealing with rust-building (which has always been terribly painful CPU-wise), or should I just move to prebuilt packages? I think I’ve never seen a piece of software as horrible to build as rust... it is terrible and I am very sorry our beloved Firefox was gone that way. It is making non-mainstreambuilds very difficult. I have issues with it not only on NetBSD, but also on Gentoo Linux. are you on 32bit or 64bit? The former has more issues! second, playing with binutils helped me out on Linux.. older version worked and the latest one worked too, but there was a series that did not. Also, RAM helps... more than 2GB seems to make thhings friendlier, 1.5GB is about the limit. Riccardo
Re: Rust, pkgsrc
Yes. I had a similar problem. The build would fill up the /tmp/ directory and die from exhausted resources. I had /tmp/ created with tmpfs and had a constraint of 64M. The answer for me was to create /tmp in /etc/fstab with tmpfs and no size constraint. Then Rust would build, but it still took a long time. On Tue, 6 Nov 2018 at 04:42, bch wrote: > The latest rust (1.30?) supporting the latest Firefox is *brutal* to > build. I’ve blown (and then resized) /tmp multiple times, and am now > exhausted on /usr for its build artifacts, before it’s even actually > installed. > > Does anybody have tips or tricks for dealing with rust-building (which has > always been terribly painful CPU-wise), or should I just move to prebuilt > packages? I think I’ve never seen a piece of software as horrible to build > as rust... > > > -bch >
