Re: -CURRENT compilation time
On 09/09/2021 00:04, Tomoaki AOKI wrote: devel/ninja/Makefile has USES= python in it, so it maybe require python to run or at least build. You could probably remove that line without anyone noticing. Ninja uses Python for precisely one thing (or, at least, did last time I looked): There is a debugging mode that will generate a visualisation of all of the dependencies in the project and run a web server that allows you to view this visualisation in your web browser. In about 10 years of using Ninja, I have used this functionality precisely once, and that was immediately after poking the code to find out why it had a Python dependency, discovering this mode existed, and looking to see what it did. Nothing on the build paths depends on Python and Ninja doesn't require Python to build itself. David
Re: -CURRENT compilation time
On Wed, Sep 8, 2021, 5:06 PM Tomoaki AOKI wrote:
> On Wed, 8 Sep 2021 14:32:16 -0600
> Warner Losh wrote:
>
> > On Wed, Sep 8, 2021, 6:33 AM David Chisnall
> wrote:
> >
> > > On 08/09/2021 11:52, Gary Jennejohn wrote:
> > > > Seems to me that there was an earlier mail about getting CMAKE to
> work
> > > > with FreeBSD builds. Could be worthwhile to look into getting ninja
> > > > to work also. But I could understand that there might be push-back,
> > > > since the project prefers to use utilities from the source tree.
> > >
> > > CMake is a build-system generator, Ninja is a build system. Usually
> the
> > > two are used together: CMake generates Ninja files, Ninja runs the
> > > build. Ninja is explicitly designed not to be written by hand.
> > >
> > > CMake can also emit other things, including POSIX Makefiles, but the
> > > Ninja build is usually the fastest.
> > >
> > > CMake and Ninja are both in package systems for Windows, macOS, *BSD,
> > > and all Linux distros that I've seen, unlike bmake, so universally
> easy
> > > to depend on for cross-builds. Cross compiling with bmake is much
> > > harder harder from anything that isn't FreeBSD.
> > >
> >
> > This may be true. I've never had a problem on mac or linux, but my
> windows
> > experience is thin.
> >
> > I got looking at the problem, btw, to see if ninja was faster than meta
> > mode. Ninja files aren't horrible to generate, but I never got past the
> > early proof of concept phase... it seemed possible to implement a make
> > ninja target... I'd wanted to use this day to day if it was better than
> > meta.
> >
> > Mason, however looked much harder to generate, but I didn't spend a lot
> of
> > time on it. I would think this sort of work would be more of a one shot.
> > Didn't look at cmake at all, but I imagine it would be similar...
> >
> > Warner
> >
> > >
>
> What about devel/samurai, ninja-compatible build tool written in C?
>
> devel/ninja/Makefile has USES= python in it, so it maybe require python
> to run or at least build.
>
> In addition, ports framework can use it instead of ninja.
> See {PORTSDIR}/Mk/Uses/ninja.mk.
>
I'll have to take a look next time I play with this stuff
Warner
--
> Tomoaki AOKI
>
>
Re: -CURRENT compilation time
On Wed, 8 Sep 2021 14:32:16 -0600
Warner Losh wrote:
> On Wed, Sep 8, 2021, 6:33 AM David Chisnall wrote:
>
> > On 08/09/2021 11:52, Gary Jennejohn wrote:
> > > Seems to me that there was an earlier mail about getting CMAKE to work
> > > with FreeBSD builds. Could be worthwhile to look into getting ninja
> > > to work also. But I could understand that there might be push-back,
> > > since the project prefers to use utilities from the source tree.
> >
> > CMake is a build-system generator, Ninja is a build system. Usually the
> > two are used together: CMake generates Ninja files, Ninja runs the
> > build. Ninja is explicitly designed not to be written by hand.
> >
> > CMake can also emit other things, including POSIX Makefiles, but the
> > Ninja build is usually the fastest.
> >
> > CMake and Ninja are both in package systems for Windows, macOS, *BSD,
> > and all Linux distros that I've seen, unlike bmake, so universally easy
> > to depend on for cross-builds. Cross compiling with bmake is much
> > harder harder from anything that isn't FreeBSD.
> >
>
> This may be true. I've never had a problem on mac or linux, but my windows
> experience is thin.
>
> I got looking at the problem, btw, to see if ninja was faster than meta
> mode. Ninja files aren't horrible to generate, but I never got past the
> early proof of concept phase... it seemed possible to implement a make
> ninja target... I'd wanted to use this day to day if it was better than
> meta.
>
> Mason, however looked much harder to generate, but I didn't spend a lot of
> time on it. I would think this sort of work would be more of a one shot.
> Didn't look at cmake at all, but I imagine it would be similar...
>
> Warner
>
> >
What about devel/samurai, ninja-compatible build tool written in C?
devel/ninja/Makefile has USES= python in it, so it maybe require python
to run or at least build.
In addition, ports framework can use it instead of ninja.
See {PORTSDIR}/Mk/Uses/ninja.mk.
--
Tomoaki AOKI
Re: -CURRENT compilation time
On Wed, Sep 8, 2021, 6:33 AM David Chisnall wrote: > On 08/09/2021 11:52, Gary Jennejohn wrote: > > Seems to me that there was an earlier mail about getting CMAKE to work > > with FreeBSD builds. Could be worthwhile to look into getting ninja > > to work also. But I could understand that there might be push-back, > > since the project prefers to use utilities from the source tree. > > CMake is a build-system generator, Ninja is a build system. Usually the > two are used together: CMake generates Ninja files, Ninja runs the > build. Ninja is explicitly designed not to be written by hand. > > CMake can also emit other things, including POSIX Makefiles, but the > Ninja build is usually the fastest. > > CMake and Ninja are both in package systems for Windows, macOS, *BSD, > and all Linux distros that I've seen, unlike bmake, so universally easy > to depend on for cross-builds. Cross compiling with bmake is much > harder harder from anything that isn't FreeBSD. > This may be true. I've never had a problem on mac or linux, but my windows experience is thin. I got looking at the problem, btw, to see if ninja was faster than meta mode. Ninja files aren't horrible to generate, but I never got past the early proof of concept phase... it seemed possible to implement a make ninja target... I'd wanted to use this day to day if it was better than meta. Mason, however looked much harder to generate, but I didn't spend a lot of time on it. I would think this sort of work would be more of a one shot. Didn't look at cmake at all, but I imagine it would be similar... Warner >
Re: -CURRENT compilation time
On Wed, Sep 8, 2021, 6:52 AM Stefan Esser wrote: > Am 08.09.21 um 10:57 schrieb David Chisnall: > > On 07/09/2021 18:02, Stefan Esser wrote: > >> Wouldn't this break META_MODE? > > > > I have never managed to get META_MODE to work but my understanding is > that > > META_MODE is addressing a problem that doesn't really exist in any other > build > > system that I've used: that dependencies are not properly tracked. > > META_MODE allows for complex interdependencies. They are no issue in the > GPL/Linux world, since components are not integrated in the same way as > has been practice in BSD for many decades. > > > When I do a build of LLVM with the upstream build system with no > changes, it > > takes Ninja approximately a tenth of a second to stat all of the > relevant files > > and tell me that I have no work to do. META_MODE apparently lets the > FreeBSD > > build system extract these dependencies and do something similar, but > it's not > > enabled by default and it's difficult to make work. > > I tend to disagree on the last 5 words of your last sentence. > > It took me just a few seconds to activate, and it has worked without fault > since. > > There are only 2 trivial steps. But it is easy to miss the fact, that > WITH_META_MODE has to be added to /etc/src-env.conf, not /etc/src.conf: > > 1) Add "WITH_META_MODE=yes" to /etc/src-env.conf (create file, if it does >not exist) > > 2) Add "device filemon" to your kernel configuration or to the kld_load >variable in /etc/rc.conf to load the kernel module > > (The kernel module can of course also be manually loaded at any time.) > > >> I'd rather be able to continue building the world within a few minutes > >> (generally much less than 10 minutes, as long as there is no major LLVM > >> upgrade) than have a faster LLVM build and then a slower build of the > world ... > > > > The rest of this thread has determined that building LLVM accounts for > half of > > the build time in a clean FreeBSD build. LLVM's CMake is not a great > example: > > it has been incrementally improved since CMake 2.8 and doesn't yet use > any of > > the modern CMake features that allow encapsulating targets and providing > import > > / export configurations. > > The build of LLVM is skipped if META_MODE is enabled, except if there > really was a change to some LLVM header that causes a complete rebuild. > > A further speed-up can be had with ccache, but I found that it does not > seem to make that much of a difference on my system. > > > In spite of that, it generates a ninja file that compiles *significantly* > > faster than the bmake-based system in FreeBSD. In other projects that > I've > > worked on with a similar-sized codebase to FreeBSD that use CMake + > Ninja, I've > > never had the same problems with build speed that I have with FreeBSD. > > Possible, but if I watch the LLVM build with top or systat, I see that > all my cores are busy, nearly throughout the full build. There are two > methods that could theoretically speed-up the build: > > 1) make use of idle CPU cores > > 2) reduce the number of object files to build > > I do not see that there is much potential for 1), since there is a high > degree of parallelism: > > >>> World build completed on Wed Sep 1 13:40:14 CEST 2021 > >>> World built in 99 seconds, ncpu: 32, make -j32 > -- >98.69 real 741.61 user 234.55 sys > > >>> World build completed on Thu Sep 2 23:22:04 CEST 2021 > >>> World built in 98 seconds, ncpu: 32, make -j32 > -- >98.34 real 780.41 user 228.67 sys > > >>> World build completed on Fri Sep 3 19:09:39 CEST 2021 > >>> World built in 165 seconds, ncpu: 32, make -j32 > -- > 164.84 real 1793.62 user 241.11 sys > > >>> World build completed on Sun Sep 5 20:23:29 CEST 2021 > >>> World built in 135 seconds, ncpu: 32, make -j32 > -- > 135.59 real 695.45 user 214.76 sys > > >>> World build completed on Mon Sep 6 21:10:44 CEST 2021 > >>> World built in 478 seconds, ncpu: 32, make -j32 > -- > 479.22 real 11374.40 user 474.19 sys > > >>> World build completed on Wed Sep 8 11:51:03 CEST 2021 > >>> World built in 652 seconds, ncpu: 32, make -j32 > -- > 652.14 real 17857.03 user 753.41 sys > > Calculating "(user + sys) / real" I get factors between 10 (in case > of only minor changes) to 28 for larger recompiles (e.g. if lots > of source files depend on an updated header), with 32 the theoretical > limit for all cores continuously active during the build. > > META_MODE does not understand that updated build tools do not always > require a
Re: -CURRENT compilation time
Am 08.09.21 um 10:57 schrieb David Chisnall: > On 07/09/2021 18:02, Stefan Esser wrote: >> Wouldn't this break META_MODE? > > I have never managed to get META_MODE to work but my understanding is that > META_MODE is addressing a problem that doesn't really exist in any other build > system that I've used: that dependencies are not properly tracked. META_MODE allows for complex interdependencies. They are no issue in the GPL/Linux world, since components are not integrated in the same way as has been practice in BSD for many decades. > When I do a build of LLVM with the upstream build system with no changes, it > takes Ninja approximately a tenth of a second to stat all of the relevant > files > and tell me that I have no work to do. META_MODE apparently lets the FreeBSD > build system extract these dependencies and do something similar, but it's not > enabled by default and it's difficult to make work. I tend to disagree on the last 5 words of your last sentence. It took me just a few seconds to activate, and it has worked without fault since. There are only 2 trivial steps. But it is easy to miss the fact, that WITH_META_MODE has to be added to /etc/src-env.conf, not /etc/src.conf: 1) Add "WITH_META_MODE=yes" to /etc/src-env.conf (create file, if it does not exist) 2) Add "device filemon" to your kernel configuration or to the kld_load variable in /etc/rc.conf to load the kernel module (The kernel module can of course also be manually loaded at any time.) >> I'd rather be able to continue building the world within a few minutes >> (generally much less than 10 minutes, as long as there is no major LLVM >> upgrade) than have a faster LLVM build and then a slower build of the world >> ... > > The rest of this thread has determined that building LLVM accounts for half of > the build time in a clean FreeBSD build. LLVM's CMake is not a great example: > it has been incrementally improved since CMake 2.8 and doesn't yet use any of > the modern CMake features that allow encapsulating targets and providing > import > / export configurations. The build of LLVM is skipped if META_MODE is enabled, except if there really was a change to some LLVM header that causes a complete rebuild. A further speed-up can be had with ccache, but I found that it does not seem to make that much of a difference on my system. > In spite of that, it generates a ninja file that compiles *significantly* > faster than the bmake-based system in FreeBSD. In other projects that I've > worked on with a similar-sized codebase to FreeBSD that use CMake + Ninja, > I've > never had the same problems with build speed that I have with FreeBSD. Possible, but if I watch the LLVM build with top or systat, I see that all my cores are busy, nearly throughout the full build. There are two methods that could theoretically speed-up the build: 1) make use of idle CPU cores 2) reduce the number of object files to build I do not see that there is much potential for 1), since there is a high degree of parallelism: >>> World build completed on Wed Sep 1 13:40:14 CEST 2021 >>> World built in 99 seconds, ncpu: 32, make -j32 -- 98.69 real 741.61 user 234.55 sys >>> World build completed on Thu Sep 2 23:22:04 CEST 2021 >>> World built in 98 seconds, ncpu: 32, make -j32 -- 98.34 real 780.41 user 228.67 sys >>> World build completed on Fri Sep 3 19:09:39 CEST 2021 >>> World built in 165 seconds, ncpu: 32, make -j32 -- 164.84 real 1793.62 user 241.11 sys >>> World build completed on Sun Sep 5 20:23:29 CEST 2021 >>> World built in 135 seconds, ncpu: 32, make -j32 -- 135.59 real 695.45 user 214.76 sys >>> World build completed on Mon Sep 6 21:10:44 CEST 2021 >>> World built in 478 seconds, ncpu: 32, make -j32 -- 479.22 real 11374.40 user 474.19 sys >>> World build completed on Wed Sep 8 11:51:03 CEST 2021 >>> World built in 652 seconds, ncpu: 32, make -j32 -- 652.14 real 17857.03 user 753.41 sys Calculating "(user + sys) / real" I get factors between 10 (in case of only minor changes) to 28 for larger recompiles (e.g. if lots of source files depend on an updated header), with 32 the theoretical limit for all cores continuously active during the build. META_MODE does not understand that updated build tools do not always require a full rebuild, but special cases have been added to the Makefile to reduce the number of unnecessary rebuilds. > Working on LLVM, I generally spend well under 10% of my time either waiting > for > builds or fighting the build system. Working on
Re: -CURRENT compilation time
On Wed, Sep 8, 2021 at 2:59 AM David Chisnall wrote: > On 07/09/2021 18:02, Stefan Esser wrote: > > Wouldn't this break META_MODE? > > I have never managed to get META_MODE to work but my understanding is > that META_MODE is addressing a problem that doesn't really exist in any > other build system that I've used: that dependencies are not properly > tracked. > It does track the dependencies. It uses filemon(4) to do this so that every dependency is tracked. This is better than what other build systems do because no dependencies are missed. > When I do a build of LLVM with the upstream build system with no > changes, it takes Ninja approximately a tenth of a second to stat all of > the relevant files and tell me that I have no work to do. META_MODE > apparently lets the FreeBSD build system extract these dependencies and > do something similar, but it's not enabled by default and it's difficult > to make work. > META_MODE does the same thing, and it's just as fast as ninja make. And it has the advantage that, unlike meson, it isn't rebuilding the makefiles all the time. And apart from loading filmon, it was quite easy to enable last time I was using it (though I did have trouble finding the right docs). > > I'd rather be able to continue building the world within a few minutes > > (generally much less than 10 minutes, as long as there is no major LLVM > > upgrade) than have a faster LLVM build and then a slower build of the > world ... > > The rest of this thread has determined that building LLVM accounts for > half of the build time in a clean FreeBSD build. LLVM's CMake is not a > great example: it has been incrementally improved since CMake 2.8 and > doesn't yet use any of the modern CMake features that allow > encapsulating targets and providing import / export configurations. > > In spite of that, it generates a ninja file that compiles > *significantly* faster than the bmake-based system in FreeBSD. In other > projects that I've worked on with a similar-sized codebase to FreeBSD > that use CMake + Ninja, I've never had the same problems with build > speed that I have with FreeBSD. > The speed is comparable to META_MODE, but much faster than the default. It suffers a bit because our buildworld stuff is a kinda sorts not quite good enough dependency tracker and it's layered on top of that. > Working on LLVM, I generally spend well under 10% of my time either > waiting for builds or fighting the build system. Working on FreeBSD, I > generally spend over 90% of my time waiting for builds or fighting the > build system. This means that my productivity contributing to FreeBSD > is almost zero. > > For reference, changes to LLVM typically build for me in under 30 > seconds with Ninja, unless I've changed a header that everything > > In particular, building FreeBSD on a 10-24 core machine has very long > periods where a number of the cores are completely idle. > > Ninja also has a few other nice features that improve performance > relative to bmake: > > - It lets you put jobs in different pools. In LLVM this is used to > put link and compile jobs in different pools because linking with LLD > uses multiple threads and a lot more memory than compilation, so a > 10-core machine may want to do 12 compile jobs in parallel but only 2 > link jobs. This makes it much easier to completely saturate the machine. > This is nice. > - Ninja provides each parallel build task with a separate pipe for > stdout and stderr, and does not print their output unless a build step > fails (or unless you build with -v). With bmake, if a parallel build > fails I have to rerun the build without -j, because the output is > interleaved with succeeding jobs and it's difficult to see what actually > failed. With ninja, the output is from each failed job, with no > interleaving. > META_MODE does this too. It's the primary reason I use it, along with the speed. The big downside of ninja make is that it requires you to have some other, higher-level build system to generate the Makefiles. And my experience to date with meson is a mixed bag: it's quite a bit better than normal bmake for some things, and quite a bit worse for others in terms of the code you need to write to get things done, or the efforts required to debug mistakes. I've not used cmake as the ninja makefile generator, so I have no comments on it. Doing the conversion to at least meson would be a quite large job, at least by my initial estimates. Hacking bmake to generate ninja makefiles looks to be quite a bit simpler... But looks can be deceiving. Warner
Re: -CURRENT compilation time
On 08/09/2021 11:52, Gary Jennejohn wrote: Seems to me that there was an earlier mail about getting CMAKE to work with FreeBSD builds. Could be worthwhile to look into getting ninja to work also. But I could understand that there might be push-back, since the project prefers to use utilities from the source tree. CMake is a build-system generator, Ninja is a build system. Usually the two are used together: CMake generates Ninja files, Ninja runs the build. Ninja is explicitly designed not to be written by hand. CMake can also emit other things, including POSIX Makefiles, but the Ninja build is usually the fastest. CMake and Ninja are both in package systems for Windows, macOS, *BSD, and all Linux distros that I've seen, unlike bmake, so universally easy to depend on for cross-builds. Cross compiling with bmake is much harder harder from anything that isn't FreeBSD. David
Re: -CURRENT compilation time
On Wed, 8 Sep 2021 09:57:50 +0100 David Chisnall wrote: > On 07/09/2021 18:02, Stefan Esser wrote: > > Wouldn't this break META_MODE? > > I have never managed to get META_MODE to work but my understanding > is that META_MODE is addressing a problem that doesn't really exist > in any other build system that I've used: that dependencies are not > properly tracked. > META_MODE requires filemon(4) to be in the kernel or loaded as a module. make(1) will use if it's available with .MAKE.MODE=meta. > When I do a build of LLVM with the upstream build system with no > changes, it takes Ninja approximately a tenth of a second to stat > all of the relevant files and tell me that I have no work to do. > META_MODE apparently lets the FreeBSD build system extract these > dependencies and do something similar, but it's not enabled by > default and it's difficult to make work. > > > I'd rather be able to continue building the world within a few > > minutes (generally much less than 10 minutes, as long as there is > > no major LLVM upgrade) than have a faster LLVM build and then a > > slower build of the world ... > > The rest of this thread has determined that building LLVM accounts > for half of the build time in a clean FreeBSD build. LLVM's CMake > is not a great example: it has been incrementally improved since > CMake 2.8 and doesn't yet use any of the modern CMake features that > allow encapsulating targets and providing import / export > configurations. > > In spite of that, it generates a ninja file that compiles > *significantly* faster than the bmake-based system in FreeBSD. In > other projects that I've worked on with a similar-sized codebase to > FreeBSD that use CMake + Ninja, I've never had the same problems > with build speed that I have with FreeBSD. > > Working on LLVM, I generally spend well under 10% of my time either > waiting for builds or fighting the build system. Working on > FreeBSD, I generally spend over 90% of my time waiting for builds or > fighting the build system. This means that my productivity > contributing to FreeBSD is almost zero. > > For reference, changes to LLVM typically build for me in under 30 > seconds with Ninja, unless I've changed a header that everything > > In particular, building FreeBSD on a 10-24 core machine has very > long periods where a number of the cores are completely idle. > > Ninja also has a few other nice features that improve performance > relative to bmake: > > - It lets you put jobs in different pools. In LLVM this is used to > put link and compile jobs in different pools because linking with > LLD uses multiple threads and a lot more memory than compilation, so > a 10-core machine may want to do 12 compile jobs in parallel but > only 2 link jobs. This makes it much easier to completely saturate > the machine. > - Ninja provides each parallel build task with a separate pipe for > stdout and stderr, and does not print their output unless a build > step fails (or unless you build with -v). With bmake, if a parallel > build fails I have to rerun the build without -j, because the output > is interleaved with succeeding jobs and it's difficult to see what > actually failed. With ninja, the output is from each failed job, > with no interleaving. > ninja sounds really neat and it's available as /usr/ports/devel/ninja. Seems to me that there was an earlier mail about getting CMAKE to work with FreeBSD builds. Could be worthwhile to look into getting ninja to work also. But I could understand that there might be push-back, since the project prefers to use utilities from the source tree. -- Gary Jennejohn
Re: -CURRENT compilation time
On 07/09/2021 18:02, Stefan Esser wrote: Wouldn't this break META_MODE? I have never managed to get META_MODE to work but my understanding is that META_MODE is addressing a problem that doesn't really exist in any other build system that I've used: that dependencies are not properly tracked. When I do a build of LLVM with the upstream build system with no changes, it takes Ninja approximately a tenth of a second to stat all of the relevant files and tell me that I have no work to do. META_MODE apparently lets the FreeBSD build system extract these dependencies and do something similar, but it's not enabled by default and it's difficult to make work. I'd rather be able to continue building the world within a few minutes (generally much less than 10 minutes, as long as there is no major LLVM upgrade) than have a faster LLVM build and then a slower build of the world ... The rest of this thread has determined that building LLVM accounts for half of the build time in a clean FreeBSD build. LLVM's CMake is not a great example: it has been incrementally improved since CMake 2.8 and doesn't yet use any of the modern CMake features that allow encapsulating targets and providing import / export configurations. In spite of that, it generates a ninja file that compiles *significantly* faster than the bmake-based system in FreeBSD. In other projects that I've worked on with a similar-sized codebase to FreeBSD that use CMake + Ninja, I've never had the same problems with build speed that I have with FreeBSD. Working on LLVM, I generally spend well under 10% of my time either waiting for builds or fighting the build system. Working on FreeBSD, I generally spend over 90% of my time waiting for builds or fighting the build system. This means that my productivity contributing to FreeBSD is almost zero. For reference, changes to LLVM typically build for me in under 30 seconds with Ninja, unless I've changed a header that everything In particular, building FreeBSD on a 10-24 core machine has very long periods where a number of the cores are completely idle. Ninja also has a few other nice features that improve performance relative to bmake: - It lets you put jobs in different pools. In LLVM this is used to put link and compile jobs in different pools because linking with LLD uses multiple threads and a lot more memory than compilation, so a 10-core machine may want to do 12 compile jobs in parallel but only 2 link jobs. This makes it much easier to completely saturate the machine. - Ninja provides each parallel build task with a separate pipe for stdout and stderr, and does not print their output unless a build step fails (or unless you build with -v). With bmake, if a parallel build fails I have to rerun the build without -j, because the output is interleaved with succeeding jobs and it's difficult to see what actually failed. With ninja, the output is from each failed job, with no interleaving. David
Re: -CURRENT compilation time
> From: David Chisnall > Date: Tue, 7 Sep 2021 14:51:21 +0100 > On 06/09/2021 20:34, Wolfram Schneider wrote: > > With the option WITHOUT_TOOLCHAIN=yes the world build time is 2.5 > > times faster (real or user+sys), down from 48 min to 19.5 min real > > time. > > Note that building LLVM with the upstream CMake + Ninja build system is > *significantly* faster on a decent multicore machine than the FreeBSD > bmake-based in-tree version. Just examples of llvm12 builds from my checking this . . . poudriere-devel llvm12 buil times, nothing else being built, all prerequisites already built (not that such are trivial): amd64 ThreadRipper 1950X with 128 GiByte of RAM, optane 1.4T media, 32 FreeBSD cpus: [00:00:07] [01] [00:00:00] Building devel/llvm12 | llvm12-12.0.1_4 [00:22:54] [01] [00:22:47] Finished devel/llvm12 | llvm12-12.0.1_4: Success for: ---Begin OPTIONS List--- ===> The following configuration options are available for llvm12-12.0.1_4: BE_AMDGPU=on: AMD GPU backend (required by mesa) CLANG=on: Build clang COMPILER_RT=on: Sanitizer libraries DOCS=on: Build and/or install documentation EXTRAS=on: Extra clang tools GOLD=on: Build the LLVM Gold plugin for LTO LIT=on: Install lit and FileCheck test tools LLD=on: Install lld, the LLVM linker LLDB=on: Install lldb, the LLVM debugger OPENMP=on: Install libomp, the LLVM OpenMP runtime library PYCLANG=off: Install python bindings to libclang > Options available for the single BACKENDS: you have to select exactly one of them BE_FREEBSD=off: Backends for FreeBSD architectures BE_NATIVE=on: Backend(s) for this architecture (X86) BE_STANDARD=off: All non-experimental backends ===> Use 'make config' to modify these settings ---End OPTIONS List--- Note the BE_NATIVE use. Also poudriere.conf has: ZPOOL=zoptb USE_TMPFS=yes ALLOW_MAKE_JOBS=yes For reference: # uname -apKU FreeBSD amd64_ZFS 13.0-RELEASE-p4 FreeBSD 13.0-RELEASE-p4 #1 releng/13.0-n244760-940681634ee1-dirty: Mon Aug 30 11:28:31 PDT 2021 root@amd64_ZFS:/usr/obj/BUILDs/13_0R-amd64-nodbg-clang/usr/13_0R-src/amd64.amd64/sys/GENERIC-NODBG amd64 amd64 1300139 1300139 The swap/paging space stayed unused. aarch64 Honycomb with 64 GiByte of RAM, optane 480 GiByte media, 16 FreeBSD cpus: [00:00:15] [01] [00:00:00] Building devel/llvm12 | llvm12-12.0.1_4 [00:58:07] [01] [00:57:52] Finished devel/llvm12 | llvm12-12.0.1_4: Success for: ---Begin OPTIONS List--- ===> The following configuration options are available for llvm12-12.0.1_4: BE_AMDGPU=on: AMD GPU backend (required by mesa) CLANG=on: Build clang DOCS=on: Build and/or install documentation EXTRAS=on: Extra clang tools LIT=on: Install lit and FileCheck test tools LLD=on: Install lld, the LLVM linker OPENMP=on: Install libomp, the LLVM OpenMP runtime library PYCLANG=off: Install python bindings to libclang > Options available for the single BACKENDS: you have to select exactly one of them BE_FREEBSD=off: Backends for FreeBSD architectures BE_NATIVE=on: Backend(s) for this architecture (AArch64) BE_STANDARD=off: All non-experimental backends ===> Use 'make config' to modify these settings ---End OPTIONS List--- Note the BE_NATIVE use. Also poudriere.conf has: ZPOOL=zopt0 USE_TMPFS="data" ALLOW_MAKE_JOBS=yes For reference: # uname -apKU FreeBSD CA72_16Gp_ZFS 13.0-RELEASE-p4 FreeBSD 13.0-RELEASE-p4 #4 releng/13.0-n244760-940681634ee1-dirty: Mon Aug 30 11:35:45 PDT 2021 root@CA72_16Gp_ZFS:/usr/obj/BUILDs/13_0R-CA72-nodbg-clang/usr/13_0R-src/arm64.aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64 1300139 1300139 The swap/paging space stayed unused. I've not yet figured out a good way to time just an analogous subset of buildworld . It would need to be stable/13 or main [so: 14] targetted to involve building an llvm12 based set of material. releng/13.0 is llvm11 based. One point is that likely the system builds libc++ and such but devel/llvm12 does not: devel/llvm12 likely is bound to the system libc++ and such. This may make "analogous subset of buildworld" harder to isolate. So I've not made it to well-measured differences in time-used yet. > One of the things I'd love to prototype if I had time is a CMake-based > build system for FreeBSD so that we could get all of the tooling > integration from the compile_commands.json, reuse LLVM's (and any other > contrib things that use CMake) build system without having to recreate > it, and be able to use ninja, to build. === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)
Re: -CURRENT compilation time
Am 07.09.21 um 15:51 schrieb David Chisnall: > On 06/09/2021 20:34, Wolfram Schneider wrote: >> With the option WITHOUT_TOOLCHAIN=yes the world build time is 2.5 >> times faster (real or user+sys), down from 48 min to 19.5 min real >> time. > > Note that building LLVM with the upstream CMake + Ninja build system is > *significantly* faster on a decent multicore machine than the FreeBSD > bmake-based in-tree version. > > One of the things I'd love to prototype if I had time is a CMake-based build > system for FreeBSD so that we could get all of the tooling integration from > the > compile_commands.json, reuse LLVM's (and any other contrib things that use > CMake) build system without having to recreate it, and be able to use ninja, > to > build. Wouldn't this break META_MODE? I'd rather be able to continue building the world within a few minutes (generally much less than 10 minutes, as long as there is no major LLVM upgrade) than have a faster LLVM build and then a slower build of the world ... Regards, STefan OpenPGP_signature Description: OpenPGP digital signature
Re: -CURRENT compilation time
On Tue, 7 Sep 2021, at 13:51, David Chisnall wrote: > One of the things I'd love to prototype if I had time is a CMake-based > build system for FreeBSD so that we could get all of the tooling > integration from the compile_commands.json, reuse LLVM's (and any other > contrib things that use CMake) build system without having to recreate > it, and be able to use ninja, to build. This would be a worthy FreeBSD foundation project IMHO. A+ Dave
Re: -CURRENT compilation time
On 06/09/2021 20:34, Wolfram Schneider wrote: With the option WITHOUT_TOOLCHAIN=yes the world build time is 2.5 times faster (real or user+sys), down from 48 min to 19.5 min real time. Note that building LLVM with the upstream CMake + Ninja build system is *significantly* faster on a decent multicore machine than the FreeBSD bmake-based in-tree version. One of the things I'd love to prototype if I had time is a CMake-based build system for FreeBSD so that we could get all of the tooling integration from the compile_commands.json, reuse LLVM's (and any other contrib things that use CMake) build system without having to recreate it, and be able to use ninja, to build. David
Re: -CURRENT compilation time
Van: David Chisnall Datum: maandag, 6 september 2021 11:43 Aan: freebsd-current@freebsd.org Onderwerp: Re: -CURRENT compilation time On 06/09/2021 09:08, Jeremie Le Hen wrote: > Compiling C++ seems > extremely CPU heavy and this is made worse by the fact LLVM is built > twice (once for build/cross tools, once for the actual world). Note that you need to build LLVM twice only if you are actively debugging LLVM reproduceable deployment images. You actually don't need to build it at all, you can use an external toolchain to skip the first build and you can compile WITHOUT_TOOLCHAIN to avoid building the version that's installed and then install a toolchain from packages: https://wiki.freebsd.org/ExternalToolchain David Hi, I'm very interested in a base without llvm because of compile times. So I tried this in a jail with 14-current and pkg llvm12 installed. /etc/make.conf: WITHOUT_TOOLCHAIN=yes CROSS_TOOLCHAIN=llvm12 Buildworld, installworld and etcupdate went fine. "yes | make delete-old" removes the toolchain from base. Afterwards you can't do buildworld anymore. # make buildworld sh: cc: not found make: "/home/ronald/dev/freebsd/share/mk/bsd.compiler.mk" line 200: warning: "cc -v 2>&1 | grep "gcc version"" returned non-zero status make: "/home/ronald/dev/freebsd/share/mk/bsd.compiler.mk" line 204: Unable to determine compiler type for CC=cc. Consider setting COMPILER_TYPE. What am I missing? Regards, Ronald.
Re: -CURRENT compilation time
On Mon, 6 Sept 2021 at 11:44, David Chisnall wrote: > > On 06/09/2021 09:08, Jeremie Le Hen wrote: > > Compiling C++ seems > > extremely CPU heavy and this is made worse by the fact LLVM is built > > twice (once for build/cross tools, once for the actual world). > > Note that you need to build LLVM twice only if you are actively > debugging LLVM reproduceable deployment images. You actually don't need > to build it at all, you can use an external toolchain to skip the first > build and you can compile WITHOUT_TOOLCHAIN to avoid building the > version that's installed and then install a toolchain from packages: > > https://wiki.freebsd.org/ExternalToolchain I did a test on a 16 core (32 VCPU) machine (Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz). With the option WITHOUT_TOOLCHAIN=yes the world build time is 2.5 times faster (real or user+sys), down from 48 min to 19.5 min real time. time make -j16 buildworld -- >>> World build completed on Mon Sep 6 12:00:45 UTC 2021 >>> World built in 2862 seconds, ncpu: 32, make -j16 -- 2862.04 real 41234.87 user 1582.66 sys time make -j16 WITHOUT_TOOLCHAIN=yes buildworld -- >>> World build completed on Mon Sep 6 11:32:41 UTC 2021 >>> World built in 1181 seconds, ncpu: 32, make -j16 -- 1180.73 real 16076.27 user 988.73 sys -Wolfram -- Wolfram Schneider https://wolfram.schneider.org
Re: -CURRENT compilation time
On Mon, 6 Sept 2021 at 10:10, Jeremie Le Hen wrote: > > Hey, > > I want to build -CURRENT again from sources. It's been a long time > since I hadn't done that. I'm shocked by the compilation time. > > I started the whole thing on Friday night and Monday morning it's > still in stage 4.2 (building libraries). Through occasional glancing > at the screen over the weekend, it seems obvious to me that the > compilation time is utterly dominated by LLVM. Compiling C++ seems > extremely CPU heavy and this is made worse by the fact LLVM is built > twice (once for build/cross tools, once for the actual world). > > So OK, my CPU is not the most powerful out there but it's still decent [1]. > > So I have a couple of questions coming to my mind: > 1. Is there any optimization I could benefit from? (I'm sure there's a > knob to use the existing compiler instead of building a > cross-compiler.) > 2. More generally, isn't this compilation time not considered as a > problem for developers? This seems to terribly slow down the iteration > time for people working on the build system. I wouldn't be surprised > if this drove people away from working on/improving that area. > > [1] > https://www.cpubenchmark.net/cpu.php?cpu=Intel+Core+i5-6260U+%40+1.80GHz=2671 Hi, I ran buildworld yesterday on a 3 CPU VM (AMD EPYC 2.4GHz) and the runtime was 2h. We spent most of the time in "stage 4.2: building libraries", in my case 62% of the CPU time and 75% of the real time. I guess the build time on your laptop should be around 6 hours if everything is ok. time make -j $(sysctl -n hw.ncpu) buildworld > log.buildworld 2>&1 tail -n 5 log.buildworld -- >>> World build completed on Sat Sep 4 20:58:00 UTC 2021 >>> World built in 7235 seconds, ncpu: 3, make -j3 -- 7235.61 real 20527.30 user 915.88 sys egrep '>>> stage| real ' log.buildworld >>> stage 1.1: legacy release compatibility shims 0.28 real 0.18 user 0.10 sys >>> stage 1.2: bootstrap tools 165.99 real 472.58 user11.56 sys >>> stage 2.1: cleaning up the object tree 21.47 real36.96 user14.14 sys 15.87 real29.14 user11.87 sys >>> stage 2.3: build tools 2.42 real 3.79 user 0.62 sys >>> stage 3: cross tools 9.92 real18.49 user 1.75 sys >>> stage 3.1: recording build metadata 0.07 real 0.01 user 0.06 sys >>> stage 4.1: building includes 16.62 real36.46 user 9.48 sys >>> stage 4.2: building libraries 5440.89 real 15724.60 user 482.58 sys >>> stage 4.3: building lib32 shim libraries 615.91 real 1654.77 user 164.58 sys >>> stage 4.4: building everything 937.23 real 2540.06 user 205.47 sys -Wolfram -- Wolfram Schneider https://wolfram.schneider.org
Re: -CURRENT compilation time
On Mon, 6 Sep 2021 10:43:06 +0100, David Chisnall wrote: > On 06/09/2021 09:08, Jeremie Le Hen wrote: > > Compiling C++ seems > > extremely CPU heavy and this is made worse by the fact LLVM is built > > twice (once for build/cross tools, once for the actual world). > > Note that you need to build LLVM twice only if you are actively > debugging LLVM reproduceable deployment images. You actually don't need > to build it at all, you can use an external toolchain to skip the first > build and you can compile WITHOUT_TOOLCHAIN to avoid building the > version that's installed and then install a toolchain from packages: > > https://wiki.freebsd.org/ExternalToolchain > > David If only that suggestion was topmost in UPDATING and repeated in each security advisory...
Re: -CURRENT compilation time
On 06/09/2021 09:08, Jeremie Le Hen wrote: Compiling C++ seems extremely CPU heavy and this is made worse by the fact LLVM is built twice (once for build/cross tools, once for the actual world). Note that you need to build LLVM twice only if you are actively debugging LLVM reproduceable deployment images. You actually don't need to build it at all, you can use an external toolchain to skip the first build and you can compile WITHOUT_TOOLCHAIN to avoid building the version that's installed and then install a toolchain from packages: https://wiki.freebsd.org/ExternalToolchain David
Re: -CURRENT compilation time
On 06/09/21 10:08, Jeremie Le Hen wrote: Hey, I want to build -CURRENT again from sources. It's been a long time since I hadn't done that. I'm shocked by the compilation time. I started the whole thing on Friday night and Monday morning it's still in stage 4.2 (building libraries). Through occasional glancing at the screen over the weekend, it seems obvious to me that the compilation time is utterly dominated by LLVM. Compiling C++ seems extremely CPU heavy and this is made worse by the fact LLVM is built twice (once for build/cross tools, once for the actual world). So OK, my CPU is not the most powerful out there but it's still decent [1]. So I have a couple of questions coming to my mind: 1. Is there any optimization I could benefit from? (I'm sure there's a knob to use the existing compiler instead of building a cross-compiler.) I'm routinely compiling head once a month or so on an "i7-6700 CPU @ 3.40GHz" (from dmesg), slightly more powerful than an i5 but this is a relatively old one so not top notch anymore. It usually takes less than 4 hours. I also build a NanoBSD image from scratch from time to time to an even older i5, which takes a little longer, but always under 6 hours, so the build times you report look anomalous. So as already suggested make sure yiu are using parallel make jobs (-j option to make) and memory is large enough (I think you need at least 2 GiB for make job is the minimum to not risk thrashing. You should really enable meta mode (look for WITH_META_MODE in src.conf(5)). It will not help the first time but will help a lot in future recompilations with an already populated /usr/obj. You could also investigate using ccache, which again will only help for successive rebuilds, and will consume a fair amount of disk space. Another consideration, my builds are happening on SSD disks, with swap on SSD, if you have everything on spinning media that is also a slowing factor. If you have plenty of ram you could build in ram, which is what I'm doing with poudriere for ports and it really speeds things up (even SSD disks tend to get slower when hit with a constant high load of mixed read/write accesses, which poudriere with parallel builds sometime causes) Hope this helps! -- Guido Falsi
Re: -CURRENT compilation time
Jeremie, a few observations (from the POV of someone who builds current ~ once a month) On Mon, Sep 6, 2021 at 10:09 AM Jeremie Le Hen wrote: > Hey, > > I want to build -CURRENT again from sources. It's been a long time > since I hadn't done that. I'm shocked by the compilation time. > > I started the whole thing on Friday night and Monday morning it's > still in stage 4.2 (building libraries). Through occasional glancing > at the screen over the weekend, it seems obvious to me that the > compilation time is utterly dominated by LLVM. Do you actually measure anything? have you looked into what your I/O is doing? How about swap? Compiling C++ seems > extremely CPU heavy and this is made worse by the fact LLVM is built > twice (once for build/cross tools, once for the actual world). > > So OK, my CPU is not the most powerful out there but it's still decent [1]. > I didn't check the specs: how many cores does your box have, and how many are you actually using (-j N)? IME, htop gives a good idea of how busy the CPUs really are. > So I have a couple of questions coming to my mind: > 1. Is there any optimization I could benefit from? (I'm sure there's a > knob to use the existing compiler instead of building a > cross-compiler.) > look at the build man page (https://www.freebsd.org/cgi/man.cgi?build(7)) for some tips on how to configure your environment. 2. More generally, isn't this compilation time not considered as a > problem for developers? This seems to terribly slow down the iteration > time for people working on the build system. I wouldn't be surprised > if this drove people away from working on/improving that area. > > [1] > https://www.cpubenchmark.net/cpu.php?cpu=Intel+Core+i5-6260U+%40+1.80GHz=2671 > > Cheers, > -- > Jeremie Le Hen > j...@freebsd.org > > regards Michael -- Michael Schuster http://recursiveramblings.wordpress.com/ recursion, n: see 'recursion'
-CURRENT compilation time
Hey, I want to build -CURRENT again from sources. It's been a long time since I hadn't done that. I'm shocked by the compilation time. I started the whole thing on Friday night and Monday morning it's still in stage 4.2 (building libraries). Through occasional glancing at the screen over the weekend, it seems obvious to me that the compilation time is utterly dominated by LLVM. Compiling C++ seems extremely CPU heavy and this is made worse by the fact LLVM is built twice (once for build/cross tools, once for the actual world). So OK, my CPU is not the most powerful out there but it's still decent [1]. So I have a couple of questions coming to my mind: 1. Is there any optimization I could benefit from? (I'm sure there's a knob to use the existing compiler instead of building a cross-compiler.) 2. More generally, isn't this compilation time not considered as a problem for developers? This seems to terribly slow down the iteration time for people working on the build system. I wouldn't be surprised if this drove people away from working on/improving that area. [1] https://www.cpubenchmark.net/cpu.php?cpu=Intel+Core+i5-6260U+%40+1.80GHz=2671 Cheers, -- Jeremie Le Hen j...@freebsd.org
