Re: Building clang in buildworld as part of the bootstrap process -- is it really necessary?
On Sep 6, 2014, at 5:32 AM, Dimitry Andric wrote: > On 06 Sep 2014, at 05:16, Warner Losh wrote: >> >> On Sep 5, 2014, at 8:21 PM, Garrett Cooper wrote: >>> >>> One of the questions that came up from a co-worker is "why do I >>> need to build clang in buildworld if I already installed it from >>> ports"? I could see some valid reasons for doing this (one needs a >>> cross-compiler, one might need specific options that might not be set >>> in the ports version), but for native builds I would tend to agree >>> with this logic. > > For one, the base version of clang has a number of patches which haven't > yet been sent upstream (and might never be). I see the port already has > a few of them, but certainly not all. > > That said, the ultimate goal is obviously to be able to build world with > a stock version of an external compiler, be it clang or gcc. There is > still quite a lot to be done to make that possible. Yes. Some use cases work, others don’t. > >>> With gcc it was wasteful building the compiler each >>> buildworld, but with clang it seems annoying continuing on this path >>> because the compile takes a long time to complete. >> >> The clang built during buildworld is used to bootstrap. So it is required >> sometimes. Usually when there’s a binary compatibility issue, which is rare >> but does happen. > > We already do something similar with BOOTSTRAPPING, where we attempt to > detect which build tools on the host system are too old, and build them > accordingly. I think something like this might also be possible for the > toolchain components, for example by using the __FreeBSD_cc_version > built-in define (which is also in our gcc, but it doesn't seem to be > used very often). Or some other system entirely. We’ve avoided playing the “optimize the build” game for buildworld. while we do make some exceptions for minor, ancillary tools, we always assume a full build will always work, and that’s the only way the project supports people. >> It is also installed as cc. > > That's actually another copy, the one built during the later stages. It > might not even run on the host system. :) True. My point in mentioning this is that you don’t get cc unless you build clang or gcc in base. > >> The ports clang may or may not build the world. However, if you want to say >> “I know what I’m doing” you can set WITHOUT_CLANG_BOOTSTRAP and >> WITHOUT_GCC_BOOTSTRAP to tell the build to do no building of bootstrap tools >> and to use the host’s instead. This usually works, but may fail from time to >> time with port-built compilers. >> >> If you don’t want buildworld to build any compiler, you can add >> WITHOUT_CLANG=t and WITHOUT_GCC=t. This will create the system without any >> compilers. You’ll need to set CC to /usr/local/bin/clang35 or whatever as >> well. There may be other settings you need as well. >> >>> Alternatively, would anyone be opposed to adding some logic to >>> automatically bypass the bootstrap compiler, i.e. add some logic to >>> Makefile.inc1 that would skip compiling clang/gcc if and only if the >>> target triplet and version met some required values? >> >> There’s enough violent opposition to this that it will never happen. >> buildworld is supposed to always be safe. Don’t mess with that. It isn’t >> designed to be fast. > > Yup, though in most cases, it should be sufficient to do an incremental > buildworld instead of a "full" one. This would also prevent rebuilding > any part of llvm and/or clang, as long as no changes occur in them. buildwolrd -DNO_CLEAN already does that... > For example, the only recent change to clang was in one of the ARM > target's .td files (to fix a problem with movw being sometimes emitted > on ARMv6). In this case, only a bunch of .inc files will be regenerated > and some of the files under lib/clang/libllvmarm* will be rebuilt, but > certainly not the entire llvm and clang codebase. > > I would really like for incremental buildworld to become more robust. :) I would too. However, every time clang gets upgraded, it clang seems to break the no_clean case :( There are also other offenders, but we’ve not had smooth sailing in the upgrade with NO_CLEAN stuff for a long time, hence my caution. jbuild shoed a lot of promise. We need to get that done if we want to get away from all this crazy. Warner signature.asc Description: Message signed with OpenPGP using GPGMail
Re: Building clang in buildworld as part of the bootstrap process -- is it really necessary?
On Sep 6, 2014, at 5:28 AM, David Chisnall wrote: > On 6 Sep 2014, at 06:47, Garrett Cooper wrote: > >> Makes sense. I'll do some poking around and see if things could potentially >> be optimized with the clang build. On beefy machines it's not a big deal, >> but as we know on machines without a ton of memory or SSDs, it can become >> painful, as expected. > > The build system for clang-in-base has improved a bit so, as you say, it is > now reasonably fast on beefy machines (release clang build with the upstream > build system takes 2-3 minutes on a fast machine, about 10 on my laptop, the > one in buildworld isn't quite as good at extracting parallelism). On slow > machines, it can be quite painful. Yes. Hence the ask for a faster way to rebuild the system. > The correct solution to this problem is likely to be to start creating > bootstrap-toolchain packages. This is also likely to be necessary for > architectures like MIPS and PowerPC before 11 anyway, because the host > compiler doesn't have the C++11 support required for bootstrapping a newer > LLVM and clang. We can work around that quite easily if we have package for > the bootstrap toolchain (possibly cross-compiled from an x86 machine). I’m not sure which problem this solves. If you are cross building, then you already are likely running on amd64 which has a good compiler. If you are doing a native build, then everybody but mips and armeb already have clang support in the base. For those what you do with clang is irrelevant, so long as gcc is around. When gcc is kicked out of the base, we’ll need to have an external toolchain story for those platforms anyway. That story might include generation of binary packages, but it will also need to include building of those packages easily. Likely without buildworld, or a modified variant that groks how to bootstrap a sysroot. Cross building for ports is getting close, we may be able to use it to generate packages for nonx86 faster than native (recent successful full builds are taking only 5x as long). I guess I’m saying that while packages may be necessary, they aren’t sufficient unless they are easily reproduced. All this also needs to be well documented, or we’re going to go off the rails trying to support it. Warner signature.asc Description: Message signed with OpenPGP using GPGMail
Re: Building clang in buildworld as part of the bootstrap process -- is it really necessary?
On 6 Sep 2014, at 06:47, Garrett Cooper wrote: > Makes sense. I'll do some poking around and see if things could potentially > be optimized with the clang build. On beefy machines it's not a big deal, but > as we know on machines without a ton of memory or SSDs, it can become > painful, as expected. The build system for clang-in-base has improved a bit so, as you say, it is now reasonably fast on beefy machines (release clang build with the upstream build system takes 2-3 minutes on a fast machine, about 10 on my laptop, the one in buildworld isn't quite as good at extracting parallelism). On slow machines, it can be quite painful. The correct solution to this problem is likely to be to start creating bootstrap-toolchain packages. This is also likely to be necessary for architectures like MIPS and PowerPC before 11 anyway, because the host compiler doesn't have the C++11 support required for bootstrapping a newer LLVM and clang. We can work around that quite easily if we have package for the bootstrap toolchain (possibly cross-compiled from an x86 machine). David ___ freebsd-toolchain@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"
Re: Building clang in buildworld as part of the bootstrap process -- is it really necessary?
On 06 Sep 2014, at 05:16, Warner Losh wrote: > > On Sep 5, 2014, at 8:21 PM, Garrett Cooper wrote: >> >> One of the questions that came up from a co-worker is "why do I >> need to build clang in buildworld if I already installed it from >> ports"? I could see some valid reasons for doing this (one needs a >> cross-compiler, one might need specific options that might not be set >> in the ports version), but for native builds I would tend to agree >> with this logic. For one, the base version of clang has a number of patches which haven't yet been sent upstream (and might never be). I see the port already has a few of them, but certainly not all. That said, the ultimate goal is obviously to be able to build world with a stock version of an external compiler, be it clang or gcc. There is still quite a lot to be done to make that possible. >> With gcc it was wasteful building the compiler each >> buildworld, but with clang it seems annoying continuing on this path >> because the compile takes a long time to complete. > > The clang built during buildworld is used to bootstrap. So it is required > sometimes. Usually when there’s a binary compatibility issue, which is rare > but does happen. We already do something similar with BOOTSTRAPPING, where we attempt to detect which build tools on the host system are too old, and build them accordingly. I think something like this might also be possible for the toolchain components, for example by using the __FreeBSD_cc_version built-in define (which is also in our gcc, but it doesn't seem to be used very often). Or some other system entirely. > It is also installed as cc. That's actually another copy, the one built during the later stages. It might not even run on the host system. :) > The ports clang may or may not build the world. However, if you want to say > “I know what I’m doing” you can set WITHOUT_CLANG_BOOTSTRAP and > WITHOUT_GCC_BOOTSTRAP to tell the build to do no building of bootstrap tools > and to use the host’s instead. This usually works, but may fail from time to > time with port-built compilers. > > If you don’t want buildworld to build any compiler, you can add > WITHOUT_CLANG=t and WITHOUT_GCC=t. This will create the system without any > compilers. You’ll need to set CC to /usr/local/bin/clang35 or whatever as > well. There may be other settings you need as well. > >> Alternatively, would anyone be opposed to adding some logic to >> automatically bypass the bootstrap compiler, i.e. add some logic to >> Makefile.inc1 that would skip compiling clang/gcc if and only if the >> target triplet and version met some required values? > > There’s enough violent opposition to this that it will never happen. > buildworld is supposed to always be safe. Don’t mess with that. It isn’t > designed to be fast. Yup, though in most cases, it should be sufficient to do an incremental buildworld instead of a "full" one. This would also prevent rebuilding any part of llvm and/or clang, as long as no changes occur in them. For example, the only recent change to clang was in one of the ARM target's .td files (to fix a problem with movw being sometimes emitted on ARMv6). In this case, only a bunch of .inc files will be regenerated and some of the files under lib/clang/libllvmarm* will be rebuilt, but certainly not the entire llvm and clang codebase. I would really like for incremental buildworld to become more robust. :) -Dimitry signature.asc Description: Message signed with OpenPGP using GPGMail
Re: Building clang in buildworld as part of the bootstrap process -- is it really necessary?
> tl;dr: NO. Makes sense. I'll do some poking around and see if things could potentially be optimized with the clang build. On beefy machines it's not a big deal, but as we know on machines without a ton of memory or SSDs, it can become painful, as expected. Thanks :)! -Garrett ___ freebsd-toolchain@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"
Re: Building clang in buildworld as part of the bootstrap process -- is it really necessary?
On Sep 5, 2014, at 8:21 PM, Garrett Cooper wrote: > Hi all, >One of the questions that came up from a co-worker is "why do I > need to build clang in buildworld if I already installed it from > ports"? I could see some valid reasons for doing this (one needs a > cross-compiler, one might need specific options that might not be set > in the ports version), but for native builds I would tend to agree > with this logic. With gcc it was wasteful building the compiler each > buildworld, but with clang it seems annoying continuing on this path > because the compile takes a long time to complete. The clang built during buildworld is used to bootstrap. So it is required sometimes. Usually when there’s a binary compatibility issue, which is rare but does happen. It is also installed as cc. The ports clang may or may not build the world. However, if you want to say “I know what I’m doing” you can set WITHOUT_CLANG_BOOTSTRAP and WITHOUT_GCC_BOOTSTRAP to tell the build to do no building of bootstrap tools and to use the host’s instead. This usually works, but may fail from time to time with port-built compilers. If you don’t want buildworld to build any compiler, you can add WITHOUT_CLANG=t and WITHOUT_GCC=t. This will create the system without any compilers. You’ll need to set CC to /usr/local/bin/clang35 or whatever as well. There may be other settings you need as well. >Alternatively, would anyone be opposed to adding some logic to > automatically bypass the bootstrap compiler, i.e. add some logic to > Makefile.inc1 that would skip compiling clang/gcc if and only if the > target triplet and version met some required values? There’s enough violent opposition to this that it will never happen. buildworld is supposed to always be safe. Don’t mess with that. It isn’t designed to be fast. If you want fast, you can tell it to be fast with different options, but it will never be fast by default because we guessed that we think it might be safe. don’t mess with buildworld. There might be support for a go-fast catch-all flag that sets a bunch of other flags to make things go fast, but once you get down that rabbit hole you’ll find one man’s optimization is another woman’s intolerable omission. tl;dr: NO. Warner signature.asc Description: Message signed with OpenPGP using GPGMail
Building clang in buildworld as part of the bootstrap process -- is it really necessary?
Hi all, One of the questions that came up from a co-worker is "why do I need to build clang in buildworld if I already installed it from ports"? I could see some valid reasons for doing this (one needs a cross-compiler, one might need specific options that might not be set in the ports version), but for native builds I would tend to agree with this logic. With gcc it was wasteful building the compiler each buildworld, but with clang it seems annoying continuing on this path because the compile takes a long time to complete. Alternatively, would anyone be opposed to adding some logic to automatically bypass the bootstrap compiler, i.e. add some logic to Makefile.inc1 that would skip compiling clang/gcc if and only if the target triplet and version met some required values? Thanks! -Garrett ___ freebsd-toolchain@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"
