Re: Buildworld and buildkernel with very slow compilation, recently
On 6/22/20, Eugene Grosbein wrote: > 22.06.2020 3:56, Dimitry Andric wrote: > >> One of the upstream problems is that there is not really any >> authoritative performance regression log being built up, so it is hard >> to see where such regressions were introduced. Somebody then has to >> spend a lot of time tracking down each and every regression, and then >> attempt to untangle it from the dozens of commits made around the same >> time. :) > > Such project as FreeBSD could become valuable source of information to track > such an issue > and it's easy to obtain, for example: > > make buildworld 2>&1 | logger -t buildworld -p user.info > > This generates timestamps for every line called in process with help of > syslogd. > It should be easy to compare outputs obtained with same FreeBSD revision > but different clang versions, calculate time diffs and sort by diffs > descending :-) Eugene, I have noticed (though quite anecdotally) a real slowdown, so I'd like to help. My mule is not busy at this time so if you have a place to put these output files, I'll be glad to crunch some over the next few days. My mule is only an i3 with 4GB with 40G of swap and an active ccache, but I should be able to run any variant of tests you need to see. I'm on 12-STABLE and I now am doing a fresh base/stable/12 pull. I have _not_ deleted my /usr/src.bak copy, although I didn't stash a copy of the old ports tree before updating it. I use tcsh everywhere, but I should be able to do what you intend (hey, it's been a lng time since I tried to speak csh!) and if you'd like me to do deltas with LLVM80 vs LLVM90 vs. LLVM100, etc, I can do this. Just tell me exactly what you want to see. -- Don Wilde * What is the Internet of Things but a system * * of systems including humans? * ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Buildworld and buildkernel with very slow compilation, recently
22.06.2020 3:56, Dimitry Andric wrote: > One of the upstream problems is that there is not really any > authoritative performance regression log being built up, so it is hard > to see where such regressions were introduced. Somebody then has to > spend a lot of time tracking down each and every regression, and then > attempt to untangle it from the dozens of commits made around the same > time. :) Such project as FreeBSD could become valuable source of information to track such an issue and it's easy to obtain, for example: make buildworld 2>&1 | logger -t buildworld -p user.info This generates timestamps for every line called in process with help of syslogd. It should be easy to compare outputs obtained with same FreeBSD revision but different clang versions, calculate time diffs and sort by diffs descending :-) Same for ports builds, maybe even quicker. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Buildworld and buildkernel with very slow compilation, recently
Dimitry Andric wrote: > > On 21 Jun 2020, at 14:36, Chris Nehren wrote: >> On Sunday, June 21, 2020 8:11:15 AM EDT Michael Grimm wrote: >>> Hi, >>> >>> I am following FreeBSD 12.1-STABLE. >>> >>> Clang has been upgraded to version 10.0.0 on May, 1st, and ever since that >>> time, I do observe a dramatic increase in compilation times of building >>> world, kernel and ports. I didn't benchmark the exact times, but >>> compilation times are at least increased by a factor of 1.5. Nothing has >>> changed of the last month besides upgrading 12.1-Stable every other week. >>> >>> Has anyone else been bitten by this? >> >> I don't have measurements to corroborate this, but here's a mailing list >> thread where folks are talking about it (split across two URLs, the OP posted >> in April and then there was a followup in May): >> >> http://lists.llvm.org/pipermail/llvm-dev/2020-April/140938.html >> https://lists.llvm.org/pipermail/llvm-dev/2020-May/141482.html >> >> So there definitely seems to be *something* going on, and you are indeed not >> crazy. :) > > Indeed, there is some upstream discussion going on about this issue. > There are some scenarios where people see non-negligible performance > loss, but apparently not everybody suffers from it. If you build the > whole ports collection, it is rather likely you may bump into it. I have > personally not seen much performance difference in building world, > kernel etc. > > One of the upstream problems is that there is not really any > authoritative performance regression log being built up, so it is hard > to see where such regressions were introduced. Somebody then has to > spend a lot of time tracking down each and every regression, and then > attempt to untangle it from the dozens of commits made around the same > time. :) > > In any case, there is at least some attention on it now, so hopefully > this will improve again. I don't think such fixes will be trivial > though, so it is not likely they will land in 10.0.1. > > -Dimitry ok, so others see some performance loss as well. In the meantime I activated META_MODE and hope, that will speed-up my weekly compilations. Thanks to all who answered, and regards, Michael ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Buildworld and buildkernel with very slow compilation, recently
On 21 Jun 2020, at 14:36, Chris Nehren wrote: > > On Sunday, June 21, 2020 8:11:15 AM EDT Michael Grimm wrote: >> Hi, >> >> I am following FreeBSD 12.1-STABLE. >> >> Clang has been upgraded to version 10.0.0 on May, 1st, and ever since that >> time, I do observe a dramatic increase in compilation times of building >> world, kernel and ports. I didn't benchmark the exact times, but >> compilation times are at least increased by a factor of 1.5. Nothing has >> changed of the last month besides upgrading 12.1-Stable every other week. >> >> Has anyone else been bitten by this? > > I don't have measurements to corroborate this, but here's a mailing list > thread where folks are talking about it (split across two URLs, the OP posted > in April and then there was a followup in May): > > http://lists.llvm.org/pipermail/llvm-dev/2020-April/140938.html > https://lists.llvm.org/pipermail/llvm-dev/2020-May/141482.html > > So there definitely seems to be *something* going on, and you are indeed not > crazy. :) Indeed, there is some upstream discussion going on about this issue. There are some scenarios where people see non-negligible performance loss, but apparently not everybody suffers from it. If you build the whole ports collection, it is rather likely you may bump into it. I have personally not seen much performance difference in building world, kernel etc. One of the upstream problems is that there is not really any authoritative performance regression log being built up, so it is hard to see where such regressions were introduced. Somebody then has to spend a lot of time tracking down each and every regression, and then attempt to untangle it from the dozens of commits made around the same time. :) In any case, there is at least some attention on it now, so hopefully this will improve again. I don't think such fixes will be trivial though, so it is not likely they will land in 10.0.1. -Dimitry signature.asc Description: Message signed with OpenPGP
Re: Buildworld and buildkernel with very slow compilation, recently
On Sun, Jun 21, 2020 at 02:48:51PM +0200, Ronald Klop wrote: > Building ports/pkgs is also significantly slower on 13 (new clang) than on > 12.1. > http://thunderx1.nyi.freebsd.org/ > 13 = 140 hours12.1 = 103 hoursFor roughly the same amount of ports. If you use stock HEAD GENERIC kernel config and stock-built libc, it is expected. Both kernel and jemalloc has extensive debugging turned on, on HEAD. > > Regards, Ronald > > Van: Michael Grimm > Datum: 21 juni 2020 14:12 > Aan: FreeBSD-STABLE Mailing List > Onderwerp: Buildworld and buildkernel with very slow compilation, recently > > > > > > > Hi, > > > > I am following FreeBSD 12.1-STABLE. > > > > Clang has been upgraded to version 10.0.0 on May, 1st, and ever since that > > time, I do observe a dramatic increase in compilation times of building > > world, kernel and ports. I didn't benchmark the exact times, but > > compilation times are at least increased by a factor of 1.5. Nothing has > > changed of the last month besides upgrading 12.1-Stable every other week. > > > > Has anyone else been bitten by this? > > > > Regards, > > Michael > > ___ > > freebsd-stable@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > > To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" > > > > > > > > > ___ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Buildworld and buildkernel with very slow compilation, recently
From: Ronald Klop Subject: Re: Buildworld and buildkernel with very slow compilation, recently Date: Sun, 21 Jun 2020 14:48:51 +0200 (CEST) > Building ports/pkgs is also significantly slower on 13 (new clang) > than on 12.1. > http://thunderx1.nyi.freebsd.org/ > 13 = 140 hours12.1 = 103 hoursFor roughly the same amount of ports. >From /usr/src/UPDATING of head: NOTE TO PEOPLE WHO THINK THAT FreeBSD 13.x IS SLOW: FreeBSD 13.x has many debugging features turned on, in both the kernel and userland. These features attempt to detect incorrect use of system primitives, and encourage loud failure through extra sanity checking and fail stop semantics. They also substantially impact system performance. If you want to do performance measurement, benchmarking, and optimization, you'll want to turn them off. This includes various WITNESS- related kernel options, INVARIANTS, malloc debugging flags in userland, and various verbose features in the kernel. Many developers choose to disable these features on build machines to maximize performance. (To completely disable malloc debugging, define MALLOC_PRODUCTION in /etc/make.conf, or to merely disable the most expensive debugging functionality run "ln -s 'abort:false,junk:false' /etc/malloc.conf".) --- Yasuhiro KIMURA ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Buildworld and buildkernel with very slow compilation, recently
Building ports/pkgs is also significantly slower on 13 (new clang) than on 12.1. http://thunderx1.nyi.freebsd.org/ 13 = 140 hours12.1 = 103 hoursFor roughly the same amount of ports. Regards, Ronald Van: Michael Grimm Datum: 21 juni 2020 14:12 Aan: FreeBSD-STABLE Mailing List Onderwerp: Buildworld and buildkernel with very slow compilation, recently Hi, I am following FreeBSD 12.1-STABLE. Clang has been upgraded to version 10.0.0 on May, 1st, and ever since that time, I do observe a dramatic increase in compilation times of building world, kernel and ports. I didn't benchmark the exact times, but compilation times are at least increased by a factor of 1.5. Nothing has changed of the last month besides upgrading 12.1-Stable every other week. Has anyone else been bitten by this? Regards, Michael ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Buildworld and buildkernel with very slow compilation, recently
On Sunday, June 21, 2020 8:11:15 AM EDT Michael Grimm wrote: > Hi, > > I am following FreeBSD 12.1-STABLE. > > Clang has been upgraded to version 10.0.0 on May, 1st, and ever since that > time, I do observe a dramatic increase in compilation times of building > world, kernel and ports. I didn't benchmark the exact times, but > compilation times are at least increased by a factor of 1.5. Nothing has > changed of the last month besides upgrading 12.1-Stable every other week. > > Has anyone else been bitten by this? I don't have measurements to corroborate this, but here's a mailing list thread where folks are talking about it (split across two URLs, the OP posted in April and then there was a followup in May): http://lists.llvm.org/pipermail/llvm-dev/2020-April/140938.html https://lists.llvm.org/pipermail/llvm-dev/2020-May/141482.html So there definitely seems to be *something* going on, and you are indeed not crazy. :) ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Buildworld and buildkernel with very slow compilation, recently
Michael Grimm wrote: Hi, I am following FreeBSD 12.1-STABLE. Clang has been upgraded to version 10.0.0 on May, 1st, and ever since that time, I do observe a dramatic increase in compilation times of building world, kernel and ports. I didn't benchmark the exact times, but compilation times are at least increased by a factor of 1.5. Nothing has changed of the last month besides upgrading 12.1-Stable every other week. Not seeing this after clang update to 10 on both stable/12 and head. I'd check if you are excessively swapping now as (not sure about that, just guessing) newer clang/llvm could consume more memory during compilation, and possibly reducing the make jobs number. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Buildworld and buildkernel with very slow compilation, recently
Hi, I am following FreeBSD 12.1-STABLE. Clang has been upgraded to version 10.0.0 on May, 1st, and ever since that time, I do observe a dramatic increase in compilation times of building world, kernel and ports. I didn't benchmark the exact times, but compilation times are at least increased by a factor of 1.5. Nothing has changed of the last month besides upgrading 12.1-Stable every other week. Has anyone else been bitten by this? Regards, Michael ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
