This is awesome - I know it took a lot of work (your section on bisection briefly alluded to that), so thanks for doing all of this - very, very useful
On Thu, 14 Nov 2019 at 11:27, Andreas Gustafsson <g...@gson.org> wrote: > > Hi all, > > Back in September, I wrote: > > I'm trying to run a bisection to determine why builds hosted on recent > > versions of NetBSD seem to be taking significantly more system time > > than they used to, building the same thing. > > I finally have some results to report. These are from builds of the > NetBSD-8/amd64 release hosted on various versions of -current/amd64, > on a HP DL360 G7 with dual Xeon L5630 CPUs (8 cores in all). The > amount of system time taken by each build was measured using time(1). > > Between a -current from September 2016 and one from October 2019, the > system time more than doubled, from 4245 seconds to 9344 seconds. > The time(1) output from the oldest and newest version was: > > 3930.86 real 15737.04 user 4245.26 sys > 4461.47 real 16687.37 user 9344.68 sys > > This means that on the recent -current, on average, roughly four of > the eight cores were executing the build tools (compilers, etc), > roughly two were executing the kernel, and the remaining two were > presumably idle. > > The increase did not happen all at once but in several smaller steps > as shown in this graph: > > http://www.gson.org/netbsd/bugs/system-time/graph.png > > For each step, finding the commits that caused it required a separate > bisection. Each bisection took 1-2 days to run, so I have only > bisected the largest steps, those of 5 percent or more. They are > listed below in order from largest to smallest, with CVS revisions > and commit messages. > > 38% increase: > > 2018.04.04.12.59.49 maxv src/sys/arch/amd64/amd64/machdep.c 1.303 > 2018.04.04.12.59.49 maxv src/sys/arch/x86/include/cpu.h 1.91 > 2018.04.04.12.59.49 maxv src/sys/arch/x86/x86/cpu.c 1.154 > 2018.04.04.12.59.49 maxv src/sys/arch/x86/x86/spectre.c 1.8 > > Enable the SpectreV2 mitigation by default at boot time. > > 12% increase: > > 2019.03.08.20.35.10 christos src/share/mk/bsd.own.mk 1.1108 > > Back to using jemalloc for x86_64; all problems have been resolved. > > 9% increase: > > 2018.02.26.05.52.50 maxv src/sys/arch/amd64/conf/GENERIC 1.485 > > Enable SVS by default. > > 7% increase: > > 2016.12.14.15.49.35 hannken src/sys/kern/vfs_vnode.c 1.63 > > Change the freelists to lrulists, all vnodes are always on one > of the lists. Speeds up namei on cached vnodes by ~3 percent. > > Merge "vrele_thread" into "vdrain_thread" so we have one thread > working on the lrulists. Adapt vfs_drainvnodes() to always wait > for a complete cycle of vdrain_thread(). > > 5% increase: > > 2018.04.07.22.39.31 christos src/external/Makefile 1.21 > 2018.04.07.22.39.31 christos src/external/README 1.16 > [302 more revisions by christos elided] > 2018.04.07.22.39.53 christos src/external/bsd/Makefile 1.59 > 2018.04.07.22.41.55 christos src/doc/3RDPARTY 1.1515 > 2018.04.07.22.41.55 christos src/doc/CHANGES 1.2376 > 2018.04.08.00.52.38 mrg src/sys/arch/amd64/conf/ALL 1.85 > 2018.04.08.00.52.38 mrg src/sys/arch/amd64/conf/GENERIC 1.489 > 2018.04.08.00.52.38 mrg src/sys/arch/i386/conf/ALL 1.437 > 2018.04.08.00.52.38 mrg src/sys/arch/i386/conf/GENERIC 1.1177 > 2018.04.08.01.30.01 christos src/external/mpl/Makefile 1.1 > > [Too many commit messages to list here, but the following from > mrg's commit of src/sys/arch/amd64/conf/GENERIC 1.489 may > be relevant] > > turn on GCC spectre v2 mitigation options. > > 5% increase: > > 2019.03.10.15.32.42 christos > src/external/bsd/jemalloc/lib/Makefile.inc 1.5 > > turn on debugging to help find problems > > 5% decrease: > > 2019.07.23.06.31.20 martin src/external/bsd/jemalloc/lib/Makefile.inc > 1.10 > > Disable JEMALLOC_DEBUG, it served us well, but now we want performance > back. Discussed with christos. > > To summarize, most of the increase was due to Spectre and Meltdown > mitigations, which I guess is not really surprising. But the 12% net > increase from jemalloc and the 7% increase from vfs_vnode.c 1.63 seem > to call for closer investigation. > -- > Andreas Gustafsson, g...@gson.org > >
