On Tue, Jan 01, 2013 at 05:58:06PM +0200, Konstantin Belousov wrote: > On Tue, Jan 01, 2013 at 02:39:44PM +0100, Dominic Fandrey wrote: > > On 01/01/2013 07:51, Konstantin Belousov wrote: > > > On Tue, Jan 01, 2013 at 02:05:11AM +0100, Dominic Fandrey wrote: > > >> On 01/01/2013 01:49, Dominic Fandrey wrote: > > >>> On 01/01/2013 01:29, Chris Rees wrote: > > >>>> On 1 Jan 2013 00:01, "Dominic Fandrey" <kamik...@bsdforen.de> wrote: > > >>>>> > > >>>>> I have a Tinderbox that I just updated to the current RELENG_9. > > >>>>> Following the update build times for packages have increased by a > > >>>>> factor between 5 and 20. I.e. I have packages that used to build in > > >>>>> 5 minutes and now take an hour. > > >>>>> > > >>>>> I'm suspecting the file system ever since I saw that the majority of > > >>>>> CPU > > >>>>> load was caused by ls when I looked at top (more than 2 minutes of CPU > > >>>>> time were counted that moment). The majority of the time most of the > > >>>>> CPU > > >>>>> load is caused by bsdtar, pkg_add, qmake-qt4, etc. Without exception > > >>>>> tools that access a lot of files. > > >>>>> > > >>>>> The file system on which packages are built is nullfs mounted from > > >>>>> an async mounted UFS. I turned async off, to no avail. > > >>>>> > > >>>>> /usr/src/UPDATING says that there were nullfs optimisations. So I > > >>>>> think this is where the problem originates. I might hack the > > >>>>> tinderbox to > > >>>>> use 'ln -s' or set it up for NFS to verify this. > > >>>> > > >>>> Is your kernel newer than the Jail? The converse causes problems. > > >>> > > >>> I ran makeJail for all jails after updating. > Did you rebuild your modules together with the new kernel ? > > > >>> > > >>> I also seem to have similar problems when building in the host-system. > > >>> The unzip for openjdk-7 has just passed the 11 minutes CPU time mark. > > >>> On my notebook it takes less than 10 seconds. > > >> > > >> Just set WRKOBJDIRPREFIX to a tmpfs on the Tinderbox host system > > >> and the extract takes less than a second. Originally WRKOBJDIRPREFIX > > >> also pointed to a nullfs mount. > > >> > > >> Afterwards I pointed WRKOBJDIRPREFIX to a UFS file system (without > > >> nullfs involvement). The entire make extract took 20s. > > >> > > >> So still faster by at least factor 30 than running it on a nullfs mount > > >> (I eventually SIGINTed so I don't know how long it would've run). > > > > > > Start providing some useful debugging information ? > > > > That one might be interesting. It's all system time: > > > > # time -lh make extract > > ===> License GPLv2 accepted by the user > > ===> Found saved configuration for openjdk-7.9.05_1 > > ===> Extracting for openjdk-7.9.05_2 > > => SHA256 Checksum OK for openjdk-7u6-fcs-src-b24-09_aug_2012.zip. > > => SHA256 Checksum OK for apache-ant-1.8.4-bin.zip. > > ===> openjdk-7.9.05_2 depends on file: /usr/local/bin/unzip - found > > ^Ctime: command terminated abnormally > > 4m29.30s real 3.03s user 4m22.55s sys > > 5008 maximum resident set size > > 135 average shared memory size > > 2932 average unshared data size > > 127 average unshared stack size > > 7772 page reclaims > > 0 page faults > > 0 swaps > > 19 block input operations > > 101 block output operations > > 0 messages sent > > 0 messages received > > 41 signals received > > 1597 voluntary context switches > > 16590 involuntary context switches > > Ok, from your mount -v output, are the three nullfs mounts the only > nullfs mount ever used ? > > Is it only unzip which demostrates the silly behaviour ? Or does it > happen with any program ? E.g., does ls(1) or sha1 on the nullfs mount > also slow ? > > Could you try some low-tech profiling on the slow program. For instance, > you could run ktrace/kdump -R to see which syscalls are slow. > > Most darkly part of your report for me, is that I also use nullfs-backed > jails both on HEAD and stable/9, with bigger scale, and I do not have > an issue. I just did > pooma32% time unzip -q > /usr/local/arch/freebsd/distfiles/openjdk-7u6-fcs-src-b24-09_aug_2012.zip > unzip -q 3.25s user 23.77s system 78% cpu 34.482 total > over nullfs mount of > /usr/home on /usr/sfw/local8/opt/pooma32/usr/home (nullfs, local). > > Please try the following patch, which changes nullfs behaviour to be > non-cached by default. You could turn on the caching with the 'mount -t > nullfs -o cache from to' mounting command. I am interested if use/non-use > of -o cache makes a difference for you.
Ping. Any update ?
pgptVju7RyzVW.pgp
Description: PGP signature