Automated report: NetBSD-current/i386 build success
The NetBSD-current/i386 build is working again. The following commits were made between the last failed build and the successful build: 2022.07.02.04.37.15 skrll src/sys/external/bsd/drm2/include/drm/drm_gem_cma_helper.h,v 1.4 Logs can be found at: http://releng.NetBSD.org/b5reports/i386/commits-2022.07.html#2022.07.02.04.37.15
daily CVS update output
Updating src tree: P src/sbin/shutdown/shutdown.8 P src/sbin/shutdown/shutdown.c P src/sys/external/bsd/drm2/drm/drm_gem_cma_helper.c P src/sys/external/bsd/drm2/include/drm/drm_gem_cma_helper.h P src/sys/kern/kern_fork.c P src/sys/kern/kern_time.c P src/sys/kern/uipc_syscalls.c P src/sys/net/rtsock_shared.c U src/tests/usr.bin/xlint/lint1/msg_138.c P src/tests/usr.bin/xlint/lint1/msg_240.c P src/usr.bin/xlint/lint1/err.c P src/usr.bin/xlint/lint1/externs1.h P src/usr.bin/xlint/lint1/lint1.h P src/usr.bin/xlint/lint1/main1.c P src/usr.bin/xlint/lint1/tree.c Updating xsrc tree: Killing core files: Updating tar files: src/top-level: collecting... replacing... done src/bin: collecting... replacing... done src/common: collecting... replacing... done src/compat: collecting... replacing... done src/crypto: collecting... replacing... done src/dist: collecting... replacing... done src/distrib: collecting... replacing... done src/doc: collecting... replacing... done src/etc: collecting... replacing... done src/external: collecting... replacing... done src/extsrc: collecting... replacing... done src/games: collecting... replacing... done src/include: collecting... replacing... done src/lib: collecting... replacing... done src/libexec: collecting... replacing... done src/regress: collecting... replacing... done src/rescue: collecting... replacing... done src/sbin: collecting... replacing... done src/share: collecting... replacing... done src/sys: collecting... replacing... done src/tests: collecting... replacing... done src/tools: collecting... replacing... done src/usr.bin: collecting... replacing... done src/usr.sbin: collecting... replacing... done src/config: collecting... replacing... done src: collecting... replacing... done xsrc/top-level: collecting... replacing... done xsrc/external: collecting... replacing... done xsrc/local: collecting... replacing... done xsrc: collecting... replacing... done Updating release-8 src tree (netbsd-8): Updating release-8 xsrc tree (netbsd-8): Updating release-8 tar files: src/top-level: collecting... replacing... done src/bin: collecting... replacing... done src/common: collecting... replacing... done src/compat: collecting... replacing... done src/crypto: collecting... replacing... done src/dist: collecting... replacing... done src/distrib: collecting... replacing... done src/doc: collecting... replacing... done src/etc: collecting... replacing... done src/external: collecting... replacing... done src/extsrc: collecting... replacing... done src/games: collecting... replacing... done src/include: collecting... replacing... done src/lib: collecting... replacing... done src/libexec: collecting... replacing... done src/regress: collecting... replacing... done src/rescue: collecting... replacing... done src/sbin: collecting... replacing... done src/share: collecting... replacing... done src/sys: collecting... replacing... done src/tests: collecting... replacing... done src/tools: collecting... replacing... done src/usr.bin: collecting... replacing... done src/usr.sbin: collecting... replacing... done src/config: collecting... replacing... done src: collecting... replacing... done xsrc/top-level: collecting... replacing... done xsrc/external: collecting... replacing... done xsrc/local: collecting... replacing... done xsrc: collecting... replacing... done Updating release-9 src tree (netbsd-9): Updating release-9 xsrc tree (netbsd-9): Updating release-9 tar files: src/top-level: collecting... replacing... done src/bin: collecting... replacing... done src/common: collecting... replacing... done src/compat: collecting... replacing... done src/crypto: collecting... replacing... done src/dist: collecting... replacing... done src/distrib: collecting... replacing... done src/doc: collecting... replacing... done src/etc: collecting... replacing... done src/external: collecting... replacing... done src/extsrc: collecting... replacing... done src/games: collecting... replacing... done src/include: collecting... replacing... done src/lib: collecting... replacing... done src/libexec: collecting... replacing... done src/regress: collecting... replacing... done src/rescue: collecting... replacing... done src/sbin: collecting... replacing... done src/share: collecting... replacing... done src/sys: collecting... replacing... done src/tests: collecting... replacing... done src/tools: collecting... replacing... done src/usr.bin: collecting... replacing... done src/usr.sbin: collecting... replacing... done src/config: collecting... replacing... done src: collecting... replacing... done xsrc/top-level: collecting... replacing... done xsrc/external: collecting... replacing... done xsrc/local: collecting... replacing... done xsrc: collecting... replacing... done Updating file list: -rw-rw-r-- 1 srcmastr netbsd 38913077 Jul 2 03:32 ls-lRA.gz
Automated report: NetBSD-current/i386 build failure
This is an automatically generated notice of a NetBSD-current/i386 build failure. The failure occurred on babylon5.netbsd.org, a NetBSD/amd64 host, using sources from CVS date 2022.07.02.00.26.07. An extract from the build.sh output follows: # compile LEGACY/drm_damage_helper.o /tmp/build/2022.07.02.00.26.07-i386/tools/bin/i486--netbsdelf-gcc -fwrapv -msoft-float -mno-mmx -mno-sse -mno-avx -mindirect-branch=thunk -mindirect-branch-register -ffreestanding -fno-zero-initialized-in-bss -fno-delete-null-pointer-checks -O2 -fno-omit-frame-pointer -fstack-protector -Wstack-protector --param ssp-buffer-size=1 -fstack-usage -Wstack-usage=3584 -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Werror -Wall -Wno-main -Wno-format-zero-length -Wpointer-arith -Wmissing-prototypes -Wstrict-prototypes -Wold-style-definition -Wswitch -Wshadow -Wcast-qual -Wwrite-strings -Wno-unreachable-code -Wno-pointer-sign -Wno-attributes -Wextra -Wno-unused-parameter -Wold-style-definition -Wno-sign-compare -Walloca -Wno-missing-field-initializers -Wno-address-of-packed-member --sysroot=/tmp/build/2022.07.02.00.26.07-i386/destdir -Di386 -I. -I/tmp/build/2022.07.02.00.26.07-i386/src/sys/external/mit/xen-include-public/dist/ -I/tmp/build/2022.07.02.00.26.07-i386/src/sys/external/bs d/libnv/dist -I/tmp/build/2022.07.02.00.26.07-i386/src/sys/external/bsd/acpica/dist -I/tmp/build/2022.07.02.00.26.07-i386/src/sys/../common/lib/libx86emu -I/tmp/build/2022.07.02.00.26.07-i386/src/sys/../common/lib/libc/misc -I/tmp/build/2022.07.02.00.26.07-i386/src/sys/../common/include -I/tmp/build/2022.07.02.00.26.07-i386/src/sys/arch -I/tmp/build/2022.07.02.00.26.07-i386/src/sys -nostdinc -DCOMPAT_UTILS -D__XEN_INTERFACE_VERSION__=0x3020a -DDIAGNOSTIC -DCOMPAT_44 -D_KERNEL -D_KERNEL_OPT -std=gnu99 -I/tmp/build/2022.07.02.00.26.07-i386/src/sys/lib/libkern/../../../common/lib/libc/quad -I/tmp/build/2022.07.02.00.26.07-i386/src/sys/lib/libkern/../../../common/lib/libc/string -I/tmp/build/2022.07.02.00.26.07-i386/src/sys/lib/libkern/../../../common/lib/libc/arch/i386/string -I/tmp/build/2022.07.02.00.26.07-i386/src/sys/lib/libkern/../../../common/lib/libc/arch/i386/atomic -I/tmp/build/2022.07.02.00.26.07-i386/src/sys/lib/libkern/../../../common/lib/libc/hash/sha3 -D_FORTIFY_SOURCE=2 -I/tmp/build/2022.07.02.00.26.07-i386/src/sys/external/isc/atheros_hal/dist -I/tmp/build/2022.07.02.00.26.07-i386/src/sys/external/isc/atheros_hal/ic -I/tmp/build/2022.07.02.00.26.07-i386/src/sys/external/bsd/drm2/include -I/tmp/build/2022.07.02.00.26.07-i386/src/sys/external/bsd/drm2/include/drm -I/tmp/build/2022.07.02.00.26.07-i386/src/sys/external/bsd/common/include -I/tmp/build/2022.07.02.00.26.07-i386/src/sys/external/bsd/drm2/dist/include -I/tmp/build/2022.07.02.00.26.07-i386/src/sys/external/bsd/drm2/dist/include/drm -I/tmp/build/2022.07.02.00.26.07-i386/src/sys/external/bsd/drm2/dist/include/uapi -D__KERNEL__ -DCONFIG_X86 -DCONFIG_X86_PAT -DCONFIG_BACKLIGHT_CLASS_DEVICE=0 -DCONFIG_BACKLIGHT_CLASS_DEVICE_MODULE=0 -DCONFIG_DRM_FBDEV_EMULATION=1 -DCONFIG_DRM_FBDEV_OVERALLOC=100 -DCONFIG_FB=0 -DCONFIG_LOCKDEP=0 -DCONFIG_PCI=1 -I/tmp/build/2022.07.02.00.26.07-i386/src/sys/../common/include -I/tmp/build/2022.07.02.00.26.07-i386/src/sys/external/bsd/acpica/dist/include -I/tmp/build /2022.07.02.00.26.07-i386/src/sys/external/bsd/libnv/dist -c /tmp/build/2022.07.02.00.26.07-i386/src/sys/external/bsd/drm2/dist/drm/drm_damage_helper.c -o drm_damage_helper.o --- kern-GENERIC --- --- drm_gem_cma_helper.o --- In file included from /tmp/build/2022.07.02.00.26.07-i386/src/sys/external/bsd/drm2/drm/drm_gem_cma_helper.c:36: /tmp/build/2022.07.02.00.26.07-i386/src/sys/external/bsd/drm2/include/drm/drm_gem_cma_helper.h:42:1: error: two or more data types in declaration specifiers 42 | struct drm_device; | ^~ --- kern-INSTALL --- --- procfs_status.o --- /tmp/build/2022.07.02.00.26.07-i386/tools/bin/nbctfconvert -g -L VERSION procfs_status.o The following commits were made between the last successful build and the failed build: 2022.07.02.00.26.07 riastradh src/sys/external/bsd/drm2/drm/drm_gem_cma_helper.c,v 1.14 2022.07.02.00.26.07 riastradh src/sys/external/bsd/drm2/include/drm/drm_gem_cma_helper.h,v 1.3 Logs can be found at: http://releng.NetBSD.org/b5reports/i386/commits-2022.07.html#2022.07.02.00.26.07
Re: pgdaemon high CPU consumption
Hi Matthias ! See PR 55707 http://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=55707 , which I do not considere fixed due to the pgdaemon issue. reverting arc.cto 1.20 will give you many xcalls, but the system stays more usable. Frank On 07/01/22 07:55, Matthias Petermann wrote: Good day, since some time I noticed that on several of my systems with NetBSD/amd64 9.99.97/98 after longer usage the kernel process pgdaemon completely claims a CPU core for itself, i.e. constantly consumes 100%. The affected systems do not have a shortage of RAM and the problem does not disappear even if all workloads are stopped, and thus no RAM is actually used by application processes. I noticed this especially in connection with accesses to the ZFS set up on the respective machines - for example after checkout from the local CVS relic hosted on ZFS. Is there already a known problem or what information would have to be collected to get to the bottom of this? I currently have such a case online, so I would be happy to pull diagnostic information this evening/afternoon. At the moment all info I have is from top. Normal view: ``` PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND 0 root 1260 0K 34M CPU/0 102:45 100% 100% [system] ``` Thread view: ``` PID LID USERNAME PRI STATE TIME WCPUCPU NAME COMMAND 0 173 root 126 CPU/1 96:57 98.93% 98.93% pgdaemon [system] ``` Kind regards Matthias
Automated report: NetBSD-current/i386 build success
The NetBSD-current/i386 build is working again. The following commits were made between the last failed build and the successful build: 2022.07.01.09.54.36 prlw1 src/sys/kern/kern_fork.c,v 1.229 Logs can be found at: http://releng.NetBSD.org/b5reports/i386/commits-2022.07.html#2022.07.01.09.54.36
Re: pgdaemon high CPU consumption
"J. Hannken-Illjes" writes: >> On 1. Jul 2022, at 07:55, Matthias Petermann wrote: >> >> Good day, >> >> since some time I noticed that on several of my systems with NetBSD/amd64 >> 9.99.97/98 after longer usage the kernel process pgdaemon completely claims >> a CPU core for itself, i.e. constantly consumes 100%. >> The affected systems do not have a shortage of RAM and the problem does not >> disappear even if all workloads are stopped, and thus no RAM is actually >> used by application processes. >> >> I noticed this especially in connection with accesses to the ZFS set up on >> the respective machines - for example after checkout from the local CVS >> relic hosted on ZFS. >> >> Is there already a known problem or what information would have to be >> collected to get to the bottom of this? >> >> I currently have such a case online, so I would be happy to pull diagnostic >> information this evening/afternoon. At the moment all info I have is from >> top. >> >> Normal view: >> >> ``` >> PID USERNAME PRI NICE SIZE RES STATE TIME WCPUCPU COMMAND >>0 root 1260 0K 34M CPU/0 102:45 100% 100% [system] >> ``` >> >> Thread view: >> >> >> ``` >> PID LID USERNAME PRI STATE TIME WCPUCPU NAME COMMAND >>0 173 root 126 CPU/1 96:57 98.93% 98.93% pgdaemon [system] >> ``` > > Looks a lot like kern/55707: ZFS seems to trigger a lot of xcalls > > Last action proposed was to back out the patch ... > > -- > J. Hannken-Illjes - hann...@mailbox.org Probably only a slightly related data point, but Ya, if you have a system / VM / Xen PV that does not have a whole lot of RAM and if you don't back out that patch your system will become unusable in a very short order if you do much at all with ZFS (tested with a recent -current building pkgsrc packages on a Xen PVHVM). The patch does fix a real bug, as NetBSD doesn't have the define that it uses, but the effect of running that code will be needed if you use ZFS at all on a "low" RAM system. I personally suspect that the ZFS ARC or some pool is allowed to consume nearly all available "something" (pools, RAM, etc..) without limit but have no specific proof (or there is a leak somewhere). I mostly run 9.x ZFS right now (which may have other problems), and have been setting maxvnodes way down for some time. If I don't do that the Xen PV will hang itself up after a couple of 'build.sh release' runs when the source and build artifacts are on ZFS filesets. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: pgdaemon high CPU consumption
> On 1. Jul 2022, at 07:55, Matthias Petermann wrote: > > Good day, > > since some time I noticed that on several of my systems with NetBSD/amd64 > 9.99.97/98 after longer usage the kernel process pgdaemon completely claims a > CPU core for itself, i.e. constantly consumes 100%. > The affected systems do not have a shortage of RAM and the problem does not > disappear even if all workloads are stopped, and thus no RAM is actually used > by application processes. > > I noticed this especially in connection with accesses to the ZFS set up on > the respective machines - for example after checkout from the local CVS relic > hosted on ZFS. > > Is there already a known problem or what information would have to be > collected to get to the bottom of this? > > I currently have such a case online, so I would be happy to pull diagnostic > information this evening/afternoon. At the moment all info I have is from top. > > Normal view: > > ``` > PID USERNAME PRI NICE SIZE RES STATE TIME WCPUCPU COMMAND >0 root 1260 0K 34M CPU/0 102:45 100% 100% [system] > ``` > > Thread view: > > > ``` > PID LID USERNAME PRI STATE TIME WCPUCPU NAME COMMAND >0 173 root 126 CPU/1 96:57 98.93% 98.93% pgdaemon [system] > ``` Looks a lot like kern/55707: ZFS seems to trigger a lot of xcalls Last action proposed was to back out the patch ... -- J. Hannken-Illjes - hann...@mailbox.org signature.asc Description: Message signed with OpenPGP
Re: pgdaemon high CPU consumption
m...@petermann-it.de (Matthias Petermann) writes: >since some time I noticed that on several of my systems with=20 >NetBSD/amd64 9.99.97/98 after longer usage the kernel process pgdaemon=20 >completely claims a CPU core for itself, i.e. constantly consumes 100%. >The affected systems do not have a shortage of RAM and the problem does=20 >not disappear even if all workloads are stopped, and thus no RAM is=20 >actually used by application processes. There is a shortage, either free RAM pages or kernel address space. The page daemon gets triggered, but if it cannot resolve the situation it will just spin until it succeeds. >I noticed this especially in connection with accesses to the ZFS set up=20 >on the respective machines - for example after checkout from the local=20 >CVS relic hosted on ZFS. Resource exhaustion could be caused by ZFS, but also something else. If you can still operate the system, a common workaround is to reduce kern.maxvnodes with sysctl (and bump it up later). If the system is not responding but you can enter DDB, setting the kernel variable desiredvnodes does the same.
