Re: For snapshot builds: armv7 chroot on aarch64 has kyua test -k /usr/tests/Kyuafile sys/kern/kern_copyin hung up [in getpid?], unkillable, prevents reboot
On Jul 7, 2023, at 10:13, Mike Karels wrote: > On 7 Jul 2023, at 11:38, Mark Millard wrote: > >> On Jul 7, 2023, at 07:36, Mark Millard wrote: >> >>> On Jul 7, 2023, at 06:50, Mike Karels wrote: >>> On 7 Jul 2023, at 6:06, John F Carr wrote: > On Jul 6, 2023, at 20:42, Mike Karels wrote: >> >> >> Thanks for isolating this. Let me know when you have the bug number. >> I just tested a fix (the compat code drops the reference on the current >> address space an extra time, probably freeing it). >> >> Mike The fix is in https://cgit.freebsd.org/src/commit/?id=be30fd3ab2e8418a696e69f54a91a7e2db5962de. > The bug was introduced in January, 2022. It allows 32 bit binaries to > crash a 64 bit system when COMPAT_FREEBSD32 is on. Test coverage of the > buggy function (sysctl_kern_proc_vm_layout) was added at the same time. > > There should be routine runs of 32 bit test suites on 64 bit systems. > Although i386 and armv7 are tier 2 systems, the tier 1 COMPAT_FREEBSD32 > kernel code needs to be exercised. This bug was only discovered by > manually running tests in the right environment, 17 months after > automated testing could have discovered it. That is not so simple currently, as the shared libraries for the test environment are not part of 32-bit compatibility package. The required bits could be extracted from the corresponding 32-bit build, but that isn't easy to automate. Fortunately, I think that very few of the tests exercise any 32-bit-specific code paths. >>> >>> One way that I demonstrated this problem on an aarch64 system >>> that supports aarch32/armv7 in user space was via the use of >>> an official snapshot armv7 image. In my case I: >>> >>> A) dd'd the image to USB3 media (after downloading it) >>> B) mounted the ufs file system on the media to a mount point >> >> I forgot to mention an important step: before chroot is >> used, I preload various kernel modules that are used in >> the Kyuafile tests --because the chroot'd activity will >> not cause the loads of themsleves but will use the >> modules if they have already been loaded. >> >>> C) used a chroot into that mount point to run the: >>> "kyua test -k /usr/tests/Kyuafile" >>> >>> (I happened to do all that as root.) >>> >>> There may be viable alternatives to dd'ing to allow an analogy to >>> (B) for (C) to use. I've not experimented with using a jail >>> instead of a chroot. >>> >>> One can also install an armv7 world into a local directory tree >>> and then use chroot (or analogous). >>> >>> How far off is an analogous sort of procedure from being reasonable >>> to automate? > > It would be easier to use the packages rather than the full image > (base.txz and tests.txz). But doing this as part of a CI setup would > mean fetching things from a different source from the install image, > and then of course doing various configuration. It's always a Small > Matter of Programming. Doing a full chroot gets into some other > problems, e.g. mdconfig doesn't currently work in compatibility > mode. It doesn't seem that automating all this would yield much; > it's hard enough to do manually. > >>> i386, of course, also has lib32 and independently testing that is >>> a messier issue, including trying to use /usr/tests/Kyuafile based >>> testing that avoids use of chroot (or analogous). I'm not claiming >>> lib32 has as simple of a potential path to automated testing. > > I think the problem is essentially the same. A chroot could be used > or a 32-bit library setup (which would test the libraries as well). > >>> I do not know if FreeBSD has powerpc64 hardware able to use a >>> powerpc world directory tree analogously. Such hardware may be too >>> old and otherwise problematical, making it not viable to automate >>> testing. > > Powerpc supports 32-bit libraries, unlike arm (so far). My understanding is that powerpc64le does not in FreeBSD: there is no powerpcle in FreeBSD. So, not even chroot style support for 32-bit little endian use. If I understand right, no 32 bit little endian ABI is defined, other than the void linux activity's material, anyway. It may be that all big endian POWER use has lib32 support, but I'm not sure if all POWER has big endian FreeBSD support. May be POWER9 (10?) still has such support in FreeBSD. === Mark Millard marklmi at yahoo.com
Re: For snapshot builds: armv7 chroot on aarch64 has kyua test -k /usr/tests/Kyuafile sys/kern/kern_copyin hung up [in getpid?], unkillable, prevents reboot
On 7 Jul 2023, at 11:38, Mark Millard wrote: > On Jul 7, 2023, at 07:36, Mark Millard wrote: > >> On Jul 7, 2023, at 06:50, Mike Karels wrote: >> >>> On 7 Jul 2023, at 6:06, John F Carr wrote: >>> On Jul 6, 2023, at 20:42, Mike Karels wrote: > > > Thanks for isolating this. Let me know when you have the bug number. > I just tested a fix (the compat code drops the reference on the current > address space an extra time, probably freeing it). > > Mike >>> >>> The fix is in >>> https://cgit.freebsd.org/src/commit/?id=be30fd3ab2e8418a696e69f54a91a7e2db5962de. >>> The bug was introduced in January, 2022. It allows 32 bit binaries to crash a 64 bit system when COMPAT_FREEBSD32 is on. Test coverage of the buggy function (sysctl_kern_proc_vm_layout) was added at the same time. There should be routine runs of 32 bit test suites on 64 bit systems. Although i386 and armv7 are tier 2 systems, the tier 1 COMPAT_FREEBSD32 kernel code needs to be exercised. This bug was only discovered by manually running tests in the right environment, 17 months after automated testing could have discovered it. >>> >>> That is not so simple currently, as the shared libraries for the >>> test environment are not part of 32-bit compatibility package. >>> The required bits could be extracted from the corresponding 32-bit >>> build, but that isn't easy to automate. Fortunately, I think that >>> very few of the tests exercise any 32-bit-specific code paths. >> >> One way that I demonstrated this problem on an aarch64 system >> that supports aarch32/armv7 in user space was via the use of >> an official snapshot armv7 image. In my case I: >> >> A) dd'd the image to USB3 media (after downloading it) >> B) mounted the ufs file system on the media to a mount point > > I forgot to mention an important step: before chroot is > used, I preload various kernel modules that are used in > the Kyuafile tests --because the chroot'd activity will > not cause the loads of themsleves but will use the > modules if they have already been loaded. > >> C) used a chroot into that mount point to run the: >>"kyua test -k /usr/tests/Kyuafile" >> >> (I happened to do all that as root.) >> >> There may be viable alternatives to dd'ing to allow an analogy to >> (B) for (C) to use. I've not experimented with using a jail >> instead of a chroot. >> >> One can also install an armv7 world into a local directory tree >> and then use chroot (or analogous). >> >> How far off is an analogous sort of procedure from being reasonable >> to automate? It would be easier to use the packages rather than the full image (base.txz and tests.txz). But doing this as part of a CI setup would mean fetching things from a different source from the install image, and then of course doing various configuration. It's always a Small Matter of Programming. Doing a full chroot gets into some other problems, e.g. mdconfig doesn't currently work in compatibility mode. It doesn't seem that automating all this would yield much; it's hard enough to do manually. >> i386, of course, also has lib32 and independently testing that is >> a messier issue, including trying to use /usr/tests/Kyuafile based >> testing that avoids use of chroot (or analogous). I'm not claiming >> lib32 has as simple of a potential path to automated testing. I think the problem is essentially the same. A chroot could be used or a 32-bit library setup (which would test the libraries as well). >> I do not know if FreeBSD has powerpc64 hardware able to use a >> powerpc world directory tree analogously. Such hardware may be too >> old and otherwise problematical, making it not viable to automate >> testing. Powerpc supports 32-bit libraries, unlike arm (so far). Mike
Re: For snapshot builds: armv7 chroot on aarch64 has kyua test -k /usr/tests/Kyuafile sys/kern/kern_copyin hung up [in getpid?], unkillable, prevents reboot
On Jul 7, 2023, at 07:36, Mark Millard wrote: > On Jul 7, 2023, at 06:50, Mike Karels wrote: > >> On 7 Jul 2023, at 6:06, John F Carr wrote: >> >>> On Jul 6, 2023, at 20:42, Mike Karels wrote: Thanks for isolating this. Let me know when you have the bug number. I just tested a fix (the compat code drops the reference on the current address space an extra time, probably freeing it). Mike >> >> The fix is in >> https://cgit.freebsd.org/src/commit/?id=be30fd3ab2e8418a696e69f54a91a7e2db5962de. >> >>> The bug was introduced in January, 2022. It allows 32 bit binaries to >>> crash a 64 bit system when COMPAT_FREEBSD32 is on. Test coverage of the >>> buggy function (sysctl_kern_proc_vm_layout) was added at the same time. >>> >>> There should be routine runs of 32 bit test suites on 64 bit systems. >>> Although i386 and armv7 are tier 2 systems, the tier 1 COMPAT_FREEBSD32 >>> kernel code needs to be exercised. This bug was only discovered by >>> manually running tests in the right environment, 17 months after automated >>> testing could have discovered it. >> >> That is not so simple currently, as the shared libraries for the >> test environment are not part of 32-bit compatibility package. >> The required bits could be extracted from the corresponding 32-bit >> build, but that isn't easy to automate. Fortunately, I think that >> very few of the tests exercise any 32-bit-specific code paths. > > One way that I demonstrated this problem on an aarch64 system > that supports aarch32/armv7 in user space was via the use of > an official snapshot armv7 image. In my case I: > > A) dd'd the image to USB3 media (after downloading it) > B) mounted the ufs file system on the media to a mount point I forgot to mention an important step: before chroot is used, I preload various kernel modules that are used in the Kyuafile tests --because the chroot'd activity will not cause the loads of themsleves but will use the modules if they have already been loaded. > C) used a chroot into that mount point to run the: >"kyua test -k /usr/tests/Kyuafile" > > (I happened to do all that as root.) > > There may be viable alternatives to dd'ing to allow an analogy to > (B) for (C) to use. I've not experimented with using a jail > instead of a chroot. > > One can also install an armv7 world into a local directory tree > and then use chroot (or analogous). > > How far off is an analogous sort of procedure from being reasonable > to automate? > > i386, of course, also has lib32 and independently testing that is > a messier issue, including trying to use /usr/tests/Kyuafile based > testing that avoids use of chroot (or analogous). I'm not claiming > lib32 has as simple of a potential path to automated testing. > > I do not know if FreeBSD has powerpc64 hardware able to use a > powerpc world directory tree analogously. Such hardware may be too > old and otherwise problematical, making it not viable to automate > testing. > === Mark Millard marklmi at yahoo.com
Re: For snapshot builds: armv7 chroot on aarch64 has kyua test -k /usr/tests/Kyuafile sys/kern/kern_copyin hung up [in getpid?], unkillable, prevents reboot
On Jul 7, 2023, at 06:50, Mike Karels wrote: > On 7 Jul 2023, at 6:06, John F Carr wrote: > >> On Jul 6, 2023, at 20:42, Mike Karels wrote: >>> >>> >>> Thanks for isolating this. Let me know when you have the bug number. >>> I just tested a fix (the compat code drops the reference on the current >>> address space an extra time, probably freeing it). >>> >>> Mike > > The fix is in > https://cgit.freebsd.org/src/commit/?id=be30fd3ab2e8418a696e69f54a91a7e2db5962de. > >> The bug was introduced in January, 2022. It allows 32 bit binaries to >> crash a 64 bit system when COMPAT_FREEBSD32 is on. Test coverage of the >> buggy function (sysctl_kern_proc_vm_layout) was added at the same time. >> >> There should be routine runs of 32 bit test suites on 64 bit systems. >> Although i386 and armv7 are tier 2 systems, the tier 1 COMPAT_FREEBSD32 >> kernel code needs to be exercised. This bug was only discovered by manually >> running tests in the right environment, 17 months after automated testing >> could have discovered it. > > That is not so simple currently, as the shared libraries for the > test environment are not part of 32-bit compatibility package. > The required bits could be extracted from the corresponding 32-bit > build, but that isn't easy to automate. Fortunately, I think that > very few of the tests exercise any 32-bit-specific code paths. One way that I demonstrated this problem on an aarch64 system that supports aarch32/armv7 in user space was via the use of an official snapshot armv7 image. In my case I: A) dd'd the image to USB3 media (after downloading it) B) mounted the ufs file system on the media to a mount point C) used a chroot into that mount point to run the: "kyua test -k /usr/tests/Kyuafile" (I happened to do all that as root.) There may be viable alternatives to dd'ing to allow an analogy to (B) for (C) to use. I've not experimented with using a jail instead of a chroot. One can also install an armv7 world into a local directory tree and then use chroot (or analogous). How far off is an analogous sort of procedure from being reasonable to automate? i386, of course, also has lib32 and independently testing that is a messier issue, including trying to use /usr/tests/Kyuafile based testing that avoids use of chroot (or analogous). I'm not claiming lib32 has as simple of a potential path to automated testing. I do not know if FreeBSD has powerpc64 hardware able to use a powerpc world directory tree analogously. Such hardware may be too old and otherwise problematical, making it not viable to automate testing. === Mark Millard marklmi at yahoo.com
Re: For snapshot builds: armv7 chroot on aarch64 has kyua test -k /usr/tests/Kyuafile sys/kern/kern_copyin hung up [in getpid?], unkillable, prevents reboot
On 7 Jul 2023, at 6:06, John F Carr wrote: > On Jul 6, 2023, at 20:42, Mike Karels wrote: >> >> >> Thanks for isolating this. Let me know when you have the bug number. >> I just tested a fix (the compat code drops the reference on the current >> address space an extra time, probably freeing it). >> >> Mike The fix is in https://cgit.freebsd.org/src/commit/?id=be30fd3ab2e8418a696e69f54a91a7e2db5962de. > The bug was introduced in January, 2022. It allows 32 bit binaries to crash > a 64 bit system when COMPAT_FREEBSD32 is on. Test coverage of the buggy > function (sysctl_kern_proc_vm_layout) was added at the same time. > > There should be routine runs of 32 bit test suites on 64 bit systems. > Although i386 and armv7 are tier 2 systems, the tier 1 COMPAT_FREEBSD32 > kernel code needs to be exercised. This bug was only discovered by manually > running tests in the right environment, 17 months after automated testing > could have discovered it. That is not so simple currently, as the shared libraries for the test environment are not part of 32-bit compatibility package. The required bits could be extracted from the corresponding 32-bit build, but that isn't easy to automate. Fortunately, I think that very few of the tests exercise any 32-bit-specific code paths. Mike
Re: For snapshot builds: armv7 chroot on aarch64 has kyua test -k /usr/tests/Kyuafile sys/kern/kern_copyin hung up [in getpid?], unkillable, prevents reboot
On Jul 6, 2023, at 20:42, Mike Karels wrote: > > > Thanks for isolating this. Let me know when you have the bug number. > I just tested a fix (the compat code drops the reference on the current > address space an extra time, probably freeing it). > > Mike The bug was introduced in January, 2022. It allows 32 bit binaries to crash a 64 bit system when COMPAT_FREEBSD32 is on. Test coverage of the buggy function (sysctl_kern_proc_vm_layout) was added at the same time. There should be routine runs of 32 bit test suites on 64 bit systems. Although i386 and armv7 are tier 2 systems, the tier 1 COMPAT_FREEBSD32 kernel code needs to be exercised. This bug was only discovered by manually running tests in the right environment, 17 months after automated testing could have discovered it.
Re: For snapshot builds: armv7 chroot on aarch64 has kyua test -k /usr/tests/Kyuafile sys/kern/kern_copyin hung up [in getpid?], unkillable, prevents reboot
On 6 Jul 2023, at 18:53, John F Carr wrote:
> The hang is caused by the sysctl call in tests/sys/kern/kern_copyin.c. The
> function below hangs when called in a 32 bit ARM process running in a chroot
> environment on a 64 bit ARM system. I will write up a bug report.
>
> static int
> get_vm_layout(struct kinfo_vm_layout *kvm)
> {
> size_t len;
> int mib[4];
>
> mib[0] = CTL_KERN;
> mib[1] = KERN_PROC;
> mib[2] = KERN_PROC_VM_LAYOUT;
> mib[3] = getpid();
> len = sizeof(*kvm);
>
> return (sysctl(mib, nitems(mib), kvm, , NULL, 0));
> }
Thanks for isolating this. Let me know when you have the bug number.
I just tested a fix (the compat code drops the reference on the current
address space an extra time, probably freeing it).
Mike
Re: For snapshot builds: armv7 chroot on aarch64 has kyua test -k /usr/tests/Kyuafile sys/kern/kern_copyin hung up [in getpid?], unkillable, prevents reboot
On Jul 6, 2023, at 16:53, John F Carr wrote:
> On Jun 25, 2023, at 20:16, Mark Millard wrote:
>>
>> . . .
>>
>
> The hang is caused by the sysctl call in tests/sys/kern/kern_copyin.c. The
> function below hangs when called in a 32 bit ARM process running in a chroot
> environment on a 64 bit ARM system. I will write up a bug report.
>
> static int
> get_vm_layout(struct kinfo_vm_layout *kvm)
> {
> size_t len;
> int mib[4];
>
> mib[0] = CTL_KERN;
> mib[1] = KERN_PROC;
> mib[2] = KERN_PROC_VM_LAYOUT;
> mib[3] = getpid();
> len = sizeof(*kvm);
>
> return (sysctl(mib, nitems(mib), kvm, , NULL, 0));
> }
>
Thanks for the tiny-reproducer analysis! That should help
make getting to a fix more actionable.
===
Mark Millard
marklmi at yahoo.com
Re: For snapshot builds: armv7 chroot on aarch64 has kyua test -k /usr/tests/Kyuafile sys/kern/kern_copyin hung up [in getpid?], unkillable, prevents reboot
> On Jun 25, 2023, at 20:16, Mark Millard wrote:
>
> Using the likes of:
>
> FreeBSD-14.0-CURRENT-arm64-aarch64-ROCK64-20230622-b95d2237af40-263748.img
> and:
> FreeBSD-14.0-CURRENT-arm-armv7-GENERICSD-20230622-b95d2237af40-263748.img
>
> I have shown the following behavior after setting up storage
> media based on them. (This was a test that my builds were not
> odd for the issue.)
>
> Boot the aarch64 media and log in. (Note: I logged in
> as root.)
>
> mount the armv7 media (-noatime is just my habit)
> and then put it to use:
>
> # mount -onoatime /dev/da1s2a /mnt
>
> # chroot /mnt/
>
> # kyua test -k /usr/tests/Kyuafile sys/kern/kern_copyin
> sys/kern/kern_copyin:kern_copyin ->
>
> On the serial console:
>
> # ps -xu
> USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
> root 11 1498.4 0.0 0 256 - RNL 23:24 542:52.92 [idle]
> root 1174 100.0 0.0 0 16 - Rs 23:37 0:00.00
> /usr/tests/sys/kern/kern_copyin -vunprivileged-user=tests
> -r/tmp/kyua.9YUttj/2/result.atf kern_copyin
> root00.0 0.0 0 1616 - DLs 23:24 0:00.50 [kernel]
> root10.0 0.0 11704 1288 - ILs 23:24 0:00.02 /sbin/init
> root20.0 0.0 0 256 - WL 23:24 0:00.26 [clock]
> root30.0 0.0 0 272 - DL 23:24 0:00.00 [crypto]
> root40.0 0.0 0 80 - DL 23:24 0:00.95 [cam]
> root50.0 0.0 0 16 - DL 23:24 0:00.00 [busdma]
> root60.0 0.0 0 16 - DL 23:24 0:00.03 [rand_harvestq]
> root70.0 0.0 0 48 - DL 23:24 0:00.06 [pagedaemon]
> root80.0 0.0 0 16 - DL 23:24 0:00.00 [vmdaemon]
> root90.0 0.0 0 160 - DL 23:24 0:00.38 [bufdaemon]
> root 100.0 0.0 0 16 - DL 23:24 0:00.00 [audit]
> root 120.0 0.0 0 880 - WL 23:24 0:11.81 [intr]
> root 130.0 0.0 0 48 - DL 23:24 0:00.04 [geom]
> root 140.0 0.0 0 16 - DL 23:24 0:00.00 [sequencer 00]
> root 150.0 0.0 0 160 - DL 23:24 0:06.42 [usb]
> root 160.0 0.0 0 16 - DL 23:24 0:00.10 [acpi_thermal]
> root 170.0 0.0 0 16 - DL 23:24 0:00.00 [acpi_cooling0]
> root 180.0 0.0 0 16 - DL 23:24 0:00.04 [syncer]
> root 190.0 0.0 0 16 - DL 23:24 0:00.00 [vnlru]
> root 6710.0 0.0 13260 2600 - Is 23:25 0:00.00 dhclient:
> system.syslog (dhclient)
> root 6740.0 0.0 13260 2752 - Is 23:25 0:00.00 dhclient: dpni0
> [priv] (dhclient)
> root 7610.0 0.0 14572 3972 - Ss 23:25 0:00.02 /sbin/devd
> root 9640.0 0.0 12832 2764 - Is 23:25 0:00.02 /usr/sbin/syslogd
> -s
> root 10330.0 0.0 13012 2604 - Ss 23:25 0:00.01 /usr/sbin/cron -s
> root 10580.0 0.0 21052 8308 - Is 23:25 0:00.01 sshd:
> /usr/sbin/sshd [listener] 0 of 10-100 startups (sshd)
> root 10780.0 0.0 21288 9304 - Is 23:26 0:00.09 sshd: root@pts/0
> (sshd)
> root 11750.0 0.0 21288 9496 - Is 23:37 0:00.04 sshd: root@pts/1
> (sshd)
> root 10740.0 0.0 13380 3008 u0 Is 23:25 0:00.01 login [pam]
> (login)
> root 10750.0 0.0 13460 3292 u0 S23:25 0:00.02 -sh (sh)
> root 12330.0 0.0 13588 3016 u0 R+ 00:00 0:00.00 ps -xu
> root 10810.0 0.0 13460 3328 0 Is 23:26 0:00.02 -sh (sh)
> root 11700.0 0.0 5788 2884 0 I23:36 0:00.02 /bin/sh -i
> root 11720.0 0.0 10408 7192 0 I+ 23:37 0:00.30 kyua test -k
> /usr/tests/Kyuafile sys/kern/kern_copyin
> root 11780.0 0.0 13460 3320 1 Is+ 23:38 0:00.01 -sh (sh)
>
> 1174 is stuck, even if one waits for 30min+.
> kill and kill -9 will not kill 1174.
>
> "shutdown -r now" hangs before the reboot happens
> and reports: "some processes would not die".
>
> An interesting property is that ps and top disagree
> about 1174 CPU usage: ps 100%, top 0%. But top also
> indicates 1174 always has CPU0 "STATE". (Across
> tests CPUn varies but within a test it has
> a fixed n.)
>
> I have also seen ps "STAT" being RXs.
>
> The following is from my earlier activity with my own
> builds involved, here 1119, not the 1174 from above.
> truss reports as the last thing for the stuck process
> as "getpid()".
>
> . . .
> 1119: 0.588983953 fstatat(AT_FDCWD,"/usr/tests/sys/kern/kern_copyin",{
> mode=-r-xr-xr-x ,inode=111756,size=9776,blksize=10240 },AT_SYMLINK_NOFOLLOW)
> = 0 (0x0)
> 1119: 0.589065030
> mmap(0x0,20480,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0)
> = 1074188288 (0x4006d000)
> 1119: 0.589227544
> openat(AT_FDCWD,"/tmp/kyua.aBQv6E/2/result.atf",O_WRONLY|O_CREAT|O_TRUNC,0644)
> = 3 (0x3)
> 1119: 0.589276503 getpid() = 1119 (0x45f)
>
>
>
> For reference, from inside an armv7 chroot session
> before doing such a test:
>
> # uname -apKU
> FreeBSD generic
Re: For snapshot builds: armv7 chroot on aarch64 has kyua test -k /usr/tests/Kyuafile sys/kern/kern_copyin hung up [in getpid?], unkillable, prevents reboot
On Jun 25, 2023, at 17:16, Mark Millard wrote:
> Using the likes of:
>
> FreeBSD-14.0-CURRENT-arm64-aarch64-ROCK64-20230622-b95d2237af40-263748.img
> and:
> FreeBSD-14.0-CURRENT-arm-armv7-GENERICSD-20230622-b95d2237af40-263748.img
>
> I have shown the following behavior after setting up storage
> media based on them. (This was a test that my builds were not
> odd for the issue.)
>
> Boot the aarch64 media and log in. (Note: I logged in
> as root.)
>
> mount the armv7 media (-noatime is just my habit)
> and then put it to use:
>
> # mount -onoatime /dev/da1s2a /mnt
>
> # chroot /mnt/
>
> # kyua test -k /usr/tests/Kyuafile sys/kern/kern_copyin
> sys/kern/kern_copyin:kern_copyin ->
>
> On the serial console:
>
> # ps -xu
> USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
> root 11 1498.4 0.0 0 256 - RNL 23:24 542:52.92 [idle]
> root 1174 100.0 0.0 0 16 - Rs 23:37 0:00.00
> /usr/tests/sys/kern/kern_copyin -vunprivileged-user=tests
> -r/tmp/kyua.9YUttj/2/result.atf kern_copyin
> root00.0 0.0 0 1616 - DLs 23:24 0:00.50 [kernel]
> root10.0 0.0 11704 1288 - ILs 23:24 0:00.02 /sbin/init
> root20.0 0.0 0 256 - WL 23:24 0:00.26 [clock]
> root30.0 0.0 0 272 - DL 23:24 0:00.00 [crypto]
> root40.0 0.0 0 80 - DL 23:24 0:00.95 [cam]
> root50.0 0.0 0 16 - DL 23:24 0:00.00 [busdma]
> root60.0 0.0 0 16 - DL 23:24 0:00.03 [rand_harvestq]
> root70.0 0.0 0 48 - DL 23:24 0:00.06 [pagedaemon]
> root80.0 0.0 0 16 - DL 23:24 0:00.00 [vmdaemon]
> root90.0 0.0 0 160 - DL 23:24 0:00.38 [bufdaemon]
> root 100.0 0.0 0 16 - DL 23:24 0:00.00 [audit]
> root 120.0 0.0 0 880 - WL 23:24 0:11.81 [intr]
> root 130.0 0.0 0 48 - DL 23:24 0:00.04 [geom]
> root 140.0 0.0 0 16 - DL 23:24 0:00.00 [sequencer 00]
> root 150.0 0.0 0 160 - DL 23:24 0:06.42 [usb]
> root 160.0 0.0 0 16 - DL 23:24 0:00.10 [acpi_thermal]
> root 170.0 0.0 0 16 - DL 23:24 0:00.00 [acpi_cooling0]
> root 180.0 0.0 0 16 - DL 23:24 0:00.04 [syncer]
> root 190.0 0.0 0 16 - DL 23:24 0:00.00 [vnlru]
> root 6710.0 0.0 13260 2600 - Is 23:25 0:00.00 dhclient:
> system.syslog (dhclient)
> root 6740.0 0.0 13260 2752 - Is 23:25 0:00.00 dhclient: dpni0
> [priv] (dhclient)
> root 7610.0 0.0 14572 3972 - Ss 23:25 0:00.02 /sbin/devd
> root 9640.0 0.0 12832 2764 - Is 23:25 0:00.02 /usr/sbin/syslogd
> -s
> root 10330.0 0.0 13012 2604 - Ss 23:25 0:00.01 /usr/sbin/cron -s
> root 10580.0 0.0 21052 8308 - Is 23:25 0:00.01 sshd:
> /usr/sbin/sshd [listener] 0 of 10-100 startups (sshd)
> root 10780.0 0.0 21288 9304 - Is 23:26 0:00.09 sshd: root@pts/0
> (sshd)
> root 11750.0 0.0 21288 9496 - Is 23:37 0:00.04 sshd: root@pts/1
> (sshd)
> root 10740.0 0.0 13380 3008 u0 Is 23:25 0:00.01 login [pam]
> (login)
> root 10750.0 0.0 13460 3292 u0 S23:25 0:00.02 -sh (sh)
> root 12330.0 0.0 13588 3016 u0 R+ 00:00 0:00.00 ps -xu
> root 10810.0 0.0 13460 3328 0 Is 23:26 0:00.02 -sh (sh)
> root 11700.0 0.0 5788 2884 0 I23:36 0:00.02 /bin/sh -i
> root 11720.0 0.0 10408 7192 0 I+ 23:37 0:00.30 kyua test -k
> /usr/tests/Kyuafile sys/kern/kern_copyin
> root 11780.0 0.0 13460 3320 1 Is+ 23:38 0:00.01 -sh (sh)
>
> 1174 is stuck, even if one waits for 30min+.
> kill and kill -9 will not kill 1174.
>
> "shutdown -r now" hangs before the reboot happens
> and reports: "some processes would not die".
>
> An interesting property is that ps and top disagree
> about 1174 CPU usage: ps 100%, top 0%. But top also
> indicates 1174 always has CPU0 "STATE". (Across
> tests CPUn varies but within a test it has
> a fixed n.)
>
> I have also seen ps "STAT" being RXs.
>
> The following is from my earlier activity with my own
> builds involved, here 1119, not the 1174 from above.
> truss reports as the last thing for the stuck process
> as "getpid()".
>
> . . .
> 1119: 0.588983953 fstatat(AT_FDCWD,"/usr/tests/sys/kern/kern_copyin",{
> mode=-r-xr-xr-x ,inode=111756,size=9776,blksize=10240 },AT_SYMLINK_NOFOLLOW)
> = 0 (0x0)
> 1119: 0.589065030
> mmap(0x0,20480,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0)
> = 1074188288 (0x4006d000)
> 1119: 0.589227544
> openat(AT_FDCWD,"/tmp/kyua.aBQv6E/2/result.atf",O_WRONLY|O_CREAT|O_TRUNC,0644)
> = 3 (0x3)
> 1119: 0.589276503 getpid() = 1119 (0x45f)
>
>
>
> For reference, from inside an armv7 chroot session
> before doing such a test:
>
> # uname -apKU
> FreeBSD generic 14.0-CURRENT
