Re: one remaining mystery about the FreeBSD domU failure on NetBSD XEN3_DOM0
On Sun, 11 Apr 2021, Greg A. Woods wrote: Anyway this does something slightly different (and better, or worse) on the FreeBSD side, but still ends up with a corrupted filesystem, as seen from both sides, though maybe not so bad from NetBSD's point of view: # mount /dev/da1 mount: /dev/da1: unknown special file or file system ... # umount /mnt # fsck /dev/da1 ** /dev/da1 ** Last Mounted on /mnt ** Phase 1 - Check Blocks and Sizes PARTIALLY TRUNCATED INODE I=325128 SALVAGE? [yn] n PARTIALLY TRUNCATED INODE I=877864 SALVAGE? [yn] n PARTIALLY TRUNCATED INODE I=877866 SALVAGE? [yn] n PARTIALLY TRUNCATED INODE I=877879 SALVAGE? [yn] ^C * FILE SYSTEM MARKED DIRTY * Don't see that on a standard block device (USB stick): $ sudo gpart add -t freebsd-ufs da0 da0p1 added $ sudo newfs -O1 /dev/da0p1 /dev/da0p1: 496.0MB (1015728 sectors) block size 32768, fragment size 4096 using 4 cylinder groups of 124.00MB, 3968 blks, 15872 inodes. super-block backups (for fsck_ffs -b #) at: 64, 254016, 507968, 761920 $ sudo fsck_ufs -EfrZz da0p1 ** /dev/da0p1 ** Last Mounted on ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 2 files, 2 used, 124907 free (27 frags, 15610 blocks, 0.0% fragmentation) * FILE SYSTEM IS CLEAN * $ sudo mount /dev/da0p1 /media $ sudo umount /media $ -RVP
Re: one remaining mystery about the FreeBSD domU failure on NetBSD XEN3_DOM0
On Sun, 11 Apr 2021, Greg A. Woods wrote: NetBSD can actually make some sense of this FreeBSD filesystem though: # fsck -n /dev/mapper/rscratch-fbsd--test.0 ** /dev/mapper/rscratch-fbsd--test.0 (NO WRITE) Invalid quota magic number CONTINUE? yes ** File system is already clean ** Last Mounted on /mnt ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups SUMMARY INFORMATION BAD SALVAGE? no BLK(S) MISSING IN BIT MAPS SALVAGE? no ** Phase 6 - Check Quotas CLEAR SUPERBLOCK QUOTA FLAG? no 2 files, 2 used, 7612693 free (21 frags, 951584 blocks, 0.0% fragmentation) * UNRESOLVED INCONSISTENCIES REMAIN * I'm not sure if those problems are to be expected with a FreeBSD-created filesystem or not. Probably the "Invalid quota magic number" is normal, but I'm not sure about the "BLK(s) MISSING IN BIT MAPS". Have FreeBSD and NetBSD FFS diverged this much? I won't try to mount it, especially not from the dom0. I've run into this before. This is a NetBSD vs. FreeBSD UFS issue. Create a UFS v1 FS on FreeBSD if you want to write it on NetBSD: newfs -O1 /dev/... Otherwise, you get that "Invalid quota magic number" error because newfs creates a UFSv2 FS on FreeBSD. -RVP
Re: one remaining mystery about the FreeBSD domU failure on NetBSD XEN3_DOM0
At Sun, 11 Apr 2021 21:13:44 + (UTC), RVP wrote: Subject: Re: one remaining mystery about the FreeBSD domU failure on NetBSD XEN3_DOM0 > > I've run into this before. This is a NetBSD vs. FreeBSD UFS issue. Create > a UFS v1 FS on FreeBSD if you want to write it on NetBSD: > > newfs -O1 /dev/... > > Otherwise, you get that "Invalid quota magic number" error because newfs > creates a UFSv2 FS on FreeBSD. While that wasn't exactly my goal (I just wanted to see from the NetBSD side if the FreeBSD side was actually writing sensible things and not missing anything or mixing anything up or corrupting anything), it could indeed help me in doing that. Anyway this does something slightly different (and better, or worse) on the FreeBSD side, but still ends up with a corrupted filesystem, as seen from both sides, though maybe not so bad from NetBSD's point of view: xbd1: 30720MB at device/vbd/2064 on xenbusb_front0 xbd1: attaching as da1 xbd1: features: flush xbd1: synchronize cache commands enabled. GEOM: new disk da1 # newfs -O1 /dev/da1 /dev/da1: 30720.0MB (62914560 sectors) block size 32768, fragment size 4096 using 121 cylinder groups of 254.00MB, 8128 blks, 32512 inodes. super-block backups (for fsck_ffs -b #) at: 64, 520256, 1040448, 1560640, 2080832, 2601024, 3121216, 3641408, 4161600, 4681792, 5201984, 5722176, 6242368, 6762560, 7282752, 7802944, 8323136, 8843328, 9363520, 9883712, 10403904, 10924096, 11444288, 11964480, 12484672, 13004864, 13525056, 14045248, 14565440, 15085632, 15605824, 16126016, 16646208, 17166400, 17686592, 18206784, 18726976, 19247168, 19767360, 20287552, 20807744, 21327936, 21848128, 22368320, 22888512, 23408704, 23928896, 24449088, 24969280, 25489472, 26009664, 26529856, 27050048, 27570240, 28090432, 28610624, 29130816, 29651008, 30171200, 30691392, 31211584, 31731776, 32251968, 32772160, 33292352, 33812544, 34332736, 34852928, 35373120, 35893312, 36413504, 36933696, 37453888, 37974080, 38494272, 39014464, 39534656, 40054848, 40575040, 41095232, 41615424, 42135616, 42655808, 43176000, 43696192, 44216384, 44736576, 45256768, 45776960, 46297152, 46817344, 47337536, 47857728, 48377920, 48898112, 49418304, 49938496, 50458688, 50978880, 51499072, 52019264, 52539456, 53059648, 53579840, 54100032, 54620224, 55140416, 55660608, 56180800, 56700992, 57221184, 57741376, 58261568, 58781760, 59301952, 59822144, 60342336, 60862528, 61382720, 61902912, 62423104 # mount /dev/da1 mount: /dev/da1: unknown special file or file system # mount /dev/da1 /mnt # df Filesystem 512-blocks UsedAvail Capacity Mounted on /dev/ufs/FreeBSD_Install 782968 737016 -16680 102%/ devfs 2 20 100%/dev tmpfs 6553661664920 1%/var tmpfs 40960 840952 0%/tmp /dev/da1 61915512 16 56962256 0%/mnt # ls -l /mnt total 8 drwxrwxr-x 2 root operator 512 Apr 11 21:45 .snap # cp /COPYRIGHT /mnt/ # ls -l /mnt/ total 24 drwxrwxr-x 2 root operator 512 Apr 11 21:45 .snap -r--r--r-- 1 root wheel 6177 Apr 11 21:46 COPYRIGHT # pax -X -rw -pe / /mnt # ls -l /mnt total 1120 -rw-r--r-- 2 root wheel 1089 Oct 23 05:56 .cshrc -rw-r--r-- 2 root wheel470 Oct 23 05:56 .profile drwxrwxr-x 2 root operator 512 Apr 11 21:52 .snap -r--r--r-- 1 root wheel 6177 Oct 23 05:56 COPYRIGHT -r--r--r-- 1 root wheel 7226 Oct 23 05:57 ERRATA.HTML -r--r--r-- 1 root wheel 3273 Oct 23 05:57 ERRATA.TXT -r--r--r-- 1 root wheel 252351 Oct 23 05:57 HARDWARE.HTML -r--r--r-- 1 root wheel 117568 Oct 23 05:57 HARDWARE.TXT -r--r--r-- 1 root wheel 23882 Oct 23 05:57 README.HTML -r--r--r-- 1 root wheel 14316 Oct 23 05:57 README.TXT -r--r--r-- 1 root wheel 36431 Oct 23 05:57 RELNOTES.HTML -r--r--r-- 1 root wheel 12343 Oct 23 05:57 RELNOTES.TXT drwxr-xr-x 2 root wheel 1024 Oct 23 05:55 bin drwxr-xr-x 9 root wheel 1536 Oct 23 05:57 boot dr-xr-xr-x 2 root wheel512 Apr 11 19:02 dev -r--r--r-- 1 root wheel 6985 Oct 23 05:57 docbook.css drwxr-xr-x 25 root wheel 2048 Oct 23 06:04 etc drwxr-xr-x 5 root wheel 1536 Oct 23 05:55 lib drwxr-xr-x 3 root wheel512 Oct 23 05:55 libexec drwxr-xr-x 2 root wheel512 Oct 23 05:55 media drwxr-xr-x 2 root wheel512 Oct 23 05:57 mnt drwxr-xr-x 2 root wheel512 Oct 23 05:55 net dr-xr-xr-x 2 root wheel512 Oct 23 05:55 proc drwxr-xr-x 2 root wheel512 Oct 23 05:55 rescue drwxr-xr-x 2 root wheel512 Oct 23 05:56 root drwxr-xr-x 2 root wheel 2560 Oct 23 05:56 sbin drwxrwxrwt 2 root wheel512 Apr 11 21:52 tmp drwxr-xr-x 13 root wheel512 Oct 23 05:57 usr drwxr-xr-x 2 root wheel512 Apr 11 19:04 var # umount /mnt # fsck /dev/da1 ** /dev/da1 ** Last Mounted on
Re: one remaining mystery about the FreeBSD domU failure on NetBSD XEN3_DOM0
At Sun, 11 Apr 2021 13:23:31 -0700, "Greg A. Woods" wrote: Subject: one remaining mystery about the FreeBSD domU failure on NetBSD XEN3_DOM0 > > In fact it only seems to be fsck that complains, possibly along > with any attempt to write to a filesystem, that causes problems. Definitely writing to a FreeBSD domU filesystem, i.e. to a FreeBSD xbd(4) with a new filesystem created on it, is impossible. I was able to write 500MB of zeros to the LVM LV backed disk, overwriting the copy of the .img file I had put there, and only see 500MB of zeros back on the NetBSD side, so writing directly to the raw /dev/da1 on FreeBSD seems to write data without problem. However then the following happens when I try to use a new FS there: # newfs /dev/da1 /dev/da1: 30720.0MB (62914560 sectors) block size 32768, fragment size 4096 using 50 cylinder groups of 626.09MB, 20035 blks, 80256 inodes. super-block backups (for fsck_ffs -b #) at: 192, 1282432, 2564672, 3846912, 5129152, 6411392, 7693632, 8975872, 10258112, 11540352, 12822592, 14104832, 15387072, 16669312, 17951552, 19233792, 20516032, 21798272, 23080512, 24362752, 25644992, 26927232, 28209472, 29491712, 30773952, 32056192, 8432, 34620672, 35902912, 37185152, 38467392, 39749632, 41031872, 42314112, 43596352, 44878592, 46160832, 47443072, 48725312, 50007552, 51289792, 52572032, 53854272, 55136512, 56418752, 57700992, 58983232, 60265472, 61547712, 62829952 # mount /dev/da1 /mnt # mount /dev/ufs/FreeBSD_Install on / (ufs, local, noatime, read-only) devfs on /dev (devfs, local, multilabel) tmpfs on /var (tmpfs, local) tmpfs on /tmp (tmpfs, local) /dev/da1 on /mnt (ufs, local) # df Filesystem 512-blocks UsedAvail Capacity Mounted on /dev/ufs/FreeBSD_Install 782968 737016 -16680 102%/ devfs 2 20 100%/dev tmpfs 6553660864928 1%/var tmpfs 40960 840952 0%/tmp /dev/da1 60901560 16 56029424 0%/mnt # cp /COPYRIGHT /mnt UFS /dev/da1 (/mnt) cylinder checksum failed: cg 0, cgp: 0xe66de1a4 != bp: 0xf433acbc UFS /dev/da1 (/mnt) cylinder checksum failed: cg 1, cgp: 0x89ba8532 != bp: 0x3491fbd0 UFS /dev/da1 (/mnt) cylinder checksum failed: cg 3, cgp: 0xdeaf87a7 != bp: 0x3a071e86 UFS /dev/da1 (/mnt) cylinder checksum failed: cg 7, cgp: 0x7085828d != bp: 0xaaae0f19 UFS /dev/da1 (/mnt) cylinder checksum failed: cg 15, cgp: 0x293dfe28 != bp: 0xe2f25f8b UFS /dev/da1 (/mnt) cylinder checksum failed: cg 31, cgp: 0x9a4d0762 != bp: 0x4119c6e [[ and on and on ]] UFS /dev/da1 (/mnt) cylinder checksum failed: cg 49, cgp: 0x931f84e5 != bp: 0xb48687df /mnt: create/symlink failed, no inodes free cp: /mnt/COPYRIGHT: No space left on device # Apr 11 20:37:28 syslogd: last message repeated 4 times Apr 11 20:37:59 kernel: pid 713 (cp), uid 0 inumber 2 on /mnt: out of inodes # df -i Filesystem 512-blocks UsedAvail Capacity iused ifree %iused Mounted on /dev/ufs/FreeBSD_Install 782968 737016 -16680 102% 12129 285 98% / devfs 2 20 100% 0 0 100% /dev tmpfs 6553660864928 1% 75 114613 0% /var tmpfs 40960 840952 0% 6 71674 0% /tmp /dev/da1 60901560 16 56029424 0% 2 4012796 0% /mnt NetBSD can actually make some sense of this FreeBSD filesystem though: # fsck -n /dev/mapper/rscratch-fbsd--test.0 ** /dev/mapper/rscratch-fbsd--test.0 (NO WRITE) Invalid quota magic number CONTINUE? yes ** File system is already clean ** Last Mounted on /mnt ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups SUMMARY INFORMATION BAD SALVAGE? no BLK(S) MISSING IN BIT MAPS SALVAGE? no ** Phase 6 - Check Quotas CLEAR SUPERBLOCK QUOTA FLAG? no 2 files, 2 used, 7612693 free (21 frags, 951584 blocks, 0.0% fragmentation) * UNRESOLVED INCONSISTENCIES REMAIN * I'm not sure if those problems are to be expected with a FreeBSD-created filesystem or not. Probably the "Invalid quota magic number" is normal, but I'm not sure about the "BLK(s) MISSING IN BIT MAPS". Have FreeBSD and NetBSD FFS diverged this much? I won't try to mount it, especially not from the dom0. Dumpfs shows the following: file system: /dev/mapper/rscratch-fbsd--test.0 format FFSv2 endian little-endian location 65536 (-b 128) magic 19540119timeSun Apr 11 13:46:15 2021 superblock location 65536 id [ 60735d32 358197c4 ] cylgrp dynamic inodes FFSv2 sblock FFSv2 fslevel 5 nbfree 951584 ndir2 nifree 4012796 nffree 21 ncg 50 size7864320 blocks 7612695 bsize 32768 shift 15 mask0x8000 fsize 4096shift
one remaining mystery about the FreeBSD domU failure on NetBSD XEN3_DOM0
So, with the vnd(4) issue more or less sorted, there seems to be one major mystery remaining w.r.t. whatever has gone wrong with the ability of NetBSD-current XEN3_DOM0 to host FreeBSD domUs. I still can't create a clean filesystem on a writeable disk. The "newfs" runs fine, but a subsequent "fsck" finds errors and cannot fix them (though the first run does change one or two things). I can't even get a clean fsck of the running system's root FS: (the "ada0: disk error" after I hit ^C is because the underlying disk (vnd0d) is exported read-only to the domU) # fsck -v /dev/ufs/FreeBSD_Install start / wait fsck_ufs /dev/ufs/FreeBSD_Install ** /dev/ufs/FreeBSD_Install SAVE DATA TO FIND ALTERNATE SUPERBLOCKS? [yn] n ADD CYLINDER GROUP CHECK-HASH PROTECTION? [yn] n ** Last Mounted on ** Root file system ** Phase 1 - Check Blocks and Sizes PARTIALLY TRUNCATED INODE I=28 SALVAGE? [yn] n PARTIALLY TRUNCATED INODE I=112 SALVAGE? [yn] ^Cada0: disk error cmd=write 8145-8152 status: fffe * FILE SYSTEM MARKED DIRTY * # Most mysteriously this filesystem is in use as the root FS and all the files in it can be found and read! Presumably they are all intact too -- no programs have failed or behaved mysteriously (except fsck) and all the human readable files I've looked at (e.g. manual pages) all seem fine. In fact it only seems to be fsck that complains, possibly along with any attempt to write to a filesystem, that causes problems. (I believe writing to a filesystem appears to corrupt it but that is only according to fsck. I do seem believe there was an eventual crashes of a system that had been running with active filesystems, but I have not got far enough again since to reproduce this, due to the fsck problem.) # mount /dev/ufs/FreeBSD_Install on / (ufs, local, noatime, read-only) devfs on /dev (devfs, local, multilabel) tmpfs on /var (tmpfs, local) tmpfs on /tmp (tmpfs, local) # df Filesystem 512-blocks Used Avail Capacity Mounted on /dev/ufs/FreeBSD_Install 782968 737016 -16680 102%/ devfs 2 2 0 100%/dev tmpfs 65536232 65304 0%/var tmpfs 40960 8 40952 0%/tmp # time -l sh -c 'find / -type f | xargs cat > /dev/null ' 38.58 real 1.36 user18.30 sys 4872 maximum resident set size 13 average shared memory size 5 average unshared data size 215 average unshared stack size 1906 page reclaims 0 page faults 0 swaps 14024 block input operations 0 block output operations 0 messages sent 0 messages received 0 signals received 12348 voluntary context switches 33 involuntary context switches In fact I can put a copy of the FreeBSD img file into an LVM LV, attach it to the running FreeBSD domU, mount it (without an FSCK, since the FreeBSD_Install filesystem comes clean from the factory), then do "diff -r -X /mnt -X /dev / /mnt" and find only the expected differences. So, what could be different about how fsck reads v.s. the kernel itself? If indeed writing to filesystem corrupts it, how and why? It seems NetBSD can make sense of the BSD label inside the FreeBSD mini-memstick.img file, e.g. when accessed through vnd(4), but it can't seem to make sense of the filesystem(s) inside (which I guess might be expected?): # file -s /dev/rvnd0f /dev/rvnd0f: DOS/MBR boot sector, BSD disklabel # disklabel vnd0 # /dev/rvnd0: type: vnd disk: vnd label: fictitious flags: bytes/sector: 512 sectors/track: 32 tracks/cylinder: 64 sectors/cylinder: 2048 cylinders: 387 total sectors: 791121 rpm: 3600 interleave: 1 trackskew: 0 cylinderskew: 0 headswitch: 0 # microseconds track-to-track seek: 0 # microseconds drivedata: 0 6 partitions: #sizeoffset fstype [fsize bsize cpg/sgs] d:791121 0 unused 0 0# (Cyl. 0 -386*) e: 1600 1unknown # (Cyl. 0*- 0*) f:789520 1601 4.2BSD 0 0 0 # (Cyl. 0*-386*) disklabel: boot block size 0 disklabel: super block size 0 # fsck -n /dev/vnd0f ** /dev/rvnd0f (NO WRITE) BAD SUPER BLOCK: CAN'T FIND SUPERBLOCK /dev/rvnd0f: CANNOT FIGURE OUT SECTORS PER CYLINDER -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpMSgeO6z7I3.pgp Description: OpenPGP Digital Signature
re: mail/sendmail not relaying on netbsd-9/sparc, problem with OpenSSL update?
Martin Husemann writes: > On Sun, Apr 11, 2021 at 10:37:21AM +1000, matthew green wrote: > > > How can you invoke a make to test this (besides a full build.sh and adding > > > some output to the makefiles)? > > > Or: can you just fix and request pullup ;-) > > > I can run sparc tests (quickly) again. > > > > cd src/compat > > nbmake-sparc64 > > BOOTSTRAP_SUBDIRS=../../../crypto/external/bsd/openssl/lib/libcrypto > > dependall > > I still have no simple way to test the sparc64 -m32 libs - does this > obfuscation really gain something in the real world? i guess you figured it out going on the commit? to be a little more verbose about this: to build any subset of the normal "src/compat" dirs, invoke the right nbmake-$arch in src/compat with BOOTSTRAP_SUBDIRS set to a series of paths that built using the provided target (so only standard targets are available -- all, dependall, depend, clean, cleandir, install, etc.) so to just test the -m32 libc, i've used this: cd src/compat nbmake-sparc64 BOOTSTRAP_SUBDIRS=../../../lib/libc dependall nbmake-sparc64 BOOTSTRAP_SUBDIRS=../../../lib/libc install DESTDIR=/export/root/sparc64 and then my nfsroot has a new /usr/lib/sparc/libc.so.12 and i test it on the target. thanks. .mrg.
Re: mail/sendmail not relaying on netbsd-9/sparc, problem with OpenSSL update?
On Sun, Apr 11, 2021 at 10:37:21AM +1000, matthew green wrote: > > How can you invoke a make to test this (besides a full build.sh and adding > > some output to the makefiles)? > > Or: can you just fix and request pullup ;-) > > I can run sparc tests (quickly) again. > > cd src/compat > nbmake-sparc64 > BOOTSTRAP_SUBDIRS=../../../crypto/external/bsd/openssl/lib/libcrypto dependall I still have no simple way to test the sparc64 -m32 libs - does this obfuscation really gain something in the real world? Martin
Re: I think I've found why Xen domUs can't mount some file-backed disk images! (vnd(4) hides labels!)
On Sat, Apr 10, 2021 at 03:17:35PM -0700, Greg A. Woods wrote: > [...] > # fdisk -F /images/FreeBSD-12.2-RELEASE-amd64-mini-memstick.img > Disk: /images/FreeBSD-12.2-RELEASE-amd64-mini-memstick.img > NetBSD disklabel disk geometry: > cylinders: 49, heads: 255, sectors/track: 63 (16065 sectors/cylinder) > total sectors: 791121, bytes/sector: 512 > > BIOS disk geometry: > cylinders: 49, heads: 255, sectors/track: 63 (16065 sectors/cylinder) > total sectors: 791121 > > Partitions aligned to 16065 sector boundaries, offset 63 > > Partition table: > 0: EFI system partition (sysid 239) > start 1, size 1600 (1 MB, Cyls 0/0/2-0/25/26) > 1: FreeBSD or 386BSD or old NetBSD (sysid 165) > start 1601, size 789520 (386 MB, Cyls 0/25/27-49/62/30), Active > 2: > 3: > First active partition: 1 > Drive serial number: 2425393296 (0x90909090) > > # fdisk vnd0 > fdisk: primary partition table invalid, no magic in sector 0 > fdisk: Cannot determine the number of heads > Disk: /dev/rvnd0d > NetBSD disklabel disk geometry: > cylinders: 4096, heads: 64, sectors/track: 32 (2048 sectors/cylinder) > total sectors: 8388608, bytes/sector: 512 > > BIOS disk geometry: > cylinders: 522, heads: 255, sectors/track: 63 (16065 sectors/cylinder) > total sectors: 8388608 > > Partitions aligned to 16065 sector boundaries, offset 63 > > Partition table: > 0: > 1: > 2: > 3: > Bootselector disabled. > No active partition. > Drive serial number: 0 (0x) I can't reproduce this fdisk/disklabel on netbsd-9 nor -current. fdisk on vnd0 gives me the same partition table as on the file. FreeBSD fails to boot with the same error message. The size of the disk is indeed 790528 in the xenstore (and the dom0's kernel message) but I don't know where this comes from. xbdback uses getdiskinfo() to get the device's size. In vnd, the size comes from a VOP_GETATTR() on the file, so it looks like VOP_GETATTR() returns the wrong size. The file is definitively 791121 sectors long: #dd if=FreeBSD-12.2-RELEASE-amd64-mini-memstick.img.orig of=FreeBSD-12.2-RELEASE-amd64-mini-memstick.img 791121+0 records in 791121+0 records out #ls -l FreeBSD-12.2-RELEASE-amd64-mini-memstick.img -rw-r--r-- 1 root wheel 405053952 Apr 11 11:56 FreeBSD-12.2-RELEASE-amd64-mini-memstick.img #expr 405053952 / 512 791121 -- Manuel Bouyer NetBSD: 26 ans d'experience feront toujours la difference --
