On Thu, Jun 07, 2018 at 05:13:06PM -0700, Philip Guenther wrote: > > The GENERIC bsd kernel is happy under vmm, but booting a GENERIC.MP kernel > hits a witness panic. I suspect some "one CPU only" optimization is > resulting in the witness code being misinformed. > > Here's the boot output in the vmm console. (Yes, the userland is out of > date, but that shouldn't lead to a witness panic either.) > > > (The weird "show witness" output for scsi_base.c mutexes is because > they're on the stack and need to be unlinked from witness before > returning; that *might* be causing the problem here, but I doubt it. I'm > starting on a diff for that part...) > > > Philip Guenther >
Is this a panic inside the guest in vmm, or is this the host panicing when you're doing something while a VM is running in vmm on that host? Can't really tell from the trace here... -ml > ------- > Copyright (c) 1982, 1986, 1989, 1991, 1993 > The Regents of the University of California. All rights reserved. > Copyright (c) 1995-2018 OpenBSD. All rights reserved. https://www.OpenBSD.org > > OpenBSD 6.3-current (GENERIC.MP) #25: Thu Jun 7 16:29:55 PDT 2018 > > guenther@morgaine.local:/usr/src/sys-realclean/arch/amd64/compile/GENERIC.MP > real mem = 520093696 (496MB) > avail mem = 485457920 (462MB) > mpath0 at root > scsibus0 at mpath0: 256 targets > mainbus0 at root > bios0 at mainbus0 > acpi at bios0 not configured > cpu0 at mainbus0: (uniprocessor) > cpu0: Intel(R) Core(TM) i7-5600U CPU @ 2.60GHz, 2594.54 MHz > cpu0: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,RDSEED,ADX,SMAP,MELTDOWN > cpu0: 256KB 64b/line 8-way L2 cache > cpu0: smt 0, core 0, package 0 > pvbus0 at mainbus0: OpenBSD > pci0 at mainbus0 bus 0 > pchb0 at pci0 dev 0 function 0 "OpenBSD VMM Host" rev 0x00 > virtio0 at pci0 dev 1 function 0 "Qumranet Virtio RNG" rev 0x00 > viornd0 at virtio0 > virtio0: irq 3 > virtio1 at pci0 dev 2 function 0 "Qumranet Virtio Storage" rev 0x00 > vioblk0 at virtio1 > scsibus1 at vioblk0: 2 targets > sd0 at scsibus1 targ 0 lun 0: <VirtIO, Block Device, > SCSI3 0/direct fixed > sd0: 4096MB, 512 bytes/sector, 8388608 sectors > virtio1: irq 5 > virtio2 at pci0 dev 3 function 0 "OpenBSD VMM Control" rev 0x00 > vmmci0 at virtio2 > virtio2: irq 6 > isa0 at mainbus0 > isadma0 at isa0 > com0 at isa0 port 0x3f8/8 irq 4: ns16450, no fifo > com0: console > vscsi0 at root > scsibus2 at vscsi0: 256 targets > softraid0 at root > scsibus3 at softraid0: 256 targets > root on sd0a (0084d990f4e53393.a) swap on sd0b dump on sd0b > Automatic boot in progress: starting file system checks. > /dev/sd0a (0084d990f4e53393.a): file system is clean; not checking > /dev/sd0e (0084d990f4e53393.e): file system is clean; not checking > /dev/sd0d (0084d990f4e53393.d): file system is clean; not checking > setting tty flags > pfctl: pfctl_rules > pfctl: DIOCXROLLBACK: Invalid argument > pf enabled > starting network > pfctl: pfctl_rules > pfctl: DIOCXROLLBACK: Invalid argument > reordering libraries:panic: acquiring blockable sleep lock with spinlock or > critical section held (kernel_lock) &kernel_lock @ > /usr/src/sys-realclean/arch/amd64/amd64/intr.c:525 > Stopped at db_enter+0x5: popq %rbp > TID PID UID PRFLAGS PFLAGS CPU COMMAND > *522028 67277 0 0x14000 0x200 0 reaper > db_enter() at db_enter+0x5 > panic() at panic+0x138 > witness_checkorder(ffffffff81b7c59c,20d,0,ffffffff81cf7ca0,ffff80000002af00) > at > witness_checkorder+0xd32 > ___mp_lock(ffff80000002af00,ffff80000e0eaca0,ffffffff81bdaff0) at > ___mp_lock+0x > 70 > intr_handler(1,ffff80000002ae80) at intr_handler+0x40 > Xintr_legacy8_untramp(ffff80000e0ead60,ffffffff81d16c60,c,10,ffff80000e0ead30,f > fffffff814562c0) at Xintr_legacy8_untramp+0x155 > Xspllower(0,282,ffffffff818c9e53,1ca9c,ffffff0002570000,10) at Xspllower+0xc > uvm_pmr_freepages(1f12000,ffffff001f75e380) at uvm_pmr_freepages+0x204 > pmap_do_remove(ffffff001bd30a18,ffffff001f75f5a0,ffff80000e0ab4d0,ffffffff81053 > c20) at pmap_do_remove+0x463 > uvm_map_teardown(0) at uvm_map_teardown+0x143 > uvmspace_free(ffff80000e0f9148) at uvmspace_free+0x36 > uvm_exit(ffff80000e0f9148) at uvm_exit+0x16 > reaper() at reaper+0x156 > end trace frame: 0x0, count: 2 > https://www.openbsd.org/ddb.html describes the minimum info required in bug > reports. Insufficient info makes it difficult to find and fix bugs. > ddb{0}> > ddb{0}> show locks > exclusive mutex &uvm.fpageqlock r = 0 (0xffffffff81d1bcc0) locked @ > /usr/src/sy > s-realclean/uvm/uvm_pmemrange.c:1124 > ddb{0}> show witness > Sleep locks: > sysctllk (type: rwlock, depth: 0) -- last acquired @ > /usr/src/sys-realclean/ker > n/kern_sysctl.c:233 > &map->lock (type: rwlock, depth: 2) -- last acquired @ > /usr/src/sys-realclean/ > uvm/uvm_map.c:1936 > netlock (type: rwlock, depth: 1) -- last acquired @ > /usr/src/sys-realclean/net > inet/igmp.c:609 > pools (type: rwlock, depth: 2) -- last acquired @ > /usr/src/sys-realclean/kern > /subr_pool.c:474 > &ar->ar_lock (type: rwlock, depth: 2) -- last acquired @ > /usr/src/sys-realcle > an/net/rtable.c:500 > swplk (type: rwlock, depth: 0) -- last acquired @ > /usr/src/sys-realclean/uvm/uv > m_swap.c:615 > &ip->i_lock (type: rrwlock, depth: 1) -- last acquired @ > /usr/src/sys-realclea > n/ufs/ufs/ufs_vnops.c:1559 > &map->lock (type: rwlock, depth: 2) -- (already displayed) > &ufsdirhash_mtx (type: rwlock, depth: 2) -- last acquired @ > /usr/src/sys-real > clean/ufs/ufs/ufs_dirhash.c:238 > &dh->dh_mtx (type: rwlock, depth: 3) -- last acquired @ > /usr/src/sys-realcle > an/ufs/ufs/ufs_dirhash.c:355 > &dh->dh_mtx (type: rwlock, depth: 3) -- (already displayed) > &mp->mnt_lock (type: rwlock, depth: 0) -- last acquired @ > /usr/src/sys-realclea > n/kern/vfs_subr.c:196 > &map->lock (type: rwlock, depth: 2) -- (already displayed) > &diskp->dk_lock (type: rwlock, depth: 1) -- last acquired @ > /usr/src/sys-realc > lean/kern/subr_disk.c:1280 > &ip->i_lock (type: rrwlock, depth: 1) -- (already displayed) > &sc->sc_lock (type: rwlock, depth: 0) -- last acquired @ > /usr/src/sys-realclean > /dev/softraid.c:1199 > &diskp->dk_lock (type: rwlock, depth: 1) -- (already displayed) > &pp->pr_lock (type: rwlock, depth: 0) -- last acquired @ > /usr/src/sys-realclean > /kern/subr_pool.c:583 > &route_cb.rcb_lk (type: rwlock, depth: 0) -- last acquired @ > /usr/src/sys-realc > lean/net/rtsock.c:267 > ifclonerslk (type: rwlock, depth: 0) -- last acquired @ > /usr/src/sys-realclean/ > net/if.c:1282 > &newfdp->fd_fd.fd_lock (type: rwlock, depth: 0) -- last acquired @ > /usr/src/sys > -realclean/kern/kern_descrip.c:712 > &map->lock (type: rwlock, depth: 2) -- (already displayed) > &diskp->dk_lock (type: rwlock, depth: 1) -- (already displayed) > &ip->i_lock (type: rrwlock, depth: 1) -- (already displayed) > vfs_stall (type: rwlock, depth: 1) -- last acquired @ > /usr/src/sys-realclean/k > ern/vfs_subr.c:1642 > ptarrlk (type: rwlock, depth: 1) -- last acquired @ > /usr/src/sys-realclean/ker > n/tty_pty.c:168 > &uidinfolk (type: rwlock, depth: 0) -- last acquired @ > /usr/src/sys-realclean/k > ern/kern_proc.c:130 > &kernel_lock (type: kernel_lock, depth: 0) -- last acquired @ > /usr/src/sys-real > clean/kern/kern_exit.c:397 > > Spin locks: > &pp->pr_lock (type: mutex, depth: 0) -- last acquired @ > /usr/src/sys-realclean/ > kern/subr_pool.c:583 > &pp->pr_lock (type: mutex, depth: 0) -- last acquired @ > /usr/src/sys-realclean/ > kern/subr_pool.c:804 > /usr/src/sys-realclean/netinet/ip_input.c:101 (type: mutex, depth: 0) -- last > a > cquired @ /usr/src/sys-realclean/netinet/ip_input.c:1013 > /usr/src/sys-realclean/netinet6/frag6.c:57 (type: mutex, depth: 0) -- last > acqu > ired @ /usr/src/sys-realclean/netinet6/frag6.c:580 > &fp->f_mtx (type: mutex, depth: 0) -- last acquired @ > /usr/src/sys-realclean/ke > rn/sys_generic.c:360 > (null) (type: mutex, depth: 0) -- last acquired @ > /usr/src/sys-realclean/scsi/s > csi_base.c:1338 > &sched_lock (type: sched_lock, depth: 3) -- last acquired @ > /usr/src/sys-realc > lean/kern/kern_synch.c:444 > /usr/src/sys-realclean/kern/kern_timeout.c:92 (type: mutex, depth: 4) -- > last > acquired @ /usr/src/sys-realclean/kern/kern_timeout.c:315 > (null) (type: mutex, depth: 0) -- last acquired @ > /usr/src/sys-realclean/scsi/s > csi_base.c:1338 > &sched_lock (type: sched_lock, depth: 3) -- (already displayed) > Hsv (type: mutex, depth: 0) -- last acquired @ > /usr/src/sys-realclean/scsi/scsi > _base.c:1338 > &sched_lock (type: sched_lock, depth: 3) -- (already displayed) > (type: mutex, depth: 0) -- last acquired @ > /usr/src/sys-realclean/scsi/scsi_ba > se.c:1338 > &sched_lock (type: sched_lock, depth: 3) -- (already displayed) > "G`/} (type: mutex, depth: 0) -- last acquired @ > /usr/src/sys-realclean/scsi/s > csi_base.c:1338 > &sched_lock (type: sched_lock, depth: 3) -- (already displayed) > 6 (type: mutex, depth: 0) -- last acquired @ > /usr/src/sys-realclean/scsi/scsi_b > ase.c:1338 > &sched_lock (type: sched_lock, depth: 3) -- (already displayed) > @ (type: mutex, depth: 0) -- last acquired @ > /usr/src/sys-realclean/scsi/scsi_b > ase.c:1338 > &sched_lock (type: sched_lock, depth: 3) -- (already displayed) > P& (type: mutex, depth: 0) -- last acquired @ > /usr/src/sys-realclean/scsi > /scsi_base.c:1338 > &sched_lock (type: sched_lock, depth: 3) -- (already displayed) > /usr/src/sys-realclean/kern/kern_synch.c (type: mutex, depth: 0) -- last > acquir > ed @ /usr/src/sys-realclean/scsi/scsi_base.c:1338 > &sched_lock (type: sched_lock, depth: 3) -- (already displayed) > uvm_fault(0xffffffff81cd9200, 0xd00, 0, 1) -> e > kernel: page fault trap, code=0 > Faulted in DDB; continuing... > ddb{0}> >
