---- On Thu, 07 Jul 2016 06:36:19 -0700 Larry Rosenman <l...@lerctr.org> wrote 
---- 
 > Thanks for that.  I've added myself to the cc list, and a comment about 
 > having 2 vmcore's.
 > 
This was introduced by  302350.  It broke the return value of 
callout_{stop,drain}. returning 1 even if the callout system did not hold a 
reference. That in turn broke the following code in lltable_free:

        LIST_FOREACH_SAFE(lle, &dchain, lle_chain, next) {
                if (callout_stop(&lle->lle_timer) > 0)
                        LLE_REMREF(lle);
                llentry_free(lle);
        }


 > 
 > On 2016-07-07 08:28, Edward Tomasz Napierała wrote:
 > > FWIW, I'm seeing this too.  I've filed a PR:
 > > 
 > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=210884
 > > 
 > > On 0707T0813, Larry Rosenman wrote:
 > >> and now it's been up for 13+ hours.  I do have both VMCORE's from the 
 > >> 2
 > >> crashes.
 > >> 
 > >> 
 > >> 
 > >> On 2016-07-06 18:22, Larry Rosenman wrote:
 > >> > Got a similar crash a few minutes later.
 > >> >
 > >> >
 > >> > On 2016-07-06 18:17, Larry Rosenman wrote:
 > >> >> First boot, and I got the following panic.  2nd boot ran just fine.
 > >> >>
 > >> >>
 > >> >> borg.lerctr.org dumped core - see /var/crash/vmcore.0
 > >> >>
 > >> >> Wed Jul  6 18:13:34 CDT 2016
 > >> >>
 > >> >> FreeBSD borg.lerctr.org 11.0-ALPHA6 FreeBSD 11.0-ALPHA6 #5 r302379:
 > >> >> Wed Jul  6 16:59:11 CDT 2016
 > >> >> r...@borg.lerctr.org:/usr/obj/usr/src/sys/VT-LER  amd64
 > >> >>
 > >> >> panic: bogus refcnt 0 on lle 0xfffff800aa941200
 > >> >>
 > >> >> GNU gdb 6.1.1 [FreeBSD]
 > >> >> Copyright 2004 Free Software Foundation, Inc.
 > >> >> GDB is free software, covered by the GNU General Public License, and
 > >> >> you are
 > >> >> welcome to change it and/or distribute copies of it under certain
 > >> >> conditions.
 > >> >> Type "show copying" to see the conditions.
 > >> >> There is absolutely no warranty for GDB.  Type "show warranty" for
 > >> >> details.
 > >> >> This GDB was configured as "amd64-marcel-freebsd"...
 > >> >>
 > >> >> Unread portion of the kernel message buffer:
 > >> >> Copyright (c) 1992-2016 The FreeBSD Project.
 > >> >> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993,
 > >> >> 1994
 > >> >>     The Regents of the University of California. All rights reserved.
 > >> >> FreeBSD is a registered trademark of The FreeBSD Foundation.
 > >> >> FreeBSD 11.0-ALPHA6 #5 r302379: Wed Jul  6 16:59:11 CDT 2016
 > >> >>     r...@borg.lerctr.org:/usr/obj/usr/src/sys/VT-LER amd64
 > >> >> FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on
 > >> >> LLVM 3.8.0)
 > >> >> can't re-use a leaf (ixl_rx_miss_bufs)!
 > >> >> MEMGUARD DEBUGGING ALLOCATOR INITIALIZED:
 > >> >>     MEMGUARD map base: 0xfffffe0000400000
 > >> >>     MEMGUARD map size: 128604256 KBytes
 > >> >> VT(vga): resolution 640x480
 > >> >> CPU: Intel(R) Xeon(R) CPU           E5410  @ 2.33GHz (2327.55-MHz
 > >> >> K8-class CPU)
 > >> >>   Origin="GenuineIntel"  Id=0x10676  Family=0x6  Model=0x17
 > >> >> Stepping=6
 > >> >>
 > >> >> Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
 > >> >>
 > >> >> Features2=0xce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1>
 > >> >>   AMD Features=0x20100800<SYSCALL,NX,LM>
 > >> >>   AMD Features2=0x1<LAHF>
 > >> >>   VT-x: HLT,PAUSE
 > >> >>   TSC: P-state invariant, performance statistics
 > >> >> real memory  = 68719476736 (65536 MB)
 > >> >> avail memory = 65382842368 (62353 MB)
 > >> >> Event timer "LAPIC" quality 400
 > >> >> ACPI APIC Table: <PTLTD       APIC  >
 > >> >> FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
 > >> >> FreeBSD/SMP: 2 package(s) x 4 core(s)
 > >> >> random: unblocking device.
 > >> >> ioapic0 <Version 2.0> irqs 0-23 on motherboard
 > >> >> ioapic1 <Version 2.0> irqs 24-47 on motherboard
 > >> >> random: entropy device external interface
 > >> >> netmap: loaded module
 > >> >> module_register_init: MOD_LOAD (vesa, 0xffffffff80f2cb40, 0) error 19
 > >> >> kbd1 at kbdmux0
 > >> >> vtvga0: <VT VGA driver> on motherboard
 > >> >> cryptosoft0: <software crypto> on motherboard
 > >> >> acpi0: <PTLTD   RSDT> on motherboard
 > >> >> acpi0: Power Button (fixed)
 > >> >> unknown: I/O range not supported
 > >> >> cpu0: <ACPI CPU> on acpi0
 > >> >> cpu1: <ACPI CPU> on acpi0
 > >> >> cpu2: <ACPI CPU> on acpi0
 > >> >> cpu3: <ACPI CPU> on acpi0
 > >> >> cpu4: <ACPI CPU> on acpi0
 > >> >> cpu5: <ACPI CPU> on acpi0
 > >> >> cpu6: <ACPI CPU> on acpi0
 > >> >> cpu7: <ACPI CPU> on acpi0
 > >> >> hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq
 > >> >> 0,8 on acpi0
 > >> >> Timecounter "HPET" frequency 14318180 Hz quality 950
 > >> >> Event timer "HPET" frequency 14318180 Hz quality 350
 > >> >> Event timer "HPET1" frequency 14318180 Hz quality 340
 > >> >> Event timer "HPET2" frequency 14318180 Hz quality 340
 > >> >> atrtc0: <AT realtime clock> port 0x70-0x71 on acpi0
 > >> >> Event timer "RTC" frequency 32768 Hz quality 0
 > >> >> attimer0: <AT timer> port 0x40-0x43,0x50-0x53 on acpi0
 > >> >> Timecounter "i8254" frequency 1193182 Hz quality 0
 > >> >> Event timer "i8254" frequency 1193182 Hz quality 100
 > >> >> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
 > >> >> acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
 > >> >> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
 > >> >> pci0: <ACPI PCI bus> on pcib0
 > >> >> pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
 > >> >> pci1: <ACPI PCI bus> on pcib1
 > >> >> pcib2: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci1
 > >> >> pci2: <ACPI PCI bus> on pcib2
 > >> >> pcib3: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci2
 > >> >> pci3: <ACPI PCI bus> on pcib3
 > >> >> pcib4: <ACPI PCI-PCI bridge> at device 0.0 on pci3
 > >> >> pci4: <ACPI PCI bus> on pcib4
 > >> >> pcib5: <ACPI PCI-PCI bridge> at device 0.2 on pci3
 > >> >> pci5: <ACPI PCI bus> on pcib5
 > >> >> pcib6: <ACPI PCI-PCI bridge> irq 18 at device 2.0 on pci2
 > >> >> pci6: <ACPI PCI bus> on pcib6
 > >> >> em0: <Intel(R) PRO/1000 Network Connection 7.6.1-k> port 0x2000-0x201f
 > >> >> mem 0xd9220000-0xd923ffff,0xd9200000-0xd921ffff irq 18 at device 0.0
 > >> >> on pci6
 > >> >> em0: Using an MSI interrupt
 > >> >> em0: Ethernet address: 00:30:48:f2:29:9c
 > >> >> em0: netmap queues/slots: TX 1/1024, RX 1/1024
 > >> >> em1: <Intel(R) PRO/1000 Network Connection 7.6.1-k> port 0x2020-0x203f
 > >> >> mem 0xd9260000-0xd927ffff,0xd9240000-0xd925ffff irq 19 at device 0.1
 > >> >> on pci6
 > >> >> em1: Using an MSI interrupt
 > >> >> em1: Ethernet address: 00:30:48:f2:29:9d
 > >> >> em1: netmap queues/slots: TX 1/1024, RX 1/1024
 > >> >> pcib7: <ACPI PCI-PCI bridge> at device 0.3 on pci1
 > >> >> pci7: <ACPI PCI bus> on pcib7
 > >> >> pcib8: <ACPI PCI-PCI bridge> at device 4.0 on pci0
 > >> >> pci8: <ACPI PCI bus> on pcib8
 > >> >> vgapci0: <VGA-compatible display> port 0x3000-0x307f mem
 > >> >> 0xd8000000-0xd8ffffff,0xc0000000-0xc7ffffff,0xc8000000-0xc9ffffff irq
 > >> >> 36 at device 0.0 on pci8
 > >> >> hdac0: <NVIDIA GF119 HDA Controller> mem 0xd9000000-0xd9003fff irq 37
 > >> >> at device 0.1 on pci8
 > >> >> pcib9: <ACPI PCI-PCI bridge> at device 6.0 on pci0
 > >> >> pci9: <ACPI PCI bus> on pcib9
 > >> >> pcib10: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
 > >> >> pcib10: [GIANT-LOCKED]
 > >> >> pci10: <ACPI PCI bus> on pcib10
 > >> >> pcib11: <PCI-PCI bridge> irq 32 at device 0.0 on pci10
 > >> >> pci11: <PCI bus> on pcib11
 > >> >> pcm0: <Envy24HT audio (Generic)> port 0x4080-0x409f,0x4000-0x407f irq
 > >> >> 32 at device 0.0 on pci11
 > >> >> pcm0: system configuration
 > >> >>   SubVendorID: 0x1412, SubDeviceID: 0x2403
 > >> >>   XIN2 Clock Source: 24.576MHz(96kHz*256)
 > >> >>   MPU-401 UART(s) #: not implemented
 > >> >>   ADC #: 1 and SPDIF receiver connected
 > >> >>   DAC #: 4
 > >> >>   Multi-track converter type: AC'97(SDATA_OUT:packed)
 > >> >>   S/PDIF(IN/OUT): 1/1 ID# 0x00
 > >> >>   GPIO(mask/dir/state): 0xff/0xff/0xff
 > >> >> uhci0: <Intel 631XESB/632XESB/3100 USB controller USB-1> port
 > >> >> 0x1800-0x181f irq 17 at device 29.0 on pci0
 > >> >> usbus0 on uhci0
 > >> >> uhci1: <Intel 631XESB/632XESB/3100 USB controller USB-2> port
 > >> >> 0x1820-0x183f irq 19 at device 29.1 on pci0
 > >> >> usbus1 on uhci1
 > >> >> uhci2: <Intel 631XESB/632XESB/3100 USB controller USB-3> port
 > >> >> 0x1840-0x185f irq 18 at device 29.2 on pci0
 > >> >> usbus2 on uhci2
 > >> >> ehci0: <Intel 63XXESB USB 2.0 controller> mem 0xd9600400-0xd96007ff
 > >> >> irq 17 at device 29.7 on pci0
 > >> >> usbus3: EHCI version 1.0
 > >> >> usbus3 on ehci0
 > >> >> pcib12: <ACPI PCI-PCI bridge> at device 30.0 on pci0
 > >> >> pci12: <ACPI PCI bus> on pcib12
 > >> >> vgapci1: <VGA-compatible display> port 0x5000-0x50ff mem
 > >> >> 0xd0000000-0xd7ffffff,0xd9300000-0xd930ffff irq 18 at device 1.0 on
 > >> >> pci12
 > >> >> vgapci1: Boot video device
 > >> >> isab0: <PCI-ISA bridge> at device 31.0 on pci0
 > >> >> isa0: <ISA bus> on isab0
 > >> >> atapci0: <Intel 63XXESB2 UDMA100 controller> port
 > >> >> 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1860-0x186f at device 31.1 on
 > >> >> pci0
 > >> >> ata0: <ATA channel> at channel 0 on atapci0
 > >> >> ahci0: <Intel ESB2 AHCI SATA controller> port
 > >> >> 0x18a0-0x18a7,0x1874-0x1877,0x1878-0x187f,0x1870-0x1873,0x1880-0x189f
 > >> >> mem 0xd9600800-0xd9600bff irq 19 at device 31.2 on pci0
 > >> >> ahci0: AHCI v1.10 with 6 3Gbps ports, Port Multiplier supported
 > >> >> ahcich0: <AHCI channel> at channel 0 on ahci0
 > >> >> ahcich1: <AHCI channel> at channel 1 on ahci0
 > >> >> ahcich2: <AHCI channel> at channel 2 on ahci0
 > >> >> ahcich3: <AHCI channel> at channel 3 on ahci0
 > >> >> ahcich4: <AHCI channel> at channel 4 on ahci0
 > >> >> ahcich5: <AHCI channel> at channel 5 on ahci0
 > >> >> ichsmb0: <Intel 631xESB/6321ESB (ESB2) SMBus controller> port
 > >> >> 0x1100-0x111f irq 19 at device 31.3 on pci0
 > >> >> smbus0: <System Management Bus> on ichsmb0
 > >> >> acpi_button0: <Power Button> on acpi0
 > >> >> ipmi0: <IPMI System Interface> port 0xca2-0xca3 on acpi0
 > >> >> ipmi0: KCS mode found at io 0xca2 on acpi
 > >> >> atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
 > >> >> atkbd0: <AT Keyboard> irq 1 on atkbdc0
 > >> >> kbd0 at atkbd0
 > >> >> atkbd0: [GIANT-LOCKED]
 > >> >> uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on
 > >> >> acpi0
 > >> >> uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
 > >> >> fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on
 > >> >> acpi0
 > >> >> fd0: <1440-KB 3.5" drive> on fdc0 drive 0
 > >> >> ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on
 > >> >> acpi0
 > >> >> ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
 > >> >> ppc0: FIFO with 16/16/9 bytes threshold
 > >> >> ppbus0: <Parallel port bus> on ppc0
 > >> >> lpt0: <Printer> on ppbus0
 > >> >> lpt0: Interrupt-driven port
 > >> >> ppi0: <Parallel I/O> on ppbus0
 > >> >> ichwd0: <Intel 63XXESB watchdog timer> on isa0
 > >> >> orm0: <ISA Option ROM> at iomem 0xc0000-0xcafff on isa0
 > >> >> coretemp0: <CPU On-Die Thermal Sensors> on cpu0
 > >> >> est0: <Enhanced SpeedStep Frequency Control> on cpu0
 > >> >> coretemp1: <CPU On-Die Thermal Sensors> on cpu1
 > >> >> est1: <Enhanced SpeedStep Frequency Control> on cpu1
 > >> >> coretemp2: <CPU On-Die Thermal Sensors> on cpu2
 > >> >> est2: <Enhanced SpeedStep Frequency Control> on cpu2
 > >> >> coretemp3: <CPU On-Die Thermal Sensors> on cpu3
 > >> >> est3: <Enhanced SpeedStep Frequency Control> on cpu3
 > >> >> coretemp4: <CPU On-Die Thermal Sensors> on cpu4
 > >> >> est4: <Enhanced SpeedStep Frequency Control> on cpu4
 > >> >> coretemp5: <CPU On-Die Thermal Sensors> on cpu5
 > >> >> est5: <Enhanced SpeedStep Frequency Control> on cpu5
 > >> >> coretemp6: <CPU On-Die Thermal Sensors> on cpu6
 > >> >> est6: <Enhanced SpeedStep Frequency Control> on cpu6
 > >> >> coretemp7: <CPU On-Die Thermal Sensors> on cpu7
 > >> >> est7: <Enhanced SpeedStep Frequency Control> on cpu7
 > >> >> usbus0: 12Mbps Full Speed USB v1.0
 > >> >> fuse-freebsd: version 0.4.4, FUSE ABI 7.8
 > >> >> ZFS filesystem version: 5
 > >> >> ZFS storage pool version: features support (5000)
 > >> >> Timecounters tick every 1.000 msec
 > >> >> nvme cam probe device init
 > >> >> hdacc0: <NVIDIA (0x001c) HDA CODEC> at cad 0 on hdac0
 > >> >> hdaa0: <NVIDIA (0x001c) Audio Function Group> at nid 1 on hdacc0
 > >> >> pcm1: <NVIDIA (0x001c) (HDMI/DP 8ch)> at nid 4 on hdaa0
 > >> >> pcm2: <NVIDIA (0x001c) (HDMI/DP 8ch)> at nid 5 on hdaa0
 > >> >> ipmi0: IPMI device rev. 1, firmware rev. 1.64, version 2.0
 > >> >> ipmi0: Number of channels 8
 > >> >> ipmi0: Attached watchdog
 > >> >> usbus1: 12Mbps Full Speed USB v1.0
 > >> >> usbus2: 12Mbps Full Speed USB v1.0
 > >> >> ugen0.1: <Intel> at usbus0
 > >> >> uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on
 > >> >> usbus0
 > >> >> ugen1.1: <Intel> at usbus1
 > >> >> uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on
 > >> >> usbus1
 > >> >> ugen2.1: <Intel> at usbus2
 > >> >> uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on
 > >> >> usbus2
 > >> >> usbus3: 480Mbps High Speed USB v2.0
 > >> >> ugen3.1: <Intel> at usbus3
 > >> >> uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on
 > >> >> usbus3
 > >> >> ata0: DMA limited to UDMA33, controller found non-ATA66 cable
 > >> >> uhub0: 2 ports with 2 removable, self powered
 > >> >> uhub1: 2 ports with 2 removable, self powered
 > >> >> uhub2: 2 ports with 2 removable, self powered
 > >> >> ada0 at ahcich0 bus 0 scbus1 target 0 lun 0
 > >> >> ada0: <ST4000DX001-1CE168 CC44> ACS-2 ATA SATA 3.x device
 > >> >> cd0 at ata0 bus 0 scbus0 target 0 lun 0
 > >> >> cd0: <Memorex DVD+-RAM 510L v1 MWS7> Removable CD-ROM SCSI device
 > >> >> cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes)
 > >> >> cd0: Attempt to query device size failed: NOT READY, Medium not
 > >> >> present
 > >> >> ada0: Serial Number Z3032MSL
 > >> >> ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
 > >> >> ada0: 3815447MB (7814037168 512 byte sectors)
 > >> >> ada1 at ahcich1 bus 0 scbus2 target 0 lun 0
 > >> >> ada1: <ST2000DL003-9VT166 CC3D> ATA8-ACS SATA 3.x device
 > >> >> ada1: Serial Number 5YD6FPLG
 > >> >> ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
 > >> >> ada1: 1907729MB (3907029168 512 byte sectors)
 > >> >> ada1: quirks=0x1<4K>
 > >> >> ada2 at ahcich2 bus 0 scbus3 target 0 lun 0
 > >> >> ada2: <ST2000DL003-9VT166 CC32> ATA8-ACS SATA 3.x device
 > >> >> ada2: Serial Number 5YDA3PC5
 > >> >> ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
 > >> >> ada2: 1907729MB (3907029168 512 byte sectors)
 > >> >> ada2: quirks=0x1<4K>
 > >> >> ada3 at ahcich3 bus 0 scbus4 target 0 lun 0
 > >> >> ada3: <TOSHIBA MD04ACA400 FP2A> ATA8-ACS SATA 3.x device
 > >> >> ada3: Serial Number Y4ODK8VHFSBA
 > >> >> ada3: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
 > >> >> ada3: 3815447MB (7814037168 512 byte sectors)
 > >> >> ada4 at ahcich4 bus 0 scbus5 target 0 lun 0
 > >> >> ada4: <ST4000DM000-1F2168 CC54> ACS-2 ATA SATA 3.x device
 > >> >> ada4: Serial Number S30144ZV
 > >> >> ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
 > >> >> ada4: 3815447MB (7814037168 512 byte sectors)
 > >> >> ada4: quirks=0x1<4K>
 > >> >> ada5 at ahcich5 bus 0 scbus6 target 0 lun 0
 > >> >> ada5: <ST2000DL003-9VT166 CC3D> ATA8-ACS SATA 3.x device
 > >> >> ada5: Serial Number 5YD5RBS8
 > >> >> ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
 > >> >> ada5: 1907729MB (3907029168 512 byte sectors)
 > >> >> ada5: quirks=0x1<4K>
 > >> >> SMP: AP CPU #6 Launched!
 > >> >> SMP: AP CPU #4 Launched!
 > >> >> SMP: AP CPU #7 Launched!
 > >> >> SMP: AP CPU #2 Launched!
 > >> >> SMP: AP CPU #5 Launched!
 > >> >> SMP: AP CPU #1 Launched!
 > >> >> SMP: AP CPU #3 Launched!
 > >> >> Timecounter "TSC-low" frequency 1163772834 Hz quality 1000
 > >> >> hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI> TSC/1/64/0x20<REA>
 > >> >> IAP/2/40/0x3ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA,PRC>
 > >> >> IAF/3/40/0x67<INT,USR,SYS,REA,WRI>
 > >> >> Trying to mount root from zfs:zroot/ROOT/oldroot []...
 > >> >> Root mount waiting for: usbus3
 > >> >> uhub3: 6 ports with 6 removable, self powered
 > >> >> Root mount waiting for: usbus3
 > >> >> Root mount waiting for: usbus3
 > >> >> ugen3.2: <Peppercon AG> at usbus3
 > >> >> ukbd0: <Peppercon AG Multidevice, class 0/0, rev 2.00/0.01, addr 2> on
 > >> >> usbus3
 > >> >> kbd2 at ukbd0
 > >> >> ugen0.2: <American Power Conversion> at usbus0
 > >> >> <118>Setting hostuuid: 53d19f64-d663-a017-8922-003048f2299c.
 > >> >> <118>Setting hostid: 0xbf916f25.
 > >> >> <118>Starting file system checks:
 > >> >> <118>Mounting local filesystems:.
 > >> >> <118>ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib
 > >> >> /usr/local/lib/compat /usr/local/lib/gcc48 /usr/local/lib/mysql
 > >> >> /usr/local/lib/perl5/5.20/mach/CORE /usr/local/lib/qt4
 > >> >> /usr/local/lib/virtualbox /usr/local/llvm36/lib /usr/local/llvm37/lib
 > >> >> <118>32-bit compatibility ldconfig path: /usr/lib32
 > >> >> /usr/local/lib32/compat
 > >> >> <118>/etc/rc: WARNING: $kldxref_enable is not set properly - see
 > >> >> rc.conf(5).
 > >> >> <118>Setting hostname: borg.lerctr.org.
 > >> >> <118>Setting up harvesting:
 > >> >> [UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,NET_ETHER,NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED
 > >> >> <118>Feeding entropy: .
 > >> >> <5>lagg0: link state changed to DOWN
 > >> >> <118>Created clone interfaces: lagg0.
 > >> >> <118>Starting Network: lo0 em0 em1 lagg0.
 > >> >> <118>lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
 > >> >> <118>    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
 > >> >> <118>    inet6 ::1 prefixlen 128
 > >> >> <118>    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
 > >> >> <118>    inet 127.0.0.1 netmask 0xff000000
 > >> >> <118>    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
 > >> >> <118>    groups: lo
 > >> >> <118>em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
 > >> >> mtu 1500
 > >> >> <118>    
 > >> >> options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
 > >> >> <118>    ether 00:30:48:f2:29:9c
 > >> >> <118>    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
 > >> >> <118>    media: Ethernet autoselect
 > >> >> <118>    status: no carrier
 > >> >> <118>em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
 > >> >> mtu 1500
 > >> >> <118>    
 > >> >> options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
 > >> >> <118>    ether 00:30:48:f2:29:9c
 > >> >> <118>    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
 > >> >> <118>    media: Ethernet autoselect
 > >> >> <118>    status: no carrier
 > >> >> <118>lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric
 > >> >> 0 mtu 1500
 > >> >> <118>    
 > >> >> options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
 > >> >> <118>    ether 00:30:48:f2:29:9c
 > >> >> <118>    inet 192.168.200.4 netmask 0xfffffc00 broadcast 
 > >> >> 192.168.203.255
 > >> >> <118>    inet6 fe80::230:48ff:fef2:299c%lagg0 prefixlen 64 scopeid 0x4
 > >> >> <118>    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
 > >> >> <118>    media: Ethernet autoselect
 > >> >> <118>    status: no carrier
 > >> >> <118>    groups: lagg
 > >> >> <118>    laggproto loadbalance lagghash l2,l3,l4
 > >> >> <118>    laggport: em0 flags=0<>
 > >> >> <118>    laggport: em1 flags=0<>
 > >> >> <118>Starting devd.
 > >> >> uhid0: <Peppercon AG Multidevice, class 0/0, rev 2.00/0.01, addr 2> on
 > >> >> usbus3
 > >> >> <118>add host 127.0.0.1: gateway lo0 fib 0: route already in table
 > >> >> <118>add net default: gateway 192.168.200.11
 > >> >> <118>add host ::1: gateway lo0 fib 0: route already in table
 > >> >> <118>add net fe80::: gateway ::1
 > >> >> <118>add net ff02::: gateway ::1
 > >> >> <118>add net ::ffff:0.0.0.0: gateway ::1
 > >> >> <118>add net ::0.0.0.0: gateway ::1
 > >> >> <118>Starting local_unbound.
 > >> >> <5>em0: link state changed to UP
 > >> >> <5>lagg0: link state changed to UP
 > >> >> <118>Waiting for nameserver to start... good
 > >> >> <118>Starting rtsold.
 > >> >> <5>em1: link state changed to UP
 > >> >> panic: bogus refcnt 0 on lle 0xfffff800aa941200
 > >> >> cpuid = 4
 > >> >> KDB: stack backtrace:
 > >> >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
 > >> >> 0xfffffe2dfe57b1a0
 > >> >> vpanic() at vpanic+0x182/frame 0xfffffe2dfe57b220
 > >> >> kassert_panic() at kassert_panic+0x126/frame 0xfffffe2dfe57b290
 > >> >> llentry_free() at llentry_free+0x136/frame 0xfffffe2dfe57b2c0
 > >> >> nd6_cache_lladdr() at nd6_cache_lladdr+0x2ff/frame 0xfffffe2dfe57b390
 > >> >> nd6_ra_input() at nd6_ra_input+0x1347/frame 0xfffffe2dfe57b670
 > >> >> icmp6_input() at icmp6_input+0xa93/frame 0xfffffe2dfe57b810
 > >> >> ip6_input() at ip6_input+0xc70/frame 0xfffffe2dfe57b8f0
 > >> >> netisr_dispatch_src() at netisr_dispatch_src+0x80/frame
 > >> >> 0xfffffe2dfe57b950
 > >> >> ether_demux() at ether_demux+0x14a/frame 0xfffffe2dfe57b980
 > >> >> ether_nh_input() at ether_nh_input+0x340/frame 0xfffffe2dfe57b9c0
 > >> >> netisr_dispatch_src() at netisr_dispatch_src+0x80/frame
 > >> >> 0xfffffe2dfe57ba20
 > >> >> ether_input() at ether_input+0x62/frame 0xfffffe2dfe57ba50
 > >> >> if_input() at if_input+0xa/frame 0xfffffe2dfe57ba60
 > >> >> em_rxeof() at em_rxeof+0x28c/frame 0xfffffe2dfe57bae0
 > >> >> em_handle_que() at em_handle_que+0x40/frame 0xfffffe2dfe57bb20
 > >> >> taskqueue_run_locked() at taskqueue_run_locked+0x13c/frame
 > >> >> 0xfffffe2dfe57bb80
 > >> >> taskqueue_thread_loop() at taskqueue_thread_loop+0x88/frame
 > >> >> 0xfffffe2dfe57bbb0
 > >> >> fork_exit() at fork_exit+0x84/frame 0xfffffe2dfe57bbf0
 > >> >> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe2dfe57bbf0
 > >> >> --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
 > >> >> Uptime: 22s
 > >> >> Dumping 3120 out of 64465
 > >> >> MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
 > >> >>
 > >> >> Reading symbols from /boot/kernel/zfs.ko...Reading symbols from
 > >> >> /usr/lib/debug//boot/kernel/zfs.ko.debug...done.
 > >> >> done.
 > >> >> Loaded symbols for /boot/kernel/zfs.ko
 > >> >> Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols
 > >> >> from /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done.
 > >> >> done.
 > >> >> Loaded symbols for /boot/kernel/opensolaris.ko
 > >> >> Reading symbols from /boot/kernel/linux.ko...Reading symbols from
 > >> >> /usr/lib/debug//boot/kernel/linux.ko.debug...done.
 > >> >> done.
 > >> >> Loaded symbols for /boot/kernel/linux.ko
 > >> >> Reading symbols from /boot/kernel/linux_common.ko...Reading symbols
 > >> >> from /usr/lib/debug//boot/kernel/linux_common.ko.debug...done.
 > >> >> done.
 > >> >> Loaded symbols for /boot/kernel/linux_common.ko
 > >> >> Reading symbols from /boot/kernel/if_lagg.ko...Reading symbols from
 > >> >> /usr/lib/debug//boot/kernel/if_lagg.ko.debug...done.
 > >> >> done.
 > >> >> Loaded symbols for /boot/kernel/if_lagg.ko
 > >> >> Reading symbols from /boot/kernel/snd_envy24ht.ko...Reading symbols
 > >> >> from /usr/lib/debug//boot/kernel/snd_envy24ht.ko.debug...done.
 > >> >> done.
 > >> >> Loaded symbols for /boot/kernel/snd_envy24ht.ko
 > >> >> Reading symbols from /boot/kernel/snd_spicds.ko...Reading symbols from
 > >> >> /usr/lib/debug//boot/kernel/snd_spicds.ko.debug...done.
 > >> >> done.
 > >> >> Loaded symbols for /boot/kernel/snd_spicds.ko
 > >> >> Reading symbols from /boot/kernel/coretemp.ko...Reading symbols from
 > >> >> /usr/lib/debug//boot/kernel/coretemp.ko.debug...done.
 > >> >> done.
 > >> >> Loaded symbols for /boot/kernel/coretemp.ko
 > >> >> Reading symbols from /boot/kernel/fuse.ko...Reading symbols from
 > >> >> /usr/lib/debug//boot/kernel/fuse.ko.debug...done.
 > >> >> done.
 > >> >> Loaded symbols for /boot/kernel/fuse.ko
 > >> >> Reading symbols from /boot/kernel/ichsmb.ko...Reading symbols from
 > >> >> /usr/lib/debug//boot/kernel/ichsmb.ko.debug...done.
 > >> >> done.
 > >> >> Loaded symbols for /boot/kernel/ichsmb.ko
 > >> >> Reading symbols from /boot/kernel/smbus.ko...Reading symbols from
 > >> >> /usr/lib/debug//boot/kernel/smbus.ko.debug...done.
 > >> >> done.
 > >> >> Loaded symbols for /boot/kernel/smbus.ko
 > >> >> Reading symbols from /boot/kernel/ichwd.ko...Reading symbols from
 > >> >> /usr/lib/debug//boot/kernel/ichwd.ko.debug...done.
 > >> >> done.
 > >> >> Loaded symbols for /boot/kernel/ichwd.ko
 > >> >> Reading symbols from /boot/kernel/cpuctl.ko...Reading symbols from
 > >> >> /usr/lib/debug//boot/kernel/cpuctl.ko.debug...done.
 > >> >> done.
 > >> >> Loaded symbols for /boot/kernel/cpuctl.ko
 > >> >> Reading symbols from /boot/kernel/cryptodev.ko...Reading symbols from
 > >> >> /usr/lib/debug//boot/kernel/cryptodev.ko.debug...done.
 > >> >> done.
 > >> >> Loaded symbols for /boot/kernel/cryptodev.ko
 > >> >> Reading symbols from /boot/kernel/dtraceall.ko...Reading symbols from
 > >> >> /usr/lib/debug//boot/kernel/dtraceall.ko.debug...done.
 > >> >> done.
 > >> >> Loaded symbols for /boot/kernel/dtraceall.ko
 > >> >> Reading symbols from /boot/kernel/profile.ko...Reading symbols from
 > >> >> /usr/lib/debug//boot/kernel/profile.ko.debug...done.
 > >> >> done.
 > >> >> Loaded symbols for /boot/kernel/profile.ko
 > >> >> Reading symbols from /boot/kernel/dtrace.ko...Reading symbols from
 > >> >> /usr/lib/debug//boot/kernel/dtrace.ko.debug...done.
 > >> >> done.
 > >> >> Loaded symbols for /boot/kernel/dtrace.ko
 > >> >> Reading symbols from /boot/kernel/systrace_freebsd32.ko...Reading
 > >> >> symbols from
 > >> >> /usr/lib/debug//boot/kernel/systrace_freebsd32.ko.debug...done.
 > >> >> done.
 > >> >> Loaded symbols for /boot/kernel/systrace_freebsd32.ko
 > >> >> Reading symbols from /boot/kernel/systrace.ko...Reading symbols from
 > >> >> /usr/lib/debug//boot/kernel/systrace.ko.debug...done.
 > >> >> done.
 > >> >> Loaded symbols for /boot/kernel/systrace.ko
 > >> >> Reading symbols from /boot/kernel/sdt.ko...Reading symbols from
 > >> >> /usr/lib/debug//boot/kernel/sdt.ko.debug...done.
 > >> >> done.
 > >> >> Loaded symbols for /boot/kernel/sdt.ko
 > >> >> Reading symbols from /boot/kernel/fasttrap.ko...Reading symbols from
 > >> >> /usr/lib/debug//boot/kernel/fasttrap.ko.debug...done.
 > >> >> done.
 > >> >> Loaded symbols for /boot/kernel/fasttrap.ko
 > >> >> Reading symbols from /boot/kernel/fbt.ko...Reading symbols from
 > >> >> /usr/lib/debug//boot/kernel/fbt.ko.debug...done.
 > >> >> done.
 > >> >> Loaded symbols for /boot/kernel/fbt.ko
 > >> >> Reading symbols from /boot/kernel/dtnfscl.ko...Reading symbols from
 > >> >> /usr/lib/debug//boot/kernel/dtnfscl.ko.debug...done.
 > >> >> done.
 > >> >> Loaded symbols for /boot/kernel/dtnfscl.ko
 > >> >> Reading symbols from /boot/kernel/dtmalloc.ko...Reading symbols from
 > >> >> /usr/lib/debug//boot/kernel/dtmalloc.ko.debug...done.
 > >> >> done.
 > >> >> Loaded symbols for /boot/kernel/dtmalloc.ko
 > >> >> Reading symbols from /boot/kernel/ipmi.ko...Reading symbols from
 > >> >> /usr/lib/debug//boot/kernel/ipmi.ko.debug...done.
 > >> >> done.
 > >> >> Loaded symbols for /boot/kernel/ipmi.ko
 > >> >> Reading symbols from /boot/kernel/ipmi_linux.ko...Reading symbols from
 > >> >> /usr/lib/debug//boot/kernel/ipmi_linux.ko.debug...done.
 > >> >> done.
 > >> >> Loaded symbols for /boot/kernel/ipmi_linux.ko
 > >> >> Reading symbols from /boot/kernel/hwpmc.ko...Reading symbols from
 > >> >> /usr/lib/debug//boot/kernel/hwpmc.ko.debug...done.
 > >> >> done.
 > >> >> Loaded symbols for /boot/kernel/hwpmc.ko
 > >> >> Reading symbols from /boot/kernel/filemon.ko...Reading symbols from
 > >> >> /usr/lib/debug//boot/kernel/filemon.ko.debug...done.
 > >> >> done.
 > >> >> Loaded symbols for /boot/kernel/filemon.ko
 > >> >> Reading symbols from /boot/kernel/uhid.ko...Reading symbols from
 > >> >> /usr/lib/debug//boot/kernel/uhid.ko.debug...done.
 > >> >> done.
 > >> >> Loaded symbols for /boot/kernel/uhid.ko
 > >> >> #0  doadump (textdump=1) at pcpu.h:221
 > >> >> 221    pcpu.h: No such file or directory.
 > >> >>     in pcpu.h
 > >> >> (kgdb) #0  doadump (textdump=1) at pcpu.h:221
 > >> >> #1  0xffffffff80a48945 in kern_reboot (howto=<value optimized out>)
 > >> >>     at /usr/src/sys/kern/kern_shutdown.c:366
 > >> >> #2  0xffffffff80a48f1b in vpanic (fmt=<value optimized out>,
 > >> >>     ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759
 > >> >> #3  0xffffffff80a48d56 in kassert_panic (fmt=<value optimized out>)
 > >> >>     at /usr/src/sys/kern/kern_shutdown.c:649
 > >> >> #4  0xffffffff80b351d6 in llentry_free (lle=<value optimized out>)
 > >> >>     at /usr/src/sys/net/if_llatbl.c:432
 > >> >> #5  0xffffffff80c5c90f in nd6_cache_lladdr (ifp=0xfffff80068d65800,
 > >> >>     from=<value optimized out>, lladdr=<value optimized out>,
 > >> >>     lladdrlen=<value optimized out>, type=134, code=0)
 > >> >>     at /usr/src/sys/netinet6/nd6.c:1972
 > >> >> #6  0xffffffff80c62f57 in nd6_ra_input (m=<value optimized out>,
 > >> >>     off=<value optimized out>, icmp6len=<value optimized out>)
 > >> >>     at /usr/src/sys/netinet6/nd6_rtr.c:437
 > >> >> #7  0xffffffff80c3ab03 in icmp6_input (mp=<value optimized out>,
 > >> >>     offp=<value optimized out>, proto=<value optimized out>)
 > >> >>     at /usr/src/sys/netinet6/icmp6.c:798
 > >> >> #8  0xffffffff80c50e90 in ip6_input (m=Cannot access memory at address
 > >> >> 0x0
 > >> >> )
 > >> >>     at /usr/src/sys/netinet6/ip6_input.c:921
 > >> >> #9  0xffffffff80b47470 in netisr_dispatch_src (proto=6, source=0,
 > >> >>     m=0xfffff80068dfc600) at /usr/src/sys/net/netisr.c:1121
 > >> >> #10 0xffffffff80b3209a in ether_demux (ifp=<value optimized out>,
 > >> >> m=0x0)
 > >> >>     at /usr/src/sys/net/if_ethersubr.c:850
 > >> >> #11 0xffffffff80b32e90 in ether_nh_input (m=<value optimized out>)
 > >> >>     at /usr/src/sys/net/if_ethersubr.c:639
 > >> >> #12 0xffffffff80b47470 in netisr_dispatch_src (proto=5, source=0,
 > >> >>     m=0xfffff80068dfc600) at /usr/src/sys/net/netisr.c:1121
 > >> >> #13 0xffffffff80b32402 in ether_input (ifp=<value optimized out>,
 > >> >> m=0x0)
 > >> >>     at /usr/src/sys/net/if_ethersubr.c:759
 > >> >> #14 0xffffffff80b2ee8a in if_input (ifp=0x0, sendmp=0x0)
 > >> >>     at /usr/src/sys/net/if.c:3956
 > >> >> #15 0xffffffff80523a6c in em_rxeof (count=99)
 > >> >>     at /usr/src/sys/dev/e1000/if_em.c:4872
 > >> >> #16 0xffffffff805250b0 in em_handle_que (context=0xfffffe1eaa699000,
 > >> >>     pending=<value optimized out>) at
 > >> >> /usr/src/sys/dev/e1000/if_em.c:1598
 > >> >> #17 0xffffffff80a9a23c in taskqueue_run_locked (queue=<value optimized
 > >> >> out>)
 > >> >>     at /usr/src/sys/kern/subr_taskqueue.c:465
 > >> >> #18 0xffffffff80a9ad38 in taskqueue_thread_loop (arg=<value optimized
 > >> >> out>)
 > >> >>     at /usr/src/sys/kern/subr_taskqueue.c:719
 > >> >> #19 0xffffffff80a0ba54 in fork_exit (
 > >> >>     callout=0xffffffff80a9acb0 <taskqueue_thread_loop>,
 > >> >>     arg=0xfffffe1eaa69b730, frame=0xfffffe2dfe57bc00)
 > >> >>     at /usr/src/sys/kern/kern_fork.c:1038
 > >> >> #20 0xffffffff80e943fe in fork_trampoline ()
 > >> >>     at /usr/src/sys/amd64/amd64/exception.S:611
 > >> >> #21 0x0000000000000000 in ?? ()
 > >> >> Current language:  auto; currently minimal
 > >> >> (kgdb)
 > >> >>
 > >> >> vmcore IS available.
 > >> 
 > >> --
 > >> Larry Rosenman                     http://www.lerctr.org/~ler
 > >> Phone: +1 214-642-9640                 E-Mail: l...@lerctr.org
 > >> US Mail: 17716 Limpia Crk, Round Rock, TX 78664-7281
 > >> _______________________________________________
 > >> freebsd-current@freebsd.org mailing list
 > >> https://lists.freebsd.org/mailman/listinfo/freebsd-current
 > >> To unsubscribe, send any mail to 
 > >> "freebsd-current-unsubscr...@freebsd.org"
 > 
 > -- 
 > Larry Rosenman                     http://www.lerctr.org/~ler
 > Phone: +1 214-642-9640                 E-Mail: l...@lerctr.org
 > US Mail: 17716 Limpia Crk, Round Rock, TX 78664-7281
 > _______________________________________________
 > freebsd-current@freebsd.org mailing list
 > https://lists.freebsd.org/mailman/listinfo/freebsd-current
 > To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

_______________________________________________
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Reply via email to