https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=225791

            Bug ID: 225791
           Summary: ena driver causing kernel panics on AWS EC2
           Product: Base System
           Version: 11.1-RELEASE
          Hardware: amd64
                OS: Any
            Status: New
          Severity: Affects Only Me
          Priority: ---
         Component: kern
          Assignee: freebsd-bugs@FreeBSD.org
          Reporter: mwal...@carbonhouse.com

Overview: 

On some of our more heavily-trafficked c5.large and c5.xlarge servers we're
seeing kernel panics & reboots as often as 2-3 times per day.

Steps to Reproduce:

Have a server with a good amount of network traffic (~3000pps) using the ena
network adapter. Wait a day or less.



Additional Information:

Here's one of the panics:

Feb  5 20:58:51 awesomehost kernel: ena0: Found a Tx that wasn't completed on
time, qid 0, index 0.
Feb  6 03:48:11 awesomehost kernel: ena0: Found a Tx that wasn't completed on
time, qid 1, index 203.
Feb  6 04:25:46 awesomehost syslogd: kernel boot file is /boot/kernel/kernel
Feb  6 04:25:46 awesomehost kernel: [zone: mbuf_jumbo_16k] kern.ipc.nmbjumbo16
limit reached
Feb  6 04:25:46 awesomehost kernel:
Feb  6 04:25:46 awesomehost kernel:
Feb  6 04:25:46 awesomehost kernel: Fatal trap 12: page fault while in kernel
mode
Feb  6 04:25:46 awesomehost kernel: cpuid = 0; apic id = 00
Feb  6 04:25:46 awesomehost kernel: fault virtual address       = 0x1c
Feb  6 04:25:46 awesomehost kernel: fault code          = supervisor write
data, page not present
Feb  6 04:25:46 awesomehost kernel: instruction pointer =
0x20:0xffffffff8218ef8c
Feb  6 04:25:46 awesomehost kernel: stack pointer               =
0x0:0xfffffe0108afe180
Feb  6 04:25:46 awesomehost kernel: frame pointer               =
0x0:0xfffffe0108afe260
Feb  6 04:25:46 awesomehost kernel: code segment                = base 0x0,
limit 0xfffff, type 0x1b
Feb  6 04:25:46 awesomehost kernel:
Feb  6 04:25:46 awesomehost kernel: = DPL 0, pres 1, long 1, def32 0, gran 1
Feb  6 04:25:46 awesomehost kernel: processor eflags    =
Feb  6 04:25:46 awesomehost kernel: Fatal trap 12: page fault while in kernel
mode
Feb  6 04:25:46 awesomehost kernel: cpuid = 1; apic id = 01
Feb  6 04:25:46 awesomehost kernel: fault virtual address       = 0x1c
Feb  6 04:25:46 awesomehost kernel: fault code          = supervisor write
data, page not present
Feb  6 04:25:46 awesomehost kernel: instruction pointer =
0x20:0xffffffff8218ef8c
Feb  6 04:25:46 awesomehost kernel: interrupt enabled, resume, IOPL = 0
Feb  6 04:25:46 awesomehost kernel: current process             = 12 (irq261:
ena0)
Feb  6 04:25:46 awesomehost kernel: trap number         = 12
Feb  6 04:25:46 awesomehost kernel: panic: page fault
Feb  6 04:25:46 awesomehost kernel: cpuid = 0
Feb  6 04:25:46 awesomehost kernel: KDB: stack backtrace:
Feb  6 04:25:46 awesomehost kernel: #0 0xffffffff80aadac7 at kdb_backtrace+0x67
Feb  6 04:25:46 awesomehost kernel: #1 0xffffffff80a6bba6 at vpanic+0x186
Feb  6 04:25:46 awesomehost kernel: #2 0xffffffff80a6ba13 at panic+0x43
Feb  6 04:25:46 awesomehost kernel: #3 0xffffffff80edf832 at trap_fatal+0x322
Feb  6 04:25:46 awesomehost kernel: #4 0xffffffff80edf889 at trap_pfault+0x49
Feb  6 04:25:46 awesomehost kernel: #5 0xffffffff80edf0c6 at trap+0x286
Feb  6 04:25:46 awesomehost kernel: #6 0xffffffff80ec36d1 at calltrap+0x8
Feb  6 04:25:46 awesomehost kernel: #7 0xffffffff80a321ec at
intr_event_execute_handlers+0xec
Feb  6 04:25:46 awesomehost kernel: #8 0xffffffff80a324d6 at ithread_loop+0xd6
Feb  6 04:25:46 awesomehost kernel: #9 0xffffffff80a2f845 at fork_exit+0x85
Feb  6 04:25:46 awesomehost kernel: #10 0xffffffff80ec3c0e at
fork_trampoline+0xe
Feb  6 04:25:46 awesomehost kernel: Uptime: 8h48m21s
Feb  6 04:25:46 awesomehost kernel: Rebooting...


Here's a full boot log:

Feb  8 22:57:59 awesomehost kernel: Copyright (c) 1992-2017 The FreeBSD
Project.                                                                       
                                                                               
                                                                         
[37/2338]
Feb  8 22:57:59 awesomehost kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988,
1989, 1991, 1992, 1993, 1994
Feb  8 22:57:59 awesomehost kernel: The Regents of the University of
California. All rights reserved.
Feb  8 22:57:59 awesomehost kernel: FreeBSD is a registered trademark of The
FreeBSD Foundation.
Feb  8 22:57:59 awesomehost kernel: FreeBSD 11.1-RELEASE-p4 #0: Tue Nov 14
06:12:40 UTC 2017
Feb  8 22:57:59 awesomehost kernel:
r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
Feb  8 22:57:59 awesomehost kernel: FreeBSD clang version 4.0.0
(tags/RELEASE_400/final 297347) (based on LLVM 4.0.0)
Feb  8 22:57:59 awesomehost kernel: VT(vga): text 80x25
Feb  8 22:57:59 awesomehost kernel: CPU: HammerEM64T (3000.07-MHz K8-class CPU)
Feb  8 22:57:59 awesomehost kernel: Origin="GenuineIntel"  Id=0x50653 
Family=0x6  Model=0x55  Stepping=3
Feb  8 22:57:59 awesomehost kernel:
Features=0x1f83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS,HTT>
Feb  8 22:57:59 awesomehost kernel:
Features2=0xfffa3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
Feb  8 22:57:59 awesomehost kernel: AMD
Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
Feb  8 22:57:59 awesomehost kernel: AMD Features2=0x121<LAHF,ABM,Prefetch>
Feb  8 22:57:59 awesomehost kernel: Structured Extended
Features=0xd11f4f3a<TSCADJ,BMI1,HLE,AVX2,BMI2,ERMS,INVPCID,RTM,MPX,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLWB,AVX512CD,AVX512BW>
Feb  8 22:57:59 awesomehost kernel: Structured Extended Features2=0x8<PKU>
Feb  8 22:57:59 awesomehost kernel: XSAVE
Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
Feb  8 22:57:59 awesomehost kernel: TSC: P-state invariant, performance
statistics
Feb  8 22:57:59 awesomehost kernel: Hypervisor: Origin = "KVMKVMKVM"
Feb  8 22:57:59 awesomehost kernel: real memory  = 5114953728 (4878 MB)
Feb  8 22:57:59 awesomehost kernel: avail memory = 3845005312 (3666 MB)
Feb  8 22:57:59 awesomehost kernel: Event timer "LAPIC" quality 600
Feb  8 22:57:59 awesomehost kernel: ACPI APIC Table: <AMAZON AMZNAPIC>
Feb  8 22:57:59 awesomehost kernel: FreeBSD/SMP: Multiprocessor System
Detected: 2 CPUs
Feb  8 22:57:59 awesomehost kernel: FreeBSD/SMP: 1 package(s) x 1 core(s) x 2
hardware threads
Feb  8 22:57:59 awesomehost kernel: random: unblocking device.
Feb  8 22:57:59 awesomehost kernel: ioapic0 <Version 1.1> irqs 0-23 on
motherboard
Feb  8 22:57:59 awesomehost kernel: SMP: AP CPU #1 Launched!
Feb  8 22:57:59 awesomehost kernel: random: entropy device external interface
Feb  8 22:57:59 awesomehost kernel: kbd1 at kbdmux0
Feb  8 22:57:59 awesomehost kernel: netmap: loaded module
Feb  8 22:57:59 awesomehost kernel: module_register_init: MOD_LOAD (vesa,
0xffffffff80f5b220, 0) error 19
Feb  8 22:57:59 awesomehost kernel: random: registering fast source Intel
Secure Key RNG
Feb  8 22:57:59 awesomehost kernel: random: fast provider: "Intel Secure Key
RNG"
Feb  8 22:57:59 awesomehost kernel: nexus0
Feb  8 22:57:59 awesomehost kernel: vtvga0: <VT VGA driver> on motherboard
Feb  8 22:57:59 awesomehost kernel: cryptosoft0: <software crypto> on
motherboard
Feb  8 22:57:59 awesomehost kernel: acpi0: <AMAZON AMZNRSDT> on motherboard
Feb  8 22:57:59 awesomehost kernel: acpi0: Power Button (fixed)
Feb  8 22:57:59 awesomehost kernel: cpu0: <ACPI CPU> on acpi0
Feb  8 22:57:59 awesomehost kernel: cpu1: <ACPI CPU> on acpi0
Feb  8 22:57:59 awesomehost kernel: atrtc0: <AT realtime clock> port
0x70-0x71,0x72-0x77 irq 8 on acpi0
Feb  8 22:57:59 awesomehost kernel: Event timer "RTC" frequency 32768 Hz
quality 0
Feb  8 22:57:59 awesomehost kernel: Timecounter "ACPI-fast" frequency 3579545
Hz quality 900
Feb  8 22:57:59 awesomehost kernel: acpi_timer0: <24-bit timer at 3.579545MHz>
port 0xb008-0xb00b on acpi0
Feb  8 22:57:59 awesomehost kernel: pcib0: <ACPI Host-PCI bridge> port
0xcf8-0xcff on acpi0
Feb  8 22:57:59 awesomehost kernel: pci0: <ACPI PCI bus> on pcib0
Feb  8 22:57:59 awesomehost kernel: isab0: <PCI-ISA bridge> at device 1.0 on
pci0
Feb  8 22:57:59 awesomehost kernel: isa0: <ISA bus> on isab0
Feb  8 22:57:59 awesomehost kernel: pci0: <old, non-VGA display device> at
device 1.3 (no driver attached)
Feb  8 22:57:59 awesomehost kernel: vgapci0: <VGA-compatible display> mem
0xfe400000-0xfe7fffff at device 3.0 on pci0
Feb  8 22:57:59 awesomehost kernel: vgapci0: Boot video device
Feb  8 22:57:59 awesomehost kernel: nvme0: <Generic NVMe Device> mem
0xfebf0000-0xfebf3fff irq 11 at device 4.0 on pci0
Feb  8 22:57:59 awesomehost kernel: ena0: <ENA adapter> mem
0xfebf4000-0xfebf7fff at device 5.0 on pci0
Feb  8 22:57:59 awesomehost kernel: ena0: Elastic Network Adapter (ENA)ena
v0.7.0
Feb  8 22:57:59 awesomehost kernel: ena0: initalize 2 io queues
Feb  8 22:57:59 awesomehost kernel: ena0: Ethernet address: 12:3d:00:e2:4f:78
Feb  8 22:57:59 awesomehost kernel: ena0: Allocated msix_entries, vectors (cnt:
3)
Feb  8 22:57:59 awesomehost kernel: nvme1: <Generic NVMe Device> mem
0xfebf8000-0xfebfbfff irq 11 at device 31.0 on pci0
Feb  8 22:57:59 awesomehost kernel: ena0: link is UP
Feb  8 22:57:59 awesomehost kernel: ena0: link state changed to UP
Feb  8 22:57:59 awesomehost kernel: atkbdc0: <Keyboard controller (i8042)> port
0x60,0x64 irq 1 on acpi0
Feb  8 22:57:59 awesomehost kernel: atkbd0: <AT Keyboard> irq 1 on atkbdc0
Feb  8 22:57:59 awesomehost kernel: kbd0 at atkbd0
Feb  8 22:57:59 awesomehost kernel: atkbd0: [GIANT-LOCKED]
Feb  8 22:57:59 awesomehost kernel: uart0: <Non-standard ns8250 class UART with
FIFOs> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
Feb  8 22:57:59 awesomehost kernel: uart0: console (115200,n,8,1)
Feb  8 22:57:59 awesomehost kernel: orm0: <ISA Option ROM> at iomem
0xef000-0xeffff on isa0
Feb  8 22:57:59 awesomehost kernel: vga0: <Generic ISA VGA> at port 0x3c0-0x3df
iomem 0xa0000-0xbffff on isa0
Feb  8 22:57:59 awesomehost kernel: attimer0: <AT timer> at port 0x40 on isa0
Feb  8 22:57:59 awesomehost kernel: Timecounter "i8254" frequency 1193182 Hz
quality 0
Feb  8 22:57:59 awesomehost kernel: attimer0: Can't map interrupt.
Feb  8 22:57:59 awesomehost kernel: ppc0: cannot reserve I/O port range
Feb  8 22:57:59 awesomehost kernel: Timecounters tick every 1.000 msec
Feb  8 22:57:59 awesomehost kernel: usb_needs_explore_all: no devclass
Feb  8 22:57:59 awesomehost kernel: nvme cam probe device init
Feb  8 22:57:59 awesomehost kernel: nvme0: temperature threshold not supported
Feb  8 22:57:59 awesomehost kernel: nvd0: <Amazon Elastic Block Store> NVMe
namespace
Feb  8 22:57:59 awesomehost kernel: nvd0: 10240MB (20971520 512 byte sectors)
Feb  8 22:57:59 awesomehost kernel: nvme1: temperature threshold not supported
Feb  8 22:57:59 awesomehost kernel: nvd1: <Amazon Elastic Block Store> NVMe
namespace
Feb  8 22:57:59 awesomehost kernel: nvd1: 40960MB (83886080 512 byte sectors)
Feb  8 22:57:59 awesomehost kernel: Trying to mount root from
ufs:/dev/gpt/rootfs [rw]...
Feb  8 22:57:59 awesomehost kernel: WARNING: / was not properly dismounted
Feb  8 22:57:59 awesomehost kernel: WARNING: /: mount pending error: blocks 0
files 2
Feb  8 22:57:59 awesomehost kernel: WARNING: /: reload pending error: blocks 0
files 2
Feb  8 22:57:59 awesomehost kernel: WARNING: / was not properly dismounted
Feb  8 22:57:59 awesomehost kernel: WARNING: /sites was not properly dismounted
Feb  8 22:57:59 awesomehost kernel: ena0: device is going UP
Feb  8 22:57:59 awesomehost kernel: ena0: queue 0 - cpu 0
Feb  8 22:57:59 awesomehost kernel: ena0: queue 1 - cpu 1
Feb  8 22:57:59 awesomehost kernel: pflog0: promiscuous mode enabled


Below are some numbers from one of our hosts experiencing this the most, it's a
c5.large server in us-east-1, booted from the official FreeBSD AMI "FreeBSD
11.1-RELEASE-amd64 (ami-70504266)"


Here's "uname -ar":

FreeBSD awesomehost 11.1-RELEASE-p4 FreeBSD 11.1-RELEASE-p4 #0: Tue Nov 14
06:12:40 UTC 2017    
r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64


Here's "uptime":

 3:29PM  up 16:31, 2 users, load averages: 0.08, 0.22, 0.23


Here's "sysctl dev.ena":

dev.ena.0.admin_stats.no_completion: 0
dev.ena.0.admin_stats.out_of_space: 0
dev.ena.0.admin_stats.completed_cmd: 58446
dev.ena.0.admin_stats.sumbitted_cmd: 58446
dev.ena.0.admin_stats.aborted_cmd: 0
dev.ena.0.hw_stats.update_stats:
dev.ena.0.hw_stats.rx_drops: 0
dev.ena.0.hw_stats.tx_bytes: 130551171301
dev.ena.0.hw_stats.rx_bytes: 129414445543
dev.ena.0.hw_stats.tx_packets: 175108516
dev.ena.0.hw_stats.rx_packets: 174924189
dev.ena.0.queue1.rx_ring.small_copy_len_pkt: 0
dev.ena.0.queue1.rx_ring.bad_desc_num: 0
dev.ena.0.queue1.rx_ring.dma_mapping_err: 0
dev.ena.0.queue1.rx_ring.mbuf_alloc_fail: 0
dev.ena.0.queue1.rx_ring.page_alloc_fail: 0
dev.ena.0.queue1.rx_ring.bad_csum: 0
dev.ena.0.queue1.rx_ring.refil_partial: 0
dev.ena.0.queue1.rx_ring.bytes: 65125566440
dev.ena.0.queue1.rx_ring.count: 87968075
dev.ena.0.queue1.tx_ring.mbuf_collapse_err: 0
dev.ena.0.queue1.tx_ring.mbuf_collapses: 0
dev.ena.0.queue1.tx_ring.stops: 0
dev.ena.0.queue1.tx_ring.bad_req_id: 0
dev.ena.0.queue1.tx_ring.missing_tx_comp: 0
dev.ena.0.queue1.tx_ring.doorbells: 87925945
dev.ena.0.queue1.tx_ring.tx_poll: 0
dev.ena.0.queue1.tx_ring.napi_comp: 0
dev.ena.0.queue1.tx_ring.unsupported_desc_num: 0
dev.ena.0.queue1.tx_ring.dma_mapping_err: 0
dev.ena.0.queue1.tx_ring.queue_wakeup: 0
dev.ena.0.queue1.tx_ring.prepare_ctx_err: 0
dev.ena.0.queue1.tx_ring.bytes: 65756583294
dev.ena.0.queue1.tx_ring.count: 87999157
dev.ena.0.queue0.rx_ring.small_copy_len_pkt: 0
dev.ena.0.queue0.rx_ring.bad_desc_num: 0
dev.ena.0.queue0.rx_ring.dma_mapping_err: 0
dev.ena.0.queue0.rx_ring.mbuf_alloc_fail: 0
dev.ena.0.queue0.rx_ring.page_alloc_fail: 0
dev.ena.0.queue0.rx_ring.bad_csum: 0
dev.ena.0.queue0.rx_ring.refil_partial: 0
dev.ena.0.queue0.rx_ring.bytes: 64288879103
dev.ena.0.queue0.rx_ring.count: 86956114
dev.ena.0.queue0.tx_ring.mbuf_collapse_err: 0
dev.ena.0.queue0.tx_ring.mbuf_collapses: 0
dev.ena.0.queue0.tx_ring.stops: 0
dev.ena.0.queue0.tx_ring.bad_req_id: 0
dev.ena.0.queue0.tx_ring.missing_tx_comp: 0
dev.ena.0.queue0.tx_ring.doorbells: 87033454
dev.ena.0.queue0.tx_ring.tx_poll: 0
dev.ena.0.queue0.tx_ring.napi_comp: 0
dev.ena.0.queue0.tx_ring.unsupported_desc_num: 0
dev.ena.0.queue0.tx_ring.dma_mapping_err: 0
dev.ena.0.queue0.tx_ring.queue_wakeup: 0
dev.ena.0.queue0.tx_ring.prepare_ctx_err: 0
dev.ena.0.queue0.tx_ring.bytes: 64794589485
dev.ena.0.queue0.tx_ring.count: 87109364
dev.ena.0.admin_q_pause: 0
dev.ena.0.interface_down: 0
dev.ena.0.interface_up: 1
dev.ena.0.wd_expired: 0
dev.ena.0.io_resume: 0
dev.ena.0.io_suspend: 0
dev.ena.0.tx_timeout: 0
dev.ena.0.missing_tx_threshold: 128
dev.ena.0.missing_tx_max_queues: 4
dev.ena.0.missing_tx_timeout: 21474836480
dev.ena.0.keep_alive_timeout: 25769803776
dev.ena.0.wd_active: 1
dev.ena.0.%parent: pci0
dev.ena.0.%pnpinfo: vendor=0x1d0f device=0xec20 subvendor=0x0000
subdevice=0x0000 class=0x020000
dev.ena.0.%location: slot=5 function=0 dbsf=pci0:0:5:0 handle=\_SB_.PCI0.S05_
dev.ena.0.%driver: ena
dev.ena.0.%desc: ENA adapter
dev.ena.%parent:


Here's "vmstat -i":

interrupt                          total       rate
irq1: atkbd0                           2          0
irq4: uart0                            6          0
cpu0:timer                      41324147        695
cpu1:timer                      26268292        442
irq256: nvme0                          9          0
irq257: nvme0                      75412          1
irq258: nvme0                      67504          1
irq259: ena0                      118511          2
irq260: ena0                   101753746       1711
irq261: ena0                   102189440       1718
irq262: nvme1                          9          0
irq263: nvme1                     125224          2
irq264: nvme1                     118774          2
Total                          272041076       4575


Here's "netstat -in":

Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
ena0   1500 <Link#1>      12:3d:00:e2:4f:78 177839782     0     0 178011394    
0     0
ena0      - 192.168.4.0/2 192.168.4.101     61087021     -     - 45501246     -
    -
lo0   16384 <Link#2>      lo0               15856955     0     0 15856955     0
    0
lo0       - ::1/128       ::1                      0     -     -        0     -
    -
lo0       - fe80::%lo0/64 fe80::1%lo0              0     -     -        0     -
    -
lo0       - 127.0.0.0/8   127.0.0.1            61418     -     - 15856955     -
    -
pflog 33160 <Link#3>      pflog0                   0     0     0     1797     0
    0


Here's "sysctl kern.ipc":

kern.ipc.maxsockbuf: 2097152
kern.ipc.sockbuf_waste_factor: 8
kern.ipc.max_linkhdr: 16
kern.ipc.max_protohdr: 60
kern.ipc.max_hdr: 76
kern.ipc.max_datalen: 92
kern.ipc.maxsockets: 122296
kern.ipc.numopensockets: 2141
kern.ipc.soacceptqueue: 65536
kern.ipc.shm_allow_removed: 1
kern.ipc.shm_use_phys: 0
kern.ipc.shmall: 131072
kern.ipc.shmseg: 128
kern.ipc.shmmni: 192
kern.ipc.shmmin: 1
kern.ipc.shmmax: 536870912
kern.ipc.semaem: 16384
kern.ipc.semvmx: 32767
kern.ipc.semusz: 632
kern.ipc.semume: 50
kern.ipc.semopm: 100
kern.ipc.semmsl: 340
kern.ipc.semmnu: 150
kern.ipc.semmns: 340
kern.ipc.semmni: 50
kern.ipc.msgseg: 2048
kern.ipc.msgssz: 8
kern.ipc.msgtql: 40
kern.ipc.msgmnb: 2048
kern.ipc.msgmni: 40
kern.ipc.msgmax: 16384
kern.ipc.aio.lifetime: 30000
kern.ipc.aio.target_procs: 4
kern.ipc.aio.num_procs: 4
kern.ipc.aio.max_procs: 32
kern.ipc.aio.empty_retries: 0
kern.ipc.aio.empty_results: 0
kern.ipc.piperesizeallowed: 1
kern.ipc.piperesizefail: 0
kern.ipc.pipeallocfail: 0
kern.ipc.pipefragretry: 0
kern.ipc.pipekva: 69632
kern.ipc.maxpipekva: 62611456
kern.ipc.umtx_max_robust: 1000
kern.ipc.umtx_vnode_persistent: 0
kern.ipc.nmbufs: 1514295
kern.ipc.nmbjumbo16: 30000
kern.ipc.nmbjumbo9: 35053
kern.ipc.nmbjumbop: 118304
kern.ipc.nmbclusters: 236608
kern.ipc.maxmbufmem: 1938294784

-- 
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
freebsd-bugs@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"

Reply via email to