Ok, Zachary noted that ivadasz had a patch. Imre and I went over it on
IRC and he committed the patch to master. I also committed some
additional changes so it would be great if anyone using master + virtio
in a virtual-hosted environment can [re]test the changes.
There are likely going to be numerous other issues with virtual hosting
not yet addressed.
-Matt
On Fri, May 27, 2016 at 10:08 AM, Matthew Dillon <[email protected]
<mailto:[email protected]>> wrote:
Virtio (for block storage devices) could be the cause. There are
known bugs in the DragonFly driver for virtio which haven't been
tracked down yet (not enough of the devs are using virtual hosting
to be able to reproduce the problem in a debugable way).
-Matt
On Fri, May 27, 2016 at 7:39 AM, Steve Petrie, P.Eng.
<[email protected] <mailto:[email protected]>> wrote:
Greetings To DragonFlyBSD List,
The subject of random server crashes with DragonFly running
running on a virtualized host machine, is of great interest (and
concern) to me. Caveat: I am an (almost) complete DragonFly newbie.
Please see my commens inline below.
Steve
----- Original Message ----- From: "Stefan Unterweger"
<[email protected]
<mailto:[email protected]>>
To: "Matthew Dillon" <[email protected]
<mailto:[email protected]>>
Cc: <[email protected] <mailto:[email protected]>>
Sent: Friday, May 27, 2016 3:38 AM
Subject: Re: Random server crashes every few weeks (smp_invltlb:
endless loop […] retrysmp_invltlb: ipi sent)
* Matthew Dillon on Thu, May 26, 2016 at 11:00:18AM -0700:
It's really hard to say from something which is
virtually hosted. It kinda
sounds like the virtual host isn't assigning enough of
its own cpus to the
virtual host. The fact that DragonFly is complaining
about smp_invltlb()
implies that the host's virtualized cpu threads are not
getting scheduled
properly.
One thing to note is that we do not do any instruction
escapes to hint to
virtual hosts when a cpu is in a tight loop waiting for
synchronization.
It would be nice if we had some support for that, it
would probably make
DFly play better on virtualized systems.
This is an interesting suggestion, which at least would
explain at least
some of the cases where I’ve experienced the crashes (the
daily HAMMER
cronjob, heavy paging under stress, I/O bursts and so on).
So in effect, could it be that the crashes are more likely
as either my
own server comes under load or some -other- server who
happens to run in
the same hypervisor?
Would this warrant opening a ticket with Profitbricks, or is
it just as
likely that I’m wasting my time and will only get a response
along the
lines of ‘Use Linux; Dragonfly BSD most certainly is not
supported’?
I suggest setting the number of cores to 1. That will
get rid of all SMP
interplay and hopefully remove the issues the virtual
host is choking on.
Interestingly enough, I have seen the opposite so far. At
first, I have
run the server on only one core, to save money and because
it doesn’t
really yet need any more. When on one core, it still
freezes, along
approximately the same pattern, but I never got a trace there.
My guess then was that perhaps there would have been some
odd race
condition between paging, HAMMER and dm_crypt—adding another
core
temporarily seemed more stable and then regressed back to
the mean.
I will try to set up another VM to see whether I can
reliably reproduce
such a crash.
After a great deal of research, I chose DragonFly as the OS for
a new website (not yet online). Three main attributes drew me to
DragonFly: 1. reputation for reliability and speed, 2. hammer
file system, 3. responsiveness of DragonFly open source community.
However, my business plan for this new website (not yet online),
requires starting out hosting it on a VM under QEMU / KVM
virtualization, because I cannot justify the much higher cost of
dedicated server hosting hardware. And I like the brutally
competitive quasi-commoditized hosting services market for QEMU
/ KVM virtualization offerings.
I do have an experimental working DragonFly installation on a
QEMU / KVM VM hosted at Elastic Hosts www.elastichosts.com
<http://www.elastichosts.com> I access this VM through TightVNC
and I get a DragonFly console using PuTTY.
But I had to suspend work on testing this DragonFly VM
installation, due to other business priorities. I hope to get
back to it later in 2016.
However, I can highly recommend Elastic Hosts for their solid
cloud infrastructure and their strong customer support.
So if Stefan wants to expand his testing of the DragonFly crash
to a VM with a (probably) different underlying architecture than
at ProfitBricks, I would recommend giving an Elastic Hosts QEMU
/ KVM VM a try.
Alternatively, if Stafan has (or develops) some simple limited
self-contained testing setup, for reproducing the DragonFly
crash he is experiencing with the ProfitBricks VM he's presently
using, I would be interested to try to set up the same testing
scenario on my current DragonFly (and later on an upgraded
version of DragonFly) on the Elastic Hosts VM where I presently
have (an outdated version of) DragonFly operational.
Steve
Thanks for your answer,
Stefan
PS: Just in case, as I’ve forgotten it previously: here’s
the dmesg from
the server in question.
| Copyright (c) 2003-2015 The DragonFly Project.
| Copyright (c) 1992-2003 The FreeBSD Project.
| Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991,
1992, 1993, 1994
| The Regents of the University of California. All rights
reserved.
| DragonFly v4.4.1-RELEASE #2: Sun Dec 6 19:10:59 EST 2015
|
[email protected]:/usr/obj/home/justin/release/4_4/sys/X86_64_GENERIC
| TSC clock: 2600054420 Hz, i8254 clock: 1193169 Hz
| CPU: AMD Opteron 62xx class CPU (2600.11-MHz K8-class CPU)
| Origin = "AuthenticAMD" Id = 0x600f12 Stepping = 2
|
Features=0x783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2>
|
Features2=0x96982203<SSE3,PCLMULQDQ,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AESNI,XSAVE,AVX,VMM>
| AMD Features=0x24500800<SYSCALL,NX,MMX+,Page1GB,LM>
| AMD
Features2=0x10be7<LAHF,CMP,SVM,ABM,SSE4A,MAS,Prefetch,OSVW,XOP,FMA4>
| MONITOR/MWAIT Features=0x2<INTBRK>
| real memory = 3219762176 (3070 MB)
| avail memory = 2990727168 (2852 MB)
| lapic: divisor index 0, frequency 500005713 Hz
| SMI Frequency (worst case): 28571 Hz (35 us)
| Initialize MI interrupts
| wdog: In-kernel automatic watchdog reset enabled
| kbd1 at kbdmux0
| md0: Preloaded image <initrd.img> 15728640 bytes at
0xffffffff82739ac0
| md1: Malloc disk
| ACPI: RSDP 0x00000000000FC980 000014 (v00 BOCHS )
| ACPI: RSDT 0x00000000BFFFBCA0 000040 (v01 BOCHS BXPCRSDT
00000001 BXPC 00000001)
| ACPI: FACP 0x00000000BFFFFF80 000074 (v01 BOCHS BXPCFACP
00000001 BXPC 00000001)
| ACPI: DSDT 0x00000000BFFFBCE0 00151D (v01 BXPC BXDSDT
00000001 INTL 20100528)
| ACPI: FACS 0x00000000BFFFFF40 000040
| ACPI: APIC 0x00000000BFFFFC60 000270 (v01 BOCHS BXPCAPIC
00000001 BXPC 00000001)
| ACPI: HPET 0x00000000BFFFFC20 000038 (v01 BOCHS BXPCHPET
00000001 BXPC 00000001)
| ACPI: SRAT 0x00000000BFFFF770 0004A8 (v01 BOCHS BXPCSRAT
00000001 BXPC 00000001)
| ACPI: SSDT 0x00000000BFFFD8E0 001E8E (v01 BOCHS BXPCSSDT
00000001 BXPC 00000001)
| ACPI: SSDT 0x00000000BFFFD870 00003D (v01 BOCHS BXPCSSDT
00000001 BXPC 00000001)
| ACPI: SSDT 0x00000000BFFFD200 00066E (v01 BXPC BXSSDTPC
00000001 INTL 20100528)
| cryptosoft0: <software crypto> on motherboard
| aesni0: <AES-CBC,AES-XTS> on motherboard
| padlock0: No ACE support.
| rdrand0: No RdRand support.
| acpi0: <BOCHS BXPCRSDT> on motherboard
| ACPI: 4 ACPI AML tables successfully acquired and loaded
| ACPI FADT: SCI testing interrupt mode ...
| ACPI FADT: SCI select level/low
| objcache_reclaimlist
| objcache_reclaimlist
| objcache_reclaimlist
| objcache_reclaimlist
| acpi0: Power Button (fixed)
| acpi_timer0 on acpi0
| acpi_hpet0: <High Precision Event Timer> iomem
0xfed00000-0xfed003ff on acpi0
| acpi_hpet0: frequency 100000000
| pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
| pci0: <ACPI PCI bus> on pcib0
| pci_link4: Unable to route IRQs: AE_NOT_FOUND
| isab0: <PCI-ISA bridge> at device 1.0 on pci0
| isa0: <ISA bus> on isab0
| atapci0: <Intel PIIX3 WDMA2 controller> port
0xc120-0xc12f,0x376,0x170-0x177,0x3f6,0x1f0-0x1f7 at device
1.1 on pci0
| ata0: <ATA channel 0> on atapci0
| ata1: <ATA channel 1> on atapci0
| acd0: DVDROM <QEMU DVD-ROM/1.0> at ata1-master WDMA2
| uhci0: <Intel 82371SB (PIIX3) USB controller> port
0xc0c0-0xc0df irq 11 at device 1.2 on pci0
| usbus0: controller did not stop
| usbus0 on uhci0
| pci0: <bridge> (vendor 0x8086, dev 0x7113) at device 1.3 irq 9
| vgapci0: <VGA-compatible display> mem
0xfd000000-0xfdffffff at device 2.0 on pci0
| vgapci0: Boot video device
| virtio_pci0: <VirtIO PCI Balloon adapter> port
0xc0e0-0xc0ff irq 11 at device 3.0 on pci0
| virtio_pci1: <VirtIO PCI Block adapter> port 0xc000-0xc03f
mem 0xfebf0000-0xfebf0fff irq 10 at device 5.0 on pci0
| vtblk0: <VirtIO Block Adapter> on virtio_pci1
| virtio_pci1: host features: 0x710006d4
<EventIdx,RingIndirect,NotifyOnEmpty,Topology,WriteCache,SCSICmds,BlockSize,DiskGeometry,MaxNumSegs>
| virtio_pci1: negotiated features: 0x254
<WriteCache,BlockSize,DiskGeometry,MaxNumSegs>
| virtio_pci2: <VirtIO PCI Block adapter> port 0xc040-0xc07f
mem 0xfebf1000-0xfebf1fff irq 10 at device 6.0 on pci0
| vtblk1: <VirtIO Block Adapter> on virtio_pci2
| virtio_pci2: host features: 0x710006d4
<EventIdx,RingIndirect,NotifyOnEmpty,Topology,WriteCache,SCSICmds,BlockSize,DiskGeometry,MaxNumSegs>
| virtio_pci2: negotiated features: 0x254
<WriteCache,BlockSize,DiskGeometry,MaxNumSegs>
| virtio_pci3: <VirtIO PCI Network adapter> port
0xc100-0xc11f mem 0xfebf2000-0xfebf2fff irq 11 at device 7.0
on pci0
| vtnet0: <VirtIO Networking Adapter> on virtio_pci3
| virtio_pci3: host features: 0x711f8060
<EventIdx,RingIndirect,NotifyOnEmpty,RxModeExtra,VLanFilter,RxMode,ControlVq,Status,MrgRxBuf,TxAllGSO,MacAddress>
| virtio_pci3: negotiated features: 0x110f8020
<RingIndirect,NotifyOnEmpty,VLanFilter,RxMode,ControlVq,Status,MrgRxBuf,MacAddress>
| usbus0: 12Mbps Full Speed USB v1.0
| vtnet0: MAC address: 02:01:06:f6:1b:63
| add dynamic link state
| virtio_pci4: <VirtIO PCI Block adapter> port 0xc080-0xc0bf
mem 0xfebf3000-0xfebf3fff irq 11 at device 8.0 on pci0
| vtblk2: <VirtIO Block Adapter> on virtio_pci4
| virtio_pci4: host features: 0x710006d4
<EventIdx,RingIndirect,NotifyOnEmpty,Topology,WriteCache,SCSICmds,BlockSize,DiskGeometry,MaxNumSegs>
| virtio_pci4: negotiated features: 0x254
<WriteCache,BlockSize,DiskGeometry,MaxNumSegs>
| atkbdc0: <Keyboard controller (i8042)> port 0x64,0x60 irq
1 on acpi0
| atkbd0: <AT Keyboard> irq 1 on atkbdc0
| kbd0 at atkbd0
| ugen0.1: <Intel> at usbus0
| uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00,
addr 1> on usbus0
| psm0: <PS/2 Mouse> irq 12 on atkbdc0
| psm0: model IntelliMouse Explorer, device ID 4
| cpu0: <ACPI CPU> on acpi0
| cpu_cst0: <ACPI CPU C-State> on cpu0
| cpu1: <ACPI CPU> on acpi0
| cpu_cst1: <ACPI CPU C-State> on cpu1
| ACPI: Enabled 16 GPEs in block 00 to 0F
| orm0: <ISA Option ROM> at iomem 0xe9800-0xeffff on isa0
| pmtimer0 on isa0
| vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem
0xa0000-0xbffff on isa0
| sc0: <System console> at flags 0x100 on isa0
| sc0: VGA <16 virtual consoles, flags=0x300>
| sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
| sio0: type 16550A
| sio1: can't drain, serial port might not exist, disabling
| hpt27xx: no controller detected.
| CAM: Configuring 2 busses
| CAM: finished configuring all busses
| cd0 at ata1 bus 0 target 0 lun 0
| cd0: <QEMU QEMU DVD-ROM 1.0> Removable CD-ROM SCSI-0 device
| cd0: 16.000MB/s transfers
| cd0: cd present [329728 x 2048 byte records]
| uhub0: 2 ports with 2 removable, self powered
| ugen0.2: <QEMU> at usbus0
| uhid0: <QEMU QEMU USB Tablet, class 0/0, rev 1.00/0.00,
addr 2> on usbus0
| no B_DEVMAGIC (bootdev=0)
| Device Mapper version 4.16.0 loaded
| dm_target_zero: Successfully initialized
| dm_target_crypt: Successfully initialized
| dm_target_error: Successfully initialized
| Mounting root from ufs:md0s0
| DMA space used: 1236k, remaining available: 131072k
| Mounting devfs
| dm_target_crypt: Setting min/max mpipe buffers: 2/30
| dm_target_crypt: Setting min/max mpipe buffers: 2/30
| HAMMER(Rhaal) recovery check seqno=055a4f51
| HAMMER(Rhaal) recovery range 300000000cc2da60-300000000cc2da60
| HAMMER(Rhaal) recovery nexto 300000000cc2da60
endseqno=055a4f52
| HAMMER(Rhaal) mounted clean, no recovery needed
| chroot_kernel: set new rootnch/rootvnode to /new_root
| dm_target_crypt: Setting min/max mpipe buffers: 2/30
| dm_target_crypt: Setting min/max mpipe buffers: 2/30
| dm_target_crypt: Setting min/max mpipe buffers: 2/30
| HAMMER: read-only -> read-write
| HAMMER(Rhaal-Daten) recovery check seqno=352f5bc4
| HAMMER(Rhaal-Daten) recovery range
3000000000d5c108-3000000000d77bc8
| HAMMER(Rhaal-Daten) recovery nexto 3000000000d77bc8
endseqno=352f5cc1
| HAMMER(Rhaal-Daten) recovery undo
3000000000d5c108-3000000000d77bc8 (113344 bytes)(RW)
| HAMMER(Rhaal-Daten) Found REDO_SYNC 3000000000cb87a0
| HAMMER(Rhaal-Daten) recovery complete
| HAMMER(Rhaal-Daten) recovery redo
3000000000d5c108-3000000000d77bc8 (113344 bytes)(RW)
| HAMMER(Rhaal-Daten) Find extended redo 3000000000cb87a0,
670056 extbytes
| HAMMER(Rhaal-Daten) End redo recovery
| dm_target_crypt: Setting min/max mpipe buffers: 2/30
| swap low/high-water marks set to 83874/125811
