Hi,
I have a pair of OpenBSD 5.2 VMs running on KVM, they have a carp interface
and are running relayd to load balancer http traffic into two webservers
(also VMs). While benchmarking the setup with ab, I noticed that the
OpenBSD VMs panic'd, I can easily reproduce the panics. Here is a typical
stack trace:
uvm_fault(0xfffffe807d0c62a8, 0x0, 0, 1) -> e
kernel: page fault trap, code=0
Stopped at somove+0x22: movq 0x78(%rdi),%r14
ddb> somove() at somove+0x22
sowwakeup() at sowwakeup+0x26
tcp_input() at tcp_input+0x2a37
ipv4_input() at ipv4_input+0x584
ipintr() at ipintr+0x7f
netintr() at netintr+0xd5
softintr_dispatch() at softintr_dispatch+0x5d
Xsoftnet() at Xsoftnet+0x28
--- interrupt ---
(null)() at 0xffff800021454e30
end of kernel
end trace frame: 0x4043c748, count: -9
ddb> PID PPID PGRP UID S FLAGS WAIT COMMAND
13819 1 13819 0 3 0x80 select sendmail
15713 1 15713 0 3 0x80 ttyin getty
3077 1 3077 0 3 0x80 ttyin getty
1982 1 1982 0 3 0x80 ttyin getty
12235 1 12235 0 3 0x80 ttyin getty
17057 1 17057 0 3 0x80 ttyin getty
23271 1 23271 0 3 0x80 select cron
4619 1 4619 0 3 0x80 select ruby18
13722 1 13722 99 3 0x80 poll sndiod
22844 18069 18069 89 3 0x80 kqread relayd
19323 18069 18069 89 3 0x80 kqread relayd
1643 18069 18069 89 3 0x80 kqread relayd
*26499 18069 18069 89 7 0 relayd
18069 9864 18069 89 3 0x80 kqread relayd
10272 9864 10272 89 3 0x80 kqread relayd
13354 9864 13354 89 3 0x80 kqread relayd
9864 1 9864 0 3 0x80 kqread relayd
22085 1 22085 0 3 0x80 select sshd
18165 18463 19253 83 3 0x80 poll ntpd
18463 19253 19253 83 3 0x80 poll ntpd
19253 1 19253 0 3 0x80 poll ntpd
26963 18156 18156 74 3 0x80 bpf pflogd
18156 1 18156 0 3 0x80 netio pflogd
30594 10090 10090 73 2 0x80 syslogd
10090 1 10090 0 3 0x80 netio syslogd
3510 1 3510 77 3 0x80 poll dhclient
20348 1 22482 0 3 0x80 poll dhclient
25124 1 25124 77 3 0x80 poll dhclient
12672 1 22482 0 3 0x80 poll dhclient
13 0 0 0 3 0x100200 aiodoned aiodoned
12 0 0 0 3 0x100200 syncer update
11 0 0 0 3 0x100200 cleaner cleaner
10 0 0 0 3 0x100200 reaper reaper
9 0 0 0 3 0x100200 pgdaemon pagedaemon
8 0 0 0 3 0x100200 bored crypto
7 0 0 0 3 0x100200 pftm pfpurge
6 0 0 0 3 0x100200 usbtsk usbtask
5 0 0 0 3 0x100200 usbatsk usbatsk
4 0 0 0 3 0x100200 acpi0 acpi0
3 0 0 0 3 0x100200 bored syswq
2 0 0 0 3 0x40100200 idle0
1 0 1 0 3 0x80 wait init
0 -1 0 0 3 0x200 scheduler swapper
ddb> rebooting...
dmesg from same machine:
OpenBSD 5.2 (GENERIC) #309: Wed Aug 1 09:58:55 MDT 2012
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC
real mem = 2146369536 (2046MB)
avail mem = 2066952192 (1971MB)
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xfbc4f (10 entries)
bios0: vendor QEMU version "QEMU" date 01/01/2007
acpi0 at bios0: rev 0
acpi0: sleep states S3 S4 S5
acpi0: tables DSDT FACP SSDT APIC
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
acpiprt0 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0
mpbios at bios0 not configured
vmt0 at mainbus0
vmware: open failed, eax=564d5868, ecx=0000001e, edx=00005658
vmt0: failed to open backdoor RPC channel (TCLO protocol)
cpu0 at mainbus0: (uniprocessor)
cpu0: QEMU Virtual CPU version 0.10.50, 2200.26 MHz
cpu0:
FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,NXE,LONG
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB
64b/line 16-way L2 cache
cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel
0 wired to compatibility, channel 1 wired to compatibility
wd0 at pciide0 channel 0 drive 0: <QEMU HARDDISK>
wd0: 16-sector PIO, LBA48, 10240MB, 20971520 sectors
wd0(pciide0:0:0): using PIO mode 0, DMA mode 2
atapiscsi0 at pciide0 channel 1 drive 0
scsibus0 at atapiscsi0: 2 targets
cd0 at scsibus0 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 0.10> ATAPI 5/cdrom
removable
cd0(pciide0:1:0): using PIO mode 0
uhci0 at pci0 dev 1 function 2 "Intel 82371SB USB" rev 0x01: irq 11
piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: irq 10
iic0 at piixpm0
iic0: addr 0x4c 48=00 words 00=0000 01=0000 02=0000 03=0000 04=0000 05=0000
06=0000 07=0000
iic0: addr 0x4e 48=00 words 00=0000 01=0000 02=0000 03=0000 04=0000 05=0000
06=0000 07=0000
vga1 at pci0 dev 2 function 0 "Cirrus Logic CL-GD5446" rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
em0 at pci0 dev 3 function 0 "Intel PRO/1000MT (82540EM)" rev 0x03: irq 11,
address 52:54:00:cd:f0:c2
em1 at pci0 dev 4 function 0 "Intel PRO/1000MT (82540EM)" rev 0x03: irq 11,
address 52:54:00:2d:c1:8d
em2 at pci0 dev 5 function 0 "Intel PRO/1000MT (82540EM)" rev 0x03: irq 10,
address 52:54:00:49:2c:e8
"Qumranet Virtio Memory" rev 0x00 at pci0 dev 6 function 0 not configured
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: probed fifo depth: 0 bytes
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 0: density unknown
fd1 at fdc0 drive 1: density unknown
usb0 at uhci0: USB revision 1.0
uhub0 at usb0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
nvram: invalid checksum
mtrr: Pentium Pro MTRR support
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
scsibus2 at softraid0: 256 targets
root on wd0a (7e2867c9d162711b.a) swap on wd0b dump on wd0b
WARNING: / was not properly unmounted
clock: unknown CMOS layout
carp0: state transition: BACKUP -> MASTER
After advice from my provider I had to do a "bsd -c" and "disable mpbios"
to get OpenBSD to boot.
relayd.conf:
ext_addr="10.201.0.3"
www1="192.168.0.4"
www2="192.168.0.5"
table <webhosts> { $www1 $www2 }
relay www {
listen on $ext_addr port http
forward to <webhosts> port http mode roundrobin check http "/" code 200
}
pf.conf:
set skip on lo
anchor "relayd/*"
pass quick on em1 proto pfsync keep state (no-sync)
pass on em1 proto carp keep state
pass # to establish keep-state
block in on ! lo0 proto tcp to port 6000:6010
Thanks,
Peter
