On 2018-02-25 22:48, Johan Huldtgren wrote:
On 2018-02-25 21:54, Alexander Bluhm wrote:
On Sat, Feb 24, 2018 at 04:20:50PM -0500, Johan Huldtgren wrote:
trying to connect to my gateway today I found the following
panic. This is 100% reproducible anytime I connect via
openvpn and then generate traffic. This first happened on
the Feb 7th snap, I updated and it happens on the latest
snap as well.
The questions is what have you used before. I have rewritten the
code at 2017/12/29. Was your working version before that?
Looking back at my logs it looks like the last time I used it was
January 20th, and the snap I had then was:
OpenBSD 6.2-current (GENERIC) #316: Sat Dec 23 11:39:17 MST 2017
[email protected]:/usr/src/sys/arch/i386/compile/GENERIC
Although a bit different, a simmilar assertion was there before.
ip_output(d2574200,0,f5275000,1,0,0,0) at ip_output+0x649
ip_forward(d2574200,d24b8000,d1f8aed8,0) at ip_forward+0x20a
ddb> show mbuf
mbuf 0xd05032f4
When you type "show mbuf" you must give the address of the mbuf.
The functions ip_output() and ip_forward() pass it at the first
argument. So "show mbuf 0xd2574200" would produces reasonable
results, your command dumped arbitrary memory.
thanks, noted for the future.
panic: kernel diagnostic assertion "skrev->reverse == NULL" failed:
file
"/usr/src/sys/net/pf.c", line 7277
pf_find_state(d2486f00,f528d8e0,2,d251b900) at pf_find_state+0x28d
This functions calls pf_state_key_link_reverse(sk, pkt_sk) with
pkt_sk->reverse != NULL.
On the way to that call we went through
pkt_sk = m->m_pkthdr.pf.statekey;
if (pkt_sk && pf_state_key_isvalid(pkt_sk->reverse))
sk = pkt_sk->reverse;
We know that pkt_sk != NULL and pkt_sk->reverse != NULL and before
doing the RB_FIND() lookup we check that sk == NULL. So
pf_state_key_isvalid(pkt_sk->reverse) must be false.
The kernel tried to use an invalid statekey. How can that happen?
Invalid means sk->removed == 1, but pf_state_key_detach() also calls
pf_state_key_unlink_reverse().
pf_test(2,3,d247d400,f528da64) at pf_test+0xb63
ip_output(d251b900,0,f528dad0,1,0,0,0) at ip_output+0x649
Here we find the outgoing state. The mbuf had a statekey before.
ip_forward(d251b900,d247d400,d201cb48,0) at ip_forward+0x20a
ip_input_if(f528dc64,f528dc50,4,0,d247d400) at ip_input_if+0x48e
ipv4_input(d247d400,d251b900) at ipv4_input+0x2b
Here pf attaches the incoming statekey to the mbuf. This is the
one with the invalid reverse.
tun_dev_write(d247d400,f528dd98,10001) at tun_dev_write+0x222
tunwrite(2800,f528dd98,11) at tunwrite+0x53
How does you pf config look like? Do you have some skip on tun?
The only thing I have skip on is lo
Was there unencrpyted traffic before you enabled openvpn?
I assume so, this is my firewall there is always some traffic
but tun0 is only used for openvpn
Were there matching pf states before you enabled openvpn?
I'm not sure exactly sure what you're asking, as this host
does more than openvpn there would have been other states,
not sure if I can find that now though?
Does it immediately crash when you start openvpn and the first
packet is sent out.
It seems so I can connect and if I then (for example) ping a
host on the inside, I get one (most I've ever seen is three)
ping back and then nothing. At that point I can also see that
the host has failed over to it's carp partner.
Do you only use the tun interface and the outgoing
interface?
No, here is the output of ifconfig, there are several
interfaces / vlans,
# ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 32768
index 6 priority 0 llprio 3
groups: lo
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
inet 127.0.0.1 netmask 0xff000000
vr0:
flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST>
mtu 1500
lladdr 00:00:24:c9:58:4c
index 1 priority 0 llprio 3
groups: egress
media: Ethernet autoselect (100baseTX full-duplex)
status: active
inet 172.16.0.2 netmask 0xffffff00 broadcast 172.16.0.255
vr1:
flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST>
mtu 1500
lladdr 00:00:24:c9:58:4d
index 2 priority 0 llprio 3
media: Ethernet autoselect (100baseTX full-duplex)
status: active
vr2:
flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST>
mtu 1500
lladdr 00:00:24:c9:58:4e
index 3 priority 0 llprio 3
media: Ethernet autoselect (100baseTX full-duplex)
status: active
vr3:
flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST>
mtu 1500
lladdr 00:00:24:c9:58:4f
index 4 priority 0 llprio 3
media: Ethernet autoselect (100baseTX full-duplex)
status: active
enc0: flags=0<>
index 5 priority 0 llprio 3
groups: enc
status: active
ral0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:12:0e:61:7f:b0
index 7 priority 4 llprio 3
groups: wlan
media: IEEE802.11 autoselect
status: no network
ieee80211: nwid ""
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:00:5e:00:01:01
index 8 priority 15 llprio 3
carp: MASTER carpdev vr0 vhid 1 advbase 1 advskew 0 carppeer
172.16.0.3
groups: carp
status: master
inet 172.16.0.103 netmask 0xffffff00 broadcast 172.16.0.255
carp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:00:5e:00:01:02
index 9 priority 15 llprio 3
carp: MASTER carpdev vlan20 vhid 2 advbase 1 advskew 0
carppeer 192.168.100.3
groups: carp
status: master
inet 192.168.100.103 netmask 0xffffff00 broadcast
192.168.100.255
carp2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:00:5e:00:01:03
index 10 priority 15 llprio 3
carp: MASTER carpdev vlan30 vhid 3 advbase 1 advskew 0
carppeer 192.168.0.3
groups: carp
status: master
inet 192.168.0.103 netmask 0xffffff00 broadcast 192.168.0.255
carp3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:00:5e:00:01:04
index 11 priority 15 llprio 3
carp: MASTER carpdev vlan666 vhid 4 advbase 1 advskew 0
carppeer 10.66.66.3
groups: carp
status: master
inet 10.66.66.103 netmask 0xffffff00 broadcast 10.66.66.255
pflow0: flags=41<UP,RUNNING> mtu 1448
index 12 priority 0 llprio 3
pflow: sender: 192.168.100.2 receiver: 192.168.100.8:9995
version: 10
groups: pflow
pfsync0: flags=41<UP,RUNNING> mtu 1500
index 13 priority 0 llprio 3
pfsync: syncdev: vlan666 syncpeer: 10.66.66.3 maxupd: 128
defer: off
groups: carp pfsync
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1500
index 14 priority 0 llprio 3
groups: tun
status: active
inet 10.6.6.1 --> 10.6.6.2 netmask 0xffffffff
vlan20: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu
1500
lladdr 00:00:24:c9:58:4d
index 15 priority 0 llprio 3
encap: vnetid 20 parent vr1
groups: vlan
media: Ethernet autoselect (100baseTX full-duplex)
status: active
inet 192.168.100.2 netmask 0xffffff00 broadcast 192.168.100.255
vlan30: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu
1500
lladdr 00:00:24:c9:58:4e
index 16 priority 0 llprio 3
encap: vnetid 30 parent vr2
groups: vlan
media: Ethernet autoselect (100baseTX full-duplex)
status: active
inet 192.168.0.2 netmask 0xffffff00 broadcast 192.168.0.255
vlan666: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu
1500
lladdr 00:00:24:c9:58:4f
index 17 priority 0 llprio 3
encap: vnetid 666 parent vr3
groups: vlan
media: Ethernet autoselect (100baseTX full-duplex)
status: active
inet 10.66.66.2 netmask 0xffffff00 broadcast 10.66.66.255
pflog0: flags=141<UP,RUNNING,PROMISC> mtu 33172
index 18 priority 0 llprio 3
groups: pflog
Do you have more forwarding paths?
I don't know, how would I check?
Do you use floating states?
pf.conf(5) says this is the default and I've not
changed it.
Does the problem go away with if-bound states?
No, I set this in pf.conf
set state-policy if-bound
and tried and it still paniced.
Is there more stuff involved like gif(4) or bridge(4) or ....
neither gif nor bridge, but vlan and carp.
Although I do not fully understand what is the root of the problem,
you can try this diff. Does it prevent the panic? Do you see the
log message I have added there? This would at least prove that my
theory is correct.
I traveling (which is why I was trying to use my vpn), unfortunately
my internet here is atrocious, just reading and writing email is
very hard. Once I get somewhere with better connections I'll get this
tested and report back.
thanks,
.jh
bluhm
Index: net/pf.c
===================================================================
RCS file: /data/mirror/openbsd/cvs/src/sys/net/pf.c,v
retrieving revision 1.1061
diff -u -p -r1.1061 pf.c
--- net/pf.c 18 Feb 2018 21:45:30 -0000 1.1061
+++ net/pf.c 26 Feb 2018 00:27:57 -0000
@@ -1070,8 +1070,20 @@ pf_find_state(struct pfi_kif *kif, struc
pkt_sk = NULL;
}
- if (pkt_sk && pf_state_key_isvalid(pkt_sk->reverse))
- sk = pkt_sk->reverse;
+ if (pkt_sk) {
+ if (pf_state_key_isvalid(pkt_sk->reverse)) {
+ sk = pkt_sk->reverse;
+ } else if (pkt_sk->reverse != NULL) {
+ log(LOG_ERR,
+ "pf: state key reverse invalid. "
+ "pkt_sk=%p, pkt_sk->reverse=%p, "
+ "pkt_sk->reverse->reverse=%p\n",
+ pkt_sk, pkt_sk->reverse,
+ pkt_sk->reverse->reverse);
+ pf_mbuf_unlink_state_key(m);
+ pkt_sk = NULL;
+ }
+ }
if (pkt_sk == NULL) {
/* here we deal with local outbound packet */
Patch does not change anything, it panics as soon as I've connected and
I try to ping something on the inside. I couldn't find any logging for
this, I assume it would be in messages or daemon?
panic: kernel diagnostic assertion "skrev->reverse == NULL" failed: file
"/usr/src/sys/net/pf.c", line 7293
Stopped at db_enter+0x4: popl %ebp
TID PID UID PRFLAGS PFLAGS CPU COMMAND
*426685 67308 577 0x10 0 0 openvpn
db_enter() at db_enter+0x4
panic() at panic+0xcc
__assert(d09c0ca8,d0a924b0,1c7d,d0a92ce7) at __assert+0x19
pf_find_state(d2486f00,f528cb20,2,d24bb700) at pf_find_state+0x2dd
pf_test_state(f528cba0,f528cb90,f528cb9e,0) at pf_test_state+0x104
pf_test(2,3,d247f400,f528cca4) at pf_test+0xb63
ip_output(d24bb700,0,f528cd10,1,0,0,0) at ip_output+0x649
ip_forward(d24bb700,d247f400,d201eab0,0) at ip_forward+0x20a
ip_input_if(f528cea4,f528ce90,4,0,d247f400) at ip_input_if+0x48e
ipv4_input(d247f400,d24bb700) at ipv4_input+0x2b
tun_dev_write(d247f400,f528cfd8,10001) at tun_dev_write+0x222
tunwrite(2800,f528cfd8,11) at tunwrite+0x53
spec_write(f528cf58) at spec_write+0x78
VOP_WRITE(d1f78ad0,f528cfd8,11,d2034f00) at VOP_WRITE+0x48
https://www.openbsd.org/ddb.html describes the minimum info required in
bug
reports. Insufficient info makes it difficult to find and fix bugs.
ddb>
ddb> trace
db_enter() at db_enter+0x4
panic() at panic+0xcc
__assert(d09c0ca8,d0a924b0,1c7d,d0a92ce7) at __assert+0x19
pf_find_state(d2486f00,f528cb20,2,d24bb700) at pf_find_state+0x2dd
pf_test_state(f528cba0,f528cb90,f528cb9e,0) at pf_test_state+0x104
pf_test(2,3,d247f400,f528cca4) at pf_test+0xb63
ip_output(d24bb700,0,f528cd10,1,0,0,0) at ip_output+0x649
ip_forward(d24bb700,d247f400,d201eab0,0) at ip_forward+0x20a
ip_input_if(f528cea4,f528ce90,4,0,d247f400) at ip_input_if+0x48e
ipv4_input(d247f400,d24bb700) at ipv4_input+0x2b
tun_dev_write(d247f400,f528cfd8,10001) at tun_dev_write+0x222
tunwrite(2800,f528cfd8,11) at tunwrite+0x53
spec_write(f528cf58) at spec_write+0x78
VOP_WRITE(d1f78ad0,f528cfd8,11,d2034f00) at VOP_WRITE+0x48
vn_write(d200dbe0,d200dbf8,f528cfd8,d2034f00) at vn_write+0xc7
dofilewritev(d1f97008,5,d200dbe0,cf7cac98,2,1,d200dbf8,f528d0b0) at
dofilewrite
v+0x1b1
sys_writev(d1f97008,f528d0b8,f528d0b0) at sys_writev+0x5a
syscall() at syscall+0x1a0
--- syscall (number -813912936) ---
end of kernel
start_phys+0x12e:
ddb>
ddb> show mbuf 0xd24bb700
mbuf 0xd24bb700
m_type: 1 m_flags: 2<M_PKTHDR>
m_next: 0x0 m_nextpkt: 0x0
m_data: 0xd24bb758 m_len: 78
m_dat: 0xd24bb718 m_pktdat: 0xd24bb754
m_ptkhdr.ph_ifidx: 14 m_pkthdr.len: 78
m_ptkhdr.ph_tags: 0x0 m_pkthdr.ph_tagsset: 0
m_pkthdr.ph_flowid: 34655 m_pkthdr.ph_loopcnt: 0
m_pkthdr.csum_flags: 0
m_pkthdr.ether_vtag: 0 m_ptkhdr.ph_rtableid: 0
m_pkthdr.pf.statekey: 0xf52db9b0 m_pkthdr.pf.inp 0x0
m_pkthdr.pf.qid: 0 m_pkthdr.pf.tag: 0
m_pkthdr.pf.flags: 80<PROCESSED>
m_pkthdr.pf.routed: 0 m_pkthdr.pf.prio: 3
ddb>
ddb> ps
PID TID PPID UID S FLAGS WAIT COMMAND
37494 334977 1 688 3 0x90 kqread
dnscrypt-proxy
78569 262057 1 559 3 0x90 select radiusd
78569 223907 1 559 3 0x4000090 thrsleep radiusd
78569 86461 1 559 3 0x4000090 thrsleep radiusd
78569 415163 1 559 3 0x4000090 thrsleep radiusd
78569 59253 1 559 3 0x4000090 thrsleep radiusd
78569 504707 1 559 3 0x4000090 thrsleep radiusd
83384 510810 1 77 3 0x100090 poll dhcpd
8001 427162 1 53 3 0x90 kqread unbound
83662 263207 1 0 3 0x100083 ttyin ksh
29432 315860 1 0 3 0x100098 poll cron
97054 64971 31708 623 3 0x90 nanosleep zabbix_agentd
14721 110975 31708 623 3 0x90 select zabbix_agentd
82957 470661 31708 623 3 0x90 netcon zabbix_agentd
61580 96345 31708 623 3 0x90 select zabbix_agentd
8157 293731 31708 623 3 0x90 nanosleep zabbix_agentd
31708 274566 1 623 3 0x90 wait zabbix_agentd
65018 293563 1 535 3 0x90 nanosleep symon
75972 80506 1 0 3 0x80 nanosleep apcupsd
75972 217074 1 0 3 0x4000088 sigwait apcupsd
75972 62068 1 0 3 0x4000080 netcon apcupsd
57051 119033 1 629 3 0x90 poll avahi-daemon
90815 22042 1 572 3 0x90 poll dbus-daemon
25072 502298 1 99 3 0x100090 poll sndiod
6111 141612 1 110 3 0x100090 poll sndiod
81476 457888 1 79 3 0x100090 kqread tftpd
13352 482910 1 109 3 0x90 kqread ftp-proxy
17797 258167 17617 95 3 0x100092 kqread smtpd
35691 98616 17617 103 3 0x100092 kqread smtpd
50891 204907 17617 95 3 0x100092 kqread smtpd
22451 235621 17617 95 3 0x100092 kqread smtpd
67790 425909 17617 95 3 0x100092 kqread smtpd
88648 171631 17617 95 3 0x100092 kqread smtpd
17617 127335 1 0 3 0x100080 kqread smtpd
48610 186699 1 0 3 0x80 select sshd
60394 333487 0 0 3 0x14200 acct acct
92161 414025 98730 0 3 0x80 netio npppd
98730 513329 1 82 3 0x90 kqread npppd
10322 369136 84716 68 3 0x90 select isakmpd
84716 67144 1 0 3 0x80 netio isakmpd
30734 509637 1 0 3 0x100080 poll ntpd
4940 247546 97778 83 3 0x100092 poll ntpd
97778 181176 1 83 3 0x100092 poll ntpd
31384 156265 25407 97 3 0x100090 kqread nsd
25407 467498 65501 97 3 0x100090 poll nsd
65501 80344 1 97 3 0x100090 kqread nsd
69998 287306 15273 74 3 0x100092 bpf pflogd
15273 489651 1 0 3 0x80 netio pflogd
95834 160543 78157 73 3 0x100090 kqread syslogd
78157 19265 1 0 3 0x100082 netio syslogd
*67308 426685 1 577 7 0x10 openvpn
49289 191175 64972 115 3 0x100092 kqread slaacd
6865 210520 64972 115 3 0x100092 kqread slaacd
64972 472377 1 0 3 0x80 kqread slaacd
37453 123053 0 0 3 0x14200 pgzero zerothread
32407 300486 0 0 3 0x14200 aiodoned aiodoned
77016 486977 0 0 3 0x14200 syncer update
48084 405859 0 0 3 0x14200 cleaner cleaner
27113 323454 0 0 3 0x14200 reaper reaper
76905 448558 0 0 3 0x14200 pgdaemon pagedaemon
14677 339690 0 0 3 0x14200 bored crynlk
20961 216802 0 0 3 0x14200 bored crypto
97734 10367 0 0 3 0x14200 usbtsk usbtask
43400 187151 0 0 3 0x14200 usbatsk usbatsk
95404 265322 0 0 2 0x14200 softnet
3895 109165 0 0 3 0x14200 bored systqmp
59037 50324 0 0 3 0x14200 bored systq
78726 466647 0 0 3 0x40014200 bored softclock
45874 438047 0 0 3 0x40014200 idle0
85719 238669 0 0 3 0x14200 kmalloc kmthread
1 85595 0 0 3 0x82 wait init
0 0 -1 0 3 0x10200 scheduler swapper
ddb>
ddb> show registers
ds 0x10
es 0x10
fs 0x20
gs 0
edi 0xd0a88f24 via8231_routing_cnfg+0x5dc
esi 0x100
ebp 0xf528ca74
ebx 0xf528ca9c
edx 0x5
ecx 0
eax 0x1
eip 0xd06e0984 db_enter+0x4
cs 0x8
eflags 0x200202 start_phys+0x12e
esp 0xf528ca74
ss 0x10
db_enter+0x4: popl %ebp
ddb>
ddb> trace /p 0t426685
uvm_fault(0xd20264c0, 0x0, 0, 1) -> e
kernel: page fault trap, code=0
Faulted in DDB; continuing...
ddb>
OpenBSD 6.2-current (GENERIC) #0: Tue Feb 27 01:39:17 EST 2018
[email protected]:/sys/arch/i386/compile/GENERIC
cpu0: Geode(TM) Integrated Processor by AMD PCS ("AuthenticAMD"
586-class) 500 MHz
cpu0: FPU,DE,PSE,TSC,MSR,CX8,SEP,PGE,CMOV,CFLUSH,MMX,MMXX,3DNOW2,3DNOW
real mem = 536363008 (511MB)
avail mem = 512655360 (488MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: date 20/71/05, BIOS32 rev. 0 @ 0xfac40
pcibios0 at bios0: rev 2.0 @ 0xf0000/0x10000
pcibios0: pcibios_get_intr_routing - function not supported
pcibios0: PCI IRQ Routing information unavailable.
pcibios0: PCI bus #0 is the last bus
bios0: ROM list: 0xc8000/0xa800
cpu0 at mainbus0: (uniprocessor)
mtrr: K6-family MTRR support (2 registers)
amdmsr0 at mainbus0
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
0:20:0: io address conflict 0x6100/0x100
0:20:0: io address conflict 0x6200/0x200
pchb0 at pci0 dev 1 function 0 "AMD Geode LX" rev 0x31
glxsb0 at pci0 dev 1 function 2 "AMD Geode LX Crypto" rev 0x00: RNG AES
vr0 at pci0 dev 6 function 0 "VIA VT6105M RhineIII" rev 0x96: irq 11,
address 00:00:24:c9:58:4c
ukphy0 at vr0 phy 1: Generic IEEE 802.3u media interface, rev. 3: OUI
0x004063, model 0x0034
vr1 at pci0 dev 7 function 0 "VIA VT6105M RhineIII" rev 0x96: irq 5,
address 00:00:24:c9:58:4d
ukphy1 at vr1 phy 1: Generic IEEE 802.3u media interface, rev. 3: OUI
0x004063, model 0x0034
vr2 at pci0 dev 8 function 0 "VIA VT6105M RhineIII" rev 0x96: irq 9,
address 00:00:24:c9:58:4e
ukphy2 at vr2 phy 1: Generic IEEE 802.3u media interface, rev. 3: OUI
0x004063, model 0x0034
vr3 at pci0 dev 9 function 0 "VIA VT6105M RhineIII" rev 0x96: irq 12,
address 00:00:24:c9:58:4f
ukphy3 at vr3 phy 1: Generic IEEE 802.3u media interface, rev. 3: OUI
0x004063, model 0x0034
ral0 at pci0 dev 17 function 0 "Ralink RT2561S" rev 0x00: irq 15,
address 00:12:0e:61:7f:b0
ral0: MAC/BBP RT2561C, RF RT5225
glxpcib0 at pci0 dev 20 function 0 "AMD CS5536 ISA" rev 0x03: rev 3,
32-bit 3579545Hz timer, watchdog, gpio, i2c
gpio0 at glxpcib0: 32 pins
iic0 at glxpcib0
pciide0 at pci0 dev 20 function 2 "AMD CS5536 IDE" rev 0x01: DMA,
channel 0 wired to compatibility, channel 1 wired to compatibility
wd0 at pciide0 channel 0 drive 0: <SanDisk SDCFX4-8192>
wd0: 4-sector PIO, LBA, 7815MB, 16007040 sectors
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 ignored (disabled)
ohci0 at pci0 dev 21 function 0 "AMD CS5536 USB" rev 0x02: irq 7,
version 1.0, legacy support
ehci0 at pci0 dev 21 function 1 "AMD CS5536 USB" rev 0x02: irq 7
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "AMD EHCI root hub" rev
2.00/1.00 addr 1
isa0 at glxpcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbc0: unable to establish interrupt for irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
nsclpcsio0 at isa0 port 0x2e/2: NSC PC87366 rev 10: GPIO VLM TMS
gpio1 at nsclpcsio0: 29 pins
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
usb1 at ohci0: USB revision 1.0
uhub1 at usb1 configuration 1 interface 0 "AMD OHCI root hub" rev
1.00/1.00 addr 1
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
scsibus2 at softraid0: 256 targets
root on wd0a (ba730608caf94ae4.a) swap on wd0b dump on wd0b
WARNING: / was not properly unmounted
carp0: state transition: BACKUP -> MASTER
carp1: state transition: BACKUP -> MASTER
carp2: state transition: BACKUP -> MASTER
carp3: state transition: BACKUP -> MASTER
carp0: state transition: MASTER -> BACKUP
carp1: state transition: MASTER -> BACKUP
carp2: state transition: MASTER -> BACKUP
carp3: state transition: MASTER -> BACKUP
carp0: state transition: BACKUP -> MASTER
carp1: state transition: BACKUP -> MASTER
carp2: state transition: BACKUP -> MASTER
carp3: state transition: BACKUP -> MASTER
