Re: em(4) watchdog timeouts

[Alexis VACHETTE]

Hi Gregor,

I use the same revision than yours :

- "Intel 82583V" rev 0x00: msi

Regards,
Alexis VACHETTE.*
*
On 16/11/2015 10:12, Alexis VACHETTE wrote:

Hi Gregor,

Thank you for your feedback.

Did you have some timeout on 5.6 ?

On amd64 version, I experienced some on heavy network load. Is it 
related ?


Regards,
Alexis VACHETTE.
On 11/11/2015 21:19, Gregor Best wrote:

Hi Alexis,

On Wed, Nov 11, 2015 at 08:11:15PM +, Alexis VACHETTE wrote:

[...]
Even with heavy network load ?
[...]

So far, yes. I've saturated the device for about 45 Minutes with
something like this (the other end is my laptop):

## on the router
$ dd if=/dev/zero bs=8k | nc 172.31.64.174 55000
## on my laptop
$ nc -l 55000 | dd of=/dev/null bs=8k

(with two or three streams in parallel). There were about 6k
interrupts per second and bandwidth was about 250Mbps, which seems
to be the maximum the tiny CPU in this router can do. No watchdog
timeouts appeared, where previously something relatively low bandwidth
(the SSDs in router and laptop suck) like this caused one every 20
or 30 seconds:

## on the router
$ pax -w /home | nc 172.31.64.174 55000

I'll keep an eye on things, but so far it looks good. Regular usage
works out so far as well. If you need me to run some special workload
for you, I'd be more than happy to do that.

Re: em(4) watchdog timeouts

[Alexis VACHETTE]

Hi Gregor,

Thank you for your feedback.

Did you have some timeout on 5.6 ?

On amd64 version, I experienced some on heavy network load. Is it related ?

Regards,
Alexis VACHETTE.
On 11/11/2015 21:19, Gregor Best wrote:

Hi Alexis,

On Wed, Nov 11, 2015 at 08:11:15PM +, Alexis VACHETTE wrote:

[...]
Even with heavy network load ?
[...]

So far, yes. I've saturated the device for about 45 Minutes with
something like this (the other end is my laptop):

## on the router
$ dd if=/dev/zero bs=8k | nc 172.31.64.174 55000
## on my laptop
$ nc -l 55000 | dd of=/dev/null bs=8k

(with two or three streams in parallel). There were about 6k
interrupts per second and bandwidth was about 250Mbps, which seems
to be the maximum the tiny CPU in this router can do. No watchdog
timeouts appeared, where previously something relatively low bandwidth
(the SSDs in router and laptop suck) like this caused one every 20
or 30 seconds:

## on the router
$ pax -w /home | nc 172.31.64.174 55000

I'll keep an eye on things, but so far it looks good. Regular usage
works out so far as well. If you need me to run some special workload
for you, I'd be more than happy to do that.

Re: em(4) watchdog timeouts

[David Gwynne]

On Fri, Nov 13, 2015 at 10:18:51AM -0500, Sonic wrote:
> On Wed, Nov 11, 2015 at 9:20 AM, Gregor Best  wrote:
> > I've done some further testing and I think I've narrowed it down to the
> > "Unlocking em(4) a bit further"-patch [0].

could you try this? its not written with the wdog stuff in mind,
but it does touch that stuff so it might help.

Index: if_em.c
===
RCS file: /cvs/src/sys/dev/pci/if_em.c,v
retrieving revision 1.310
diff -u -p -r1.310 if_em.c
--- if_em.c 29 Oct 2015 03:19:42 -  1.310
+++ if_em.c 15 Nov 2015 14:01:39 -
@@ -605,16 +605,20 @@ em_start(struct ifnet *ifp)
}
 
for (;;) {
-   IFQ_POLL(>if_snd, m_head);
-   if (m_head == NULL)
-   break;
-
-   if (em_encap(sc, m_head)) {
+   if (sc->num_tx_desc_avail < EM_MAX_SCATTER + 2) {
ifp->if_flags |= IFF_OACTIVE;
break;
}
 
IFQ_DEQUEUE(>if_snd, m_head);
+   if (m_head == NULL)
+   break;
+
+   if (em_encap(sc, m_head)) {
+   m_freem(m_head);
+   ifp->if_oerrors++;
+   continue;
+   }
 
 #if NBPFILTER > 0
/* Send a copy of the frame to the BPF listener */
@@ -622,9 +626,6 @@ em_start(struct ifnet *ifp)
bpf_mtap_ether(ifp->if_bpf, m_head, BPF_DIRECTION_OUT);
 #endif
 
-   /* Set timeout in case hardware has problems transmitting */
-   ifp->if_timer = EM_TX_TIMEOUT;
-
post = 1;
}
 
@@ -637,8 +638,11 @@ em_start(struct ifnet *ifp)
 * this tells the E1000 that this frame is
 * available to transmit.
 */
-   if (post)
+   if (post) {
E1000_WRITE_REG(>hw, TDT, sc->next_avail_tx_desc);
+
+   ifp->if_timer = EM_TX_TIMEOUT;
+   }
}
 }
 
@@ -1104,12 +1108,6 @@ em_encap(struct em_softc *sc, struct mbu
struct em_buffer   *tx_buffer, *tx_buffer_mapped;
struct em_tx_desc *current_tx_desc = NULL;
 
-   /* Check that we have least the minimal number of TX descriptors. */
-   if (sc->num_tx_desc_avail <= EM_TX_OP_THRESHOLD) {
-   sc->no_tx_desc_avail1++;
-   return (ENOBUFS);
-   }
-
if (sc->hw.mac_type == em_82547) {
bus_dmamap_sync(sc->txdma.dma_tag, sc->txdma.dma_map, 0,
sc->txdma.dma_map->dm_mapsize,
@@ -1147,9 +1145,6 @@ em_encap(struct em_softc *sc, struct mbu
 
EM_KASSERT(map->dm_nsegs!= 0, ("em_encap: empty packet"));
 
-   if (map->dm_nsegs > sc->num_tx_desc_avail - 2)
-   goto fail;
-
if (sc->hw.mac_type >= em_82543 && sc->hw.mac_type != em_82575 &&
sc->hw.mac_type != em_82580 && sc->hw.mac_type != em_i210 &&
sc->hw.mac_type != em_i350)
@@ -1168,9 +1163,9 @@ em_encap(struct em_softc *sc, struct mbu
 * Check the Address and Length combination and
 * split the data accordingly
 */
-   array_elements = 
em_fill_descriptors(map->dm_segs[j].ds_addr,
-
map->dm_segs[j].ds_len,
-_array);
+   array_elements = em_fill_descriptors(
+   map->dm_segs[j].ds_addr,
+   map->dm_segs[j].ds_len, _array);
for (counter = 0; counter < array_elements; counter++) {
if (txd_used == sc->num_tx_desc_avail) {
sc->next_avail_tx_desc = txd_saved;
@@ -2481,8 +2476,7 @@ em_txeof(struct em_softc *sc)
 * If we have enough room, clear IFF_OACTIVE to tell the stack
 * that it is OK to send packets.
 */
-   if (ISSET(ifp->if_flags, IFF_OACTIVE) &&
-   num_avail > EM_TX_OP_THRESHOLD) {
+   if (num_avail > 0 && ISSET(ifp->if_flags, IFF_OACTIVE)) {
KERNEL_LOCK();
CLR(ifp->if_flags, IFF_OACTIVE);
em_start(ifp);

Re: em(4) watchdog timeouts

[Gregor Best]

On Mon, Nov 16, 2015 at 12:05:12AM +1000, David Gwynne wrote:
> On Fri, Nov 13, 2015 at 10:18:51AM -0500, Sonic wrote:
> > On Wed, Nov 11, 2015 at 9:20 AM, Gregor Best  wrote:
> > > I've done some further testing and I think I've narrowed it down to the
> > > "Unlocking em(4) a bit further"-patch [0].
> 
> could you try this? its not written with the wdog stuff in mind,
> but it does touch that stuff so it might help.
> [...]

Just tried it, sadly it doesn't seem to help :/

To be sure, I enabled debug on the routers em's, but apart from the watchdog 
timeout, there's nothing in there.

-- 
Gregor

Re: em(4) watchdog timeouts

[Sonic]

On Wed, Nov 11, 2015 at 9:20 AM, Gregor Best  wrote:
> I've done some further testing and I think I've narrowed it down to the
> "Unlocking em(4) a bit further"-patch [0].

That was the start of it for me.

When I could revert to rev 1.305 for if_em.c and rev 1.57 for if_em.h
all was fine. But the kernel no longer builds with if_em.c rev 1.305
due to:
cc1: warnings being treated as errors
../../../../dev/pci/if_em.c: In function 'em_ioctl':
../../../../dev/pci/if_em.c:674: warning: implicit declaration of function
'arp_ifinit'
*** Error 1 in /usr/src/sys/arch/amd64/compile/GENERIC.MP (Makefile:937
'if_em.o')

I'm wondering if the patch that moved if_em.c from rev 1.308 to rev
1.309 could be applied to rev 1.305 and if that would the kernel to
build and work?

These timeouts are killing me.
Nov 13 08:21:11 stargate /bsd: em0: watchdog timeout -- resetting
Nov 13 09:47:36 stargate unbound: [12687:1] notice: sendto failed: No
buffer space available
Nov 13 09:47:36 stargate unbound: [22477:0] notice: sendto failed: No
buffer space available
Nov 13 09:47:36 stargate unbound: [22477:0] notice: remote address is
172.27.12.66 port 34281
Nov 13 09:47:36 stargate unbound: [12687:1] notice: remote address is
172.27.12.66 port 34281
Nov 13 09:47:37 stargate /bsd: em0: watchdog timeout -- resetting
Nov 13 09:49:42 stargate /bsd: em0: watchdog timeout -- resetting
Nov 13 09:51:55 stargate unbound: [22477:0] notice: sendto failed: No
buffer space available
Nov 13 09:51:55 stargate unbound: [22477:0] notice: remote address is
172.27.12.66 port 57280
Nov 13 09:52:00 stargate /bsd: em0: watchdog timeout -- resetting
Nov 13 09:53:18 stargate /bsd: em0: watchdog timeout -- resetting
Nov 13 09:54:32 stargate unbound: [12687:1] notice: sendto failed: No
buffer space available
Nov 13 09:54:32 stargate unbound: [22477:0] notice: sendto failed: No
buffer space available
Nov 13 09:54:32 stargate unbound: [22477:0] notice: remote address is
172.27.12.66 port 57438
Nov 13 09:54:32 stargate unbound: [12687:1] notice: remote address is
172.27.12.66 port 57438
Nov 13 09:54:35 stargate /bsd: em0: watchdog timeout -- resetting
Nov 13 09:55:54 stargate /bsd: em0: watchdog timeout -- resetting
Nov 13 09:58:08 stargate last message repeated 2 times
Nov 13 09:58:53 stargate /bsd: em1: watchdog timeout -- resetting
Nov 13 09:59:05 stargate ftp-proxy[18455]: #1 client command too long
or not clean
Nov 13 09:59:09 stargate unbound: [12687:1] notice: sendto failed: No
buffer space available
Nov 13 09:59:09 stargate unbound: [12687:1] notice: remote address is
172.27.12.66 port 22970
Nov 13 09:59:09 stargate unbound: [12687:1] notice: sendto failed: No
buffer space available
Nov 13 09:59:09 stargate unbound: [12687:1] notice: remote address is
172.27.12.66 port 60307
Nov 13 09:59:11 stargate unbound: [12687:1] notice: sendto failed: No
buffer space available
Nov 13 09:59:11 stargate unbound: [12687:1] notice: remote address is
172.27.12.66 port 41198
Nov 13 09:59:11 stargate /bsd: em0: watchdog timeout -- resetting
Nov 13 10:01:15 stargate /bsd: em0: watchdog timeout -- resetting

Chris

Re: em(4) watchdog timeouts

[Gregor Best]

I've done some further testing and I think I've narrowed it down to the
"Unlocking em(4) a bit further"-patch [0]. With the patch reverted, I
haven't seen any watchdog timeouts yet. I'm currently running the router
with the patch reverted to make sure the timeouts don't happen again.

[0]: https://www.marc.info/?l=openbsd-tech=144347723907388=4

-- 
Gregor

Re: em(4) watchdog timeouts

[Gregor Best]

Hi Alexis,

On Wed, Nov 11, 2015 at 08:11:15PM +, Alexis VACHETTE wrote:
> [...]
> Even with heavy network load ?
> [...]

So far, yes. I've saturated the device for about 45 Minutes with
something like this (the other end is my laptop):

## on the router
$ dd if=/dev/zero bs=8k | nc 172.31.64.174 55000
## on my laptop
$ nc -l 55000 | dd of=/dev/null bs=8k

(with two or three streams in parallel). There were about 6k
interrupts per second and bandwidth was about 250Mbps, which seems
to be the maximum the tiny CPU in this router can do. No watchdog
timeouts appeared, where previously something relatively low bandwidth
(the SSDs in router and laptop suck) like this caused one every 20
or 30 seconds:

## on the router
$ pax -w /home | nc 172.31.64.174 55000

I'll keep an eye on things, but so far it looks good. Regular usage
works out so far as well. If you need me to run some special workload
for you, I'd be more than happy to do that.

-- 
Gregor

Re: em(4) watchdog timeouts

[Alexis VACHETTE]

Hi Gregor,

Even with heavy network load ?

Regards,
Alexis.


De : owner-t...@openbsd.org <owner-t...@openbsd.org> de la part de Gregor Best 
<g...@unobtanium.de>
Envoyé : mercredi 11 novembre 2015 15:20
À : Mark Kettenis
Cc : tech@openbsd.org; m...@openbsd.org
Objet : Re: em(4) watchdog timeouts

I've done some further testing and I think I've narrowed it down to the
"Unlocking em(4) a bit further"-patch [0]. With the patch reverted, I
haven't seen any watchdog timeouts yet. I'm currently running the router
with the patch reverted to make sure the timeouts don't happen again.

[0]: https://www.marc.info/?l=openbsd-tech=144347723907388=4

--
Gregor

Re: em(4) watchdog timeouts

[Gregor Best]

On Mon, Nov 02, 2015 at 09:29:20PM +0100, Gregor Best wrote:
> [...]
> Looks good so far. I've run a few light tests and the usual load that
> caused the timeouts before, haven't seen any yet.
> [...]

I just checked back on the router and it seems that the patch doesn't
help after all :( The number of watchdog timeouts went down, but they
are still there, about 35 in the last two days with network (and other)
load on the router almost nonexistant.

If it helps debugging this, I can give SSH access to the router,
provided that reboots don't happen between 18:00 and 02:00 German time
too often, since that's when we have larger amounts of visitors in our
hackerspace.

-- 
Gregor

Re: em(4) watchdog timeouts

[Gregor Best]

On Sun, Nov 08, 2015 at 06:57:23PM +0100, Gregor Best wrote:
> [...]
> If it helps debugging this, I can give SSH access to the router,
> provided that reboots don't happen between 18:00 and 02:00 German time
> too often, since that's when we have larger amounts of visitors in our
> hackerspace.
> [...]

Forgot to mention, the SSH access includes a push button monkey with a
console cable at hand (me) in case something goes wrong.

-- 
Gregor

Re: em(4) watchdog timeouts

[Sonic]

On Wed, Nov 4, 2015 at 2:51 PM, Sonic  wrote:
> Is there anything else I can provide to assist in finding a cure for this 
> issue?

Not sure this helps at all but the specific error I get is "em0:
watchdog timeout -- resetting". In this case em0 is the nic on the
internal network. I do not see the errors on the external network nic
(em1) which connects to the cable modem. The internal network nic
(em0) connects directly to an HP 2520 switch.

# dmesg |grep em1
em1 at pci3 dev 0 function 0 "Intel 82574L" rev 0x00: msi, address
00:25:90:92:d4:f9
spdmem1 at iic0 addr 0x51: 2GB DDR3 SDRAM PC3-10600 SO-DIMM
# dmesg |grep em0
em0 at pci2 dev 0 function 0 "Intel 82574L" rev 0x00: msi, address
00:25:90:92:d4:f8
spdmem0 at iic0 addr 0x50: 2GB DDR3 SDRAM PC3-10600 SO-DIMM
em0: watchdog timeout -- resetting
em0: watchdog timeout -- resetting
em0: watchdog timeout -- resetting
em0: watchdog timeout -- resetting
em0: watchdog timeout -- resetting
em0: watchdog timeout -- resetting

Chris

Re: em(4) watchdog timeouts

[Sonic]

On Mon, Nov 2, 2015 at 11:19 PM, Sonic  wrote:
> Sorry to report that the diff does not solve the timeout problem here.
>
> All was working fine with the if_em* versions from 2015/09/29 (I
> downgraded to this version per Stuarts post on 10-14):
> "try backing out the last commits to
> if_em.c and if_em.h ("cd /sys/dev/pci; cvs up -D 2015/09/29 if_em*") to
> see if it makes a difference."
>
> However, that version no longer compiles with -current and the
> watchdog timeouts are back (even with the diff).

Is there anything else I can provide to assist in finding a cure for this issue?
I get sporadic timeouts even under normal usage, but starting a
bittorrent on a client brings the firewall to its knees. And as all
the firewalls I manage use the em driver I cannot take a chance on
upgrading any of them to -current.

Thank you,

Chris

em(4) watchdog timeouts

[Mark Kettenis]

Can those that are experiencing watchdog timeouts check if the diff
below gets rid of them?


Index: if_em.h
===
RCS file: /home/cvs/src/sys/dev/pci/if_em.h,v
retrieving revision 1.58
diff -u -p -r1.58 if_em.h
--- if_em.h 30 Sep 2015 11:25:08 -  1.58
+++ if_em.h 2 Nov 2015 19:07:55 -
@@ -190,7 +190,7 @@ typedef int boolean_t;
  * Thise parameter controls the minimum number of available transmit
  * descriptors needed before we attempt transmission of a packet.
  */
-#define EM_TX_OP_THRESHOLD (sc->num_tx_desc / 32)
+#define EM_TX_OP_THRESHOLD (EM_MAX_SCATTER + 6)
 
 /*
  * This parameter controls whether or not autonegotiation is enabled.
@@ -271,7 +271,7 @@ typedef int boolean_t;
 #define EM_MCLBYTESEM_RXBUFFER_2048
 #endif
 
-#define EM_MAX_SCATTER 64
+#define EM_MAX_SCATTER 32
 #define EM_TSO_SIZE65535
 
 struct em_buffer {

Re: em(4) watchdog timeouts

[Gregor Best]

On Mon, Nov 02, 2015 at 08:11:30PM +0100, Mark Kettenis wrote:
> Can those that are experiencing watchdog timeouts check if the diff
> below gets rid of them?
> [...]

Looks good so far. I've run a few light tests and the usual load that
caused the timeouts before, haven't seen any yet.

For the record, this is with

em0 at pci1 dev 0 function 0 "Intel 82583V" rev 0x00: msi, address 
00:03:2d:20:cf:84
em1 at pci2 dev 0 function 0 "Intel 82583V" rev 0x00: msi, address 
00:03:2d:20:cf:85
em2 at pci3 dev 0 function 0 "Intel 82583V" rev 0x00: msi, address 
00:03:2d:20:cf:86
em3 at pci4 dev 0 function 0 "Intel 82583V" rev 0x00: msi, address 
00:03:2d:20:cf:87

on i386 (GENERIC.MP).

-- 
Gregor

Re: em(4) watchdog timeouts

[Sonic]

On Mon, Nov 2, 2015 at 2:11 PM, Mark Kettenis  wrote:
> Can those that are experiencing watchdog timeouts check if the diff
> below gets rid of them?

Sorry to report that the diff does not solve the timeout problem here.

All was working fine with the if_em* versions from 2015/09/29 (I
downgraded to this version per Stuarts post on 10-14):
"try backing out the last commits to
if_em.c and if_em.h ("cd /sys/dev/pci; cvs up -D 2015/09/29 if_em*") to
see if it makes a difference."

However, that version no longer compiles with -current and the
watchdog timeouts are back (even with the diff).

$ dmesg
OpenBSD 5.8-current (GENERIC.MP) #12: Mon Nov  2 20:29:08 EST 2015
   r...@stargate.grizzly.bear:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 4277665792 (4079MB)
avail mem = 4143894528 (3951MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.6 @ 0x9f000 (19 entries)
bios0: vendor American Megatrends Inc. version "1.2b" date 07/19/13
bios0: Supermicro X7SPA-HF
acpi0 at bios0: rev 2
acpi0: sleep states S0 S1 S4 S5
acpi0: tables DSDT FACP APIC MCFG OEMB HPET EINJ BERT ERST HEST
acpi0: wakeup devices P0P1(S4) PS2K(S4) PS2M(S4) USB0(S4) USB1(S4)
USB2(S4) USB5(S4) EUSB(S4) USB3(S4) USB4(S4) USB6(S4) USBE(S4) P0P4(S
4) P0P5(S4) P0P6(S4) P0P7(S4) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Atom(TM) CPU D525 @ 1.80GHz, 1800.24 MHz
cpu0: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64
,MWAIT,DS-CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE,NXE,LONG,LAHF,PERF,SENSOR
cpu0: 512KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 199MHz
cpu0: mwait min=64, max=64, C-substates=0.1, IBE
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Atom(TM) CPU D525 @ 1.80GHz, 1800.00 MHz
cpu1: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64
,MWAIT,DS-CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE,NXE,LONG,LAHF,PERF,SENSOR
cpu1: 512KB 64b/line 8-way L2 cache
cpu1: smt 0, core 1, package 0
ioapic0 at mainbus0: apid 3 pa 0xfec0, version 20, 24 pins
ioapic0: misconfigured as apic 1, remapped to apid 3
acpimcfg0 at acpi0 addr 0xe000, bus 0-255
acpihpet0 at acpi0: 14318179 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 4 (P0P1)
acpiprt2 at acpi0: bus 1 (P0P4)
acpiprt3 at acpi0: bus 2 (P0P8)
acpiprt4 at acpi0: bus 3 (P0P9)
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
acpibtn0 at acpi0: SLPB
acpibtn1 at acpi0: PWRB
ipmi at mainbus0 not configured
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel Pineview DMI" rev 0x02
uhci0 at pci0 dev 26 function 0 "Intel 82801I USB" rev 0x02: apic 3 int 16
uhci1 at pci0 dev 26 function 1 "Intel 82801I USB" rev 0x02: apic 3 int 21
uhci2 at pci0 dev 26 function 2 "Intel 82801I USB" rev 0x02: apic 3 int 19
ehci0 at pci0 dev 26 function 7 "Intel 82801I USB" rev 0x02: apic 3 int 18
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb0 at pci0 dev 28 function 0 "Intel 82801I PCIE" rev 0x02: msi
pci1 at ppb0 bus 1
ppb1 at pci0 dev 28 function 4 "Intel 82801I PCIE" rev 0x02: msi
pci2 at ppb1 bus 2
em0 at pci2 dev 0 function 0 "Intel 82574L" rev 0x00: msi, address
00:25:90:92:d4:f8
ppb2 at pci0 dev 28 function 5 "Intel 82801I PCIE" rev 0x02: msi
pci3 at ppb2 bus 3
em1 at pci3 dev 0 function 0 "Intel 82574L" rev 0x00: msi, address
00:25:90:92:d4:f9
uhci3 at pci0 dev 29 function 0 "Intel 82801I USB" rev 0x02: apic 3 int 23
uhci4 at pci0 dev 29 function 1 "Intel 82801I USB" rev 0x02: apic 3 int 19
uhci5 at pci0 dev 29 function 2 "Intel 82801I USB" rev 0x02: apic 3 int 18
ehci1 at pci0 dev 29 function 7 "Intel 82801I USB" rev 0x02: apic 3 int 23
usb1 at ehci1: USB revision 2.0
uhub1 at usb1 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb3 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x92
pci4 at ppb3 bus 4
vga1 at pci4 dev 4 function 0 "Matrox MGA G200eW" rev 0x0a
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
pcib0 at pci0 dev 31 function 0 "Intel 82801IR LPC" rev 0x02
ahci0 at pci0 dev 31 function 2 "Intel 82801I AHCI" rev 0x02: msi, AHCI 1.2
ahci0: port 0: 3.0Gb/s
scsibus1 at ahci0: 32 targets
sd0 at scsibus1 targ 0 lun 0:  SCSI3
0/direct fixed naa.5001517959323666
sd0: 76319MB, 512 bytes/sector, 156301488 sectors, thin
ichiic0 at pci0 dev 31 function 3 "Intel 82801I SMBus" rev 0x02: apic 3 int 18
iic0 at ichiic0
lm1 at iic0 addr 0x2d: W83627DHG
spdmem0 at iic0 addr 0x50: 2GB DDR3 SDRAM PC3-10600 SO-DIMM
spdmem1 at iic0 addr 0x51: 2GB DDR3 SDRAM PC3-10600 SO-DIMM
usb2 at uhci0: USB revision