On 03/10/2011 03:45 PM, Tom Murphy wrote:
> Ryan McBride wrote:
>> On Thu, Mar 10, 2011 at 12:18:32PM +0000, Tom Murphy wrote:
>>> I had a pair of Dell PowerEdge R200s that have both em(4) and bge(4)s
>>> in them, however, it's the em(4) doing the heavy lifting. Roughly 30-40
>>> megabits/s sustained and doing anywhere between 3000-4000 packets/s.
>>>
>>> On OpenBSD 4.4, it happily forwards packets along. I upgraded one of
>>> the firewalls to 4.8 and switched CARP over to it (yes, I know the
>>> redundancy is broken anyway now.) and it couldn't seem to handle the
>>> traffic. Any inbound connections would stall and I have no idea why.
>>
Hi folks,
Sorry for hijacking this thread.
I also have a Dell machine with em(4)'s.
When I upgraded a machine from 4.3 or 4.4 to 4.7 the kernel is leaking
memory I've been looking at it ever since. This was just before 4.8 came
out so it didn't get 4.8.
I disabled everything I could find to figure out if I did something
wrong, ranging from openvpn with briding setup and to the new setup I
made with relayd. Anything I could think of in userspace.
And I also reverted back to the stock kernel instead of one with errata
patches applied. I've set the interfaces to use a full duplex instead of
automatic. Disabled the use of IPv6 (which wasn't used before the upgrade).
Nothing seems to have worked so far.
It isn't a big machine and it doesn't need to handle a lot of traffic
but at the current rate it is leaking memory all day long and I have to
reboot the machine every 1 or 2 weeks or it will stop working.
Which obvously is very sad.
When it gets to about 8000+ mbufs the machine starts to exhibit really
weird behaviour but does not lockup. It can setup client connections on
TCP but TCP or Unix server sockets can not receive any new connections.
I keep a log of the output of netstat -m.
There is part of the output of the log and the dmesg at the end of this
email.
The one part I haven't tried disabling is the dynamic routing, it does
get frequent route updates.
I have an other machine which runs exactly the same binaries but the
hardware is a bit different.
I looked at a lot of changes in CVS and I didn't see anything special in
the related drivers that I could find which warranted an upgrade to 4.8.
Doing an upgrade would take quite a
bit of time I don't have right now and I also didn't want to make the
problem worse. ;-)
If you have any tips I could to further investigate or fix problem I
would really appreciate it.
If you need any extra information let me know.
I keep wondering what changed between 4.3/4.4 and 4.7/4.8 in
correspondance with Dell and em(4).
At this point I'm thinking wasn't there a big update in how ACPI works
on OpenBSD or something like that which might effect how interrupts and
drivers work ?
Anyway have a nice day,
Leen.
___________________________
OpenBSD 4.7 (GENERIC) #558: Wed Mar 17 20:46:15 MDT 2010
dera...@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel(R) Pentium(R) 4 CPU 3.06GHz ("GenuineIntel" 686-class) 3.07 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,CNXT-ID,xTPR
real mem = 1073184768 (1023MB)
avail mem = 1031110656 (983MB)
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 10/08/03, BIOS32 rev. 0 @ 0xffe90,
SMBIOS rev. 2.3 @ 0xfae10 (77 entries)
bios0: vendor Dell Computer Corporation version "A04" date 10/08/2003
bios0: Dell Computer Corporation PowerEdge 650
acpi0 at bios0: rev 0
acpi0: tables DSDT FACP APIC SPCR
acpi0: wakeup devices PCI0(S5) PCI1(S5) PCI2(S5)
acpitimer0 at acpi0: 3579545 Hz, 32 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: apic clock running at 133MHz
cpu at mainbus0: not configured
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 11, 16 pins
ioapic0: misconfigured as apic 0, remapped to apid 2
ioapic1 at mainbus0: apid 3 pa 0xfec01000, version 11, 16 pins
ioapic1: misconfigured as apic 0, remapped to apid 3
ioapic2 at mainbus0: apid 4 pa 0xfec02000, version 11, 16 pins
ioapic2: misconfigured as apic 0, remapped to apid 4
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 1 (PCI1)
acpiprt2 at acpi0: bus 2 (PCI2)
acpicpu0 at acpi0
bios0: ROM list: 0xc0000/0x8000 0xc8000/0x4800 0xcc800/0x1800
0xce000/0x1800 0xec000/0x4000!
pci0 at mainbus0 bus 0: configuration mode 1 (bios)
pchb0 at pci0 dev 0 function 0 "ServerWorks GCNB-LE Host" rev 0x32
pchb1 at pci0 dev 0 function 1 "ServerWorks GCNB-LE Host" rev 0x00
pci1 at pchb1 bus 1
em0 at pci1 dev 3 function 0 "Intel PRO/1000MT (82546EB)" rev 0x01: apic
3 int 3 (irq 5), address 00:04:23:9f:24:56
em1 at pci1 dev 3 function 1 "Intel PRO/1000MT (82546EB)" rev 0x01: apic
3 int 4 (irq 3), address 00:04:23:9f:24:57
em2 at pci0 dev 3 function 0 "Intel PRO/1000MT (82546EB)" rev 0x01: apic
3 int 1 (irq 15), address 00:04:23:5f:1c:b2
em3 at pci0 dev 3 function 1 "Intel PRO/1000MT (82546EB)" rev 0x01: apic
3 int 2 (irq 11), address 00:04:23:5f:1c:b3
vga1 at pci0 dev 4 function 0 "ATI Rage XL" rev 0x27
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
pciide0 at pci0 dev 5 function 0 "CMD Technology PCI0680" rev 0x02
pciide0: bus-master DMA support present
pciide0: channel 0 wired to native-PCI mode
pciide0: using apic 3 int 7 (irq 10) for native-PCI interrupt
wd0 at pciide0 channel 0 drive 0: <WDC WD2000JB-00GVC0>
wd0: 16-sector PIO, LBA48, 190782MB, 390721968 sectors
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5
pciide0: channel 1 wired to native-PCI mode
piixpm0 at pci0 dev 15 function 0 "ServerWorks CSB6" rev 0xa0: SMBus
disabled
pciide1 at pci0 dev 15 function 1 "ServerWorks CSB6 RAID/IDE" rev 0xa0: DMA
atapiscsi0 at pciide1 channel 0 drive 0
scsibus0 at atapiscsi0: 2 targets
cd0 at scsibus0 targ 0 lun 0: <TEAC, CD-224E, K.9A> ATAPI 5/cdrom removable
cd0(pciide1:0:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 2
ohci0 at pci0 dev 15 function 2 "ServerWorks CSB6 USB" rev 0x05: apic 2
int 7 (irq 7), version 1.0, legacy support
pcib0 at pci0 dev 15 function 3 "ServerWorks GCLE-2 Host" rev 0x00
pchb2 at pci0 dev 16 function 0 "ServerWorks CIOB-E" rev 0x12
pchb3 at pci0 dev 16 function 2 "ServerWorks CIOB-E" rev 0x12
pci2 at pchb3 bus 2
usb0 at ohci0: USB revision 1.0
uhub0 at usb0 "ServerWorks OHCI root hub" rev 1.00/1.00 addr 1
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
Sat Mar 5 05:53:00 CET 2011
215 mbufs in use:
142 mbufs allocated to data
15 mbufs allocated to packet headers
58 mbufs allocated to socket names and addresses
84/180/6144 mbuf 2048 byte clusters in use (current/peak/max)
0/8/6144 mbuf 4096 byte clusters in use (current/peak/max)
0/8/6144 mbuf 8192 byte clusters in use (current/peak/max)
0/8/6144 mbuf 9216 byte clusters in use (current/peak/max)
0/8/6144 mbuf 12288 byte clusters in use (current/peak/max)
0/8/6144 mbuf 16384 byte clusters in use (current/peak/max)
0/8/6144 mbuf 65536 byte clusters in use (current/peak/max)
700 Kbytes allocated to network (31% in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
Sat Mar 5 06:53:02 CET 2011
231 mbufs in use:
158 mbufs allocated to data
15 mbufs allocated to packet headers
58 mbufs allocated to socket names and addresses
95/180/6144 mbuf 2048 byte clusters in use (current/peak/max)
0/8/6144 mbuf 4096 byte clusters in use (current/peak/max)
0/8/6144 mbuf 8192 byte clusters in use (current/peak/max)
0/8/6144 mbuf 9216 byte clusters in use (current/peak/max)
0/8/6144 mbuf 12288 byte clusters in use (current/peak/max)
0/8/6144 mbuf 16384 byte clusters in use (current/peak/max)
0/8/6144 mbuf 65536 byte clusters in use (current/peak/max)
700 Kbytes allocated to network (35% in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
Sat Mar 5 07:53:01 CET 2011
262 mbufs in use:
189 mbufs allocated to data
15 mbufs allocated to packet headers
58 mbufs allocated to socket names and addresses
109/180/6144 mbuf 2048 byte clusters in use (current/peak/max)
0/8/6144 mbuf 4096 byte clusters in use (current/peak/max)
0/8/6144 mbuf 8192 byte clusters in use (current/peak/max)
0/8/6144 mbuf 9216 byte clusters in use (current/peak/max)
0/8/6144 mbuf 12288 byte clusters in use (current/peak/max)
0/8/6144 mbuf 16384 byte clusters in use (current/peak/max)
0/8/6144 mbuf 65536 byte clusters in use (current/peak/max)
700 Kbytes allocated to network (40% in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
Sat Mar 5 08:53:01 CET 2011
293 mbufs in use:
220 mbufs allocated to data
15 mbufs allocated to packet headers
58 mbufs allocated to socket names and addresses
123/180/6144 mbuf 2048 byte clusters in use (current/peak/max)
0/8/6144 mbuf 4096 byte clusters in use (current/peak/max)
0/8/6144 mbuf 8192 byte clusters in use (current/peak/max)
0/8/6144 mbuf 9216 byte clusters in use (current/peak/max)
0/8/6144 mbuf 12288 byte clusters in use (current/peak/max)
0/8/6144 mbuf 16384 byte clusters in use (current/peak/max)
0/8/6144 mbuf 65536 byte clusters in use (current/peak/max)
700 Kbytes allocated to network (45% in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
Sat Mar 5 09:53:01 CET 2011
316 mbufs in use:
243 mbufs allocated to data
15 mbufs allocated to packet headers
58 mbufs allocated to socket names and addresses
135/180/6144 mbuf 2048 byte clusters in use (current/peak/max)
0/8/6144 mbuf 4096 byte clusters in use (current/peak/max)
0/8/6144 mbuf 8192 byte clusters in use (current/peak/max)
0/8/6144 mbuf 9216 byte clusters in use (current/peak/max)
0/8/6144 mbuf 12288 byte clusters in use (current/peak/max)
0/8/6144 mbuf 16384 byte clusters in use (current/peak/max)
0/8/6144 mbuf 65536 byte clusters in use (current/peak/max)
700 Kbytes allocated to network (49% in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
Sat Mar 5 10:53:01 CET 2011
340 mbufs in use:
265 mbufs allocated to data
17 mbufs allocated to packet headers
58 mbufs allocated to socket names and addresses
146/180/6144 mbuf 2048 byte clusters in use (current/peak/max)
0/8/6144 mbuf 4096 byte clusters in use (current/peak/max)
0/8/6144 mbuf 8192 byte clusters in use (current/peak/max)
0/8/6144 mbuf 9216 byte clusters in use (current/peak/max)
0/8/6144 mbuf 12288 byte clusters in use (current/peak/max)
0/8/6144 mbuf 16384 byte clusters in use (current/peak/max)
0/8/6144 mbuf 65536 byte clusters in use (current/peak/max)
700 Kbytes allocated to network (53% in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
Sat Mar 5 11:53:01 CET 2011
355 mbufs in use:
281 mbufs allocated to data
16 mbufs allocated to packet headers
58 mbufs allocated to socket names and addresses
154/180/6144 mbuf 2048 byte clusters in use (current/peak/max)
0/8/6144 mbuf 4096 byte clusters in use (current/peak/max)
0/8/6144 mbuf 8192 byte clusters in use (current/peak/max)
0/8/6144 mbuf 9216 byte clusters in use (current/peak/max)
0/8/6144 mbuf 12288 byte clusters in use (current/peak/max)
0/8/6144 mbuf 16384 byte clusters in use (current/peak/max)
0/8/6144 mbuf 65536 byte clusters in use (current/peak/max)
700 Kbytes allocated to network (56% in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
[..skip..]
Thu Mar 10 21:53:01 CET 2011
5376 mbufs in use:
5303 mbufs allocated to data
15 mbufs allocated to packet headers
58 mbufs allocated to socket names and addresses
2679/2702/6144 mbuf 2048 byte clusters in use (current/peak/max)
0/8/6144 mbuf 4096 byte clusters in use (current/peak/max)
0/8/6144 mbuf 8192 byte clusters in use (current/peak/max)
0/8/6144 mbuf 9216 byte clusters in use (current/peak/max)
0/8/6144 mbuf 12288 byte clusters in use (current/peak/max)
0/8/6144 mbuf 16384 byte clusters in use (current/peak/max)
0/8/6144 mbuf 65536 byte clusters in use (current/peak/max)
6952 Kbytes allocated to network (96% in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
Thu Mar 10 22:53:01 CET 2011
5388 mbufs in use:
5315 mbufs allocated to data
15 mbufs allocated to packet headers
58 mbufs allocated to socket names and addresses
2688/2710/6144 mbuf 2048 byte clusters in use (current/peak/max)
0/8/6144 mbuf 4096 byte clusters in use (current/peak/max)
0/8/6144 mbuf 8192 byte clusters in use (current/peak/max)
0/8/6144 mbuf 9216 byte clusters in use (current/peak/max)
0/8/6144 mbuf 12288 byte clusters in use (current/peak/max)
0/8/6144 mbuf 16384 byte clusters in use (current/peak/max)
0/8/6144 mbuf 65536 byte clusters in use (current/peak/max)
6972 Kbytes allocated to network (96% in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
Thu Mar 10 23:53:01 CET 2011
5410 mbufs in use:
5334 mbufs allocated to data
18 mbufs allocated to packet headers
58 mbufs allocated to socket names and addresses
2696/2728/6144 mbuf 2048 byte clusters in use (current/peak/max)
0/8/6144 mbuf 4096 byte clusters in use (current/peak/max)
0/8/6144 mbuf 8192 byte clusters in use (current/peak/max)
0/8/6144 mbuf 9216 byte clusters in use (current/peak/max)
0/8/6144 mbuf 12288 byte clusters in use (current/peak/max)
0/8/6144 mbuf 16384 byte clusters in use (current/peak/max)
0/8/6144 mbuf 65536 byte clusters in use (current/peak/max)
7016 Kbytes allocated to network (96% in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
