Re: Memory problems on 4.5

[Wade, Daniel]

 -Original Message-
 From: owner-m...@openbsd.org [mailto:owner-m...@openbsd.org] On
 Behalf Of Lars Kotthoff
 Sent: Saturday, June 06, 2009 9:17 AM
 To: misc@openbsd.org
 Subject: Memory problems on 4.5

 Hi all,

  after upgrading to 4.5, I've had problems with the memory usage of
 the box
 increasing continuously until if starts to swap and performance
 becomes so bad
 that I have to reboot the box. This does not seem to be related to
 a specific
 program, at least the usual tools don't indicate that one process
 uses a lot of
 memory or that its memory usage increases steadily.

 Here's a picture of what I'm talking about --
 http://www.larsko.net/mem.jpg
 As you can see, memory usage increases quite significantly. The Wed
 to Sat part
 is what it's like now, in the part before there was some other
 stuff going on as
 well. This didn't occur with 4.4.

 Does anybody have any suggestions what could cause this and how to
 fix it? Dmesg
 below.




I get similar problems on my computer after a few weeks of uptime.  Where it's
swapping continually and I can't even log in on the console.  I only have 32MB
of RAM so I think the effects are more pronounced for me.  I also have some
output from ddb if that would be helpful.  I'll post a dmesg later but I'm
running a snapshot from early May I think, and have been seeing this problem
for a few months.

Re: Memory problems on 4.5

[Lars Kotthoff]

 please send vmstat -m and netstat -m at a point when the memory
 use is medium or high.

See below. Total memory use is about 170MB.

 if you're using symon to collect your stats, it might be worth
 recording and plotting the mbuf stats it can save.

That doesn't work anymore with 4.5 -- I got loads of symon: mbuf() failed (flag
!= 3) in the system log, so I disabled it.

 what's the box doing?

Nothing particularly exciting, routing some traffic and doing DNS for the local
network. The actual activity of the box doesn't seem to have any influence on
the memory usage though.

Thanks,

Lars

netstat -m
9405 mbufs in use:
9393 mbufs allocated to data
3 mbufs allocated to packet headers
9 mbufs allocated to socket names and addresses
128/202/6144 mbuf 2048 byte clusters in use (current/peak/max)
0/8/6144 mbuf 4096 byte clusters in use (current/peak/max)
0/8/6144 mbuf 8192 byte clusters in use (current/peak/max)
0/8/6144 mbuf 9216 byte clusters in use (current/peak/max)
0/8/6144 mbuf 12288 byte clusters in use (current/peak/max)
0/8/6144 mbuf 16384 byte clusters in use (current/peak/max)
0/8/6144 mbuf 65536 byte clusters in use (current/peak/max)
52108 Kbytes allocated to network (99% in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines

vmstat -m
Memory statistics by bucket size
Size   In Use   Free   Requests  HighWater  Couldfree
  16 2669   168330948411280   6403
  32  607161 592073 640  0
  64 1217   1279 264720 320  16018
 128  312200   8545 160 73
 256  257143 244144  80  11467
 512  273 79  38071  40582
1024  323 41 220518  20  80293
2048  144  4287  10  0
4096   27  4   1440   5  0
81925  1   1329   5  0
   163849  0 12   5  0
   327681  0  2   5  0
   655362  0  2   5  0
  5242888  0  8   5  0

Memory usage type by bucket size
Size  Type(s)
  16  devbuf, pcb, routetbl, sysctl, UFS mount, sem, dirhash, in_multi,
  exec, miscfs mount, xform_data, VM swap, UVM amap, UVM aobj, USB,
  packet tags, temp
  32  devbuf, pcb, routetbl, ifaddr, sem, dirhash, proc, VFS cluster,
  NFS daemon, in_multi, ether_multi, exec, xform_data, VM swap,
  UVM amap, USB, temp
  64  devbuf, pcb, routetbl, ifaddr, vnodes, sem, dirhash, in_multi,
  pfkey data, UVM amap, UVM aobj, USB, NDP, temp
 128  devbuf, routetbl, ifaddr, sysctl, vnodes, dirhash, NFS srvsock, ttys,
  UVM amap, USB, USB device, ip6_options, NDP, temp
 256  devbuf, routetbl, ifaddr, sysctl, ioctlops, NFS mount, vnodes,
  UFS mount, shm, VM map, sem, dirhash, file desc, Export Host,
  NFS daemon, exec, newblk, UVM amap, temp
 512  devbuf, pcb, ifaddr, ioctlops, UFS mount, shm, dirhash, file desc,
  proc, ttys, exec, UVM amap, USB device, temp
1024  devbuf, ioctlops, mount, UFS mount, dirhash, file desc, ttys, exec,
  UVM amap, UVM aobj, crypto data, temp
2048  devbuf, ifaddr, ioctlops, namecache, UFS mount, shm, dirhash,
  file desc, proc, VM swap, UVM amap, temp
4096  devbuf, ioctlops, UFS mount, file desc, pagedep, UVM amap, temp
8192  devbuf, namecache, MSDOSFS mount, temp
   16384  devbuf, NFS node, namecache, UFS quota, UFS mount, ISOFS mount,
  inodedep, temp
   32768  devbuf, UVM amap
   65536  VM swap
  524288  devbuf

Memory statistics by type   Type  Kern
  Type InUse MemUse HighUse  Limit Requests Limit Limit Size(s)
devbuf   702 16836K  16836K 37990K  8990 0  
16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,524288
   pcb48 5K  6K 37990K 34790 0  16,32,64,512
  routetbl   100 7K  8K 37990K   2157960 0  16,32,64,128,256
ifaddr   20020K 20K 37990K  2020 0  
32,64,128,256,512,2048
sysctl 2 1K  1K 37990K40 0  16,128,256
  ioctlops 0 0K  4K 37990K 30450 0  
256,512,1024,2048,4096
 mount 6 6K  6K 37990K60 0  1024
 NFS mount 1 1K  1K 37990K10 0  256
  NFS node 116K 16K 37990K10 0  16384
vnodes   40930K101K 37990K 45550 0  64,128,256
 namecache 326K 26K 37990K30 0  2048,8192,16384
 UFS quota 116K 16K 37990K10 0  16384
 UFS mount1759K 59K 

Re: Memory problems on 4.5

[Stuart Henderson]

On 2009-06-07, Lars Kotthoff li...@larsko.org wrote:
 please send vmstat -m and netstat -m at a point when the memory
 use is medium or high.

 See below. Total memory use is about 170MB.

 if you're using symon to collect your stats, it might be worth
 recording and plotting the mbuf stats it can save.

 That doesn't work anymore with 4.5 -- I got loads of symon: mbuf() failed 
 (flag
 != 3) in the system log, so I disabled it.

oh right, that was fixed after 4.5; you can build from the -current port
on 4.5, that will fix it.

 netstat -m
 9405 mbufs in use:
   9393 mbufs allocated to data

that is way too high for what the system is doing.

any way to try it with the ral(4) wireless card disabled to see
if that makes any difference? (I have a feeling the wired RTL8139
rl(4) is more likely to have the problem but I guess those are
going to be harder to disable..)

Re: Memory problems on 4.5

[Lars Kotthoff]

 that is way too high for what the system is doing.
 
 any way to try it with the ral(4) wireless card disabled to see
 if that makes any difference? (I have a feeling the wired RTL8139
 rl(4) is more likely to have the problem but I guess those are
 going to be harder to disable..)

Oh no, not the darned wireless again. I'll try removing the wireless card
altogether as soon as I get back from holiday (just a quick update before
leaving...) and report back.

Lars

Memory problems on 4.5

[Lars Kotthoff]

Hi all,

 after upgrading to 4.5, I've had problems with the memory usage of the box
increasing continuously until if starts to swap and performance becomes so bad
that I have to reboot the box. This does not seem to be related to a specific
program, at least the usual tools don't indicate that one process uses a lot of
memory or that its memory usage increases steadily.

Here's a picture of what I'm talking about -- http://www.larsko.net/mem.jpg
As you can see, memory usage increases quite significantly. The Wed to Sat part
is what it's like now, in the part before there was some other stuff going on as
well. This didn't occur with 4.4.

Does anybody have any suggestions what could cause this and how to fix it? Dmesg
below.

Thanks,

Lars


OpenBSD 4.5 (GENERIC) #1749: Sat Feb 28 14:51:18 MST 2009
dera...@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Geode(TM) Integrated Processor by National Semi (Geode by NSC 
586-class) 334 MHz
cpu0: FPU,DE,PSE,TSC,MSR,CX8,PGE,CMOV,MMX
real mem  = 259350528 (247MB)
avail mem = 242470912 (231MB)
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 09/01/06, BIOS32 rev. 0 @ 0xfadd0, SMBIOS 
rev. 2.2 @ 0xf (28 entries)
bios0: vendor Phoenix Technologies, LTD version 6.00 PG date 09/01/2006
apm0 at bios0: Power Management spec V1.2 (slowidle)
apm0: AC on, battery charge unknown
acpi at bios0 function 0x0 not configured
pcibios0 at bios0: rev 2.1 @ 0xf/0xdfb4
pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xfdf20/144 (7 entries)
pcibios0: bad IRQ table checksum
pcibios0: PCI BIOS has 7 Interrupt Routing table entries
pcibios0: PCI Exclusive IRQs: 5 7 10 11
pcibios0: no compatible PCI ICU found
pcibios0: Warning, unable to fix up PCI interrupt routing
pcibios0: PCI bus #0 is the last bus
bios0: ROM list: 0xc/0x8000 0xef000/0x1000!
cpu0 at mainbus0: (uniprocessor)
pci0 at mainbus0 bus 0: configuration mode 1 (bios)
pchb0 at pci0 dev 1 function 0 NS CS5535 Host rev 0x21
vga1 at pci0 dev 1 function 1 NS CS5535 VIDEO rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
pcib0 at pci0 dev 15 function 0 NS CS5535 ISA rev 0x13
pciide0 at pci0 dev 15 function 2 NS CS5535 IDE rev 0x00: DMA, channel 0 
wired to compatibility
wd0 at pciide0 channel 0 drive 0: SAMSUNG HM160HC
wd0: 16-sector PIO, LBA48, 152627MB, 312581808 sectors
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 4
ohci0 at pci0 dev 15 function 4 NS CS5535 USB rev 0x06: irq 10, version 1.0, 
legacy support
ohci1 at pci0 dev 15 function 5 NS CS5535 USB rev 0x06: irq 7, version 1.0, 
legacy support
rl0 at pci0 dev 18 function 0 Realtek 8139 rev 0x10: irq 11, address 
00:00:00:00:00:00
rlphy0 at rl0 phy 0: RTL internal PHY
rl1 at pci0 dev 19 function 0 Realtek 8139 rev 0x10: irq 5, address 
00:00:00:00:00:00
rlphy1 at rl1 phy 0: RTL internal PHY
rl2 at pci0 dev 20 function 0 Realtek 8139 rev 0x10: irq 10, address 
00:00:00:00:00:00
rlphy2 at rl2 phy 0: RTL internal PHY
ral0 at pci0 dev 21 function 0 Ralink RT2860 rev 0x00: irq 11, address 
00:00:00:00:00:00
ral0: MAC/BBP RT2860 (rev 0x0101), RF RT2820 (MIMO 2T3R)
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: PC speaker
spkr0 at pcppi0
it0 at isa0 port 0x2e/2: IT8712F rev 7, EC port 0x290
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
usb0 at ohci0: USB revision 1.0
uhub0 at usb0 NS OHCI root hub rev 1.00/1.00 addr 1
usb1 at ohci1: USB revision 1.0
uhub1 at usb1 NS OHCI root hub rev 1.00/1.00 addr 1
biomask f7cd netmask ffed ttymask 
softraid0 at root
root on wd0a swap on wd0b dump on wd0b

Re: Memory problems on 4.5

[Stuart Henderson]

On 2009-06-06, Lars Kotthoff li...@larsko.org wrote:
 Hi all,

  after upgrading to 4.5, I've had problems with the memory usage of the box
 increasing continuously until if starts to swap and performance becomes so bad
 that I have to reboot the box. This does not seem to be related to a specific
 program, at least the usual tools don't indicate that one process uses a lot 
 of
 memory or that its memory usage increases steadily.

 Here's a picture of what I'm talking about -- http://www.larsko.net/mem.jpg
 As you can see, memory usage increases quite significantly. The Wed to Sat 
 part
 is what it's like now, in the part before there was some other stuff going on 
 as
 well. This didn't occur with 4.4.

please send vmstat -m and netstat -m at a point when the memory
use is medium or high.

if you're using symon to collect your stats, it might be worth
recording and plotting the mbuf stats it can save.

what's the box doing?


 Does anybody have any suggestions what could cause this and how to fix it? 
 Dmesg
 below.

 Thanks,

 Lars


 OpenBSD 4.5 (GENERIC) #1749: Sat Feb 28 14:51:18 MST 2009
 dera...@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC
 cpu0: Geode(TM) Integrated Processor by National Semi (Geode by NSC 
 586-class) 334 MHz
 cpu0: FPU,DE,PSE,TSC,MSR,CX8,PGE,CMOV,MMX
 real mem  = 259350528 (247MB)
 avail mem = 242470912 (231MB)
 mainbus0 at root
 bios0 at mainbus0: AT/286+ BIOS, date 09/01/06, BIOS32 rev. 0 @ 0xfadd0, 
 SMBIOS rev. 2.2 @ 0xf (28 entries)
 bios0: vendor Phoenix Technologies, LTD version 6.00 PG date 09/01/2006
 apm0 at bios0: Power Management spec V1.2 (slowidle)
 apm0: AC on, battery charge unknown
 acpi at bios0 function 0x0 not configured
 pcibios0 at bios0: rev 2.1 @ 0xf/0xdfb4
 pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xfdf20/144 (7 entries)
 pcibios0: bad IRQ table checksum
 pcibios0: PCI BIOS has 7 Interrupt Routing table entries
 pcibios0: PCI Exclusive IRQs: 5 7 10 11
 pcibios0: no compatible PCI ICU found
 pcibios0: Warning, unable to fix up PCI interrupt routing
 pcibios0: PCI bus #0 is the last bus
 bios0: ROM list: 0xc/0x8000 0xef000/0x1000!
 cpu0 at mainbus0: (uniprocessor)
 pci0 at mainbus0 bus 0: configuration mode 1 (bios)
 pchb0 at pci0 dev 1 function 0 NS CS5535 Host rev 0x21
 vga1 at pci0 dev 1 function 1 NS CS5535 VIDEO rev 0x00
 wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
 wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
 pcib0 at pci0 dev 15 function 0 NS CS5535 ISA rev 0x13
 pciide0 at pci0 dev 15 function 2 NS CS5535 IDE rev 0x00: DMA, channel 0 
 wired to compatibility
 wd0 at pciide0 channel 0 drive 0: SAMSUNG HM160HC
 wd0: 16-sector PIO, LBA48, 152627MB, 312581808 sectors
 wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 4
 ohci0 at pci0 dev 15 function 4 NS CS5535 USB rev 0x06: irq 10, version 
 1.0, legacy support
 ohci1 at pci0 dev 15 function 5 NS CS5535 USB rev 0x06: irq 7, version 1.0, 
 legacy support
 rl0 at pci0 dev 18 function 0 Realtek 8139 rev 0x10: irq 11, address 
 00:00:00:00:00:00
 rlphy0 at rl0 phy 0: RTL internal PHY
 rl1 at pci0 dev 19 function 0 Realtek 8139 rev 0x10: irq 5, address 
 00:00:00:00:00:00
 rlphy1 at rl1 phy 0: RTL internal PHY
 rl2 at pci0 dev 20 function 0 Realtek 8139 rev 0x10: irq 10, address 
 00:00:00:00:00:00
 rlphy2 at rl2 phy 0: RTL internal PHY
 ral0 at pci0 dev 21 function 0 Ralink RT2860 rev 0x00: irq 11, address 
 00:00:00:00:00:00
 ral0: MAC/BBP RT2860 (rev 0x0101), RF RT2820 (MIMO 2T3R)
 isa0 at pcib0
 isadma0 at isa0
 com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
 pckbc0 at isa0 port 0x60/5
 pckbd0 at pckbc0 (kbd slot)
 pckbc0: using irq 1 for kbd slot
 wskbd0 at pckbd0: console keyboard, using wsdisplay0
 pcppi0 at isa0 port 0x61
 midi0 at pcppi0: PC speaker
 spkr0 at pcppi0
 it0 at isa0 port 0x2e/2: IT8712F rev 7, EC port 0x290
 npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
 usb0 at ohci0: USB revision 1.0
 uhub0 at usb0 NS OHCI root hub rev 1.00/1.00 addr 1
 usb1 at ohci1: USB revision 1.0
 uhub1 at usb1 NS OHCI root hub rev 1.00/1.00 addr 1
 biomask f7cd netmask ffed ttymask 
 softraid0 at root
 root on wd0a swap on wd0b dump on wd0b