Jonathan Adams wrote:
On Mon, Aug 24, 2009 at 08:16:42PM +1000, Gavin Maltby wrote:
Hi
Ian Collins wrote:
I'm looking for pointers towards kernel memory debug code.
I had a system that periodically ran out of physical memory and froze.
It turns out the system had kmem_flags of 0xf and
kmem_bufctl_audit_cache usage was getting to about 1.3GB. I'm
interested is seeing how this value is used and how it might cause the
physical memory exhaustion problem.
There's a chapter on kmem debugging in the mdb (modular debugger) guide
on docs.sun.com
The basic summary, memory-usage wise, is:
When debugging is enabled, we allocate a 216-byte chunk of memory
*per buffer* in the system. This holds the allocation stack trace.
Every buffer in the system gets a "buffer tag" after the end, which
we use to detect buffer overruns and get fast access to the auditing
chunk, above.
Two large circular logs using up a total of 1/25th of the memory
in the system are allocated, and are used to hold recent activity
and freed buffer contents.
Together, this usually approximately doubles the kernel memory footprint.
The trade-off is that a huge class of bugs are now much easier to find and
diagnose.
Thanks Jonathan.
What puzzles me is this system ran out of physical memory and hung
without any evidence of paging. In use swap remained pretty steady
right up until the the last sample. My suspicion is that it thought it
had more free memory than it actually did.
The summary of last ::kmastat sample I logged was:
------------------------- ------ ------ ------ --------- --------- -----
Total [hat_memload] 5517312 4014783 0
Total [kmem_msb] 1604435968 179293909 0
Total [kmem_firewall] 103231488 73146293 0
Total [kmem_va] 3325034496 1076037 0
Total [kmem_default] 3605368832 1629234898 0
Total [kmem_io_64G] 138686464 17805 0
Total [kmem_io_4G] 81920 19788246 0
Total [kmem_io_2G] 8192 4 0
Total [bp_map] 0 1156 0
Total [umem_np] 0 1035 0
Total [id32] 4096 25283 0
Total [zio_buf] 2379313152 3675123 0
Total [segkp] 524288 6392 0
Total [ip_minor_arena_sa] 64 558 0
Total [ip_minor_arena_la] 192 1793 0
Total [spdsock] 0 1 0
Total [namefs_inodes] 64 36 0
------------------------- ------ ------ ------ --------- --------- -----
vmem memory memory memory alloc alloc
name in use total import succeed fail
------------------------- --------- ---------- --------- --------- -----
heap 12359630848 1627851325440 0
73448968 0
vmem_metadata 1543135232 1543241728 1543241728 92948 0
vmem_seg 1520795648 1520795648 1520795648 92822 0
vmem_hash 22065152 22069248 22069248 70 0
vmem_vmem 276640 306736 270336 82 0
static 0 0 0 0 0
static_alloc 0 0 0 0 0
hat_memload 5517312 5517312 5517312 1469 0
kstat 519184 528384 462848 1537 0
kmem_metadata 6020997120 6021054464 6021054464 1441051 0
kmem_msb 5899403264 5899403264 5899403264 1440304 0
kmem_cache 990176 1589248 1589248 440 0
kmem_hash 119980544 120004608 120004608 911 0
kmem_log 44050624 44056576 44056576 12 0
kmem_firewall_va 208740352 208740352 208740352 73148475 0
kmem_firewall 103223296 103223296 103223296 73148833 0
kmem_oversize 105306706 105517056 105517056 480 0
mod_sysfile 255 4096 4096 7 0
kmem_va 3911598080 3911598080 3911598080 84568 0
kmem_default 3605368832 3605368832 3605368832 1110786 0
kmem_io_64G 138686464 138686464 138686464 17251 0
kmem_io_4G 81920 81920 81920 23 0
kmem_io_2G 40960 40960 40960 23957 0
kmem_io_16M 0 0 0 0 0
bp_map 0 0 0 1731 0
ksyms 2308478 2588672 2588672 295 0
ctf 3543658 3850240 3850240 293 0
umem_np 131072 131072 131072 1070 0
heap_core 7237632 794820608 0 59 0
heaptext 12816384 67108864 0 168 0
module_text 13642064 13832192 12816384 294 0
id32 4096 4096 4096 1 0
module_data 3401311 7540736 7233536 414 0
logminor_space 55 262137 0 106 0
taskq_id_arena 187 2147483647 0 236 0
zio 2379313152 17054040064 0 60480 0
zio_buf 2379313152 2379313152 2379313152 60480 0
device 3010560 1073741824 0 330 0
segkp 27164672 2147483648 0 3819 0
rctl_ids 36 32767 0 36 0
zoneid_space 1 9998 0 1 0
taskid_space 64 999999 0 182 0
pool_ids 0 999998 0 0 0
contracts 71 2147483646 0 231 0
ip_minor_arena_sa 64 262140 0 1 0
ip_minor_arena_la 192 4294705152 0 15 0
dls_minor_arena 4 4294967295 0 9 0
dld_ctl 0 4294967295 0 0 0
tl_minor_space 90 262138 0 668 0
keysock 0 4294967295 0 0 0
spdsock 0 4294967295 0 1 0
namefs_inodes 64 65536 0 1 0
Hex0xffffffffa0ca3638_minor 0 4294967294 0 0 0
Hex0xffffffffa1082588_minor 0 4294967294 0 0 0
ptms_minor 3 16 0 14 0
devfsadm_event_channel 1 101 0 1 0
devfsadm_event_channel 1 2 0 1 0
syseventconfd_event_channel 0 101 0 0 0
syseventconfd_event_channel 1 2 0 1 0
syseventd_channel 3 101 0 3 0
syseventd_channel 1 2 0 1 0
dtrace 20996 4294967295 0 27148 0
dtrace_minor 0 4294967293 0 0 0
lmsysid_space 2 16383 0 11 0
Client_id_space 1 131072 0 10 0
OpenOwner_id_space 1 1048576 0 10 0
OpenStateID_id_space 1 1048576 0 10 0
LockStateID_id_space 0 1048576 0 0 0
Lockowner_id_space 0 1048576 0 0 0
DelegStateID_id_space 1 1048576 0 4 0
shmids 1 64 0 12 0
crypto 0 16 0 34 0
lxa_minor_id 0 1021 0 0 0
logdmux_minor 0 256 0 0 0
sppptun_minor 0 16 0 0 0
semids 1 64 0 9 0
------------------------- --------- ---------- --------- --------- -----
--
Ian.
_______________________________________________
opensolaris-code mailing list
opensolaris-code@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/opensolaris-code
