Re: FreeBSD 6.0 panic: kmem_malloc(16384): kmem_map too small: 172728320 total allocated [solved]
Kris Kennaway [EMAIL PROTECTED] wrote: On Wed, Dec 14, 2005 at 05:32:34PM +0100, Fabian Keil wrote: I guess you're right. I can fill a 256MB swap-backed disk without panic and without swapping. FYI, this is documented in the manpage. I think the panic potential should be mentioned in md(4) as well. I used a script not written by me, the commands used were working and after the panic I only read man md. Of course mdconfig(8) is mentioned twice, but I didn't think I needed more information about it. Fabian -- http://www.fabiankeil.de/ signature.asc Description: PGP signature
FreeBSD 6.0 panic: kmem_malloc(16384): kmem_map too small: 172728320 total allocated
I triggered a few reproducible panics on FreeBSD 6.0-STABLE.
I created a ramdisk with:
/sbin/mdconfig -a -t malloc -s 256M -u 10
/sbin/newfs -U /dev/md10
/sbin/mount /dev/md10 /mnt/ramdisk
The system has avail memory = 515932160 (492 MB)
and 1GB swap space.
While copying to /mnt/ramdisk trough ftp localhost
it got:
[EMAIL PROTECTED] ~/crashdump #kgdb kernel-GENERIC.debug vmcore.3
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so:
Undefined symbol ps_pglobal_lookup]
GNU gdb 6.1.1 [FreeBSD]
[...]
Unread portion of the kernel message buffer:
panic: kmem_malloc(16384): kmem_map too small: 172728320 total allocated
Uptime: 2m57s
Dumping 511 MB (2 chunks)
chunk 0: 1MB (158 pages) ... ok
chunk 1: 511MB (130800 pages) 495 479 463 447 431 415 399 383 367 351 335 319
303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15
#0 doadump () at pcpu.h:165
165 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) where
#0 doadump () at pcpu.h:165
#1 0xc063a4ee in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:399
#2 0xc063a784 in panic (fmt=0xc0880846 kmem_malloc(%ld): kmem_map too small:
%ld total allocated)
at /usr/src/sys/kern/kern_shutdown.c:555
#3 0xc07a44bd in kmem_malloc (map=0xc10430c0, size=16384, flags=1026) at
/usr/src/sys/vm/vm_kern.c:299
#4 0xc079c0c6 in page_alloc (zone=0x0, bytes=16384, pflag=0x0, wait=1026) at
/usr/src/sys/vm/uma_core.c:958
#5 0xc079e41f in uma_large_malloc (size=16384, wait=1026) at
/usr/src/sys/vm/uma_core.c:2702
#6 0xc0630085 in malloc (size=16384, mtp=0xc08ffe40, flags=1026) at
/usr/src/sys/kern/kern_malloc.c:329
#7 0xc078365e in softdep_disk_io_initiation (bp=0xcd899658) at
/usr/src/sys/ufs/ffs/ffs_softdep.c:3630
#8 0xc078b1fe in ffs_geom_strategy (bo=0xc3593e90, bp=0xcd899658) at buf.h:422
#9 0xc0796869 in ufs_strategy (ap=0x0) at /usr/src/sys/ufs/ufs/ufs_vnops.c:1926
#10 0xc081c645 in VOP_STRATEGY_APV (vop=0xc09012a0, a=0xdd93ec0c) at
vnode_if.c:1796
#11 0xc06841d0 in bufstrategy (bo=0xc35f7720, bp=0x0) at vnode_if.h:928
#12 0xc067eda8 in bufwrite (bp=0xcd899658) at buf.h:415
#13 0xc067f397 in bawrite (bp=0x0) at buf.h:399
#14 0xc078b53d in ffs_syncvnode (vp=0xc35f7660, waitfor=1) at
/usr/src/sys/ufs/ffs/ffs_vnops.c:256
#15 0xc078b28e in ffs_fsync (ap=0xdd93ecc0) at
/usr/src/sys/ufs/ffs/ffs_vnops.c:179
#16 0xc081c05c in VOP_FSYNC_APV (vop=0x0, a=0x0) at vnode_if.c:1020
#17 0xc0698278 in fsync (td=0xc3460d80, uap=0x0) at vnode_if.h:537
#18 0xc080b6eb in syscall (frame=
{tf_fs = 59, tf_es = 59, tf_ds = 59, tf_edi = 64, tf_esi = 134572032,
tf_ebp = -1077940680, tf_isp = -5775079
96, tf_ebx = 134561920, tf_edx = 1, tf_ecx = 6, tf_eax = 95, tf_trapno = 0,
tf_err = 2, tf_eip = 672366947, tf_cs =
51, tf_eflags = 662, tf_esp = -1077945572, tf_ss = 59}) at
/usr/src/sys/i386/i386/trap.c:981
#19 0xc07fa57f in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:200
#20 0x0033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
By simply copying to /mnt/ramdisk with cp I got:
[EMAIL PROTECTED] ~/crashdump #kgdb kernel-GENERIC.debug vmcore.4
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so:
Undefined symbol ps_pglobal_lookup]
GNU gdb 6.1.1 [FreeBSD]
[...]
Unread portion of the kernel message buffer:
g_vfs_done():md10[WRITE(offset=206372864, length=131072)]error = 28
g_vfs_done():md10[WRITE(offset=206503936, length=131072)]error = 28
g_vfs_done():md10[WRITE(offset=206635008, length=131072)]error = 28
g_vfs_done():md10[WRITE(offset=206766080, length=131072)]error = 28
g_vfs_done():md10[WRITE(offset=206897152, length=131072)]error = 28
g_vfs_done():md10[WRITE(offset=207028224, length=131072)]error = 28
g_vfs_done():md10[WRITE(offset=207159296, length=131072)]error = 28
g_vfs_done():md10[WRITE(offset=207290368, length=131072)]error = 28
g_vfs_done():md10[WRITE(offset=207421440, length=131072)]error = 28
g_vfs_done():md10[WRITE(offset=207552512, length=131072)]error = 28
g_vfs_done():md10[WRITE(offset=207683584, length=131072)]error = 28
g_vfs_done():md10[WRITE(offset=207814656, length=131072)]error = 28
g_vfs_done():md10[WRITE(offset=207945728, length=131072)]error = 28
g_vfs_done():md10[WRITE(offset=208076800, length=131072)]error = 28
g_vfs_done():md10[WRITE(offset=208207872, length=131072)]error = 28
g_vfs_done():md10[WRITE(offset=208338944, length=131072)]error = 28
panic: kmem_malloc(4096): kmem_map too small: 172728320 total allocated
Uptime: 11m23s
Dumping 511 MB (2 chunks)
chunk 0: 1MB (158 pages) ... ok
chunk 1: 511MB (130800 pages) 495 479 463 447 431 415 399 383 367 351 335 319
303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15
#0 doadump () at pcpu.h:165
165 pcpu.h: No such file or directory.
in pcpu.h
#0 doadump () at pcpu.h:165
#1 0xc063a4ee in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:399
#2 0xc063a784 in panic (fmt=0xc0880846 kmem_malloc(%ld): kmem_map too
Re: FreeBSD 6.0 panic: kmem_malloc(16384): kmem_map too small: 172728320 total allocated
On Wed, Dec 14, 2005 at 01:25:30PM +0100, Fabian Keil wrote: F I triggered a few reproducible panics on FreeBSD 6.0-STABLE. F F I created a ramdisk with: F F /sbin/mdconfig -a -t malloc -s 256M -u 10 F /sbin/newfs -U /dev/md10 F /sbin/mount /dev/md10 /mnt/ramdisk F F The system has avail memory = 515932160 (492 MB) F and 1GB swap space. F F While copying to /mnt/ramdisk trough ftp localhost F it got: This usually exposes some memory leak in kernel. Can you please do the following - copy some amount of data to /mnt/ramdisk trough ftp localhost, and cancel the operation before it panics. Then run vmstat -m and vmstat -z, to determine what kind of memory allocation is leaking. -- Totus tuus, Glebius. GLEBIUS-RIPN GLEB-RIPE ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 6.0 panic: kmem_malloc(16384): kmem_map too small: 172728320 total allocated
Gleb Smirnoff [EMAIL PROTECTED] wrote: On Wed, Dec 14, 2005 at 01:25:30PM +0100, Fabian Keil wrote: F I triggered a few reproducible panics on FreeBSD 6.0-STABLE. F F I created a ramdisk with: F F /sbin/mdconfig -a -t malloc -s 256M -u 10 F /sbin/newfs -U /dev/md10 F /sbin/mount /dev/md10 /mnt/ramdisk F F The system has avail memory = 515932160 (492 MB) F and 1GB swap space. F F While copying to /mnt/ramdisk trough ftp localhost F it got: This usually exposes some memory leak in kernel. Can you please do the following - copy some amount of data to /mnt/ramdisk trough ftp localhost, and cancel the operation before it panics. Then run vmstat -m and vmstat -z, to determine what kind of memory allocation is leaking. I had loops with vmstat -m and vmstat -z in the background while copying to /mnt/ramdisk. The last output before the panic was: Type InUse MemUse HighUse Requests Size(s) DEVFS22 1K - 23 16,128 pfs_nodes20 3K - 20 128 GEOM 18926K - 858 16,32,64,128,256,512,1024,2048,4096 isadev17 2K - 17 64 ATA DMA 4 1K -4 128 cdev27 4K - 27 128 AR driver 0 0K - 11 512,2048 ACD driver 3 6K -3 2048 file desc 12046K - 1611 16,32,256,512,2048 sigio 2 1K -3 32 kenv96 7K - 97 16,32,64,4096 kqueue 0 0K - 62 256,1024 proc-args43 2K - 797 16,32,64,128 zombie 0 0K - 907 128 ithread48 5K - 49 64,128 KTRACE 10013K - 100 128 CAM SIM 1 1K -1 64 linker68 3K - 99 16,32,256 CAM XPT10 1K - 17 16,64,512 lockf 3 1K -3 64 devbuf 1346 3177K - 1816 16,32,64,128,256,512,1024,2048,4096 temp16 171K - 6266 16,32,64,128,256,512,1024,2048,4096 ip6opt 1 1K - 1 128 ip6ndp 6 1K -7 64,128 module 37124K - 371 64,128 mtx_pool 1 8K -1 pgrp36 3K - 623 64 session29 4K - 47 128 proc 2 4K -2 2048 subproc 209 413K - 1116 256,4096 cred35 5K - 4132 128 plimit18 5K - 400 256 uidinfo 4 1K - 20 32,512 sysctl 0 0K - 619 16,32,64 sysctloid 256777K - 2567 16,32,64 sysctltmp 0 0K - 280 16,32,128 umtx 120 8K - 120 64 SWAP 2 141K -2 64 bus 95938K - 3599 16,32,64,128,1024 bus-sc5727K - 1537 16,32,64,128,256,512,1024,2048,4096 devstat1837K - 18 16,4096 eventhandler37 3K - 37 32,128 kobj 248 496K - 299 2048 MD disk 294 7K - 294 16,2048 MD sectors 293 1172K - 293 4096 rman 14910K - 570 16,64 sbuf 0 0K - 440 16,32,64,128,256,512,1024,2048,4096 sleep queues 121 4K - 121 32 taskqueue 6 1K -6 128 turnstiles 121 8K - 121 64 Unitno 7 1K -9 16,64 ioctlops 0 0K - 2757 16,32,64,256,512,1024,4096 iov 0 0K - 487 16,64,128 msg 425K -4 1024,4096 sem 4 7K -4 512,1024,4096 shm 112K -1 ttys 1228 174K - 3223 128,1024 ptys 3 1K -3 128 mbuf_tag 0 0K -6 32,64 soname 6 1K - 735 16,32,128 pcb29 5K - 81 16,32,64,2048 BIO buffer 0 0K - 99 2048 vfscache 1 256K -1 cluster_save buffer 0 0K - 19 32,64 Export Host 1 1K -2 256 VFS hash 1 128K -1 vnodes 1 1K -1 128 mount 13012K - 641 16,32,64,128,512,1024,2048 CAM periph 1 1K -1 128 BPF 4 1K -4 64 ifnet 5 5K -5 256,1024 ifaddr4010K - 40 16,32,64,256,512,2048 ether_multi40 2K - 46 16,32,64 clone 416K -4 4096 arpcom 2 1K -
Re: FreeBSD 6.0 panic: kmem_malloc(16384): kmem_map too small: 172728320 total allocated
Gleb Smirnoff wrote: On Wed, Dec 14, 2005 at 01:25:30PM +0100, Fabian Keil wrote: F I triggered a few reproducible panics on FreeBSD 6.0-STABLE. F F I created a ramdisk with: F F /sbin/mdconfig -a -t malloc -s 256M -u 10 F /sbin/newfs -U /dev/md10 F /sbin/mount /dev/md10 /mnt/ramdisk F F The system has avail memory = 515932160 (492 MB) F and 1GB swap space. F F While copying to /mnt/ramdisk trough ftp localhost F it got: This usually exposes some memory leak in kernel. Can you please do the following - copy some amount of data to /mnt/ramdisk trough ftp localhost, and cancel the operation before it panics. Then run vmstat -m and vmstat -z, to determine what kind of memory allocation is leaking. While it can mean a memory leak in the kernel, I don't think that's the case here. On i386, only 320MB can be allocated to kernel malloc memory. Much of this space can get consumed with vnodes and other filesystem structures, so trying to allocate 256MB to a ramdisk is likely putting you over the max. I'd suggest instead to use a swap-back disk. It doesn't necessarily mean that the ramdisk pages will live in swap, it just means that they will get managed directly in the bufcache, eliminating the 320MB restriction. Scott ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 6.0 panic: kmem_malloc(16384): kmem_map too small: 172728320 total allocated [solved]
Scott Long [EMAIL PROTECTED] wrote: Gleb Smirnoff wrote: On Wed, Dec 14, 2005 at 01:25:30PM +0100, Fabian Keil wrote: F I triggered a few reproducible panics on FreeBSD 6.0-STABLE. F F I created a ramdisk with: F F /sbin/mdconfig -a -t malloc -s 256M -u 10 F /sbin/newfs -U /dev/md10 F /sbin/mount /dev/md10 /mnt/ramdisk F F The system has avail memory = 515932160 (492 MB) F and 1GB swap space. F F While copying to /mnt/ramdisk trough ftp localhost F it got: This usually exposes some memory leak in kernel. Can you please do the following - copy some amount of data to /mnt/ramdisk trough ftp localhost, and cancel the operation before it panics. Then run vmstat -m and vmstat -z, to determine what kind of memory allocation is leaking. While it can mean a memory leak in the kernel, I don't think that's the case here. On i386, only 320MB can be allocated to kernel malloc memory. Much of this space can get consumed with vnodes and other filesystem structures, so trying to allocate 256MB to a ramdisk is likely putting you over the max. I'd suggest instead to use a swap-back disk. It doesn't necessarily mean that the ramdisk pages will live in swap, it just means that they will get managed directly in the bufcache, eliminating the 320MB restriction. I guess you're right. I can fill a 256MB swap-backed disk without panic and without swapping. Before ftp localhost: last pid: 652; load averages: 0.02, 0.09, 0.07up 0+00:07:16 17:12:05 37 processes: 1 running, 36 sleeping CPU states: 0.0% user, 0.0% nice, 0.0% system, 0.4% interrupt, 99.6% idle Mem: 11M Active, 12M Inact, 18M Wired, 11M Buf, 453M Free Swap: 999M Total, 999M Free After ftp localhost: last pid: 666; load averages: 0.20, 0.12, 0.08up 0+00:09:05 17:13:54 36 processes: 1 running, 35 sleeping CPU states: 0.0% user, 0.0% nice, 0.0% system, 0.4% interrupt, 99.6% idle Mem: 244M Active, 150M Inact, 73M Wired, 27M Cache, 60M Buf, 984K Free Swap: 999M Total, 999M Free After removal of the swap-backed disk: last pid: 690; load averages: 0.00, 0.01, 0.03up 0+00:17:53 17:22:42 34 processes: 1 running, 33 sleeping CPU states: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle Mem: 15M Active, 76M Inact, 43M Wired, 13M Cache, 60M Buf, 347M Free Swap: 999M Total, 999M Free Thanks for your time Gleb and Scott. Fabian -- http://www.fabiankeil.de/ signature.asc Description: PGP signature
Re: FreeBSD 6.0 panic: kmem_malloc(16384): kmem_map too small: 172728320 total allocated [solved]
On Wed, Dec 14, 2005 at 05:32:34PM +0100, Fabian Keil wrote: I guess you're right. I can fill a 256MB swap-backed disk without panic and without swapping. FYI, this is documented in the manpage. Kris pgpJ4IsKT7ayY.pgp Description: PGP signature
