Re: 6.1-RC2: strange kernel panic! [SOLVED]
On Wed, May 03, 2006 at 09:09:41PM +0200, [EMAIL PROTECTED]@mgEDV.net wrote: > > original error: > > panic: kmem_malloc(4096): kmem_map too small: 335544320 total allocated > kernel panic issue seems to be solved by changing our memory disk > from malloc(9) backed to a swap-backed disk. > > thx 4 helpin', guys! > > and yes, i could have tested it because this was already in the archives > (shame on me) ;-) Great, it's nice when problems are resolved :) Kris pgppqRHtO1XQn.pgp Description: PGP signature
RE: 6.1-RC2: strange kernel panic! [SOLVED]
original error: > panic: kmem_malloc(4096): kmem_map too small: 335544320 total allocated kernel panic issue seems to be solved by changing our memory disk from malloc(9) backed to a swap-backed disk. thx 4 helpin', guys! and yes, i could have tested it because this was already in the archives (shame on me) ;-) ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: 6.1-RC2: strange kernel panic!
On Wed, May 03, 2006 at 11:55:26AM +0200, [EMAIL PROTECTED]@mgEDV.net wrote: > > > Don't you think you should test it instead of guessing? :-) I suggested > > it because it *is* a possibility (that is why I have it in my kernel). > yes, but doesn't it make sense to find memory consuming things > before adding more? > btw. how can we check for such things? > > > Are you sure you are using swap backing and not malloc? > nope, i'm not sure if it was that, but -M was passed to mdmfs, > so malloc(9) was used. we changed the code to swap-based, let's > see if that fixes our problem. Since you were using malloc backing that is what used up all your kernel memory. Increasing memory per my suggestion would have fixed it. Chances are you don't want to use malloc backing anyway though, because it's slower (unless you're swapping). Kris pgplZzr0kvIPo.pgp Description: PGP signature
RE: 6.1-RC2: strange kernel panic!
> Don't you think you should test it instead of guessing? :-) I suggested > it because it *is* a possibility (that is why I have it in my kernel). yes, but doesn't it make sense to find memory consuming things before adding more? btw. how can we check for such things? > Are you sure you are using swap backing and not malloc? nope, i'm not sure if it was that, but -M was passed to mdmfs, so malloc(9) was used. we changed the code to swap-based, let's see if that fixes our problem. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: 6.1-RC2: strange kernel panic!
On Tue, May 02, 2006 at 08:59:38PM +0200, [EMAIL PROTECTED]@mgEDV.net wrote: > > > Your kernel ran out of memory. Either you are using a workload that > > is too heavy for your current settings, or there is a memory leak > > somewhere in a kernel subsystem you are using. > > Try to increase VM_KMEM_SIZE_MAX in your kernel, e.g. > > options VM_KMEM_SIZE_MAX=524288000 #500MB > > You may need to increase it further. > > i'm not sure, but probably this does not solve our problem. Don't you think you should test it instead of guessing? :-) I suggested it because it *is* a possibility (that is why I have it in my kernel). > this system > is used as a compilation host only (currently) and therefore there are > no permanently running things like databases, huge daemons, etc... only > ssh and syslog is up in userland. so the main question to me is, where > the memory goes on this server, and how i can prevent this type of leak. > (and even maybe help you fixin' it ;-) > > our current settings are (default in GENERIC): > vm.kmem_size: 335544320 > vm.kmem_size_max: 335544320 > > the compilation system uses a 350MB swap-based memory-disk for compilation, > the whole disks are encrypted using GELI (AES256). network traffic is low > (only ssh commandline stuff, no huge transfers). > > when i issued the "du -sk" the panic occurred. Could be to do with GELI, I don't know about memory requirements. > 5min ago, the system panic'd again, this time some more was logged: > (originally, there have been >200 of these messages, numbers change, > error=same) > g_vfs_done():md0[WRITE(offset=346742784, length=6144)]error = 28 > g_vfs_done():md0[WRITE(offset=346750976, length=8192)]error = 28 > g_vfs_done():md0[WRITE(offset=346761216, length=6144)]error = 28 > g_vfs_done():md0[WRITE(offset=346767360, length=6144)]error = 28 > g_vfs_done():md0[WRITE(offset=346773504, length=6144)]error = 28 This is suspicious: #define ENOSPC 28 /* No space left on device */ Are you sure you are using swap backing and not malloc? Kris pgplfoAV2n5lE.pgp Description: PGP signature
RE: 6.1-RC2: strange kernel panic!
> Your kernel ran out of memory. Either you are using a workload that > is too heavy for your current settings, or there is a memory leak > somewhere in a kernel subsystem you are using. > Try to increase VM_KMEM_SIZE_MAX in your kernel, e.g. > options VM_KMEM_SIZE_MAX=524288000 #500MB > You may need to increase it further. i'm not sure, but probably this does not solve our problem. this system is used as a compilation host only (currently) and therefore there are no permanently running things like databases, huge daemons, etc... only ssh and syslog is up in userland. so the main question to me is, where the memory goes on this server, and how i can prevent this type of leak. (and even maybe help you fixin' it ;-) our current settings are (default in GENERIC): vm.kmem_size: 335544320 vm.kmem_size_max: 335544320 the compilation system uses a 350MB swap-based memory-disk for compilation, the whole disks are encrypted using GELI (AES256). network traffic is low (only ssh commandline stuff, no huge transfers). when i issued the "du -sk" the panic occurred. 5min ago, the system panic'd again, this time some more was logged: (originally, there have been >200 of these messages, numbers change, error=same) g_vfs_done():md0[WRITE(offset=346742784, length=6144)]error = 28 g_vfs_done():md0[WRITE(offset=346750976, length=8192)]error = 28 g_vfs_done():md0[WRITE(offset=346761216, length=6144)]error = 28 g_vfs_done():md0[WRITE(offset=346767360, length=6144)]error = 28 g_vfs_done():md0[WRITE(offset=346773504, length=6144)]error = 28 this time the panic occurred while transferring data from the hdd's to the md-device: panic: kmem_malloc(4096): kmem_map too small: 335544320 total allocated Uptime: 1h13m18s is there any way (which is suitable for a non-c-guru like me) how i can at least monitor, which statements cause the memory leaks? givin' it more memory could only raise the uptime, because at this time there are no permanently running processes except the os and ssh. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: 6.1-RC2: strange kernel panic!
On Tue, May 02, 2006 at 07:34:58PM +0200, [EMAIL PROTECTED]@mgEDV.net wrote: > > hi together! > > this is the 4th time the server died since last week (and the 1st time we > catched the error!). > it happened during an "du -sk ." of some large directory structure. > > panic: kmem_malloc(4096): kmem_map too small: 335544320 total allocated > > any ideas on this? this system should go live soon, so we definitely need to > fix this! Your kernel ran out of memory. Either you are using a workload that is too heavy for your current settings, or there is a memory leak somewhere in a kernel subsystem you are using. Try to increase VM_KMEM_SIZE_MAX in your kernel, e.g. options VM_KMEM_SIZE_MAX=524288000 #500MB You may need to increase it further. Kris pgpJ0NLcfNKa2.pgp Description: PGP signature
6.1-RC2: strange kernel panic!
hi together! this is the 4th time the server died since last week (and the 1st time we catched the error!). it happened during an "du -sk ." of some large directory structure. panic: kmem_malloc(4096): kmem_map too small: 335544320 total allocated any ideas on this? this system should go live soon, so we definitely need to fix this! the dmesg/bootlog of the system: /boot/kernel/kernel text=0x4a8ee8 data=0x97ae0+0x4eb0c syms=[0x4+0x63aa0+0x4+0x7b0d7] /boot/kernel/umass.ko text=0x5034 data=0x488+0x4 syms=[0x4+0x7f0+0x4+0x811] /boot/kernel/if_myk.ko text=0x2c738 data=0x13c8+0x4 syms=[0x4+0x1d00+0x4+0x198] loading required module 'libmbpool' /boot/kernel/libmbpool.ko text=0xd2c data=0x10c syms=[0x4+0x390+0x4+0x2f4] /boot/kernel/geom_eli.ko text=0x8bf4 data=0x3a8+0x10 syms=[0x4+0xc50+0x4+0xd31] loading required module 'crypto' /boot/kernel/crypto.ko text=0x16df8 data=0x6d8+0x1ac syms=[0x4+0x13e0+0x4+0x12d8] loading required module 'zlib' /boot/kernel/zlib.ko text=0x70bc data=0x410+0x19d0 syms=[0x4+0x7e0+0x4+0x67a] / Booting [/boot/kernel/kernel]... /boot/kernel/acpi.ko text=0x42ed4 data=0x2280+0x10f0 syms=[0x4+0x7ad0+0x4+0xa709] Copyright (c) 1992-2006 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 6.1-RC2 #0: Sun Apr 30 09:42:38 UTC 2006 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/GENERIC module_register: module uhub/umass already exists! Module uhub/umass failed to register: 17 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2793.01-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf43 Stepping = 3 Features=0xbfebfbff Features2=0x641d> AMD Features=0x2010 Logical CPUs per core: 2 real memory = 2147274752 (2047 MB) avail memory = 2096517120 (1999 MB) ACPI APIC Table: ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 cpu0: on acpi0 acpi_throttle0: on cpu0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pci0: at device 0.1 (no driver attached) pcib1: irq 16 at device 2.0 on pci0 pci1: on pcib1 pcib2: irq 16 at device 3.0 on pci0 pci2: on pcib2 myk0: port 0xdf00-0 xdfff mem 0xdedfc000-0xdedf irq 16 at device 0.0 on pci2 myk0: Ethernet address: 00:0e:0c:6a:a6:40 pcib3: at device 28.0 on pci0 pci3: on pcib3 amr0: mem 0xddff-0xddff irq 24 at device 1.0 on pci3 amr0: Firmware 713Q, BIOS G401, 64MB RAM uhci0: port 0xcf00-0xcf1f irq 16 at device 29.0 on pci0 uhci0: [GIANT-LOCKED] usb0: on uhci0 usb0: USB revision 1.0 uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 2 ports with 2 removable, self powered uhci1: port 0xcf80-0xcf9f irq 19 at device 29.1 on pci0 uhci1: [GIANT-LOCKED] usb1: on uhci1 usb1: USB revision 1.0 uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub1: 2 ports with 2 removable, self powered pci0: at device 29.4 (no driver attached) pci0: at device 29.5 (no driver attached) ehci0: mem 0xdecffc00-0xdecf irq 23 at device 29.7 on pci0 ehci0: [GIANT-LOCKED] usb2: EHCI version 1.0 usb2: companion controllers, 2 ports each: usb0 usb1 usb2: on ehci0 usb2: USB revision 2.0 uhub2: Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1 uhub2: 4 ports with 4 removable, self powered umass0: Kingston DataTraveler 2.0, rev 2.00/1.00, addr 2 pcib4: at device 30.0 on pci0 pci4: on pcib4 pci4: at device 2.0 (no driver attached) em0: port 0xee80-0xeebf mem 0xdefa-0xdefb irq 17 at device 3.0 on pci4 em0: Ethernet address: 00:0e:0c:6a:a6:41 isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfc00-0xfc0f at device 31.1 on pci0 ata0: on atapci0 ata1: on atapci0 pci0: at device 31.3 (no driver attached) acpi_button0: on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A, console sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0 sio1: type 16550A fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FAST] pmtimer0 on isa0 orm0: at iomem 0xc-0xc7fff on isa0 ppc0: parallel port not found. sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x100> vga0: at port 0x3c0-0x3df iomem 0xa-0xb on isa0 Timecounter "TSC" frequency 2793013356 Hz quality 800 Timecounters tick every 1.000 msec acd0: CDROM at ata1-slave UDMA33 amrd0: on amr0 amrd0: 381469MB (781248512 sectors) RAID 1 (optimal) amrd1: on amr0 amrd1: 381469MB (78124
