RE: 6.1-RC2: strange kernel panic!
Don't you think you should test it instead of guessing? :-) I suggested it because it *is* a possibility (that is why I have it in my kernel). yes, but doesn't it make sense to find memory consuming things before adding more? btw. how can we check for such things? Are you sure you are using swap backing and not malloc? nope, i'm not sure if it was that, but -M was passed to mdmfs, so malloc(9) was used. we changed the code to swap-based, let's see if that fixes our problem. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: 6.1-RC2: strange kernel panic!
On Wed, May 03, 2006 at 11:55:26AM +0200, [EMAIL PROTECTED]@mgEDV.net wrote: Don't you think you should test it instead of guessing? :-) I suggested it because it *is* a possibility (that is why I have it in my kernel). yes, but doesn't it make sense to find memory consuming things before adding more? btw. how can we check for such things? Are you sure you are using swap backing and not malloc? nope, i'm not sure if it was that, but -M was passed to mdmfs, so malloc(9) was used. we changed the code to swap-based, let's see if that fixes our problem. Since you were using malloc backing that is what used up all your kernel memory. Increasing memory per my suggestion would have fixed it. Chances are you don't want to use malloc backing anyway though, because it's slower (unless you're swapping). Kris pgplZzr0kvIPo.pgp Description: PGP signature
RE: 6.1-RC2: strange kernel panic! [SOLVED]
original error: panic: kmem_malloc(4096): kmem_map too small: 335544320 total allocated kernel panic issue seems to be solved by changing our memory disk from malloc(9) backed to a swap-backed disk. thx 4 helpin', guys! and yes, i could have tested it because this was already in the archives (shame on me) ;-) ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: 6.1-RC2: strange kernel panic! [SOLVED]
On Wed, May 03, 2006 at 09:09:41PM +0200, [EMAIL PROTECTED]@mgEDV.net wrote: original error: panic: kmem_malloc(4096): kmem_map too small: 335544320 total allocated kernel panic issue seems to be solved by changing our memory disk from malloc(9) backed to a swap-backed disk. thx 4 helpin', guys! and yes, i could have tested it because this was already in the archives (shame on me) ;-) Great, it's nice when problems are resolved :) Kris pgppqRHtO1XQn.pgp Description: PGP signature
6.1-RC2: strange kernel panic!
hi together! this is the 4th time the server died since last week (and the 1st time we catched the error!). it happened during an du -sk . of some large directory structure. panic: kmem_malloc(4096): kmem_map too small: 335544320 total allocated any ideas on this? this system should go live soon, so we definitely need to fix this! the dmesg/bootlog of the system: /boot/kernel/kernel text=0x4a8ee8 data=0x97ae0+0x4eb0c syms=[0x4+0x63aa0+0x4+0x7b0d7] /boot/kernel/umass.ko text=0x5034 data=0x488+0x4 syms=[0x4+0x7f0+0x4+0x811] /boot/kernel/if_myk.ko text=0x2c738 data=0x13c8+0x4 syms=[0x4+0x1d00+0x4+0x198] loading required module 'libmbpool' /boot/kernel/libmbpool.ko text=0xd2c data=0x10c syms=[0x4+0x390+0x4+0x2f4] /boot/kernel/geom_eli.ko text=0x8bf4 data=0x3a8+0x10 syms=[0x4+0xc50+0x4+0xd31] loading required module 'crypto' /boot/kernel/crypto.ko text=0x16df8 data=0x6d8+0x1ac syms=[0x4+0x13e0+0x4+0x12d8] loading required module 'zlib' /boot/kernel/zlib.ko text=0x70bc data=0x410+0x19d0 syms=[0x4+0x7e0+0x4+0x67a] / Booting [/boot/kernel/kernel]... /boot/kernel/acpi.ko text=0x42ed4 data=0x2280+0x10f0 syms=[0x4+0x7ad0+0x4+0xa709] Copyright (c) 1992-2006 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 6.1-RC2 #0: Sun Apr 30 09:42:38 UTC 2006 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/GENERIC module_register: module uhub/umass already exists! Module uhub/umass failed to register: 17 Timecounter i8254 frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2793.01-MHz 686-class CPU) Origin = GenuineIntel Id = 0xf43 Stepping = 3 Features=0xbfebfbffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA ,C MOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE Features2=0x641dSSE3,RSVD2,MON,DS_CPL,CNTX-ID,CX16,b14 AMD Features=0x2010NX,LM Logical CPUs per core: 2 real memory = 2147274752 (2047 MB) avail memory = 2096517120 (1999 MB) ACPI APIC Table: A M I OEMAPIC ioapic0 Version 2.0 irqs 0-23 on motherboard ioapic1 Version 2.0 irqs 24-47 on motherboard kbd1 at kbdmux0 acpi0: A M I OEMRSDT on motherboard acpi0: Power Button (fixed) Timecounter ACPI-fast frequency 3579545 Hz quality 1000 acpi_timer0: 24-bit timer at 3.579545MHz port 0x808-0x80b on acpi0 cpu0: ACPI CPU on acpi0 acpi_throttle0: ACPI CPU Throttling on cpu0 pcib0: ACPI Host-PCI bridge port 0xcf8-0xcff on acpi0 pci0: ACPI PCI bus on pcib0 pci0: unknown at device 0.1 (no driver attached) pcib1: ACPI PCI-PCI bridge irq 16 at device 2.0 on pci0 pci1: ACPI PCI bus on pcib1 pcib2: ACPI PCI-PCI bridge irq 16 at device 3.0 on pci0 pci2: ACPI PCI bus on pcib2 myk0: Marvell 88E8050 Gigabit Ethernet Controller, Ver. 8.12.2.3 port 0xdf00-0 xdfff mem 0xdedfc000-0xdedf irq 16 at device 0.0 on pci2 myk0: Ethernet address: 00:0e:0c:6a:a6:40 pcib3: ACPI PCI-PCI bridge at device 28.0 on pci0 pci3: ACPI PCI bus on pcib3 amr0: LSILogic MegaRAID 1.53 mem 0xddff-0xddff irq 24 at device 1.0 on pci3 amr0: LSILogic Intel(R) RAID Controller SRCS16 Firmware 713Q, BIOS G401, 64MB RAM uhci0: UHCI (generic) USB controller port 0xcf00-0xcf1f irq 16 at device 29.0 on pci0 uhci0: [GIANT-LOCKED] usb0: UHCI (generic) USB controller on uhci0 usb0: USB revision 1.0 uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 2 ports with 2 removable, self powered uhci1: UHCI (generic) USB controller port 0xcf80-0xcf9f irq 19 at device 29.1 on pci0 uhci1: [GIANT-LOCKED] usb1: UHCI (generic) USB controller on uhci1 usb1: USB revision 1.0 uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub1: 2 ports with 2 removable, self powered pci0: base peripheral at device 29.4 (no driver attached) pci0: base peripheral, interrupt controller at device 29.5 (no driver attached) ehci0: Intel 6300ESB USB 2.0 controller mem 0xdecffc00-0xdecf irq 23 at device 29.7 on pci0 ehci0: [GIANT-LOCKED] usb2: EHCI version 1.0 usb2: companion controllers, 2 ports each: usb0 usb1 usb2: Intel 6300ESB USB 2.0 controller on ehci0 usb2: USB revision 2.0 uhub2: Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1 uhub2: 4 ports with 4 removable, self powered umass0: Kingston DataTraveler 2.0, rev 2.00/1.00, addr 2 pcib4: ACPI PCI-PCI bridge at device 30.0 on pci0 pci4: ACPI PCI bus on pcib4 pci4: display, VGA at device 2.0 (no driver attached) em0: Intel(R) PRO/1000 Network Connection Version - 3.2.18 port 0xee80-0xeebf mem 0xdefa-0xdefb irq 17 at device 3.0 on pci4 em0: Ethernet address: 00:0e:0c:6a:a6:41 isab0: PCI-ISA bridge at device 31.0 on pci0 isa0: ISA bus on isab0 atapci0: Intel 6300ESB UDMA100 controller port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfc00-0xfc0f at device 31.1 on pci0 ata0: ATA channel 0 on atapci0 ata1: ATA channel 1 on atapci0 pci0: serial bus, SMBus at device 31.3 (no driver attached) acpi_button0: Power Button on acpi0 atkbdc0: Keyboard controller (i8042) port 0x60,0x64
Re: 6.1-RC2: strange kernel panic!
On Tue, May 02, 2006 at 07:34:58PM +0200, [EMAIL PROTECTED]@mgEDV.net wrote: hi together! this is the 4th time the server died since last week (and the 1st time we catched the error!). it happened during an du -sk . of some large directory structure. panic: kmem_malloc(4096): kmem_map too small: 335544320 total allocated any ideas on this? this system should go live soon, so we definitely need to fix this! Your kernel ran out of memory. Either you are using a workload that is too heavy for your current settings, or there is a memory leak somewhere in a kernel subsystem you are using. Try to increase VM_KMEM_SIZE_MAX in your kernel, e.g. options VM_KMEM_SIZE_MAX=524288000 #500MB You may need to increase it further. Kris pgpJ0NLcfNKa2.pgp Description: PGP signature
RE: 6.1-RC2: strange kernel panic!
Your kernel ran out of memory. Either you are using a workload that is too heavy for your current settings, or there is a memory leak somewhere in a kernel subsystem you are using. Try to increase VM_KMEM_SIZE_MAX in your kernel, e.g. options VM_KMEM_SIZE_MAX=524288000 #500MB You may need to increase it further. i'm not sure, but probably this does not solve our problem. this system is used as a compilation host only (currently) and therefore there are no permanently running things like databases, huge daemons, etc... only ssh and syslog is up in userland. so the main question to me is, where the memory goes on this server, and how i can prevent this type of leak. (and even maybe help you fixin' it ;-) our current settings are (default in GENERIC): vm.kmem_size: 335544320 vm.kmem_size_max: 335544320 the compilation system uses a 350MB swap-based memory-disk for compilation, the whole disks are encrypted using GELI (AES256). network traffic is low (only ssh commandline stuff, no huge transfers). when i issued the du -sk the panic occurred. 5min ago, the system panic'd again, this time some more was logged: (originally, there have been 200 of these messages, numbers change, error=same) g_vfs_done():md0[WRITE(offset=346742784, length=6144)]error = 28 g_vfs_done():md0[WRITE(offset=346750976, length=8192)]error = 28 g_vfs_done():md0[WRITE(offset=346761216, length=6144)]error = 28 g_vfs_done():md0[WRITE(offset=346767360, length=6144)]error = 28 g_vfs_done():md0[WRITE(offset=346773504, length=6144)]error = 28 this time the panic occurred while transferring data from the hdd's to the md-device: panic: kmem_malloc(4096): kmem_map too small: 335544320 total allocated Uptime: 1h13m18s is there any way (which is suitable for a non-c-guru like me) how i can at least monitor, which statements cause the memory leaks? givin' it more memory could only raise the uptime, because at this time there are no permanently running processes except the os and ssh. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: 6.1-RC2: strange kernel panic!
On Tue, May 02, 2006 at 08:59:38PM +0200, [EMAIL PROTECTED]@mgEDV.net wrote: Your kernel ran out of memory. Either you are using a workload that is too heavy for your current settings, or there is a memory leak somewhere in a kernel subsystem you are using. Try to increase VM_KMEM_SIZE_MAX in your kernel, e.g. options VM_KMEM_SIZE_MAX=524288000 #500MB You may need to increase it further. i'm not sure, but probably this does not solve our problem. Don't you think you should test it instead of guessing? :-) I suggested it because it *is* a possibility (that is why I have it in my kernel). this system is used as a compilation host only (currently) and therefore there are no permanently running things like databases, huge daemons, etc... only ssh and syslog is up in userland. so the main question to me is, where the memory goes on this server, and how i can prevent this type of leak. (and even maybe help you fixin' it ;-) our current settings are (default in GENERIC): vm.kmem_size: 335544320 vm.kmem_size_max: 335544320 the compilation system uses a 350MB swap-based memory-disk for compilation, the whole disks are encrypted using GELI (AES256). network traffic is low (only ssh commandline stuff, no huge transfers). when i issued the du -sk the panic occurred. Could be to do with GELI, I don't know about memory requirements. 5min ago, the system panic'd again, this time some more was logged: (originally, there have been 200 of these messages, numbers change, error=same) g_vfs_done():md0[WRITE(offset=346742784, length=6144)]error = 28 g_vfs_done():md0[WRITE(offset=346750976, length=8192)]error = 28 g_vfs_done():md0[WRITE(offset=346761216, length=6144)]error = 28 g_vfs_done():md0[WRITE(offset=346767360, length=6144)]error = 28 g_vfs_done():md0[WRITE(offset=346773504, length=6144)]error = 28 This is suspicious: #define ENOSPC 28 /* No space left on device */ Are you sure you are using swap backing and not malloc? Kris pgplfoAV2n5lE.pgp Description: PGP signature
