STABLE kernel panic: privileged instruction fault
Hello. I have a couple of Supermicro servers which got the similar kernel panic with all FreeBSD versions I tried since 6.4. Now I want to investigate into the problem. The servers get into panic with similar workload: file server with a lot of files and connections. Web server software is nginx. File system is UFS+GJOURNAL. Outgoing traffic on each server is ~10 MB/s. I think it is not software problem, because when I've installed Linux with such configuration there were no kernel panics. Here is the short overview of the hardware: CPU: Intel(R) Pentium(R) 4 CPU 3.00GHz (2992.51-MHz K8-class CPU) Origin = GenuineIntel Id = 0xf65 Family = f Model = 6 Stepping = 5 Features=0xbfebfbffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE Features2=0xe59dSSE3,DTES64,MON,DS_CPL,EST,TM2,CNXT-ID,CX16,xTPR,PDCM AMD Features=0x20100800SYSCALL,NX,LM AMD Features2=0x1LAHF TSC: P-state invariant real memory = 2147483648 (2048 MB) avail memory = 2054619136 (1959 MB) DMESG: http://lexasoft.ru/m/dmesg.txt CORE: http://lexasoft.ru/m/core.txt Fatal trap 1: privileged instruction fault while in kernel mode cpuid = 1; apic id = 01 instruction pointer = 0x20:0xff8040d2cc83 stack pointer = 0x28:0xff8040d2ca80 frame pointer = 0x28:0xff0060c0b740 code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags= interrupt enabled, resume, IOPL = 0 current process = 9388 (nginx) trap number = 1 panic: privileged instruction fault cpuid = 1 Uptime: 17d15h48m49s Physical memory: 2032 MB Dumping 1485 MB: 1470 1454 1438 1422 1406 1390 1374 1358 1342 1326 1310 1294 1278 1262 1246 1230 1214 1198 1182 1166 1150 1134 1118 1102 1086 1070 1054 1038 1022 1006 990 974 958 942 926 910 894 878 862 846 830 814 798 782 766 750 734 718 702 686 670 654 638 622 606 590 574 558 542 526 510 494 478 462 446 430 414 398 382 366 350 334 318 302 286 270 254 238 222 206 190 174 158 142 126 110 94 78 62 46 30 14 (kgdb) #0 doadump () at pcpu.h:223 #1 0x80590c59 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:416 #2 0x8059108c in panic (fmt=0x80951fc4 %s) at /usr/src/sys/kern/kern_shutdown.c:579 #3 0x80878fd8 in trap_fatal (frame=0xff0060c0b740, eva=Variable eva is not available. ) at /usr/src/sys/amd64/amd64/trap.c:857 #4 0x808799ea in trap (frame=0xff8040d2c9d0) at /usr/src/sys/amd64/amd64/trap.c:644 #5 0x8085f983 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:224 #6 0xff8040d2cc83 in ?? () #7 0xff8040d2cb50 in ?? () #8 0xff8040d2caf0 in ?? () #9 0xff8040d2cbf0 in ?? () #10 0xff0060c0b740 in ?? () #11 0x80b83c60 in sysent () #12 0xff8040d2cc80 in ?? () #13 0xff8040d2cae0 in ?? () #14 0x8059c431 in bintime (bt=0x80ad3140) at /usr/src/sys/kern/kern_tc.c:200 Previous frame inner to this frame (corrupt stack?) (kgdb) -- Alexey Tarasov (\__/) (='.'=) E[: | | | | :]З ()_() ___ freebsd-curr...@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: STABLE kernel panic: privileged instruction fault
On Mon, Aug 16, 2010 at 07:15:16PM +0400, Alexey Tarasov wrote: Hello. I have a couple of Supermicro servers which got the similar kernel panic with all FreeBSD versions I tried since 6.4. Now I want to investigate into the problem. The servers get into panic with similar workload: file server with a lot of files and connections. Web server software is nginx. File system is UFS+GJOURNAL. Outgoing traffic on each server is ~10 MB/s. I think it is not software problem, because when I've installed Linux with such configuration there were no kernel panics. Here is the short overview of the hardware: CPU: Intel(R) Pentium(R) 4 CPU 3.00GHz (2992.51-MHz K8-class CPU) Origin = GenuineIntel Id = 0xf65 Family = f Model = 6 Stepping = 5 Features=0xbfebfbffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE Features2=0xe59dSSE3,DTES64,MON,DS_CPL,EST,TM2,CNXT-ID,CX16,xTPR,PDCM AMD Features=0x20100800SYSCALL,NX,LM AMD Features2=0x1LAHF TSC: P-state invariant real memory = 2147483648 (2048 MB) avail memory = 2054619136 (1959 MB) DMESG: http://lexasoft.ru/m/dmesg.txt CORE: http://lexasoft.ru/m/core.txt Fatal trap 1: privileged instruction fault while in kernel mode cpuid = 1; apic id = 01 instruction pointer = 0x20:0xff8040d2cc83 stack pointer = 0x28:0xff8040d2ca80 frame pointer = 0x28:0xff0060c0b740 code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags= interrupt enabled, resume, IOPL = 0 current process = 9388 (nginx) trap number = 1 panic: privileged instruction fault cpuid = 1 Uptime: 17d15h48m49s Physical memory: 2032 MB Dumping 1485 MB: 1470 1454 1438 1422 1406 1390 1374 1358 1342 1326 1310 1294 1278 1262 1246 1230 1214 1198 1182 1166 1150 1134 1118 1102 1086 1070 1054 1038 1022 1006 990 974 958 942 926 910 894 878 862 846 830 814 798 782 766 750 734 718 702 686 670 654 638 622 606 590 574 558 542 526 510 494 478 462 446 430 414 398 382 366 350 334 318 302 286 270 254 238 222 206 190 174 158 142 126 110 94 78 62 46 30 14 (kgdb) #0 doadump () at pcpu.h:223 #1 0x80590c59 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:416 #2 0x8059108c in panic (fmt=0x80951fc4 %s) at /usr/src/sys/kern/kern_shutdown.c:579 #3 0x80878fd8 in trap_fatal (frame=0xff0060c0b740, eva=Variable eva is not available. ) at /usr/src/sys/amd64/amd64/trap.c:857 #4 0x808799ea in trap (frame=0xff8040d2c9d0) at /usr/src/sys/amd64/amd64/trap.c:644 #5 0x8085f983 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:224 #6 0xff8040d2cc83 in ?? () #7 0xff8040d2cb50 in ?? () #8 0xff8040d2caf0 in ?? () #9 0xff8040d2cbf0 in ?? () #10 0xff0060c0b740 in ?? () #11 0x80b83c60 in sysent () #12 0xff8040d2cc80 in ?? () #13 0xff8040d2cae0 in ?? () #14 0x8059c431 in bintime (bt=0x80ad3140) at /usr/src/sys/kern/kern_tc.c:200 Previous frame inner to this frame (corrupt stack?) (kgdb) The backtrace make absolutely no sense. I would not trust kgdb anyway. Compile ddb in and do backtrace in console on the panic. Also, disassemble the kernel at the fault address. I am very curious which instruction causes this. This is stock GENERIC on the bare metal booted, right ? pgp0KYiAf9rFf.pgp Description: PGP signature
Re: STABLE kernel panic: privileged instruction fault
Hello Kostik! On Aug 16, 2010, at 10:48 PM, Kostik Belousov wrote: The backtrace make absolutely no sense. I would not trust kgdb anyway. Compile ddb in and do backtrace in console on the panic. Also, disassemble the kernel at the fault address. I am very curious which instruction causes this. This is stock GENERIC on the bare metal booted, right ? Yes, stock GENERIC. Please, check this out: Dump of assembler code from 0xff0060c0b700 to 0xff0060c0b780: 0xff0060c0b700: add%al,(%rax) 0xff0060c0b702: add%al,(%rax) 0xff0060c0b704: add%al,(%rax) 0xff0060c0b706: add%al,(%rax) 0xff0060c0b708: add%al,(%rax) 0xff0060c0b70a: add%al,(%rax) 0xff0060c0b70c: add%al,(%rax) 0xff0060c0b70e: add%al,(%rax) 0xff0060c0b710: or %dh,0xffc2(%rax) 0xff0060c0b713: cmp$0xff,%bh 0xff0060c0b716: (bad) 0xff0060c0b717: incl (%rax) 0xff0060c0b719: add%al,(%rcx) 0xff0060c0b71b: add%cl,%bh 0xff0060c0b71d: pop%rsp 0xff0060c0b71e: out%al,(%dx) 0xff0060c0b71f: pop%rdx 0xff0060c0b720: or $0x0,%al 0xff0060c0b722: add%al,(%rax) 0xff0060c0b724: or %al,%fs:(%rax) 0xff0060c0b727: add%ah,%bl 0xff0060c0b729: int3 0xff0060c0b72a: (bad) 0xff0060c0b72b: add%cl,%bh 0xff0060c0b72d: pop%rsp 0xff0060c0b72e: out%al,(%dx) 0xff0060c0b72f: pop%rdx 0xff0060c0b730: iret 0xff0060c0b731: pop%rsp 0xff0060c0b732: out%al,(%dx) 0xff0060c0b733: pop%rdx 0xff0060c0b734: rex xor$0x9f105aee,%eax 0xff0060c0b73a: add%al,(%rax) 0xff0060c0b73c: add%al,(%rax) 0xff0060c0b73e: add%al,(%rax) 0xff0060c0b740: rex pop%rdi 0xff0060c0b742: retq $0xff80 0xff0060c0b745: (bad) 0xff0060c0b746: (bad) 0xff0060c0b747: incl (%rax) 0xff0060c0b749: push %rax 0xff0060c0b74a: loop 0xff0060c0b78e 0xff0060c0b74c: add%bh,%bh 0xff0060c0b74e: (bad) 0xff0060c0b74f: incl (%rax) 0xff0060c0b751: add%al,(%rax) 0xff0060c0b753: add%al,(%rax) 0xff0060c0b755: add%al,(%rax) 0xff0060c0b757: add%dl,(%rax) 0xff0060c0b759: push %rax 0xff0060c0b75a: loop 0xff0060c0b79e 0xff0060c0b75c: add%bh,%bh 0xff0060c0b75e: (bad) 0xff0060c0b75f: incl (%rax) 0xff0060c0b761: add%al,(%rax) 0xff0060c0b763: add%al,(%rax) 0xff0060c0b765: add%al,(%rax) 0xff0060c0b767: add%bl,%al 0xff0060c0b769: pop%rdi 0xff0060c0b76a: retq $0xff80 0xff0060c0b76d: (bad) 0xff0060c0b76e: (bad) 0xff0060c0b76f: incl (%rax) 0xff0060c0b771: add%al,(%rax) 0xff0060c0b773: add%al,(%rax) 0xff0060c0b775: add%al,(%rax) 0xff0060c0b777: add%al,0x290c55(%rax) 0xff0060c0b77d: (bad) 0xff0060c0b77e: (bad) 0xff0060c0b77f: incl (%rax) -- Alexey Tarasov (\__/) (='.'=) E[: | | | | :]З ()_() ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: STABLE kernel panic: privileged instruction fault
On Mon, Aug 16, 2010 at 11:21:15PM +0400, Alexey Tarasov wrote: Hello Kostik! On Aug 16, 2010, at 10:48 PM, Kostik Belousov wrote: The backtrace make absolutely no sense. I would not trust kgdb anyway. Compile ddb in and do backtrace in console on the panic. Also, disassemble the kernel at the fault address. I am very curious which instruction causes this. This is stock GENERIC on the bare metal booted, right ? Yes, stock GENERIC. Please, check this out: Dump of assembler code from 0xff0060c0b700 to 0xff0060c0b780: Would be nice if you keep all requested data in one place, so that we do not need to search for the old mails to see the context. According to your previous mail, the fault happen at the address instruction pointer = 0x20:0xff8040d2cc83 Your disassembled the stack instead. Please just do disass 0xff8040d2cc83,0xff8040d2cca0 in kgdb. But also, I want to see the backtrace and disassembly output from ddb. pgp2XZZMDRqkp.pgp Description: PGP signature
Re: STABLE kernel panic: privileged instruction fault
On Aug 16, 2010, at 11:31 PM, Kostik Belousov wrote: On Mon, Aug 16, 2010 at 11:21:15PM +0400, Alexey Tarasov wrote: Hello Kostik! On Aug 16, 2010, at 10:48 PM, Kostik Belousov wrote: The backtrace make absolutely no sense. I would not trust kgdb anyway. Compile ddb in and do backtrace in console on the panic. Also, disassemble the kernel at the fault address. I am very curious which instruction causes this. This is stock GENERIC on the bare metal booted, right ? Yes, stock GENERIC. Please, check this out: Dump of assembler code from 0xff0060c0b700 to 0xff0060c0b780: Would be nice if you keep all requested data in one place, so that we do not need to search for the old mails to see the context. According to your previous mail, the fault happen at the address instruction pointer = 0x20:0xff8040d2cc83 Your disassembled the stack instead. Please just do disass 0xff8040d2cc83,0xff8040d2cca0 in kgdb. But also, I want to see the backtrace and disassembly output from ddb. (kgdb) disass 0xff8040d2cc83,0xff8040d2cca0 No function contains specified address. I will build kernel with DDB tomorrow, install it on some servers and wait for the panic occurs. -- Alexey Tarasov (\__/) (='.'=) E[: | | | | :]З ()_() ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: STABLE kernel panic: privileged instruction fault
On Mon, Aug 16, 2010 at 11:35:36PM +0400, Alexey Tarasov wrote: On Aug 16, 2010, at 11:31 PM, Kostik Belousov wrote: On Mon, Aug 16, 2010 at 11:21:15PM +0400, Alexey Tarasov wrote: Hello Kostik! On Aug 16, 2010, at 10:48 PM, Kostik Belousov wrote: The backtrace make absolutely no sense. I would not trust kgdb anyway. Compile ddb in and do backtrace in console on the panic. Also, disassemble the kernel at the fault address. I am very curious which instruction causes this. This is stock GENERIC on the bare metal booted, right ? Yes, stock GENERIC. Please, check this out: Dump of assembler code from 0xff0060c0b700 to 0xff0060c0b780: Would be nice if you keep all requested data in one place, so that we do not need to search for the old mails to see the context. According to your previous mail, the fault happen at the address instruction pointer = 0x20:0xff8040d2cc83 Your disassembled the stack instead. Please just do disass 0xff8040d2cc83,0xff8040d2cca0 in kgdb. But also, I want to see the backtrace and disassembly output from ddb. (kgdb) disass 0xff8040d2cc83,0xff8040d2cca0 No function contains specified address. Err, it seems that old gdb accepts only spaces. Please try disass 0xff8040d2cc83 0xff8040d2cca0 instead. I will build kernel with DDB tomorrow, install it on some servers and wait for the panic occurs. Ok. Did you checked for such things as rootkits ? pgpTly6pt0t7A.pgp Description: PGP signature
Re: STABLE kernel panic: privileged instruction fault
On Aug 16, 2010, at 11:39 PM, Kostik Belousov wrote: On Mon, Aug 16, 2010 at 11:35:36PM +0400, Alexey Tarasov wrote: On Aug 16, 2010, at 11:31 PM, Kostik Belousov wrote: On Mon, Aug 16, 2010 at 11:21:15PM +0400, Alexey Tarasov wrote: Hello Kostik! On Aug 16, 2010, at 10:48 PM, Kostik Belousov wrote: The backtrace make absolutely no sense. I would not trust kgdb anyway. Compile ddb in and do backtrace in console on the panic. Also, disassemble the kernel at the fault address. I am very curious which instruction causes this. This is stock GENERIC on the bare metal booted, right ? Yes, stock GENERIC. Please, check this out: Dump of assembler code from 0xff0060c0b700 to 0xff0060c0b780: Would be nice if you keep all requested data in one place, so that we do not need to search for the old mails to see the context. According to your previous mail, the fault happen at the address instruction pointer = 0x20:0xff8040d2cc83 Your disassembled the stack instead. Please just do disass 0xff8040d2cc83,0xff8040d2cca0 in kgdb. But also, I want to see the backtrace and disassembly output from ddb. (kgdb) disass 0xff8040d2cc83,0xff8040d2cca0 No function contains specified address. Err, it seems that old gdb accepts only spaces. Please try disass 0xff8040d2cc83 0xff8040d2cca0 instead. (kgdb) disass 0xff8040d2cc83 0xff8040d2cca0 Dump of assembler code from 0xff8040d2cc83 to 0xff8040d2cca0: 0xff8040d2cc83: (bad) 0xff8040d2cc84: (bad) 0xff8040d2cc85: jg 0xff8040d2cc87 0xff8040d2cc87: add%al,(%rax) 0xff8040d2cc89: add%al,(%rax) 0xff8040d2cc8b: add%al,(%rax) 0xff8040d2cc8d: add%al,(%rax) 0xff8040d2cc8f: add%al,(%rax) 0xff8040d2cc91: add%al,(%rax) 0xff8040d2cc93: add%al,(%rax) 0xff8040d2cc95: add%al,(%rax) 0xff8040d2cc97: add%al,(%rcx) 0xff8040d2cc99: add%al,(%rax) 0xff8040d2cc9b: add%al,(%rax) 0xff8040d2cc9d: add%al,(%rax) 0xff8040d2cc9f: add%al,(%rax) End of assembler dump. I will build kernel with DDB tomorrow, install it on some servers and wait for the panic occurs. Ok. Did you checked for such things as rootkits ? I am noticing such panics only on this model of supermicro servers for a long! time under FreeBSD. This servers were tested on huge workload under Linux and there were no problems. I've installed and run chkrootkit now, there are no rootkits. -- Alexey Tarasov (\__/) (='.'=) E[: | | | | :]З ()_() ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
