Virtio network: poor network with KVM hypervisor (latest Proxmox)
Hi all!
I am using the latest Proxmox 4.1 with all updates installed.
I have several VM's with FreeBSD guests and 1 VM with Ubuntu 14 (all KVM).
Host system file download speed: 60 MBps.
FreeBSD guest download speed: 2 MBps on virtio network with TSO enabled, 5-9
MBps with TSO disabled; 12 MBps on e1000 network.
Ubuntu guest: 60 MBps with virtio.
I've tried the following:
1) Different FreeBSD versions: 9.3, 10.2, 10.3-BETA3.
2) Different TSO settings, enabling/disabling RXCSUM.
3) Different TSO settings on host system.
The best results I got described above :(
Does anyone have any ideas how to get full network performance inside FreeBSD
guests?
--
Alexey Tarasov
(\__/)
(='.'=)
E[: | | | | :]З
(")_(")
smime.p7s
Description: S/MIME cryptographic signature
STABLE kernel panic: privileged instruction fault
Hello. I have a couple of Supermicro servers which got the similar kernel panic with all FreeBSD versions I tried since 6.4. Now I want to investigate into the problem. The servers get into panic with similar workload: file server with a lot of files and connections. Web server software is nginx. File system is UFS+GJOURNAL. Outgoing traffic on each server is ~10 MB/s. I think it is not software problem, because when I've installed Linux with such configuration there were no kernel panics. Here is the short overview of the hardware: CPU: Intel(R) Pentium(R) 4 CPU 3.00GHz (2992.51-MHz K8-class CPU) Origin = GenuineIntel Id = 0xf65 Family = f Model = 6 Stepping = 5 Features=0xbfebfbffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE Features2=0xe59dSSE3,DTES64,MON,DS_CPL,EST,TM2,CNXT-ID,CX16,xTPR,PDCM AMD Features=0x20100800SYSCALL,NX,LM AMD Features2=0x1LAHF TSC: P-state invariant real memory = 2147483648 (2048 MB) avail memory = 2054619136 (1959 MB) DMESG: http://lexasoft.ru/m/dmesg.txt CORE: http://lexasoft.ru/m/core.txt Fatal trap 1: privileged instruction fault while in kernel mode cpuid = 1; apic id = 01 instruction pointer = 0x20:0xff8040d2cc83 stack pointer = 0x28:0xff8040d2ca80 frame pointer = 0x28:0xff0060c0b740 code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags= interrupt enabled, resume, IOPL = 0 current process = 9388 (nginx) trap number = 1 panic: privileged instruction fault cpuid = 1 Uptime: 17d15h48m49s Physical memory: 2032 MB Dumping 1485 MB: 1470 1454 1438 1422 1406 1390 1374 1358 1342 1326 1310 1294 1278 1262 1246 1230 1214 1198 1182 1166 1150 1134 1118 1102 1086 1070 1054 1038 1022 1006 990 974 958 942 926 910 894 878 862 846 830 814 798 782 766 750 734 718 702 686 670 654 638 622 606 590 574 558 542 526 510 494 478 462 446 430 414 398 382 366 350 334 318 302 286 270 254 238 222 206 190 174 158 142 126 110 94 78 62 46 30 14 (kgdb) #0 doadump () at pcpu.h:223 #1 0x80590c59 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:416 #2 0x8059108c in panic (fmt=0x80951fc4 %s) at /usr/src/sys/kern/kern_shutdown.c:579 #3 0x80878fd8 in trap_fatal (frame=0xff0060c0b740, eva=Variable eva is not available. ) at /usr/src/sys/amd64/amd64/trap.c:857 #4 0x808799ea in trap (frame=0xff8040d2c9d0) at /usr/src/sys/amd64/amd64/trap.c:644 #5 0x8085f983 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:224 #6 0xff8040d2cc83 in ?? () #7 0xff8040d2cb50 in ?? () #8 0xff8040d2caf0 in ?? () #9 0xff8040d2cbf0 in ?? () #10 0xff0060c0b740 in ?? () #11 0x80b83c60 in sysent () #12 0xff8040d2cc80 in ?? () #13 0xff8040d2cae0 in ?? () #14 0x8059c431 in bintime (bt=0x80ad3140) at /usr/src/sys/kern/kern_tc.c:200 Previous frame inner to this frame (corrupt stack?) (kgdb) -- Alexey Tarasov (\__/) (='.'=) E[: | | | | :]З ()_() ___ freebsd-curr...@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: STABLE kernel panic: privileged instruction fault
Hello Kostik! On Aug 16, 2010, at 10:48 PM, Kostik Belousov wrote: The backtrace make absolutely no sense. I would not trust kgdb anyway. Compile ddb in and do backtrace in console on the panic. Also, disassemble the kernel at the fault address. I am very curious which instruction causes this. This is stock GENERIC on the bare metal booted, right ? Yes, stock GENERIC. Please, check this out: Dump of assembler code from 0xff0060c0b700 to 0xff0060c0b780: 0xff0060c0b700: add%al,(%rax) 0xff0060c0b702: add%al,(%rax) 0xff0060c0b704: add%al,(%rax) 0xff0060c0b706: add%al,(%rax) 0xff0060c0b708: add%al,(%rax) 0xff0060c0b70a: add%al,(%rax) 0xff0060c0b70c: add%al,(%rax) 0xff0060c0b70e: add%al,(%rax) 0xff0060c0b710: or %dh,0xffc2(%rax) 0xff0060c0b713: cmp$0xff,%bh 0xff0060c0b716: (bad) 0xff0060c0b717: incl (%rax) 0xff0060c0b719: add%al,(%rcx) 0xff0060c0b71b: add%cl,%bh 0xff0060c0b71d: pop%rsp 0xff0060c0b71e: out%al,(%dx) 0xff0060c0b71f: pop%rdx 0xff0060c0b720: or $0x0,%al 0xff0060c0b722: add%al,(%rax) 0xff0060c0b724: or %al,%fs:(%rax) 0xff0060c0b727: add%ah,%bl 0xff0060c0b729: int3 0xff0060c0b72a: (bad) 0xff0060c0b72b: add%cl,%bh 0xff0060c0b72d: pop%rsp 0xff0060c0b72e: out%al,(%dx) 0xff0060c0b72f: pop%rdx 0xff0060c0b730: iret 0xff0060c0b731: pop%rsp 0xff0060c0b732: out%al,(%dx) 0xff0060c0b733: pop%rdx 0xff0060c0b734: rex xor$0x9f105aee,%eax 0xff0060c0b73a: add%al,(%rax) 0xff0060c0b73c: add%al,(%rax) 0xff0060c0b73e: add%al,(%rax) 0xff0060c0b740: rex pop%rdi 0xff0060c0b742: retq $0xff80 0xff0060c0b745: (bad) 0xff0060c0b746: (bad) 0xff0060c0b747: incl (%rax) 0xff0060c0b749: push %rax 0xff0060c0b74a: loop 0xff0060c0b78e 0xff0060c0b74c: add%bh,%bh 0xff0060c0b74e: (bad) 0xff0060c0b74f: incl (%rax) 0xff0060c0b751: add%al,(%rax) 0xff0060c0b753: add%al,(%rax) 0xff0060c0b755: add%al,(%rax) 0xff0060c0b757: add%dl,(%rax) 0xff0060c0b759: push %rax 0xff0060c0b75a: loop 0xff0060c0b79e 0xff0060c0b75c: add%bh,%bh 0xff0060c0b75e: (bad) 0xff0060c0b75f: incl (%rax) 0xff0060c0b761: add%al,(%rax) 0xff0060c0b763: add%al,(%rax) 0xff0060c0b765: add%al,(%rax) 0xff0060c0b767: add%bl,%al 0xff0060c0b769: pop%rdi 0xff0060c0b76a: retq $0xff80 0xff0060c0b76d: (bad) 0xff0060c0b76e: (bad) 0xff0060c0b76f: incl (%rax) 0xff0060c0b771: add%al,(%rax) 0xff0060c0b773: add%al,(%rax) 0xff0060c0b775: add%al,(%rax) 0xff0060c0b777: add%al,0x290c55(%rax) 0xff0060c0b77d: (bad) 0xff0060c0b77e: (bad) 0xff0060c0b77f: incl (%rax) -- Alexey Tarasov (\__/) (='.'=) E[: | | | | :]З ()_() ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: STABLE kernel panic: privileged instruction fault
On Aug 16, 2010, at 11:31 PM, Kostik Belousov wrote: On Mon, Aug 16, 2010 at 11:21:15PM +0400, Alexey Tarasov wrote: Hello Kostik! On Aug 16, 2010, at 10:48 PM, Kostik Belousov wrote: The backtrace make absolutely no sense. I would not trust kgdb anyway. Compile ddb in and do backtrace in console on the panic. Also, disassemble the kernel at the fault address. I am very curious which instruction causes this. This is stock GENERIC on the bare metal booted, right ? Yes, stock GENERIC. Please, check this out: Dump of assembler code from 0xff0060c0b700 to 0xff0060c0b780: Would be nice if you keep all requested data in one place, so that we do not need to search for the old mails to see the context. According to your previous mail, the fault happen at the address instruction pointer = 0x20:0xff8040d2cc83 Your disassembled the stack instead. Please just do disass 0xff8040d2cc83,0xff8040d2cca0 in kgdb. But also, I want to see the backtrace and disassembly output from ddb. (kgdb) disass 0xff8040d2cc83,0xff8040d2cca0 No function contains specified address. I will build kernel with DDB tomorrow, install it on some servers and wait for the panic occurs. -- Alexey Tarasov (\__/) (='.'=) E[: | | | | :]З ()_() ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: STABLE kernel panic: privileged instruction fault
On Aug 16, 2010, at 11:39 PM, Kostik Belousov wrote: On Mon, Aug 16, 2010 at 11:35:36PM +0400, Alexey Tarasov wrote: On Aug 16, 2010, at 11:31 PM, Kostik Belousov wrote: On Mon, Aug 16, 2010 at 11:21:15PM +0400, Alexey Tarasov wrote: Hello Kostik! On Aug 16, 2010, at 10:48 PM, Kostik Belousov wrote: The backtrace make absolutely no sense. I would not trust kgdb anyway. Compile ddb in and do backtrace in console on the panic. Also, disassemble the kernel at the fault address. I am very curious which instruction causes this. This is stock GENERIC on the bare metal booted, right ? Yes, stock GENERIC. Please, check this out: Dump of assembler code from 0xff0060c0b700 to 0xff0060c0b780: Would be nice if you keep all requested data in one place, so that we do not need to search for the old mails to see the context. According to your previous mail, the fault happen at the address instruction pointer = 0x20:0xff8040d2cc83 Your disassembled the stack instead. Please just do disass 0xff8040d2cc83,0xff8040d2cca0 in kgdb. But also, I want to see the backtrace and disassembly output from ddb. (kgdb) disass 0xff8040d2cc83,0xff8040d2cca0 No function contains specified address. Err, it seems that old gdb accepts only spaces. Please try disass 0xff8040d2cc83 0xff8040d2cca0 instead. (kgdb) disass 0xff8040d2cc83 0xff8040d2cca0 Dump of assembler code from 0xff8040d2cc83 to 0xff8040d2cca0: 0xff8040d2cc83: (bad) 0xff8040d2cc84: (bad) 0xff8040d2cc85: jg 0xff8040d2cc87 0xff8040d2cc87: add%al,(%rax) 0xff8040d2cc89: add%al,(%rax) 0xff8040d2cc8b: add%al,(%rax) 0xff8040d2cc8d: add%al,(%rax) 0xff8040d2cc8f: add%al,(%rax) 0xff8040d2cc91: add%al,(%rax) 0xff8040d2cc93: add%al,(%rax) 0xff8040d2cc95: add%al,(%rax) 0xff8040d2cc97: add%al,(%rcx) 0xff8040d2cc99: add%al,(%rax) 0xff8040d2cc9b: add%al,(%rax) 0xff8040d2cc9d: add%al,(%rax) 0xff8040d2cc9f: add%al,(%rax) End of assembler dump. I will build kernel with DDB tomorrow, install it on some servers and wait for the panic occurs. Ok. Did you checked for such things as rootkits ? I am noticing such panics only on this model of supermicro servers for a long! time under FreeBSD. This servers were tested on huge workload under Linux and there were no problems. I've installed and run chkrootkit now, there are no rootkits. -- Alexey Tarasov (\__/) (='.'=) E[: | | | | :]З ()_() ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Kernel panic on 7.2-STABLE
: PCI-ISA bridge at device 31.0 on pci0 isa0: ISA bus on isab0 atapci0: Intel ICH7 UDMA100 controller port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x30a0-0x30af at device 31.1 on pci0 ata0: ATA channel 0 on atapci0 ata0: [ITHREAD] atapci1: Intel ICH7 SATA300 controller port 0x30e8-0x30ef,0x30dc-0x30df,0x30e0-0x30e7,0x30d8-0x30db,0x30b0-0x30bf mem 0xe8600400-0xe86007ff irq 19 at device 31.2 on pci0 atapci1: [ITHREAD] ata2: ATA channel 0 on atapci1 ata2: [ITHREAD] ata3: ATA channel 1 on atapci1 ata3: [ITHREAD] pci0: serial bus, SMBus at device 31.3 (no driver attached) acpi_button0: Power Button on acpi0 atkbdc0: Keyboard controller (i8042) port 0x60,0x64 irq 1 on acpi0 atkbd0: AT Keyboard irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: 16550A-compatible COM port port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A sio0: [FILTER] sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled sio1: 16550A-compatible COM port port 0x2f8-0x2ff irq 3 on acpi0 sio1: type 16550A sio1: [FILTER] fdc0: floppy drive controller port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: does not respond device_attach: fdc0 attach returned 6 ppc0: Parallel port port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0 ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode ppc0: FIFO with 16/16/9 bytes threshold ppbus0: Parallel port bus on ppc0 ppbus0: [ITHREAD] plip0: PLIP network interface on ppbus0 plip0: WARNING: using obsoleted IFF_NEEDSGIANT flag lpt0: Printer on ppbus0 lpt0: Interrupt-driven port ppi0: Parallel I/O on ppbus0 ppc0: [GIANT-LOCKED] ppc0: [ITHREAD] cpu0: ACPI CPU on acpi0 est0: Enhanced SpeedStep Frequency Control on cpu0 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr f270f27 device_attach: est0 attach returned 6 p4tcc0: CPU Frequency Thermal Control on cpu0 cpu1: ACPI CPU on acpi0 est1: Enhanced SpeedStep Frequency Control on cpu1 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr f270f27 device_attach: est1 attach returned 6 p4tcc1: CPU Frequency Thermal Control on cpu1 fdc0: floppy drive controller port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: does not respond device_attach: fdc0 attach returned 6 orm0: ISA Option ROMs at iomem 0xc-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff on isa0 sc0: System console at flags 0x100 on isa0 sc0: VGA 16 virtual consoles, flags=0x300 vga0: Generic ISA VGA at port 0x3c0-0x3df iomem 0xa-0xb on isa0 Timecounters tick every 1.000 msec acd0: CDROM CD-224E-N/1.AA at ata0-slave UDMA33 ad4: 381554MB Seagate ST3400632NS 5.01 at ata2-master SATA150 ad5: 381554MB Seagate ST3400632NS 5.01 at ata2-slave SATA150 ad6: 381554MB Seagate ST3400632NS 5.01 at ata3-master SATA150 ad7: 476940MB Seagate ST3500630AS 3.AAK at ata3-slave SATA150 SMP: AP CPU #1 Launched! WARNING: WITNESS option enabled, expect reduced performance. Trying to mount root from ufs:/dev/ad4s1a WARNING: / was not properly dismounted WARNING: /data1 was not properly dismounted WARNING: /data2 was not properly dismounted WARNING: /data3 was not properly dismounted WARNING: /data4 was not properly dismounted WARNING: /tmp was not properly dismounted WARNING: /usr was not properly dismounted /usr: mount pending error: blocks 176 files 26 WARNING: /var was not properly dismounted /var: mount pending error: blocks 44 files 11 em0: link state changed to UP I have vmcore in /var/crash folder. What debug information is needed to debug this problem? -- (\__/) (='.'=) E[: | | | | :]З ()_() Alexey Tarasov ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
