Re: large amount of NMI_INTERRUPT disgrade winxp VM performance much.
On 08/11/2011 11:59 AM, ya su wrote: Hi, Avi: Your guess is right, the fast server is AMD with NPT. this slow server is Intel's 7430 with no EPT, I now understand the reserved bit come from kvm's virtual soft-mmu. But there is still one confusing problem: why a FC14 VM has a much better storage IO performance on the same host? Hard to tell. Please post a much larger log somewhere (not in email - it wraps horribly). -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: large amount of NMI_INTERRUPT disgrade winxp VM performance much.
Hi, Avi: Your guess is right, the fast server is AMD with NPT. this slow server is Intel's 7430 with no EPT, I now understand the reserved bit come from kvm's virtual soft-mmu. But there is still one confusing problem: why a FC14 VM has a much better storage IO performance on the same host? I always check the IO on host with iotop when copying files or running fio in VM, when running with FC14 VM guest, it can reach 30Mbps; while copying file or running fio in winxp VM guest, it is about 2-3Mbps. FC14's trace-cmd output is as the following: qemu-kvm-7636 [006] 897.452208: kvm_entry:vcpu 0 qemu-kvm-7636 [006] 897.452213: kvm_exit: reason EXCEPTION_NMI rip 0x8100b5fa qemu-kvm-7636 [006] 897.452217: kvm_entry:vcpu 0 qemu-kvm-7636 [006] 897.452408: kvm_exit: reason EXTERNAL_INTERRUPT rip 0x81009ddd qemu-kvm-7636 [006] 897.452411: kvm_entry:vcpu 0 qemu-kvm-7636 [006] 897.452437: kvm_exit: reason CR_ACCESS rip 0x8103fadd qemu-kvm-7636 [006] 897.452437: kvm_cr: cr_write 3 = 0x7a709000 qemu-kvm-7636 [006] 897.452442: kvm_entry:vcpu 0 qemu-kvm-7636 [006] 897.453113: kvm_exit: reason EXTERNAL_INTERRUPT rip 0x8103d12e qemu-kvm-7636 [006] 897.453116: kvm_apic_accept_irq: apicid 0 vec 239 (Fixed|edge) qemu-kvm-7636 [006] 897.453120: kvm_inj_virq: irq 239 qemu-kvm-7636 [006] 897.453121: kvm_entry:vcpu 0 qemu-kvm-7636 [006] 897.453126: kvm_exit: reason APIC_ACCESS rip 0x81026239 qemu-kvm-7636 [006] 897.453134: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-kvm-7636 [006] 897.453135: kvm_apic: apic_write APIC_EOI = 0x0 qemu-kvm-7636 [006] 897.453137: kvm_entry:vcpu 0 qemu-kvm-7636 [006] 897.453155: kvm_exit: reason APIC_ACCESS rip 0x81026239 qemu-kvm-7636 [006] 897.453159: kvm_mmio: mmio write len 4 gpa 0xfee00380 val 0xe6f5 qemu-kvm-7636 [006] 897.453160: kvm_apic: apic_write APIC_TMICT = 0xe6f5 qemu-kvm-7636 [006] 897.453164: kvm_entry:vcpu 0 qemu-kvm-7636 [006] 897.453373: kvm_exit: reason IO_INSTRUCTION rip 0x812243e4 qemu-kvm-7636 [006] 897.453378: kvm_pio: pio_write at 0xc050 size 2 count 1 qemu-kvm-7636 [006] 897.453625: kvm_entry:vcpu 0 qemu-kvm-7636 [006] 897.453984: kvm_exit: reason IO_INSTRUCTION rip 0x812243e4 qemu-kvm-7636 [006] 897.453984: kvm_pio: pio_write at 0xc050 size 2 count 1 qemu-kvm-7636 [006] 897.454198: kvm_apic_accept_irq: apicid 0 vec 239 (Fixed|edge) qemu-kvm-7636 [006] 897.454201: kvm_entry:vcpu 0 qemu-kvm-7636 [006] 897.454206: kvm_exit: reason PENDING_INTERRUPT rip 0x81201c95 qemu-kvm-7636 [006] 897.454209: kvm_inj_virq: irq 239 qemu-kvm-7636 [006] 897.454209: kvm_entry:vcpu 0 qemu-kvm-7636 [006] 897.454212: kvm_exit: reason APIC_ACCESS rip 0x81026239 qemu-kvm-7636 [006] 897.454220: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-kvm-7636 [006] 897.454222: kvm_apic: apic_write APIC_EOI = 0x0 qemu-kvm-7636 [006] 897.454225: kvm_entry:vcpu 0 qemu-kvm-7636 [006] 897.454238: kvm_exit: reason APIC_ACCESS rip 0x81026239 qemu-kvm-7636 [006] 897.454243: kvm_mmio: mmio write len 4 gpa 0xfee00380 val 0xd29f qemu-kvm-7636 [006] 897.454243: kvm_apic: apic_write APIC_TMICT = 0xd29f qemu-kvm-7636 [006] 897.454247: kvm_entry:vcpu 0 qemu-kvm-7636 [006] 897.454405: kvm_exit: reason EXTERNAL_INTERRUPT rip 0x8113e91f qemu-kvm-7636 [006] 897.454410: kvm_entry:vcpu 0 qemu-kvm-7636 [006] 897.454547: kvm_exit: reason IO_INSTRUCTION rip 0x812243e4 qemu-kvm-7636 [006] 897.454548: kvm_pio: pio_write at 0xc050 size 2 count 1 qemu-kvm-7636 [006] 897.454690: kvm_entry:vcpu 0 qemu-kvm-7636 [006] 897.454714: kvm_exit: reason APIC_ACCESS rip 0x81026239 qemu-kvm-7636 [006] 897.454720: kvm_mmio: mmio write len 4 gpa 0xfee00380 val 0x7e qemu-kvm-7636 [006] 897.454721: kvm_apic: apic_write APIC_TMICT = 0x7e qemu-kvm-7636 [006] 897.454725: kvm_entry:vcpu 0 qemu-kvm-7636 [006] 897.454730: kvm_exit: reason APIC_ACCESS rip 0x81026239 qemu-kvm-7636 [006] 897.454733: kvm_mmio: mmio write len 4 gpa 0xfee00380 val 0x1c040d qemu-kvm-7636 [006] 897.454735: kvm_apic: apic_write APIC_TMICT = 0x1c040d qemu-kvm-7636 [006] 897.454737: kvm_entry:vcpu 0
Re: large amount of NMI_INTERRUPT disgrade winxp VM performance much.
On 08/11/2011 08:41 AM, Tian, Kevin wrote: > qemu-system-x86-4454 [004] 549.958172: kvm_exit: > reason EXCEPTION_NMI rip 0x8051d5e1 > qemu-system-x86-4454 [004] 549.958172: kvm_page_fault: > address c8f8a000 error_code b error_code 'b' means the page fault is caused by a write access in kernel space, but related page entry has reserved bit set. This is usually used by OS for some special tricks, e.g. to handle page swaps. You may check related setting in win guest. In this case it was kvm setting the reserved bit. Looking at your other trace, that server is probably an AMD with the NPT feature, while your first one is an Intel without the equivalent (EPT). For this type of workload, you need EPT or NPT. -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: large amount of NMI_INTERRUPT disgrade winxp VM performance much.
To clear the problem from guest settings, I run the same winxp image on another server with the same kernel/qemu-kvm/command. the copy is fast. so I think this problem relates only with some kind of host's special hardware. the fast server's trace-cmd output as the following: qemu-system-x86-7681 [001] 20054.604841: kvm_entry:vcpu 0 qemu-system-x86-7681 [001] 20054.604842: kvm_exit: reason UNKNOWN rip 0x806e7d33 qemu-system-x86-7681 [001] 20054.604842: kvm_page_fault: address fee000b0 error_code 6 qemu-system-x86-7681 [001] 20054.604843: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 qemu-system-x86-7681 [001] 20054.604843: kvm_apic: apic_write APIC_EOI = 0x0 qemu-system-x86-7681 [001] 20054.604844: kvm_entry:vcpu 0 qemu-system-x86-7681 [001] 20054.604917: kvm_exit: reason UNKNOWN rip 0xbff63b14 qemu-system-x86-7681 [001] 20054.604917: kvm_page_fault: address b8040 error_code 4 qemu-system-x86-7681 [001] 20054.604920: kvm_mmio: mmio unsatisfied-read len 1 gpa 0xb8040 val 0x0 qemu-system-x86-7681 [001] 20054.604923: kvm_mmio: mmio read len 1 gpa 0xb8040 val 0x0 qemu-system-x86-7681 [001] 20054.604924: kvm_mmio: mmio write len 1 gpa 0xb8040 val 0x0 qemu-system-x86-7681 [001] 20054.604925: kvm_entry:vcpu 0 qemu-system-x86-7681 [001] 20054.604926: kvm_exit: reason UNKNOWN rip 0xbff63b1a qemu-system-x86-7681 [001] 20054.604927: kvm_page_fault: address b801a error_code 6 qemu-system-x86-7681 [001] 20054.604928: kvm_mmio: mmio write len 1 gpa 0xb801a val 0xd qemu-system-x86-7681 [001] 20054.604928: kvm_entry:vcpu 0 qemu-system-x86-7681 [001] 20054.604929: kvm_exit: reason UNKNOWN rip 0xbff63b23 qemu-system-x86-7681 [001] 20054.604929: kvm_page_fault: address b8014 error_code 6 qemu-system-x86-7681 [001] 20054.604930: kvm_mmio: mmio write len 4 gpa 0xb8014 val 0x15f900 According to Tian's suggest, the NMI is produced by guest's write to reservd page, Is there any way to find out why the slow-copy server reserve the memory page? I checked the server's memory, there is large free space, no swap is used. and I have tested with server with kernel 2.6.39, the problem remains. Regards. Suya. 2011/8/11 Tian, Kevin : >> From: ya su >> Sent: Thursday, August 11, 2011 11:57 AM >> >> When I run winxp guest on one server, copy one file about 4G will >> take a time of 40-50 min; if I run a FC14 guest, it will take about >> 2-3 min; >> >> I copy and run the winxp image on another server, it works well, take >> about 3min. >> >> I run trace-cmd while copying files, the main difference of the two >> outputs is that: the slow one's output have many NMI_INTERRUPT >> vm_exit, while the fast output has no such vm_exit. both of the two >> servers have NMI enabled default. the slow one's output as the >> following: >> qemu-system-x86-4454 [004] 549.958147: kvm_entry: >> vcpu 0 >> qemu-system-x86-4454 [004] 549.958172: kvm_exit: >> reason EXCEPTION_NMI rip 0x8051d5e1 >> qemu-system-x86-4454 [004] 549.958172: kvm_page_fault: >> address c8f8a000 error_code b >> qemu-system-x86-4454 [004] 549.958177: kvm_entry: >> vcpu 0 >> qemu-system-x86-4454 [004] 549.958202: kvm_exit: >> reason EXCEPTION_NMI rip 0x8051d5e1 >> qemu-system-x86-4454 [004] 549.958204: kvm_page_fault: >> address c8f8b000 error_code b >> qemu-system-x86-4454 [004] 549.958209: kvm_entry: >> vcpu 0 >> qemu-system-x86-4454 [004] 549.958234: kvm_exit: >> reason EXCEPTION_NMI rip 0x8051d5e1 >> qemu-system-x86-4454 [004] 549.958234: kvm_page_fault: >> address c8f8c000 error_code b >> qemu-system-x86-4454 [004] 549.958239: kvm_entry: >> vcpu 0 >> qemu-system-x86-4454 [004] 549.958264: kvm_exit: >> reason EXCEPTION_NMI rip 0x8051d5e1 >> qemu-system-x86-4454 [004] 549.958264: kvm_page_fault: >> address c8f8d000 error_code b >> qemu-system-x86-4454 [004] 549.958267: kvm_entry: >> vcpu 0 >> qemu-system-x86-4454 [004] 549.958292: kvm_exit: >> reason EXCEPTION_NMI rip 0x8051d5e1 >> qemu-system-x86-4454 [004] 549.958294: kvm_page_fault: >> address c8f8e000 error_code b >> qemu-system-x86-4454 [004] 549.958299: kvm_entry: >> vcpu 0 >> qemu-system-x86-4454 [004] 549.958324: kvm_exit: >> reason EXCEPTION_NMI rip 0x8051d5e1 >> qemu-system-x86-4454 [004] 549.958324: kvm_page_fault: >> address c8f8f000 error_code b >> qemu-system-x86-4454 [004] 549.958329: kvm_entry: >> vcpu 0 >> qemu-system-x86-4454 [004] 549.958447: kvm_exit: >> reason EXTERNAL_INTERRUPT rip 0x80547ac8 >> qemu-system-x86-4454 [004] 549.958450: kvm_entry: >> vcpu 0 >> qemu-system-x86-4454 [004] 549.958461: kvm_exit: >> reason CR_ACCESS rip 0x8054428c >> qemu-system-x86-4454 [004] 549.958461: kvm_cr: >> cr_write 0 = 0x80010031 >> qemu-system-x86-4454 [004] 549.958541: kvm_entry: >> vcpu 0 >> qemu-system-x86-4454 [004] 549.958573: kvm_exit: >> reason
RE: large amount of NMI_INTERRUPT disgrade winxp VM performance much.
> From: ya su > Sent: Thursday, August 11, 2011 11:57 AM > > When I run winxp guest on one server, copy one file about 4G will > take a time of 40-50 min; if I run a FC14 guest, it will take about > 2-3 min; > > I copy and run the winxp image on another server, it works well, take > about 3min. > > I run trace-cmd while copying files, the main difference of the two > outputs is that: the slow one's output have many NMI_INTERRUPT > vm_exit, while the fast output has no such vm_exit. both of the two > servers have NMI enabled default. the slow one's output as the > following: > qemu-system-x86-4454 [004] 549.958147: kvm_entry: > vcpu 0 > qemu-system-x86-4454 [004] 549.958172: kvm_exit: > reason EXCEPTION_NMI rip 0x8051d5e1 > qemu-system-x86-4454 [004] 549.958172: kvm_page_fault: > address c8f8a000 error_code b > qemu-system-x86-4454 [004] 549.958177: kvm_entry: > vcpu 0 > qemu-system-x86-4454 [004] 549.958202: kvm_exit: > reason EXCEPTION_NMI rip 0x8051d5e1 > qemu-system-x86-4454 [004] 549.958204: kvm_page_fault: > address c8f8b000 error_code b > qemu-system-x86-4454 [004] 549.958209: kvm_entry: > vcpu 0 > qemu-system-x86-4454 [004] 549.958234: kvm_exit: > reason EXCEPTION_NMI rip 0x8051d5e1 > qemu-system-x86-4454 [004] 549.958234: kvm_page_fault: > address c8f8c000 error_code b > qemu-system-x86-4454 [004] 549.958239: kvm_entry: > vcpu 0 > qemu-system-x86-4454 [004] 549.958264: kvm_exit: > reason EXCEPTION_NMI rip 0x8051d5e1 > qemu-system-x86-4454 [004] 549.958264: kvm_page_fault: > address c8f8d000 error_code b > qemu-system-x86-4454 [004] 549.958267: kvm_entry: > vcpu 0 > qemu-system-x86-4454 [004] 549.958292: kvm_exit: > reason EXCEPTION_NMI rip 0x8051d5e1 > qemu-system-x86-4454 [004] 549.958294: kvm_page_fault: > address c8f8e000 error_code b > qemu-system-x86-4454 [004] 549.958299: kvm_entry: > vcpu 0 > qemu-system-x86-4454 [004] 549.958324: kvm_exit: > reason EXCEPTION_NMI rip 0x8051d5e1 > qemu-system-x86-4454 [004] 549.958324: kvm_page_fault: > address c8f8f000 error_code b > qemu-system-x86-4454 [004] 549.958329: kvm_entry: > vcpu 0 > qemu-system-x86-4454 [004] 549.958447: kvm_exit: > reason EXTERNAL_INTERRUPT rip 0x80547ac8 > qemu-system-x86-4454 [004] 549.958450: kvm_entry: > vcpu 0 > qemu-system-x86-4454 [004] 549.958461: kvm_exit: > reason CR_ACCESS rip 0x8054428c > qemu-system-x86-4454 [004] 549.958461: kvm_cr: > cr_write 0 = 0x80010031 > qemu-system-x86-4454 [004] 549.958541: kvm_entry: > vcpu 0 > qemu-system-x86-4454 [004] 549.958573: kvm_exit: > reason CR_ACCESS rip 0x80546beb > qemu-system-x86-4454 [004] 549.958575: kvm_cr: > cr_write 0 = 0x8001003b > qemu-system-x86-4454 [004] 549.958585: kvm_entry: > vcpu 0 > qemu-system-x86-4454 [004] 549.958610: kvm_exit: > reason CR_ACCESS rip 0x80546b6c > qemu-system-x86-4454 [004] 549.958610: kvm_cr: > cr_write 3 = 0x6e00020 > qemu-system-x86-4454 [004] 549.958621: kvm_entry: > vcpu 0 > qemu-system-x86-4454 [004] 549.958645: kvm_exit: > reason EXCEPTION_NMI rip 0x8051d7f4 > qemu-system-x86-4454 [004] 549.958645: kvm_page_fault: > address c0648200 error_code 3 > qemu-system-x86-4454 [004] 549.958653: kvm_entry: > vcpu 0 > qemu-system-x86-4454 [004] 549.958725: kvm_exit: > reason EXCEPTION_NMI rip 0x8050a26a > qemu-system-x86-4454 [004] 549.958726: kvm_page_fault: > address c0796994 error_code 3 > qemu-system-x86-4454 [004] 549.958738: kvm_entry: > vcpu 0 > qemu-system-x86-4454 [004] 549.958750: kvm_exit: > reason IO_INSTRUCTION rip 0x806edad0 > qemu-system-x86-4454 [004] 549.958750: kvm_pio: > pio_write at 0xc050 size 2 count 1 > qemu-system-x86-4454 [004] 549.958838: kvm_entry: > vcpu 0 > qemu-system-x86-4454 [004] 549.958844: kvm_exit: > reason APIC_ACCESS rip 0x806e7b85 > qemu-system-x86-4454 [004] 549.958852: kvm_apic: > apic_read APIC_ICR = 0x40041 > qemu-system-x86-4454 [004] 549.958855: kvm_mmio: > mmio > read len 4 gpa 0xfee00300 val 0x40041 > qemu-system-x86-4454 [004] 549.958857: kvm_mmio: > mmio > write len 4 gpa 0xfee00300 val 0x40041 > qemu-system-x86-4454 [004] 549.958858: kvm_apic: > apic_write APIC_ICR = 0x40041 > qemu-system-x86-4454 [004] 549.958860: kvm_apic_ipi: dst 1 > vec 65 (Fixed|physical|de-assert|edge|self) > qemu-system-x86-4454 [004] 549.958860: kvm_apic_accept_irq: > apicid 0 vec 65 (Fixed|edge) > > Even I disable the NMI when I boot the kernel with nmi_watchdog=0, > the trace-cmd output still show there are many NMI_INTERRUPT. I find > that in /proc/interrupts, the amount of NMI is 0. Does this mean that > NMI is produced in winxp guest OS, or this setting can not hinder kvm > to catch NMI interrupt? > > I think the difference between FC14 and winxp is that: fc14 process > the NMI interrupt correctly, but winxp can not, is this right? > > I run qemu-kvm with version 0.14.0, kernel version 2.6.32-131.6.4. I > change kvm
