Re: [kvm-devel] performance with guests running 2.4 kernels (specifically RHEL3)

2008-05-16 Thread David S. Ahern
Avi Kivity wrote: Okay, I committed the patch without the flood count == 5. I've continued testing the RHEL3 guests with the flood count at 3, and I am right back to where I started. With the patch and the flood count at 3, I had 2 runs totaling around 24 hours that looked really good. Now,

Re: [kvm-devel] performance with guests running 2.4 kernels (specifically RHEL3)

2008-05-15 Thread Avi Kivity
David S. Ahern wrote: Avi Kivity wrote: Not so fast... the patch updates the flood count to 5. Can you check if a lower value still works? Also, whether updating the flood count to 5 (without the rest of the patch) works? Unconditionally bumping the flood count to 5 will likely cause

Re: [kvm-devel] performance with guests running 2.4 kernels (specifically RHEL3)

2008-05-14 Thread David S. Ahern
Avi Kivity wrote: Not so fast... the patch updates the flood count to 5. Can you check if a lower value still works? Also, whether updating the flood count to 5 (without the rest of the patch) works? Unconditionally bumping the flood count to 5 will likely cause a performance regression

Re: [kvm-devel] performance with guests running 2.4 kernels (specifically RHEL3)

2008-05-13 Thread Avi Kivity
David S. Ahern wrote: That does the trick with kscand. Not so fast... the patch updates the flood count to 5. Can you check if a lower value still works? Also, whether updating the flood count to 5 (without the rest of the patch) works? Unconditionally bumping the flood count to 5

Re: [kvm-devel] performance with guests running 2.4 kernels (specifically RHEL3)

2008-05-12 Thread David S. Ahern
That does the trick with kscand. Do you have recommendations for clock source settings? For example in my test case for this patch the guest gained 73 seconds (ahead of real time) after only 3 hours, 5 min of uptime. thanks, david Avi Kivity wrote: Avi Kivity wrote: Avi Kivity wrote: I

Re: [kvm-devel] performance with guests running 2.4 kernels (specifically RHEL3)

2008-05-11 Thread Avi Kivity
Avi Kivity wrote: I asked fo this thinking bypass_guest_pf may help show more information. But thinking a bit more, it will not. I think I do know what the problem is. I will try it out. Is there a free clone (like centos) available somewhere? This patch tracks down emulated accesses

Re: [kvm-devel] performance with guests running 2.4 kernels (specifically RHEL3)

2008-05-11 Thread Avi Kivity
Avi Kivity wrote: Avi Kivity wrote: I asked fo this thinking bypass_guest_pf may help show more information. But thinking a bit more, it will not. I think I do know what the problem is. I will try it out. Is there a free clone (like centos) available somewhere? This patch tracks down

Re: [kvm-devel] performance with guests running 2.4 kernels (specifically RHEL3)

2008-04-30 Thread Avi Kivity
David S. Ahern wrote: Another tidbit for you guys as I make my way through various permutations: I installed the RHEL3 hugemem kernel and the guest behavior is *much* better. System time still has some regular hiccups that are higher than xen and esx (e.g., 1 minute samples out of 5 show

Re: [kvm-devel] performance with guests running 2.4 kernels (specifically RHEL3)

2008-04-30 Thread David S. Ahern
Avi Kivity wrote: David S. Ahern wrote: Another tidbit for you guys as I make my way through various permutations: I installed the RHEL3 hugemem kernel and the guest behavior is *much* better. System time still has some regular hiccups that are higher than xen and esx (e.g., 1 minute

Re: [kvm-devel] performance with guests running 2.4 kernels (specifically RHEL3)

2008-04-30 Thread Avi Kivity
David S. Ahern wrote: Avi Kivity wrote: David S. Ahern wrote: Another tidbit for you guys as I make my way through various permutations: I installed the RHEL3 hugemem kernel and the guest behavior is *much* better. System time still has some regular hiccups that are higher than

Re: [kvm-devel] performance with guests running 2.4 kernels (specifically RHEL3)

2008-04-30 Thread Daniel P. Berrange
On Wed, Apr 30, 2008 at 07:39:53AM -0600, David S. Ahern wrote: Avi Kivity wrote: David S. Ahern wrote: Another tidbit for you guys as I make my way through various permutations: I installed the RHEL3 hugemem kernel and the guest behavior is *much* better. System time still has some

Re: [kvm-devel] performance with guests running 2.4 kernels (specifically RHEL3)

2008-04-30 Thread David S. Ahern
Yes, the 4G/4G patch and the 64G options are both enabled for the hugemem kernel: CONFIG_HIGHMEM64G=y CONFIG_X86_4G=y Differences between the standard kernel and the hugemem kernel: # diff config-2.4.21-47.ELsmp config-2.4.21-47.ELhugemem 2157,2158c2157,2158 CONFIG_M686=y #

Re: [kvm-devel] performance with guests running 2.4 kernels (specifically RHEL3)

2008-04-29 Thread David S. Ahern
Another tidbit for you guys as I make my way through various permutations: I installed the RHEL3 hugemem kernel and the guest behavior is *much* better. System time still has some regular hiccups that are higher than xen and esx (e.g., 1 minute samples out of 5 show system time between 10 and

Re: [kvm-devel] performance with guests running 2.4 kernels (specifically RHEL3)

2008-04-28 Thread Marcelo Tosatti
On Fri, Apr 25, 2008 at 11:33:18AM -0600, David S. Ahern wrote: Most of the cycles (~80% of that 54k+) are spent in paging64_prefetch_page(): for (i = 0; i PT64_ENT_PER_PAGE; ++i) { gpa_t pte_gpa = gfn_to_gpa(sp-gfn); pte_gpa += (i+offset) *

Re: [kvm-devel] performance with guests running 2.4 kernels (specifically RHEL3)

2008-04-26 Thread Avi Kivity
David S. Ahern wrote: Avi Kivity wrote: Ah! The flood detector is not seeing the access through the kmap_atomic() pte, because that access has gone through the emulator. last_updated_pte_accessed(vcpu) will never return true. Can you verify that last_updated_pte_accessed(vcpu) indeed

Re: [kvm-devel] performance with guests running 2.4 kernels (specifically RHEL3)

2008-04-26 Thread Avi Kivity
David S. Ahern wrote: What is the rip (GUEST_RIP) value in the VMEXIT trace output? Is that the current instruction pointer for the guest? Yes. I take it the virt in the PAGE_FAULT trace output is the virtual address the guest was referencing when the page fault occurred. What I don't

Re: [kvm-devel] performance with guests running 2.4 kernels (specifically RHEL3)

2008-04-26 Thread Avi Kivity
David S. Ahern wrote: David S. Ahern wrote: Avi Kivity wrote: David S. Ahern wrote: I added the traces and captured data over another apparent lockup of the guest. This seems to be representative of the sequence (pid/vcpu removed). (+4776) VMEXIT [ exitcode =

Re: [kvm-devel] performance with guests running 2.4 kernels (specifically RHEL3)

2008-04-25 Thread David S. Ahern
David S. Ahern wrote: Avi Kivity wrote: David S. Ahern wrote: I added the traces and captured data over another apparent lockup of the guest. This seems to be representative of the sequence (pid/vcpu removed). (+4776) VMEXIT [ exitcode = 0x, rip = 0x c016127c ]

Re: [kvm-devel] performance with guests running 2.4 kernels (specifically RHEL3)

2008-04-24 Thread David S. Ahern
What is the rip (GUEST_RIP) value in the VMEXIT trace output? Is that the current instruction pointer for the guest? I take it the virt in the PAGE_FAULT trace output is the virtual address the guest was referencing when the page fault occurred. What I don't understand (one of many things

Re: [kvm-devel] performance with guests running 2.4 kernels (specifically RHEL3)

2008-04-23 Thread Avi Kivity
David S. Ahern wrote: kvm_stat -1 is practically impossible to time correctly to get a good snippet. I've added a --log option to get vmstat-like output. I've also added --fields to select which fields are of interest, to avoid the need for 280-column displays. That's now pushed to

Re: [kvm-devel] performance with guests running 2.4 kernels (specifically RHEL3)

2008-04-23 Thread Avi Kivity
David S. Ahern wrote: I added tracers to kvm_mmu_page_fault() that include collecting tsc cycles: 1. before vcpu-arch.mmu.page_fault() 2. after vcpu-arch.mmu.page_fault() 3. after mmu_topup_memory_caches() 4. after emulate_instruction() So the delta in the trace reports show: - cycles

Re: [kvm-devel] performance with guests running 2.4 kernels (specifically RHEL3)

2008-04-23 Thread Avi Kivity
David S. Ahern wrote: I've continued drilling down with the tracers to answer that question. I have done runs with tracers in paging64_page_fault and it showed the overhead is with the fetch() function. On my last run the tracers are in paging64_fetch() as follows: 1. after

Re: [kvm-devel] performance with guests running 2.4 kernels (specifically RHEL3)

2008-04-23 Thread David S. Ahern
Avi Kivity wrote: Ah! The flood detector is not seeing the access through the kmap_atomic() pte, because that access has gone through the emulator. last_updated_pte_accessed(vcpu) will never return true. Can you verify that last_updated_pte_accessed(vcpu) indeed always returns false?

Re: [kvm-devel] performance with guests running 2.4 kernels (specifically RHEL3)

2008-04-22 Thread David S. Ahern
I added tracers to kvm_mmu_page_fault() that include collecting tsc cycles: 1. before vcpu-arch.mmu.page_fault() 2. after vcpu-arch.mmu.page_fault() 3. after mmu_topup_memory_caches() 4. after emulate_instruction() So the delta in the trace reports show: - cycles required for arch.mmu.page_fault

Re: [kvm-devel] performance with guests running 2.4 kernels (specifically RHEL3)

2008-04-21 Thread Avi Kivity
David S. Ahern wrote: I added the traces and captured data over another apparent lockup of the guest. This seems to be representative of the sequence (pid/vcpu removed). (+4776) VMEXIT [ exitcode = 0x, rip = 0x c016127c ] (+ 0) PAGE_FAULT [ errorcode =

Re: [kvm-devel] performance with guests running 2.4 kernels (specifically RHEL3)

2008-04-20 Thread David S. Ahern
I added the traces and captured data over another apparent lockup of the guest. This seems to be representative of the sequence (pid/vcpu removed). (+4776) VMEXIT [ exitcode = 0x, rip = 0x c016127c ] (+ 0) PAGE_FAULT [ errorcode = 0x0003, virt = 0x

Re: [kvm-devel] performance with guests running 2.4 kernels (specifically RHEL3)

2008-04-17 Thread David S. Ahern
kvm_stat -1 is practically impossible to time correctly to get a good snippet. kvmtrace is a fascinating tool. I captured trace data that encompassed one intense period where the VM appeared to freeze (no terminal response for a few seconds). After converting to text I examined an arbitrary

Re: [kvm-devel] performance with guests running 2.4 kernels (specifically RHEL3)

2008-04-16 Thread Avi Kivity
David S. Ahern wrote: I have been looking at RHEL3 based guests lately, and to say the least the performance is horrible. Rather than write a long tome on what I've done and observed, I'd like to find out if anyone has some insights or known problem areas running 2.4 guests. The short of it is

[kvm-devel] performance with guests running 2.4 kernels (specifically RHEL3)

2008-04-15 Thread David S. Ahern
I have been looking at RHEL3 based guests lately, and to say the least the performance is horrible. Rather than write a long tome on what I've done and observed, I'd like to find out if anyone has some insights or known problem areas running 2.4 guests. The short of it is that % system time