Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-03-06 Thread Zachary Amsden
On 03/05/2011 02:21 AM, Nikola Ciprich wrote: Can you try this patch to see if it fixes the problem? You haven't read my replies, did you? ;-) kvm_request_guest_time_update seems to have been removed, and kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu) seems to be used instead, adding it

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-03-06 Thread Nikola Ciprich
I did read your mail, but I was working on an old tree... because of that transformation, this fix will unfortunately have to be back and forward ported by hand. OK, sorry, I didn't mean to be adverse... Did you try just that change right applied on top of the patch

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-03-04 Thread Zachary Amsden
On 03/03/2011 05:01 PM, Nikola Ciprich wrote: That sounds like a kernel which will be vulnerable to broken KVM clock on 32-bit. There's a kernel side fix that is needed, but why the server side change triggers the problem needs more investigation. OK, it's important for me that I can fix

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-03-04 Thread Nikola Ciprich
Hello Zachary, You don't see any messages about TSC being unstable or switching clocksource after loading the KVM module? And you are not suspending the host or anything? no messages, no suspending, nothing. Can you try using processor.max_cstate=1 on the host as a kernel parameter

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-03-04 Thread Glauber Costa
On Fri, 2011-03-04 at 19:27 +0100, Nikola Ciprich wrote: Hello Zachary, You don't see any messages about TSC being unstable or switching clocksource after loading the KVM module? And you are not suspending the host or anything? no messages, no suspending, nothing. Can you try

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-03-04 Thread Nikola Ciprich
Zach, I don't understand 100 % the logic behind all your tsc changes. But kvm-clock-wise, most of the problems we had in the past were related to the difference in resolution between the tsc and the host clocksource (hpet, acpi_pm, etc), which in his case, it is a non-issue. It does seem

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-03-04 Thread Glauber Costa
On Fri, 2011-03-04 at 21:55 +0100, Nikola Ciprich wrote: Zach, I don't understand 100 % the logic behind all your tsc changes. But kvm-clock-wise, most of the problems we had in the past were related to the difference in resolution between the tsc and the host clocksource (hpet,

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-03-04 Thread Nikola Ciprich
I think although the long term plan is to just do this update once in your case (stable tsc), this update is needed. Why don't you send a patch to re-include it ? Yes, I'll gladly submit patch, one question, is this OK to just add calling kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu)

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-03-04 Thread Zachary Amsden
On 03/04/2011 02:09 PM, Glauber Costa wrote: On Fri, 2011-03-04 at 19:27 +0100, Nikola Ciprich wrote: Hello Zachary, You don't see any messages about TSC being unstable or switching clocksource after loading the KVM module? And you are not suspending the host or anything?

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-03-04 Thread Zachary Amsden
On 03/04/2011 05:36 PM, Nikola Ciprich wrote: I think although the long term plan is to just do this update once in your case (stable tsc), this update is needed. Why don't you send a patch to re-include it ? Yes, I'll gladly submit patch, one question, is this OK to just add calling

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-03-04 Thread Zachary Amsden
On 03/04/2011 05:36 PM, Nikola Ciprich wrote: I think although the long term plan is to just do this update once in your case (stable tsc), this update is needed. Why don't you send a patch to re-include it ? Yes, I'll gladly submit patch, one question, is this OK to just add calling

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-03-04 Thread Nikola Ciprich
Can you try this patch to see if it fixes the problem? You haven't read my replies, did you? ;-) kvm_request_guest_time_update seems to have been removed, and kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu) seems to be used instead, adding it fixes the problem. That's what I was going to use in

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-03-03 Thread Zachary Amsden
On 03/03/2011 02:06 AM, Nikola Ciprich wrote: No worries. What mess? twice sending the same mail, nevermind :) I have two things you can try: first is running a single VCPU guest, if you have not done so already. yup, UP guest is fine, just SMP doesn't work. Second is

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-03-03 Thread Nikola Ciprich
What is the exact kernel version you are using in the guest. It's latest centos (2.6.18-194.32.1.el5), so I guess there are a lot of fixes, but it's possible the kvm-clock is broken in it. I can't influence what kernel is used there (at least not on customer's guests), but I guess asking for

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-03-03 Thread Zachary Amsden
On 03/03/2011 04:06 PM, Nikola Ciprich wrote: What is the exact kernel version you are using in the guest. It's latest centos (2.6.18-194.32.1.el5), so I guess there are a lot of fixes, but it's possible the kvm-clock is broken in it. I can't influence what kernel is used there (at least

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-03-03 Thread Nikola Ciprich
That sounds like a kernel which will be vulnerable to broken KVM clock on 32-bit. There's a kernel side fix that is needed, but why the server side change triggers the problem needs more investigation. OK, it's important for me that I can fix this by kernel parameter, but if I can help

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-03-02 Thread Zachary Amsden
(resend, sorry for the mess) No worries. What mess? I have two things you can try: first is running a single VCPU guest, if you have not done so already. Second is adding the bootparameter clocksource=acpi_pm to your guest kernel. If either of those fixes the problem, it very well

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-03-02 Thread Nikola Ciprich
No worries. What mess? twice sending the same mail, nevermind :) I have two things you can try: first is running a single VCPU guest, if you have not done so already. yup, UP guest is fine, just SMP doesn't work. Second is adding the bootparameter clocksource=acpi_pm to your guest

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-02-28 Thread Zachary Amsden
On 02/27/2011 12:20 PM, Nikola Ciprich wrote: I was not aware of the thread. Please cc me directly, or add a keyword I track - timekeeping, TSC.. Hello Zachary, thanks for Your time looking at this! That change alone may not bisect well; without further fixes on top of it, you may

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-02-28 Thread Nikola Ciprich
Does the bug you are hitting manifest on both Intel and AMD platforms? I don't have any AMD box here, I'll try this out at my home box. Further, do the systems you are hitting this on have stable or unstable TSCs? how do I find this out? I don't see any warning about TSC in guest, but I've

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-02-28 Thread Zachary Amsden
On 02/28/2011 09:32 AM, Nikola Ciprich wrote: Does the bug you are hitting manifest on both Intel and AMD platforms? I don't have any AMD box here, I'll try this out at my home box. Further, do the systems you are hitting this on have stable or unstable TSCs? how do I find

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-02-28 Thread Nikola Ciprich
On Mon, Feb 28, 2011 at 10:17:24AM -0500, Zachary Amsden wrote: On 02/28/2011 09:32 AM, Nikola Ciprich wrote: Does the bug you are hitting manifest on both Intel and AMD platforms? I don't have any AMD box here, I'll try this out at my home box. Further, do the systems you are

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-02-28 Thread Zachary Amsden
On Mon, Feb 28, 2011 at 10:17:24AM -0500, Zachary Amsden wrote: On 02/28/2011 09:32 AM, Nikola Ciprich wrote: Does the bug you are hitting manifest on both Intel and AMD platforms? I don't have any AMD box here, I'll try this out at my home box. Further, do the

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-02-28 Thread Nikola Ciprich
(resend, sorry for the mess) cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 30 model name: Intel(R) Xeon(R) CPU X3440 @ 2.53GHz stepping : 5 cpu MHz : 2533.185 cache size : 8192 KB

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-02-28 Thread Nikola Ciprich
cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 30 model name: Intel(R) Xeon(R) CPU X3440 @ 2.53GHz stepping : 5 cpu MHz : 2533.185 cache size : 8192 KB physical id : 0 siblings :

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-02-27 Thread Nikola Ciprich
I was not aware of the thread. Please cc me directly, or add a keyword I track - timekeeping, TSC.. Hello Zachary, thanks for Your time looking at this! That change alone may not bisect well; without further fixes on top of it, you may end up with a hang or stall, which is likely to

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-02-25 Thread Nikola Ciprich
(CC: Zachary) Hello, Zachary, in case You haven't noticed the thread, we're trying to find out the reason why 32bit SMP guests stopped working in 2.6.37. bisect shows this as the culprit: e48672fa25e879f7ae21785c7efd187738139593 is first bad commit commit

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-02-25 Thread Zachary Amsden
On 02/25/2011 05:48 AM, Nikola Ciprich wrote: (CC: Zachary) Hello, Zachary, in case You haven't noticed the thread, we're trying to find out the reason why 32bit SMP guests stopped working in 2.6.37. bisect shows this as the culprit: I was not aware of the thread. Please cc me directly,

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-02-24 Thread Avi Kivity
On 02/24/2011 01:42 AM, Nikola Ciprich wrote: Hello Avi et al, seems like I've hit regression in 2.6.37: 32bit SMP centos guest stopped booting, they just hang during initrd phase. (haven't tried different distros) UP guest are OK. when I (forcibly) compiled kvm-kmod-2.6.36.2 and used it in

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-02-24 Thread Nikola Ciprich
On Thu, Feb 24, 2011 at 12:17:40PM +0200, Avi Kivity wrote: On 02/24/2011 01:42 AM, Nikola Ciprich wrote: Hello Avi et al, seems like I've hit regression in 2.6.37: 32bit SMP centos guest stopped booting, they just hang during initrd phase. (haven't tried different distros) UP guest are

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-02-24 Thread Avi Kivity
On 02/24/2011 12:48 PM, Nikola Ciprich wrote: On Thu, Feb 24, 2011 at 12:17:40PM +0200, Avi Kivity wrote: On 02/24/2011 01:42 AM, Nikola Ciprich wrote: Hello Avi et al, seems like I've hit regression in 2.6.37: 32bit SMP centos guest stopped booting, they just hang during initrd phase.

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-02-24 Thread Nikola Ciprich
Not very useful when the guest is making progress, I'm afraid. can perf report help here? Can you try a little trace-cmd -e kvm -b 2? ugh, I'm afraid I'll have some dumb questions here :-[ You mean this: git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git ? and then

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-02-24 Thread Avi Kivity
On 02/24/2011 01:27 PM, Nikola Ciprich wrote: Not very useful when the guest is making progress, I'm afraid. can perf report help here? Can you try a little trace-cmd -e kvm -b 2? ugh, I'm afraid I'll have some dumb questions here :-[ You mean this:

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-02-24 Thread Avi Kivity
On 02/24/2011 02:41 PM, Nikola Ciprich wrote: Yes. If you have udis86 and udis86-devel installed when building it, it's even better. yes, now I remember! I've done some tracing for You already.. You don't have to execute qemu-kvm under it, if you have a running instance you can run

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-02-24 Thread Nikola Ciprich
The only activity I can see is the timer interrupt, so I'm afraid a bisect is needed. OK, nevermind, it's easy to reproduce, so I'll just bisect it and report. n. If you let git bisect just kvm, it'll be a bit faster: $ git bisect $BAD $GOOD virt/kvm arch/x86/kvm -- error compiling

Re: regression - 2.6.36 - 2.6.37 - kvm - 32bit SMP guests don't boot

2011-02-24 Thread Nikola Ciprich
Yes. If you have udis86 and udis86-devel installed when building it, it's even better. yes, now I remember! I've done some tracing for You already.. You don't have to execute qemu-kvm under it, if you have a running instance you can run trace-cmd in parallel and it will record whatever's