[kvm-devel] KVM & VMX root mode
Hi, We have had several reports of users saying our product VirtualBox crashes their (Linux) host. It turned out that KVM is responsible, because it leaves the CPU in VMX root mode. Doing this changes the way the cpu operates: updates to reserved bits in control registers cause general protection faults. We hit such cases on two occasions; clearing the X86_CR4_VMXE bit in CR4 and turning off paging. I have added a workaround to detect VMX root mode, simply refuse to run and recommend that the user disables KVM. I would however appreciate if you could fix KVM instead. -- Kind regards / mit freundlichen Gruessen / Met vriendelijke groet Sander van Leeuwen innoTek GmbH http://www.innotek.de Germany - This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ ___ kvm-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] KVM & VMX root mode
Hi Avi, Our non-vmx mode fails, because the cpu is in vmx root mode. Two products that use vt-x for virtualization could perfectly co-exist if both comply with the way Intel recommends people to use vt-x. See figure 19.1 in chapter 19.4 of the 'Intel 64 and IA-32 Architectures Software Developer's Manual'. VirtualBox is programmed to follow these rules and therefor allows any other virtualizer to run side-by-side. Currently KVM prevents us from using our generic virtualization engine and does not allow anybody else to use the vt-x extensions (without explicitely leaving vmx root mode). As your product is included in the mainline Linux kernel and enabled by default, it would be nice if you could follow Intel's recommendations. Thank you. Avi Kivity wrote: > Sander van Leeuwen wrote: >> Hi, >> >> We have had several reports of users saying our product VirtualBox >> crashes their (Linux) host. It turned out that KVM is responsible, >> because >> it leaves the CPU in VMX root mode. >> >> Doing this changes the way the cpu operates: updates to reserved bits >> in control registers cause general protection faults. >> We hit such cases on two occasions; clearing the X86_CR4_VMXE bit in >> CR4 and turning off paging. >> >> I have added a workaround to detect VMX root mode, simply refuse to >> run and recommend that the user disables KVM. >> I would however appreciate if you could fix KVM instead. >> >> > > Simply '/sbin/rmmod kvm-intel' and kvm will exit vmx root mode. > > If VirtualBox uses vmx, then it can't be run side-by-side with kvm. -- Kind regards / mit freundlichen Gruessen / Met vriendelijke groet Sander van Leeuwen innoTek GmbH [EMAIL PROTECTED] http://www.innotek.de Germany - This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ ___ kvm-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] KVM & VMX root mode
Avi,
I'd like to make clear that we are talking about the situation where kvm
is *not* being used by any application.
It's my understanding that in some (many?) distributions kvm is actually
active on a permanent basis. I can't say whether this was done
by default or explicitly by the user. There is however no need to
activate vmx root mode when there is no client program that is using
the kvm extension.
From your comments I understand that you enable vmx root mode when the
kvm module is loaded. Regardless of whether an application
actually wishes to use the kvm extension.
Avi Kivity wrote:
> Sander van Leeuwen wrote:
>> Hi Avi,
>>
>> Our non-vmx mode fails, because the cpu is in vmx root mode.
>>
>> Two products that use vt-x for virtualization could perfectly
>> co-exist if both comply with the way Intel recommends people to use
>> vt-x.
>> See figure 19.1 in chapter 19.4 of the 'Intel 64 and IA-32
>> Architectures Software Developer's Manual'. VirtualBox is programmed to
>> follow these rules and therefor allows any other virtualizer to run
>> side-by-side.
>
> As far as I understand, kvm follows these rules. It enables vmx when
> loaded and disables then when unloaded.
>
>>
>> Currently KVM prevents us from using our generic virtualization
>> engine and does not allow anybody else to use the vt-x extensions
>> (without
>> explicitely leaving vmx root mode).
>
> Well, obviously kvm can't operate if you disable cr4.vmxe and/or
> switch paging off. The two solutions are not run-time compatible. I
> don't see why this is a problem as you should simply not run the
> product you aren't using, and everything should just work.
>
>>
>> As your product is included in the mainline Linux kernel and enabled
>> by default, it would be nice if you could follow Intel's
>> recommendations.
>
> kvm isn't enabled by default. It requires explicit user action to
> enter vmx mode ('modprobe kvm-intel').
>
>
--
Kind regards / mit freundlichen Gruessen / Met vriendelijke groet
Sander van Leeuwen
innoTek GmbH
[EMAIL PROTECTED]
http://www.innotek.de
Germany
-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
___
kvm-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] KVM & VMX root mode
Daniel P. Berrange wrote: > While one could in theory change KVM to only activate VMX when a guest is > started this wouldn't give a significant improvement to the user experiance. > Instead of being able to rely on KVM working, it now may or may not work > depending on whether you happen to have a VirtualBox guest running or not & > vica-verca. You still would not be able to run guests from both concurrently > which should surely be the real goal. The only way I see the latter being > achieved is if VirtualBox were to leverage the existing KVM kernel APIs to > access the underlying hardware virt capabilities. > You are mistaken. VT-x allows multiple clients to work concurrently. There's only one catch: all those clients should follow certain guidelines. -- Kind regards / mit freundlichen Gruessen / Met vriendelijke groet Sander van Leeuwen innoTek GmbH http://www.innotek.de Germany - This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ ___ kvm-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] KVM & VMX root mode
Avi Kivity wrote: Sander van Leeuwen wrote: You are mistaken. VT-x allows multiple clients to work concurrently. There's only one catch: all those clients should follow certain guidelines. Currently, kvm assumes that it is the only user that issues vmptrld instructions. There may me other such assumptions in the code. There is also the question of who issues the vmxon/vmxoff instructions and set the magic msr bits that enable kvm. You can't have two hypervisors doing that; that would be racy. It should be possible to abstract those bits out to something outside kvm, but I don't see that happening. VirtualBox currently always executes vmxon when it wants to execute some guest code. When leaving ring 0 it issues the corresponding vmxoff. While that may increase world switch overhead, it has not turned out to be an issue for us. I think it's a bit risky to use somebody else's vmxon pointer. You have a point with the msr updates though. -- Kind regards / mit freundlichen Gruessen / Met vriendelijke groet Sander van Leeuwen innoTek GmbH http://www.innotek.de Germany - This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/___ kvm-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] KVM & VMX root mode
Avi Kivity wrote: >> I think it's a bit risky to use somebody else's vmxon pointer. >> >> You have a point with the msr updates though. > I think the easiest way forward is for you to unload kvm-intel (or > kvm-amd) when you load, and to reload it when you unload. > You may want to reconsider as KVM breaks VMWare server 1.3 on 64 bits linux hosts as well. In exactly the same way as it breaks VirtualBox. We have tried this ourselves and others have reported the same: - http://www.vmware.com/community/message.jspa?messageID=718184 - http://www.redhat.com/archives/rhl-list/2007-June/msg02501.html -- Kind regards / mit freundlichen Gruessen / Met vriendelijke groet Sander van Leeuwen innoTek GmbH http://www.innotek.de Germany - This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ ___ kvm-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] KVM & VMX root mode
Avi Kivity wrote: > Sander van Leeuwen wrote: >> You may want to reconsider as KVM breaks VMWare server 1.3 on 64 bits >> linux hosts as well. In exactly the same way >> as it breaks VirtualBox. >> >> We have tried this ourselves and others have reported the same: >> - http://www.vmware.com/community/message.jspa?messageID=718184 >> - http://www.redhat.com/archives/rhl-list/2007-June/msg02501.html >> > > This is unsurprising. But I believe the same solution (rmmod > kvm-intel) will work for VMware as well as for VirtualBox, or anything > else that wants to use VT. > > Can you explain what is wrong with it? It is simple, requires no > kernel changes, and has no present and potential future performance > impact. It will work on all kernel versions, unlike any change you > may wish us to make. > This was just an added note. More meant for distribution maintainers who feel it is necessary to activate KVM by default. -- Kind regards / mit freundlichen Gruessen / Met vriendelijke groet Sander van Leeuwen innoTek GmbH http://www.innotek.de Germany - This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ ___ kvm-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] KVM & VMX root mode
Anthony Liguori wrote: On Tue, 2007-09-04 at 11:51 +0200, Sander van Leeuwen wrote: Avi Kivity wrote: I think it's a bit risky to use somebody else's vmxon pointer. You have a point with the msr updates though. I think the easiest way forward is for you to unload kvm-intel (or kvm-amd) when you load, and to reload it when you unload. You may want to reconsider as KVM breaks VMWare server 1.3 on 64 bits linux hosts as well. In exactly the same way as it breaks VirtualBox. Can VirtualBox and VMware server 1.3 on 64 bits run at the same time? In theory they can, although I haven't tried myself. We've heard from people that they managed to run the 32 bits version side by side. -- Kind regards / mit freundlichen Gruessen / Met vriendelijke groet Sander van Leeuwen innoTek GmbH http://www.innotek.de Germany - This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/___ kvm-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/kvm-devel
