Re: Enhance perf to support KVM
On 03/02/2010 07:17 PM, Ingo Molnar wrote: * Paolo Bonzini wrote: On 02/26/2010 03:23 PM, Ingo Molnar wrote: I do think tools/X and tools/libc would make quite a bit of sense - this is one of the better design aspects of FreeBSD et al. It's a mistake that it's not being done. I don't see what it would buy to have tools/libc. You cannot force users to update kernel-space and user-space in lockstep (Apple forced you to do that sometimes when I used Macs at work, and it was very very inconvenient), so it's not like libc would be able to always assume the latest system calls. There is (relatively) a lot of backwards-compatibility code in libc; it's ugly code, but you have to live with it. If glibc was part of the kernel repo with a klibc alike approach we wouldnt have such problems: the kernel would provide the system library and that's it. You'd not want to upgrade them separately, just like you generally wouldnt want to upgrade your memory management code separately from the scheduler either. The same argument could be made in a reverse fashion with just about any part of the kernel: 'it would be better to upgrade the ext3 driver separately' - etc. It's similar to all the classic microkernel versus monolithic kernel arguments. No, that's Documentation/stable_api_nonsense.txt. It's perfectly possible to have a filesystem driver that is decoupled from the kernel, yet both are in the same address space. There are costs with increasing size and increasing integration - but fact is that we can manage a 13+ MLOC kernel just fine and the benefits of an integrated approach far outweigh the costs. In my experience it's far better to have one project for 'infrastructure' bits: developed, tested and offered as one coherent entity in essence. A bit like how Xorg does it. These many splintered kernel facilities are historical legacies in essence (from the times when there was no single usable free OS available), and over-modularization has many costs and few advantages. You're probably right for core libraries, but I don't think a GUI for kvm qualifies. A server oriented application, maybe. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Enhance perf to support KVM
* Paolo Bonzini wrote: > On 03/02/2010 06:12 PM, Arnaldo Carvalho de Melo wrote: > >You imply lockstep updates because both are on the same source tree? I > >don't see why that would be required, there is an ABI contract to be > >respected no matter where the sources for the signatories live. > > It's not about ABIs, it's about not being able to rely exclusively on new > syscalls or flags (one for all: SOCK_NONBLOCK) because people can run new > glibc on old kernels. That is not what i was suggesting though. I am suggesting a klibc-alike approach, where glibc is not on the filesystem but mapped by the kernel, basically as an extension of the standard vDSO. We always have most of glibc paged in all the time, so there's no real RAM usage difference. (and ultra-embedded would sure make use of this to trim RAM usage in fact.) That way the C library comes with the kernel. New kernel, new/updated system library. Ingo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Enhance perf to support KVM
On 03/02/2010 06:12 PM, Arnaldo Carvalho de Melo wrote:
You imply lockstep updates because both are on the same source tree? I
don't see why that would be required, there is an ABI contract to be
respected no matter where the sources for the signatories live.
It's not about ABIs, it's about not being able to rely exclusively on
new syscalls or flags (one for all: SOCK_NONBLOCK) because people can
run new glibc on old kernels.
If libc had this luxury, it would possibly make a lot more sense to have
it in the kernel tree. But it doesn't.
Also, setting up a testing environment for glibc is not trivial, and the
intersection of people that care about developing the kernel _and_ glibc
is probably too small and opinionated to make this easier.
I could've done both the kernel and userspace bits in the same patch.
You could add a glibc submodule to your git tree, and ask on the git
list for help adding some kind of multi-destination support to
git-{format-patch,send-email}.
Paolo
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Enhance perf to support KVM
* Paolo Bonzini wrote: > On 02/26/2010 03:23 PM, Ingo Molnar wrote: > >I do think tools/X and tools/libc would make quite a bit of sense - this is > >one of the better design aspects of FreeBSD et al. It's a mistake that it's > >not being done. > > I don't see what it would buy to have tools/libc. You cannot force users to > update kernel-space and user-space in lockstep (Apple forced you to do that > sometimes when I used Macs at work, and it was very very inconvenient), so > it's not like libc would be able to always assume the latest system calls. > There is (relatively) a lot of backwards-compatibility code in libc; it's > ugly code, but you have to live with it. If glibc was part of the kernel repo with a klibc alike approach we wouldnt have such problems: the kernel would provide the system library and that's it. You'd not want to upgrade them separately, just like you generally wouldnt want to upgrade your memory management code separately from the scheduler either. The same argument could be made in a reverse fashion with just about any part of the kernel: 'it would be better to upgrade the ext3 driver separately' - etc. It's similar to all the classic microkernel versus monolithic kernel arguments. There are costs with increasing size and increasing integration - but fact is that we can manage a 13+ MLOC kernel just fine and the benefits of an integrated approach far outweigh the costs. In my experience it's far better to have one project for 'infrastructure' bits: developed, tested and offered as one coherent entity in essence. A bit like how Xorg does it. These many splintered kernel facilities are historical legacies in essence (from the times when there was no single usable free OS available), and over-modularization has many costs and few advantages. Thanks, Ingo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Enhance perf to support KVM
Em Tue, Mar 02, 2010 at 05:46:03PM +0100, Paolo Bonzini escreveu: > On 02/26/2010 03:23 PM, Ingo Molnar wrote: >> I do think tools/X and tools/libc would make quite a bit of sense - this is >> one of the better design aspects of FreeBSD et al. It's a mistake that it's >> not being done. > > I don't see what it would buy to have tools/libc. You cannot force > users to update kernel-space and user-space in lockstep (Apple forced > you to do that sometimes when I used Macs at work, and it was very very > inconvenient), so it's not like libc would be able to always assume the > latest system calls. There is (relatively) a lot of > backwards-compatibility code in libc; it's ugly code, but you have to > live with it. > > Any case in which you need lockstep upgrades of kernel and libc for > bisectability is a bug in libc, and I haven't seen it happen enough to > be a problem. You imply lockstep updates because both are on the same source tree? I don't see why that would be required, there is an ABI contract to be respected no matter where the sources for the signatories live. perf lives in the kernel sources and while developing it I use all sorts of combinations of userland and kernel perf bits so as not to be rebooting the many machines of different arches where I test the userspace bits all the time with a new kernel, even if it indeed has new in-kernel perf bits. As an instance where I could have worked on tools/libc recently was the recvmmsg syscall I introduced. Till it got into glibc (IIRC it was, but I wasn't involved in it, google alert told me) interested parties had to use 'man syscall' and pick the syscall number for their specific architecture, etc. I could've done both the kernel and userspace bits in the same patch. - Arnaldo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Enhance perf to support KVM
On 02/26/2010 03:23 PM, Ingo Molnar wrote: I do think tools/X and tools/libc would make quite a bit of sense - this is one of the better design aspects of FreeBSD et al. It's a mistake that it's not being done. I don't see what it would buy to have tools/libc. You cannot force users to update kernel-space and user-space in lockstep (Apple forced you to do that sometimes when I used Macs at work, and it was very very inconvenient), so it's not like libc would be able to always assume the latest system calls. There is (relatively) a lot of backwards-compatibility code in libc; it's ugly code, but you have to live with it. Any case in which you need lockstep upgrades of kernel and libc for bisectability is a bug in libc, and I haven't seen it happen enough to be a problem. Paolo ps: does it make sense to keep such a long list of recipients??? -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Enhance perf to support KVM
On 02/26/2010 04:35 AM, Ingo Molnar wrote: Basically what is needed is plain filesystem access - properly privileged. So doing this via a vmchannel would be nice, but for the symbol extraction it would be a glorified NFS server in essence. Do you have (or plan) any turn-key 'access to all files of the guest' kind of guest-transparent facility that could be used for such purposes? That would have various advantages over a traditional explicit file server approach: - it would not contaminate the guest port space - no guest side configuration needed (the various oprofile remote daemons always sucked as they needed extra setup) - it might even be used with a guest that does no networking If you look at VMware, they provide a guest daemon that runs in userspace and communicates with the host to allow the host to do operations like execute a binary or read a file. We've been trying to build this for years and virtio-console is what we've finally settled on. The userspace bits are super easy to build. The problem is that we need guest kernel support. We've not had a lot of success there as we've had a number of things Nacked. The approach we originally started with was to introduce a new socket family for virtualization. This is what s390 does and this is what VMware does. But Dave Miller nacked that. VMware happily carries their own module so they don't have to deal with problems like this. I guess s390 was able to get away with it because noone pays attention to what they do. Rusty had a proposal very similar to vhost-net that he posted two years ago but Andrew Morton nacked that because it wasn't generic enough (serves us right for trying to do anything generic in the first place). Honestly, what's hardest for us to build a turn key solution is that unlike our competition, we've got to deal with getting people who don't necessarily care about virtualization to agree with what we're doing as opposed to just maintaining our own set of kernel modules. Things limited to userspace have not slowed us down compared to getting things into the kernel. Look at how difficult it was to get MMU notifiers merged as another example. If you're volunteering to commit the things that we need to build a turn key solution, then that would certainly be a big help to us. Regards, Anthony Liguori -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Enhance perf to support KVM
On 02/26/2010 01:17 PM, Ingo Molnar wrote: Nobody is really 'in charge' of how KVM gets delivered to the user. You isolated the fun kernel part for you and pushed out the boring bits to user-space. So if mundane things like mouse integration sucks 'hey that's a user-space tooling problem', if file integration sucks then 'hey, that's an admin problem', if it cannot be used over the network 'hey, that's an Xorg problem', etc. etc. btw, mouse integration works with -usbdevice tablet and recent Fedoras, 'it was an X.org driver problem'. Really, I don't understand your problems. -- Do not meddle in the internals of kernels, for they are subtle and quick to panic. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Enhance perf to support KVM
On 02/26/2010 04:23 PM, Ingo Molnar wrote: * Avi Kivity wrote: On 02/26/2010 03:16 PM, Ingo Molnar wrote: * Avi Kivity wrote: That was not what i suggested tho. tools/kvm/ would work plenty fine. I'll wait until we have tools/libc and tools/X. After all, they affect a lot more people and are concerned with a lot more kernel/user interfaces than kvm. So your answer can be summed up as: 'we wont do what makes sense technically because others suck even more' ? I can sum up your this remark as 'whenever you disagree with me, I will rephrase your words to make you look like an idiot'. Two points: 1) You can try to ridicule me if you want, I'd much prefer it if if no ridiculing was employed on either side. but do you actually claim that my summary is inaccurate? I do claim it's a substantially accurate summary: you said you will (quote:) "wait with tools/kvm/ until we have tools/libc and tools/X". I do think tools/X and tools/libc would make quite a bit of sense - this is one of the better design aspects of FreeBSD et al. It's a mistake that it's not being done. There are arguments for libc to be developed in linux-2.6.git, and arguments against. The fact is that they are not, so presumably the arguments against plus inertia outweigh the arguments for. The same logic holds for kvm, except that there are less arguments for development in linux-2.6.git. Only a small part of qemu is actually concerned with kvm; most of it is mucking around with X, emulating old devices, emulating instruction sets (irrelevant for tools/kvm) and doing boring managementy stuff. Do we really want to add several hundered thousand lines to Linux, only a few thousand or of which talk to the kernel? 2) I used a question mark (the sentence was not a statement of fact), and you have no obligation to agree with the summary i provided. Thanks. I hope you don't agree with it either. -- Do not meddle in the internals of kernels, for they are subtle and quick to panic. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Enhance perf to support KVM
* Avi Kivity wrote: > On 02/26/2010 03:16 PM, Ingo Molnar wrote: > >* Avi Kivity wrote: > > > >>> That was not what i suggested tho. tools/kvm/ would work plenty fine. > >>> > >> I'll wait until we have tools/libc and tools/X. After all, they affect a > >> lot more people and are concerned with a lot more kernel/user interfaces > >> than kvm. > > > > So your answer can be summed up as: 'we wont do what makes sense > > technically because others suck even more' ? > > I can sum up your this remark as 'whenever you disagree with me, I will > rephrase your words to make you look like an idiot'. Two points: 1) You can try to ridicule me if you want, but do you actually claim that my summary is inaccurate? I do claim it's a substantially accurate summary: you said you will (quote:) "wait with tools/kvm/ until we have tools/libc and tools/X". I do think tools/X and tools/libc would make quite a bit of sense - this is one of the better design aspects of FreeBSD et al. It's a mistake that it's not being done. 2) I used a question mark (the sentence was not a statement of fact), and you have no obligation to agree with the summary i provided. Ingo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Enhance perf to support KVM
On 02/26/2010 03:16 PM, Ingo Molnar wrote: * Avi Kivity wrote: That was not what i suggested tho. tools/kvm/ would work plenty fine. I'll wait until we have tools/libc and tools/X. After all, they affect a lot more people and are concerned with a lot more kernel/user interfaces than kvm. So your answer can be summed up as: 'we wont do what makes sense technically because others suck even more' ? I can sum up your this remark as 'whenever you disagree with me, I will rephrase your words to make you look like an idiot'. If you believe I'm an idiot, there's no need to have this (or any) conversation. If not, please refrain from this type of verbal gymnastics. And it's not just the kernel<->user interface (which btw., for the case of X is far narrower than what KVM currently has to Qemu). The issue is a basic question of software design: does kvm-qemu really make as much sense without the kernel component as with it? The answer is: it will borderline-work with CPU emulation (and i'm sure there are people making use of it that way), but 90%+ of the userbase uses it with KVM and vice versa. It is really a single logical component as far as maintenance goes, and tools/kvm/ would make quite a bit of sense. There are two separate questions. Is there room for a kvm-only userspace component? I believe so, but throwing away the momentum behind qemu would be foolish. Does it make sense for such a component to live in linux.git? IMO, no, and certainly a lot less than libc and X. -- Do not meddle in the internals of kernels, for they are subtle and quick to panic. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Enhance perf to support KVM
On 02/26/10 14:16, Ingo Molnar wrote: * Avi Kivity wrote: That was not what i suggested tho. tools/kvm/ would work plenty fine. I'll wait until we have tools/libc and tools/X. After all, they affect a lot more people and are concerned with a lot more kernel/user interfaces than kvm. So your answer can be summed up as: 'we wont do what makes sense technically because others suck even more' ? Well in this discussion what makes sense technically differs depending on who you ask. I will argue that emulating the MSR access doesn't make sense technically because there is no fixed specification we can rely on, since the spec seems to change randomly with every cpu family release from Inte. In addition the overhead is making the resulting numbers less if at all interesting. Jes -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Enhance perf to support KVM
* Avi Kivity wrote: > > That was not what i suggested tho. tools/kvm/ would work plenty fine. > > I'll wait until we have tools/libc and tools/X. After all, they affect a > lot more people and are concerned with a lot more kernel/user interfaces > than kvm. So your answer can be summed up as: 'we wont do what makes sense technically because others suck even more' ? And it's not just the kernel<->user interface (which btw., for the case of X is far narrower than what KVM currently has to Qemu). The issue is a basic question of software design: does kvm-qemu really make as much sense without the kernel component as with it? The answer is: it will borderline-work with CPU emulation (and i'm sure there are people making use of it that way), but 90%+ of the userbase uses it with KVM and vice versa. It is really a single logical component as far as maintenance goes, and tools/kvm/ would make quite a bit of sense. But i digress. Ingo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Enhance perf to support KVM
On 02/26/2010 02:46 PM, Ingo Molnar wrote: * Avi Kivity wrote: You basically have given up control over the quality of KVM by pushing so many aspects of it to user-space and letting it rot there. That's wrong on so many levels. First, nothing is rotting in userspace, qemu is evolving faster than kvm is. If I pushed it into the kernel then development pace would be much slower (since kernel development is harder), quality would be lower (less infrastructure, any bug is a host crash or security issue), and I personally would be totally swamped. That was not what i suggested tho. tools/kvm/ would work plenty fine. I'll wait until we have tools/libc and tools/X. After all, they affect a lot more people and are concerned with a lot more kernel/user interfaces than kvm. As i said: [...] You are pushing _way_ too much to user-space into different modules and maintenance domains, [...] ( Note that i dont mind user-space tooling per se, as long as it sits together with the kernel bits and gets developed, packaged and given to the user in the same domain. ) [...] Sure the design looks somewhat cleaner on paper, but if the end result is not helped by it then over-modularization sure can hurt ... Run 'rpm -qa' one of these days. Modern software is modular, that's the only way to manage it. Of course rpm -qa shows cases where modularization works. But my point was over-modularization, which due to the KVM/qemu split we all suffer from. You're the only one who suffers from it. Everyone else is happy with adding features in the modules that implements them, be it kvm, qemu, libvirt, or virt-manager (to name one tool stack out of several). Modularizing along the wrong interface is worse than not modularizing something that could be. So when designing software you generally want to err on the side of _under_-modularizing. It's always very easy to split stuff up, when there's a really strong technical argument for it. It's very hard to pull the broken pieces back together though once they are in difference domains of maintanence - as then it's usually social integration that has to happen, which is always harder than a technical split-up. As it happens, the kvm and qemu development community has a large overlap. Many developers read both lists, contribute to both projects, and participate on the same weekly call. While we had difficulties pushing patches to qemu in the past, that's behind us, and qemu is now accepting patches at a much higher rate than kvm. Technically, it is obvious that the userspace and kernel components are separate projects. All that remains is the social divide. Since everyone (except you) is mostly happy, I see no reason to change. -- Do not meddle in the internals of kernels, for they are subtle and quick to panic. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Enhance perf to support KVM
* Avi Kivity wrote: > > You basically have given up control over the quality of KVM by pushing so > > many aspects of it to user-space and letting it rot there. > > That's wrong on so many levels. First, nothing is rotting in userspace, > qemu is evolving faster than kvm is. If I pushed it into the kernel then > development pace would be much slower (since kernel development is harder), > quality would be lower (less infrastructure, any bug is a host crash or > security issue), and I personally would be totally swamped. That was not what i suggested tho. tools/kvm/ would work plenty fine. As i said: > > [...] You are pushing _way_ too much to user-space into different modules > > and maintenance domains, [...] > > > > ( Note that i dont mind user-space tooling per se, as long as it sits > > together > > with the kernel bits and gets developed, packaged and given to the user > > in the same domain. ) [...] > > Sure the design looks somewhat cleaner on paper, but if the end result is > > not helped by it then over-modularization sure can hurt ... > > Run 'rpm -qa' one of these days. Modern software is modular, that's the > only way to manage it. Of course rpm -qa shows cases where modularization works. But my point was over-modularization, which due to the KVM/qemu split we all suffer from. Modularizing along the wrong interface is worse than not modularizing something that could be. So when designing software you generally want to err on the side of _under_-modularizing. It's always very easy to split stuff up, when there's a really strong technical argument for it. It's very hard to pull the broken pieces back together though once they are in difference domains of maintanence - as then it's usually social integration that has to happen, which is always harder than a technical split-up. Ingo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Enhance perf to support KVM
On 02/26/2010 01:48 PM, Peter Zijlstra wrote: On Fri, 2010-02-26 at 12:47 +0200, Avi Kivity wrote: Not really. The guest and host admins are usually different people, who may, being admins, even actively hate each other. The guest admin would probably regard it as a security hole. It's probably useful for the single-host scenario, and of course for developers. LOL, let me be the malicious host admin, then you can be the guest, there is no way you can protect yourself. If you don't trust the host, don't use it. All your IO flows through the host, all your sekrit keys are in memory, there is no security. That's true. But guest admins are going to be unhappy about a file server serving their data to the host all the same. -- Do not meddle in the internals of kernels, for they are subtle and quick to panic. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Enhance perf to support KVM
On Fri, 2010-02-26 at 12:47 +0200, Avi Kivity wrote: > Not really. The guest and host admins are usually different people, who > may, being admins, even actively hate each other. The guest admin would > probably regard it as a security hole. It's probably useful for the > single-host scenario, and of course for developers. LOL, let me be the malicious host admin, then you can be the guest, there is no way you can protect yourself. If you don't trust the host, don't use it. All your IO flows through the host, all your sekrit keys are in memory, there is no security. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Enhance perf to support KVM
On 02/26/2010 01:17 PM, Ingo Molnar wrote: * Avi Kivity wrote: Do you have (or plan) any turn-key 'access to all files of the guest' kind of guest-transparent facility that could be used for such purposes? Not really. The guest and host admins are usually different people, who may, being admins, even actively hate each other. The guest admin would probably regard it as a security hole. It's probably useful for the single-host scenario, and of course for developers. Sounds like an exceedingly silly argument to me - the host admin is the king in any case. Your argument boils down to: 'dont offer transparent, turn-key solutions because some might object to the functionality they offer for all the wrong reasons'. Which does not withstand elementary scrutiny. Again, the host admin and the guest admin are different people. What would the host admin do with guest files? Why would the guest admin want to run any code that exposes their files? This is a basic usability issue, and affects many parts of the KVM universe. Really, it's by far the most fubar-ed notion of KVM. You are pushing _way_ too much to user-space into different modules and maintenance domains, and user-space forks those bits, fragments, diverts, delays and messes up basic features in the usual fashion. The result is a basic out-of-box virtualization experience that sucks even these days. Nobody is really 'in charge' of how KVM gets delivered to the user. You isolated the fun kernel part for you and pushed out the boring bits to user-space. So if mundane things like mouse integration sucks 'hey that's a user-space tooling problem', if file integration sucks then 'hey, that's an admin problem', if it cannot be used over the network 'hey, that's an Xorg problem', etc. etc. What would you have me do? Push 200K lines of device emulation code into the kernel? Write an X client, toolkit, and display in the kernel so that mouse integration works out of the box when you install Linux 2.6.653? As to "nobody is in charge", that's really insulting to the people who are in charge of the userspace components. Perhaps the problems that we see are not the same problems that you see. It might be that direct access to guest files from the host is only a pressing problem for you, but nobody else. If there are features that you miss, post patches, if you will deign to code for lowly user space. You basically have given up control over the quality of KVM by pushing so many aspects of it to user-space and letting it rot there. That's wrong on so many levels. First, nothing is rotting in userspace, qemu is evolving faster than kvm is. If I pushed it into the kernel then development pace would be much slower (since kernel development is harder), quality would be lower (less infrastructure, any bug is a host crash or security issue), and I personally would be totally swamped. Sure the design looks somewhat cleaner on paper, but if the end result is not helped by it then over-modularization sure can hurt ... Run 'rpm -qa' one of these days. Modern software is modular, that's the only way to manage it. ( Note that i dont mind user-space tooling per se, as long as it sits together with the kernel bits and gets developed, packaged and given to the user in the same domain. ) Call me when glibc, the X servers and clients, and everything else qemu now uses is developed, packaged, and given to the user in the same domain. And that's a key conceptual area were tools/perf/ differs: it's an integrated, turn-key solution that you can really rely on. We take responsibility for the full thing, no ifs and when. And if you cannot rely on your instrumentation tooling as a single unit you cannot use it, simple as that. (that is a key mistake Oprofile made a decade ago too btw.) perf is a tool written by developers for developers. kvm is written for users (most of them hidden behind management interfaces). There's no point at all in shipping it as part of the kernel, users don't install and use kernels, they install and use distributions. So i can see some upcoming culture friction with standing KVM principles there ;-) No friction at all - I don't think any kvm developer agrees with you (but if anyone does please speak up). -- Do not meddle in the internals of kernels, for they are subtle and quick to panic. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Enhance perf to support KVM
* Avi Kivity wrote: > > Do you have (or plan) any turn-key 'access to all files of the guest' kind > > of guest-transparent facility that could be used for such purposes? > > Not really. The guest and host admins are usually different people, who > may, being admins, even actively hate each other. The guest admin would > probably regard it as a security hole. It's probably useful for the > single-host scenario, and of course for developers. Sounds like an exceedingly silly argument to me - the host admin is the king in any case. Your argument boils down to: 'dont offer transparent, turn-key solutions because some might object to the functionality they offer for all the wrong reasons'. Which does not withstand elementary scrutiny. This is a basic usability issue, and affects many parts of the KVM universe. Really, it's by far the most fubar-ed notion of KVM. You are pushing _way_ too much to user-space into different modules and maintenance domains, and user-space forks those bits, fragments, diverts, delays and messes up basic features in the usual fashion. The result is a basic out-of-box virtualization experience that sucks even these days. Nobody is really 'in charge' of how KVM gets delivered to the user. You isolated the fun kernel part for you and pushed out the boring bits to user-space. So if mundane things like mouse integration sucks 'hey that's a user-space tooling problem', if file integration sucks then 'hey, that's an admin problem', if it cannot be used over the network 'hey, that's an Xorg problem', etc. etc. You basically have given up control over the quality of KVM by pushing so many aspects of it to user-space and letting it rot there. Sure the design looks somewhat cleaner on paper, but if the end result is not helped by it then over-modularization sure can hurt ... ( Note that i dont mind user-space tooling per se, as long as it sits together with the kernel bits and gets developed, packaged and given to the user in the same domain. ) And that's a key conceptual area were tools/perf/ differs: it's an integrated, turn-key solution that you can really rely on. We take responsibility for the full thing, no ifs and when. And if you cannot rely on your instrumentation tooling as a single unit you cannot use it, simple as that. (that is a key mistake Oprofile made a decade ago too btw.) So i can see some upcoming culture friction with standing KVM principles there ;-) Ingo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Enhance perf to support KVM
On 02/26/2010 12:35 PM, Ingo Molnar wrote: One additional step needed is to get symbol information from the guest, and to integrate it into the symbol cache on the host side in ~/.debug. We already support cross-arch symbols and 'perf archive', so the basic facilities are there for that. So you can profile on 32-bit PA-RISC and type 'perf report' on 64-bit x86 and get all the right info. For this to work across a guest, a gateway is needed towards the guest. There's several ways to achieve this. The most practical would be two steps: - a user-space facility to access guest images/libraries. (say via ssh, or just a plain TCP port) This would be useful for general 'remote profiling' sessions as well, so it's not KVM specific - it would be useful for remote debugging. - The guest /proc/kallsyms (and vmlinux) could be accessed via that channel as well. (Note that this is purely for guest symbol space access - all the profiling data itself comes via the host kernel.) In theory we could build some sort of 'symbol server' facility into the kernel, which could be enabled in guest kernels too - but i suspect existing, user-space transports go most of the way already. There is also vmchannel aka virtio-serial, a guest-to-host communication channel. Basically what is needed is plain filesystem access - properly privileged. So doing this via a vmchannel would be nice, but for the symbol extraction it would be a glorified NFS server in essence. Well, we could run an nfs server over vmchannel, or over a private network interface. Do you have (or plan) any turn-key 'access to all files of the guest' kind of guest-transparent facility that could be used for such purposes? Not really. The guest and host admins are usually different people, who may, being admins, even actively hate each other. The guest admin would probably regard it as a security hole. It's probably useful for the single-host scenario, and of course for developers. I guess sshfs can fill this role, with one command it gives you secure access to all guest files, provided you have the proper credentials. That would have various advantages over a traditional explicit file server approach: - it would not contaminate the guest port space - no guest side configuration needed (the various oprofile remote daemons always sucked as they needed extra setup) - it might even be used with a guest that does no networking - if done fully in the kernel it could be done with a fully 'unaware' guest, etc. Seems sshfs fulfils the first two. For the latter, we could do a vmchannelfs, but it seems quite a bit of work, and would require fairly new guest kernels, whereas sshfs would work out of the box on 10 year old guests and can be easily made to work on Windows. Somewhat related, see libguestfs/guestfish, though that provides offline access only. -- Do not meddle in the internals of kernels, for they are subtle and quick to panic. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Enhance perf to support KVM
* Avi Kivity wrote: > On 02/26/2010 11:01 AM, Ingo Molnar wrote: > >* Zhang, Yanmin wrote: > > > >>2) We couldn't get guest os kernel/user stack data in an easy way, so we > >>might not support callchain feature of tool perf. A work around is KVM > >>copies kernel stack data out, so we could at least support guest os kernel > >>callchain. > >If the guest is Linux, KVM can get all the info we need. > > > >While the PMU event itself might trigger in an NMI (where we cannot access > >most of KVM's data structures safely), for this specific case of KVM > >instrumentation we can delay the processing to a more appropriate time - in > >fact we can do it in the KVM thread itself. > > The nmi will be a synchronous event: it happens in guest context, > and we program the hardware to intercept nmis, so we just get an > exit telling us that an nmi has happened. > > (would also be interesting to allow the guest to process the nmi > directly in some scenarios, though that would require that there be > no nmi sources on the host). > > >We can do that because we just triggered a VM exit, so the VM state is for > >all > >purposes frozen (as far as this virtual CPU goes). > > Yes. > > >Which egives us plenty of time and opportunity to piggy back to the KVM > >thread, look up the guest stack, process/fill the MMU cache as we walk the > >guest page tables, etc. etc. > > > >It would need some minimal callback facility towards KVM, triggered by a perf > >event PMI. > > Since the event is synchronous and kvm is aware of it we don't need > a callback; kvm can call directly into perf with all the > information. Yes - it's still a "callback" in the abstract sense. Much of it already all existing. > >One additional step needed is to get symbol information from the guest, and > >to > >integrate it into the symbol cache on the host side in ~/.debug. We already > >support cross-arch symbols and 'perf archive', so the basic facilities are > >there for that. So you can profile on 32-bit PA-RISC and type 'perf report' > >on > >64-bit x86 and get all the right info. > > > >For this to work across a guest, a gateway is needed towards the guest. > >There's several ways to achieve this. The most practical would be two steps: > > > > - a user-space facility to access guest images/libraries. (say via ssh, or > >just a plain TCP port) This would be useful for general 'remote > > profiling' > >sessions as well, so it's not KVM specific - it would be useful for > > remote > >debugging. > > > > - The guest /proc/kallsyms (and vmlinux) could be accessed via that channel > >as well. > > > >(Note that this is purely for guest symbol space access - all the profiling > >data itself comes via the host kernel.) > > > >In theory we could build some sort of 'symbol server' facility into the > >kernel, which could be enabled in guest kernels too - but i suspect existing, > >user-space transports go most of the way already. > > There is also vmchannel aka virtio-serial, a guest-to-host communication > channel. Basically what is needed is plain filesystem access - properly privileged. So doing this via a vmchannel would be nice, but for the symbol extraction it would be a glorified NFS server in essence. Do you have (or plan) any turn-key 'access to all files of the guest' kind of guest-transparent facility that could be used for such purposes? That would have various advantages over a traditional explicit file server approach: - it would not contaminate the guest port space - no guest side configuration needed (the various oprofile remote daemons always sucked as they needed extra setup) - it might even be used with a guest that does no networking - if done fully in the kernel it could be done with a fully 'unaware' guest, etc. Thanks, Ingo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Enhance perf to support KVM
On 02/26/2010 11:01 AM, Ingo Molnar wrote: * Zhang, Yanmin wrote: 2) We couldn't get guest os kernel/user stack data in an easy way, so we might not support callchain feature of tool perf. A work around is KVM copies kernel stack data out, so we could at least support guest os kernel callchain. If the guest is Linux, KVM can get all the info we need. While the PMU event itself might trigger in an NMI (where we cannot access most of KVM's data structures safely), for this specific case of KVM instrumentation we can delay the processing to a more appropriate time - in fact we can do it in the KVM thread itself. The nmi will be a synchronous event: it happens in guest context, and we program the hardware to intercept nmis, so we just get an exit telling us that an nmi has happened. (would also be interesting to allow the guest to process the nmi directly in some scenarios, though that would require that there be no nmi sources on the host). We can do that because we just triggered a VM exit, so the VM state is for all purposes frozen (as far as this virtual CPU goes). Yes. Which egives us plenty of time and opportunity to piggy back to the KVM thread, look up the guest stack, process/fill the MMU cache as we walk the guest page tables, etc. etc. It would need some minimal callback facility towards KVM, triggered by a perf event PMI. Since the event is synchronous and kvm is aware of it we don't need a callback; kvm can call directly into perf with all the information. One additional step needed is to get symbol information from the guest, and to integrate it into the symbol cache on the host side in ~/.debug. We already support cross-arch symbols and 'perf archive', so the basic facilities are there for that. So you can profile on 32-bit PA-RISC and type 'perf report' on 64-bit x86 and get all the right info. For this to work across a guest, a gateway is needed towards the guest. There's several ways to achieve this. The most practical would be two steps: - a user-space facility to access guest images/libraries. (say via ssh, or just a plain TCP port) This would be useful for general 'remote profiling' sessions as well, so it's not KVM specific - it would be useful for remote debugging. - The guest /proc/kallsyms (and vmlinux) could be accessed via that channel as well. (Note that this is purely for guest symbol space access - all the profiling data itself comes via the host kernel.) In theory we could build some sort of 'symbol server' facility into the kernel, which could be enabled in guest kernels too - but i suspect existing, user-space transports go most of the way already. There is also vmchannel aka virtio-serial, a guest-to-host communication channel. -- Do not meddle in the internals of kernels, for they are subtle and quick to panic. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Enhance perf to support KVM
* Zhang, Yanmin wrote: > 2) We couldn't get guest os kernel/user stack data in an easy way, so we > might not support callchain feature of tool perf. A work around is KVM > copies kernel stack data out, so we could at least support guest os kernel > callchain. If the guest is Linux, KVM can get all the info we need. While the PMU event itself might trigger in an NMI (where we cannot access most of KVM's data structures safely), for this specific case of KVM instrumentation we can delay the processing to a more appropriate time - in fact we can do it in the KVM thread itself. We can do that because we just triggered a VM exit, so the VM state is for all purposes frozen (as far as this virtual CPU goes). Which egives us plenty of time and opportunity to piggy back to the KVM thread, look up the guest stack, process/fill the MMU cache as we walk the guest page tables, etc. etc. It would need some minimal callback facility towards KVM, triggered by a perf event PMI. One additional step needed is to get symbol information from the guest, and to integrate it into the symbol cache on the host side in ~/.debug. We already support cross-arch symbols and 'perf archive', so the basic facilities are there for that. So you can profile on 32-bit PA-RISC and type 'perf report' on 64-bit x86 and get all the right info. For this to work across a guest, a gateway is needed towards the guest. There's several ways to achieve this. The most practical would be two steps: - a user-space facility to access guest images/libraries. (say via ssh, or just a plain TCP port) This would be useful for general 'remote profiling' sessions as well, so it's not KVM specific - it would be useful for remote debugging. - The guest /proc/kallsyms (and vmlinux) could be accessed via that channel as well. (Note that this is purely for guest symbol space access - all the profiling data itself comes via the host kernel.) In theory we could build some sort of 'symbol server' facility into the kernel, which could be enabled in guest kernels too - but i suspect existing, user-space transports go most of the way already. (the only disadvantage of existing transports is that they all have to be configured, enabled and made user-accessible, which is one of the few weak points of KVM in general.) Thanks, Ingo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Enhance perf to support KVM
On Thu, 2010-02-25 at 10:20 +0100, Peter Zijlstra wrote: > On Thu, 2010-02-25 at 11:27 +0800, Zhang, Yanmin wrote: > > Ingo, > > > > I did some testing with KVM virtualization. perf shows vmx_vcpu_run > > consumes more than 50% cpu time. Actually, the info is incorrect because > > when perf counter overflows and NMI is triggered, vm exit to function > > vmx_vcpu_run, then vmx_vcpu_run triggers a software NMI so perf event is > > notified. perf just checks regs which just saves the address of > > vmx_vcpu_run. > > > > I want to enhance perf to collect real guest os address. > > > > Below is the design. > > KVM uses multi-thread model. Every guest os is a process of multi-thread. > > > > 1) Kernel: > > Add a per_cpu var and some functions, so KVM records interrupted > > guest os address before triggering the software NMI. perf event would check > > the per_cpu var to use it if it's not zero, or just goes though the old > > path. > > > > 2) User space: Add a new parameter to perf-top and perf-report, such like > > -g pid:guest_os_vmlinux_path. Command perf parses the guest os kernel image > > to collect symbols. Change perf to summarize results based on pid. > > Another direction is to use the new parameter -g only when old parameter > > -p is defined. Perf just needs separate native kernel and guest os kernel. > I really appreciate your kind comments, and will contact you again in the future for help. > -g is already taken :-) We could use other flag or just -G. > > One thing I worry about is making sense of the guest data, it might be > possible to sorta make sense of the main kernel image, but after that > its going to be 'interesting' in deed. > > You're going to have to extend PERF_RECORD_MISC_* though, perhaps you > can reuse CPUMODE_UNKNOWN for GUEST. > > The callchain stuff already has GUEST context identifiers, however > determining KERNEL/USER context might be hard and interpreting it is > going to be harder still since we don't have map information for the > guest. Right. As for side #1 pointed in Ingo' email, we assume guest os is linux. We couldn't support all capabilities of perf on KVM from host side. 1) We couldn't get module and process mapping info in guest os in an easy way, so we can't support to collect guest kernel module and user space hot functions. A work around is user could get guest os /proc/kallsym and pass it to tool perf at host side so we could analyze module host functions. 2) We couldn't get guest os kernel/user stack data in an easy way, so we might not support callchain feature of tool perf. A work around is KVM copies kernel stack data out, so we could at least support guest os kernel callchain. So the host side perf support on guest os: perf kvm list perf kvm record# records the first running guest perf kvm stat # stats the first running KVM guest perf kvm top # shows the profile of the first running guest perf kvm trace # active the KVM specific tracepoints As for record, doesn't support to record guest os user space stack callchain and guest os user space hot functions. Yanmin -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
