RE: [Xen-devel] Does dom0 see all physical processors? (RE:[Xen-ia64-devel] SAL INFO virtualization)
(Orran/Jimi cc'ed, see question below...) I understand and sympathize with the need for dom0 to sometimes get and use information from each processor that is only available if dom0 is running on each processor. However, AFAIK, SMP guests are always gang-scheduled, correct? No, there's no need to strictly gang schedule, and the current scheduler makes no attempt to do so. It may generally be a decent thing to do, though. (If not, aren't there some very knotty research issues related to locking and forward progress?) You could end up preempting a vCPU holding a lock which could lead to daft behaviour of naïve spin locks. A number of possible workarounds have been prototyped, but since it doesn't seem to be much of a problem in practice nothing has been checked in. I wonder if not a problem in practice is more of an indication of lack of practice than lack of problem. I can see that the problem would be unlikely to occur with small numbers of processors and one SMP guests running a highly scalable SMP app (such as a web server), but I'll bet a real enterprise load of home-grown SMP apps running in a IT shop that's had big SMP boxes for years would see the problem more quickly, especially after multiple SMP guests are consolidated onto a single box. I believe ppc has paravirtualized spinlocks in their Linux kernel, though even this won't necessarily help with a poorly written SMP application. No data, admittedly, but perhaps our good buddies at Watson could comment? So on a 16-processor system, every time dom0 needs to run (e.g. to handle backend I/O for any one of perhaps hundreds of domains), *every* domain gets descheduled so that dom0 can be (gang-)scheduled on all 16 processors? If true, this sounds like a _horrible_ performance hit, so I hope I'm misunderstanding something... This isn't an issue. After booting you probably want dom0 to give up all but 1 vCPU anyway. Unless of course the PCPU's have data that change over time, such as variable cycle rate (for power management) or hot-plug memory... Ian Dan ___ Xen-ia64-devel mailing list Xen-ia64-devel@lists.xensource.com http://lists.xensource.com/xen-ia64-devel
Re: [Xen-devel] Does dom0 see all physical processors? (RE:[Xen-ia64-devel] SAL INFO virtualization)
On 5 Apr 2006, at 15:07, Magenheimer, Dan (HP Labs Fort Collins) wrote: I believe ppc has paravirtualized spinlocks in their Linux kernel, though even this won't necessarily help with a poorly written SMP application. No data, admittedly, but perhaps our good buddies at Watson could comment? IBM did some investigation into this a while back. The conclusion then was that, even in a microbenchmark somewhat contrived to try and indicate 'worst case' spinlock behaviour, the benefit of smarter spinlocks was negligible. Maybe the benchmark was bad, maybe there are other pathological worst cases out there that the benchmark did not represent, or maybe the smart spinlocks should have been smarter. Whatever: the numbers we have so far do not indicate that there's a problem to be solved, and these potential multi-processor optimisations have to be empirically driven because intuition is so frequently off the mark. -- Keir ___ Xen-ia64-devel mailing list Xen-ia64-devel@lists.xensource.com http://lists.xensource.com/xen-ia64-devel
RE: [Xen-devel] Does dom0 see all physical processors? (RE:[Xen-ia64-devel] SAL INFO virtualization)
I believe ppc has paravirtualized spinlocks in their Linux kernel, though even this won't necessarily help with a poorly written SMP application. We have an equivalent of this (bad pre-emption mitigation), along with an alternative (bad pre-emption avoidance). Both have various pros and cons, and can be shown to offer significant benefits in various contrived benchmarks. We have benchmark code that records when these bad pre-emptions happen, and the workloads we've looked at we haven't been moved to check either scheme in. That's not to say that we won't have to at some point in the future, but the limits to scalability are currently elsewhere. So on a 16-processor system, every time dom0 needs to run (e.g. to handle backend I/O for any one of perhaps hundreds of domains), *every* domain gets descheduled so that dom0 can be (gang-)scheduled on all 16 processors? If true, this sounds like a _horrible_ performance hit, so I hope I'm misunderstanding something... This isn't an issue. After booting you probably want dom0 to give up all but 1 vCPU anyway. Unless of course the PCPU's have data that change over time, such as variable cycle rate (for power management) or hot-plug memory... Such events don't happen very often -- dom0 is still free to run something on a given CPU whenever it wants to. Ian ___ Xen-ia64-devel mailing list Xen-ia64-devel@lists.xensource.com http://lists.xensource.com/xen-ia64-devel
Re: [Xen-devel] Does dom0 see all physical processors? (RE: [Xen-ia64-devel] SAL INFO virtualization)
On 4 Apr 2006, at 03:17, Tian, Kevin wrote: Then consider your question about a large box with many processors. How about the real environment? Is it the case to provide a 16-way SMP box, or a 16-way NUMA box? I prefer to the latter. If it's a NUMA box, dom0 sees physical ACPI table and can be configured as NUMA aware. This is a model we must support if we are to have domain0 handle other processor-related ACPI activities (e.g., power management). The power information and available settings won't make much sense to the user unless there's an equivalence between VCPUs and PCPUs for domain0. -- Keir ___ Xen-ia64-devel mailing list Xen-ia64-devel@lists.xensource.com http://lists.xensource.com/xen-ia64-devel
RE: [Xen-devel] Does dom0 see all physical processors? (RE: [Xen-ia64-devel] SAL INFO virtualization)
From: Keir Fraser [mailto:[EMAIL PROTECTED] Sent: 2006年4月4日 15:26 On 4 Apr 2006, at 03:17, Tian, Kevin wrote: Then consider your question about a large box with many processors. How about the real environment? Is it the case to provide a 16-way SMP box, or a 16-way NUMA box? I prefer to the latter. If it's a NUMA box, dom0 sees physical ACPI table and can be configured as NUMA aware. This is a model we must support if we are to have domain0 handle other processor-related ACPI activities (e.g., power management). The power information and available settings won't make much sense to the user unless there's an equivalence between VCPUs and PCPUs for domain0. -- Keir Yes, and thanks for making it clearer. (Originally I only came up performance reason for this model.) Thanks, Kevin ___ Xen-ia64-devel mailing list Xen-ia64-devel@lists.xensource.com http://lists.xensource.com/xen-ia64-devel
RE: [Xen-devel] Does dom0 see all physical processors? (RE: [Xen-ia64-devel] SAL INFO virtualization)
I understand and sympathize with the need for dom0 to sometimes get and use information from each processor that is only available if dom0 is running on each processor. However, AFAIK, SMP guests are always gang-scheduled, correct? (If not, aren't there some very knotty research issues related to locking and forward progress?) So on a 16-processor system, every time dom0 needs to run (e.g. to handle backend I/O for any one of perhaps hundreds of domains), *every* domain gets descheduled so that dom0 can be (gang-)scheduled on all 16 processors? If true, this sounds like a _horrible_ performance hit, so I hope I'm misunderstanding something... Dan -Original Message- From: Keir Fraser [mailto:[EMAIL PROTECTED] Sent: Tuesday, April 04, 2006 1:26 AM To: Tian, Kevin Cc: xen-devel; xen-ia64-devel@lists.xensource.com; Magenheimer, Dan (HP Labs Fort Collins); Tristan Gingold Subject: Re: [Xen-devel] Does dom0 see all physical processors? (RE: [Xen-ia64-devel] SAL INFO virtualization) On 4 Apr 2006, at 03:17, Tian, Kevin wrote: Then consider your question about a large box with many processors. How about the real environment? Is it the case to provide a 16-way SMP box, or a 16-way NUMA box? I prefer to the latter. If it's a NUMA box, dom0 sees physical ACPI table and can be configured as NUMA aware. This is a model we must support if we are to have domain0 handle other processor-related ACPI activities (e.g., power management). The power information and available settings won't make much sense to the user unless there's an equivalence between VCPUs and PCPUs for domain0. -- Keir ___ Xen-ia64-devel mailing list Xen-ia64-devel@lists.xensource.com http://lists.xensource.com/xen-ia64-devel
RE: [Xen-devel] Does dom0 see all physical processors? (RE:[Xen-ia64-devel] SAL INFO virtualization)
I understand and sympathize with the need for dom0 to sometimes get and use information from each processor that is only available if dom0 is running on each processor. However, AFAIK, SMP guests are always gang-scheduled, correct? No, there's no need to strictly gang schedule, and the current scheduler makes no attempt to do so. It may generally be a decent thing to do, though. (If not, aren't there some very knotty research issues related to locking and forward progress?) You could end up preempting a vCPU holding a lock which could lead to daft behaviour of naïve spin locks. A number of possible workarounds have been prototyped, but since it doesn't seem to be much of a problem in practice nothing has been checked in. So on a 16-processor system, every time dom0 needs to run (e.g. to handle backend I/O for any one of perhaps hundreds of domains), *every* domain gets descheduled so that dom0 can be (gang-)scheduled on all 16 processors? If true, this sounds like a _horrible_ performance hit, so I hope I'm misunderstanding something... This isn't an issue. After booting you probably want dom0 to give up all but 1 vCPU anyway. Ian ___ Xen-ia64-devel mailing list Xen-ia64-devel@lists.xensource.com http://lists.xensource.com/xen-ia64-devel
