Re: Xen (and others Hypervisors) how do they handle IPIs?
On 9/26/13 5:50 PM, Roger Pau Monné wrote: On 26/09/13 03:48, Julian Elischer wrote: On 9/25/13 11:04 PM, Roger Pau Monné wrote: On 25/09/13 16:56, Julian Elischer wrote: If CPUs are mapped around, how are IPIs handled? I assume they must be emulated? I've noticed that under Xen (on both Amazon EC2 and a Redhat server) whenever you schedule a thread it always sits on the run queue for 20 uSecs before it starts running. It looks to me like it's the IPI taking a long time to be emulated. This has been improved on the FreeBSD Xen PVHVM port by using PV IPIs instead of the emulated ones, see r255331. It should be faster than the previous emulated implementation. I missed that.. thanks! Do you (or anyone else) know if this can be used on Amazon EC2? And do you need a specific version/configuration of Xen to be able to use it? I ran a new GENERIC kernel on an Amazon Xen 4.2 system and it performed a bit better. delays in the running of threads reduced from a constant 20uSecs to a variable amount between 1.5uSecs and 9uSecs. It's still very slow when compared with real hardware. (timing viewed using Schedgraph) Is there a way to see whether PV IPIs are active? (it looks it from the results but that could be so many other things too). Julian The PV IPIs require Xen version 4.0 or greater, and the PV timer requires 4.0.1 or greater if I'm not mistaken. If you are lucky to get an Amazon instance that uses this Xen version (or any superior one) they will be activated by default, if not FreeBSD will switch to the old event delivery method and PV IPIs and PV timer will be disabled in favour of the emulated ones. I guess it's just a matter of time before Amazon switches all their servers to Xen 4.x. Roger. ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: Xen (and others Hypervisors) how do they handle IPIs?
On 27/09/13 09:44, Julian Elischer wrote: On 9/26/13 5:50 PM, Roger Pau Monné wrote: On 26/09/13 03:48, Julian Elischer wrote: On 9/25/13 11:04 PM, Roger Pau Monné wrote: On 25/09/13 16:56, Julian Elischer wrote: If CPUs are mapped around, how are IPIs handled? I assume they must be emulated? I've noticed that under Xen (on both Amazon EC2 and a Redhat server) whenever you schedule a thread it always sits on the run queue for 20 uSecs before it starts running. It looks to me like it's the IPI taking a long time to be emulated. This has been improved on the FreeBSD Xen PVHVM port by using PV IPIs instead of the emulated ones, see r255331. It should be faster than the previous emulated implementation. I missed that.. thanks! Do you (or anyone else) know if this can be used on Amazon EC2? And do you need a specific version/configuration of Xen to be able to use it? I ran a new GENERIC kernel on an Amazon Xen 4.2 system and it performed a bit better. delays in the running of threads reduced from a constant 20uSecs to a variable amount between 1.5uSecs and 9uSecs. It's still very slow when compared with real hardware. (timing viewed using Schedgraph) Is there a way to see whether PV IPIs are active? (it looks it from the results but that could be so many other things too). The easiest way to see if PV IPIs are active is to run vmstat -i inside of the guest and search for the following lines: irq836: cpu6:r 2 0 irq839: cpu6:irg2530 70 irq841: cpu6:b 2175 60 This is the number of the different kinds of PV IPIs received on each CPU. You can find the correlation between this names and the IPIs on top of the file x86/xen/hvm.c. Roger. ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: Xen (and others Hypervisors) how do they handle IPIs?
On 26/09/13 03:48, Julian Elischer wrote: On 9/25/13 11:04 PM, Roger Pau Monné wrote: On 25/09/13 16:56, Julian Elischer wrote: If CPUs are mapped around, how are IPIs handled? I assume they must be emulated? I've noticed that under Xen (on both Amazon EC2 and a Redhat server) whenever you schedule a thread it always sits on the run queue for 20 uSecs before it starts running. It looks to me like it's the IPI taking a long time to be emulated. This has been improved on the FreeBSD Xen PVHVM port by using PV IPIs instead of the emulated ones, see r255331. It should be faster than the previous emulated implementation. I missed that.. thanks! Do you (or anyone else) know if this can be used on Amazon EC2? And do you need a specific version/configuration of Xen to be able to use it? The PV IPIs require Xen version 4.0 or greater, and the PV timer requires 4.0.1 or greater if I'm not mistaken. If you are lucky to get an Amazon instance that uses this Xen version (or any superior one) they will be activated by default, if not FreeBSD will switch to the old event delivery method and PV IPIs and PV timer will be disabled in favour of the emulated ones. I guess it's just a matter of time before Amazon switches all their servers to Xen 4.x. Roger. ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: Xen (and others Hypervisors) how do they handle IPIs?
On 9/26/13 4:16 PM, Peter Grehan wrote: Hi Julian, If CPUs are mapped around, how are IPIs handled? I assume they must be emulated? For bhyve, if the target vCPU of an IPI is running, a null IPI is sent on the host to force it to exit so the IPI can be injected (vmm.c:vm_interrupt_hostcpu()). If the target is asleep due to being idle, it is woken up. And, if the target is running but not in vCPU context, an interrupt is queued up so it will be injected on the next vmenter (subject to the emulated local APIC interrupt priority) Does anyone know if BHyVe or HyperV also have this problem? Hmmm, hard to say. Would you be able to try your workload out on a bhyve system ? possibly in a few weeks.. I'll try remember when the time is right. later, Peter. ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Xen (and others Hypervisors) how do they handle IPIs?
If CPUs are mapped around, how are IPIs handled? I assume they must be emulated? I've noticed that under Xen (on both Amazon EC2 and a Redhat server) whenever you schedule a thread it always sits on the run queue for 20 uSecs before it starts running. It looks to me like it's the IPI taking a long time to be emulated. We have some workloads where there is a lot of flipping back and forth between threads and they are slowed down by an order of magnitude due to this.. turning off NOADAPTIVE(mumble) seems to help a bit as some of the reschedules go away, but it's still a problem. Does anyone know if BHyVe or HyperV also have this problem? I have not yet gone to the source of Xen to see what it does but it would be educational to know what the other Hypervisors do. ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: Xen (and others Hypervisors) how do they handle IPIs?
On 25/09/13 16:56, Julian Elischer wrote: If CPUs are mapped around, how are IPIs handled? I assume they must be emulated? I've noticed that under Xen (on both Amazon EC2 and a Redhat server) whenever you schedule a thread it always sits on the run queue for 20 uSecs before it starts running. It looks to me like it's the IPI taking a long time to be emulated. This has been improved on the FreeBSD Xen PVHVM port by using PV IPIs instead of the emulated ones, see r255331. It should be faster than the previous emulated implementation. ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
