Re: [PATCH v2 0/5] rcu,nohz,kvm: use RCU extended quiescent state when running KVM guest
On Thu, Feb 05, 2015 at 03:23:47PM -0500, r...@redhat.com wrote: When running a KVM guest on a system with NOHZ_FULL enabled I just need to clarify the motivation first, does the above situation really happen? Ok some distros enable NOHZ_FULL to let the user stop the tick in userspace. So most of the time, CONFIG_NOHZ_FULL=y but nohz full is runtime disabled (we need to pass a nohz_full= boot parameter to enable it). And when it is runtime disabled, there should be no rcu nocb CPU. (Although not setting CPUs in nocb mode when nohz full is runtime disabled is perhaps a recent change.) So for the problem to arise, one need to enable nohz_full and run KVM guest. And I never heard about such workloads. That said it's potentially interesting to turn off the tick on the host when the guest runs. , and the KVM guest running with idle=poll mode, we still get wakeups of the rcuos/N threads. So we need nohz_full on the host and idle=poll mode on the guest. Is it likely to happen? (sorry, again I'm just trying to make sure we agree on why we do this change). This problem has already been solved for user space by telling the RCU subsystem that the CPU is in an extended quiescent state while running user space code. This patch series extends that code a little bit to make it usable to track KVM guest space, too. I tested the code by booting a KVM guest with idle=poll, on a system with NOHZ_FULL enabled on most CPUs, and a VCPU thread bound to a CPU. In a 10 second interval, rcuos/N threads on other CPUs got woken up several times, while the rcuos thread on the CPU running the bound and alwasy running VCPU thread never got woken up once. So what you're describing is to set RCU in extended quiescent state, right? This doesn't include stopping the tick while running in guest mode? Those are indeed two different thing, although stopping the tick most often requires to set RCU in extended quiescent state. Thanks to Christian Borntraeger and Paul McKenney for reviewing the first version of this patch series, and helping optimize patch 4/5. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 0/5] rcu,nohz,kvm: use RCU extended quiescent state when running KVM guest
On Fri, Feb 06, 2015 at 02:50:44PM +0100, Paolo Bonzini wrote: On 06/02/2015 14:46, Frederic Weisbecker wrote: When running a KVM guest on a system with NOHZ_FULL enabled I just need to clarify the motivation first, does the above situation really happen? Ok some distros enable NOHZ_FULL to let the user stop the tick in userspace. So most of the time, CONFIG_NOHZ_FULL=y but nohz full is runtime disabled (we need to pass a nohz_full= boot parameter to enable it). And when it is runtime disabled, there should be no rcu nocb CPU. (Although not setting CPUs in nocb mode when nohz full is runtime disabled is perhaps a recent change.) So for the problem to arise, one need to enable nohz_full and run KVM guest. And I never heard about such workloads. Yeah, it's a new thing but Marcelo, Luiz and Rik have been having a lot of fun with them (with PREEMPT_RT too). They're getting pretty good results given the right tuning. I'll let Paul queue the patches for 3.21 then! Frederic, given the background from Paolo, Rik, and Christian, are you OK with these patches? Thanx, Paul Paolo That said it's potentially interesting to turn off the tick on the host when the guest runs. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 0/5] rcu,nohz,kvm: use RCU extended quiescent state when running KVM guest
On 06/02/2015 14:46, Frederic Weisbecker wrote: When running a KVM guest on a system with NOHZ_FULL enabled I just need to clarify the motivation first, does the above situation really happen? Ok some distros enable NOHZ_FULL to let the user stop the tick in userspace. So most of the time, CONFIG_NOHZ_FULL=y but nohz full is runtime disabled (we need to pass a nohz_full= boot parameter to enable it). And when it is runtime disabled, there should be no rcu nocb CPU. (Although not setting CPUs in nocb mode when nohz full is runtime disabled is perhaps a recent change.) So for the problem to arise, one need to enable nohz_full and run KVM guest. And I never heard about such workloads. Yeah, it's a new thing but Marcelo, Luiz and Rik have been having a lot of fun with them (with PREEMPT_RT too). They're getting pretty good results given the right tuning. I'll let Paul queue the patches for 3.21 then! Paolo That said it's potentially interesting to turn off the tick on the host when the guest runs. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 0/5] rcu,nohz,kvm: use RCU extended quiescent state when running KVM guest
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/06/2015 08:46 AM, Frederic Weisbecker wrote: On Thu, Feb 05, 2015 at 03:23:47PM -0500, r...@redhat.com wrote: When running a KVM guest on a system with NOHZ_FULL enabled I just need to clarify the motivation first, does the above situation really happen? Ok some distros enable NOHZ_FULL to let the user stop the tick in userspace. So most of the time, CONFIG_NOHZ_FULL=y but nohz full is runtime disabled (we need to pass a nohz_full= boot parameter to enable it). And when it is runtime disabled, there should be no rcu nocb CPU. (Although not setting CPUs in nocb mode when nohz full is runtime disabled is perhaps a recent change.) So for the problem to arise, one need to enable nohz_full and run KVM guest. And I never heard about such workloads. That said it's potentially interesting to turn off the tick on the host when the guest runs. , and the KVM guest running with idle=poll mode, we still get wakeups of the rcuos/N threads. So we need nohz_full on the host and idle=poll mode on the guest. Is it likely to happen? (sorry, again I'm just trying to make sure we agree on why we do this change). We have users interested in doing just that, in order to run KVM guests with the least amount of perturbation to the guest. - -- All rights reversed -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQEcBAEBAgAGBQJU1NYrAAoJEM553pKExN6DMgcIAIKoBBxsFEt7yOP+k32uqm+W S/VbP5dIHE5OnYeBoitgNkia1U4rsAX6AVAuVFvKc7Y8aixENGzubPWHe0NuHida VIaQmK92Jym4FH8Xsnj09MhgLV+ZEG/cCzdUFZfShJq3hHwzedZx+cC7uQMB6kd4 iuo7CtgTjzTgBce29Fc147azXlJbfFfwFt3a6YVxbv25IYpDL9ETulh34h6NrNLz nB0snDjq8FHKcyjlD3XnJpT/tbaZcrZctExq4JrespcBMe6prMnoWvVoXWX/fVVG TIR1hp2xfKWoS4gc56PnLazIIB9SRmlC/SzSMwAaSgf1dWa5BcwpuMbYVFIEeME= =+BrK -END PGP SIGNATURE- -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 0/5] rcu,nohz,kvm: use RCU extended quiescent state when running KVM guest
Am 05.02.2015 um 21:23 schrieb r...@redhat.com: When running a KVM guest on a system with NOHZ_FULL enabled, and the KVM guest running with idle=poll mode, we still get wakeups of the rcuos/N threads. This problem has already been solved for user space by telling the RCU subsystem that the CPU is in an extended quiescent state while running user space code. This patch series extends that code a little bit to make it usable to track KVM guest space, too. I tested the code by booting a KVM guest with idle=poll, on a system with NOHZ_FULL enabled on most CPUs, and a VCPU thread bound to a CPU. In a 10 second interval, rcuos/N threads on other CPUs got woken up several times, while the rcuos thread on the CPU running the bound and alwasy running VCPU thread never got woken up once. Thanks to Christian Borntraeger and Paul McKenney for reviewing the first version of this patch series, and helping optimize patch 4/5. I gave it a quick run on s390/kvm and everything still seem to be running fine. A also I like the idea of this patch set. We have seen several cases were the fact that we are in guest context a full tick for cpu bound guests (10ms on s390) caused significant latencies for host synchronize-rcu heavy workload - e.g. getting rid of macvtap devices on guest shutdown, adding hundreds of irq routes for many guest devices s390 has no context tracking infrastructure yet (no nohz_full), but this series looks like that the current case (nohz_idle) still works. With this in place, having hohz==full on s390 now even makes more sense, as KVM hosts with cpu bound guests should have get much quicker rcu response times when most host CPUs are in an extended quiescant state. Christian -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 0/5] rcu,nohz,kvm: use RCU extended quiescent state when running KVM guest
On Fri, Feb 06, 2015 at 09:56:43AM -0500, Rik van Riel wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/06/2015 08:46 AM, Frederic Weisbecker wrote: On Thu, Feb 05, 2015 at 03:23:47PM -0500, r...@redhat.com wrote: When running a KVM guest on a system with NOHZ_FULL enabled I just need to clarify the motivation first, does the above situation really happen? Ok some distros enable NOHZ_FULL to let the user stop the tick in userspace. So most of the time, CONFIG_NOHZ_FULL=y but nohz full is runtime disabled (we need to pass a nohz_full= boot parameter to enable it). And when it is runtime disabled, there should be no rcu nocb CPU. (Although not setting CPUs in nocb mode when nohz full is runtime disabled is perhaps a recent change.) So for the problem to arise, one need to enable nohz_full and run KVM guest. And I never heard about such workloads. That said it's potentially interesting to turn off the tick on the host when the guest runs. , and the KVM guest running with idle=poll mode, we still get wakeups of the rcuos/N threads. So we need nohz_full on the host and idle=poll mode on the guest. Is it likely to happen? (sorry, again I'm just trying to make sure we agree on why we do this change). We have users interested in doing just that, in order to run KVM guests with the least amount of perturbation to the guest. So what are you interested in exactly? Only RCU extended quiescent state or also full dynticks? Because your cover letter only points to disturbing RCU nocb and quiescent states. Also I'm curious why you run guests in idle=poll. Maybe that avoids host/guest context switches? Thanks. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 0/5] rcu,nohz,kvm: use RCU extended quiescent state when running KVM guest
On Fri, Feb 06, 2015 at 02:50:44PM +0100, Paolo Bonzini wrote: On 06/02/2015 14:46, Frederic Weisbecker wrote: When running a KVM guest on a system with NOHZ_FULL enabled I just need to clarify the motivation first, does the above situation really happen? Ok some distros enable NOHZ_FULL to let the user stop the tick in userspace. So most of the time, CONFIG_NOHZ_FULL=y but nohz full is runtime disabled (we need to pass a nohz_full= boot parameter to enable it). And when it is runtime disabled, there should be no rcu nocb CPU. (Although not setting CPUs in nocb mode when nohz full is runtime disabled is perhaps a recent change.) So for the problem to arise, one need to enable nohz_full and run KVM guest. And I never heard about such workloads. Yeah, it's a new thing but Marcelo, Luiz and Rik have been having a lot of fun with them (with PREEMPT_RT too). They're getting pretty good results given the right tuning. Ok but, I'm still not sure about the details of what you're trying to do. Whether it's only about RCU or it also involves ticks. What kind of tuning you're doing and what kind of performance gain? Thanks. I'll let Paul queue the patches for 3.21 then! Paolo That said it's potentially interesting to turn off the tick on the host when the guest runs. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 0/5] rcu,nohz,kvm: use RCU extended quiescent state when running KVM guest
On Fri, Feb 06, 2015 at 04:00:27PM +0100, Christian Borntraeger wrote: Am 05.02.2015 um 21:23 schrieb r...@redhat.com: When running a KVM guest on a system with NOHZ_FULL enabled, and the KVM guest running with idle=poll mode, we still get wakeups of the rcuos/N threads. This problem has already been solved for user space by telling the RCU subsystem that the CPU is in an extended quiescent state while running user space code. This patch series extends that code a little bit to make it usable to track KVM guest space, too. I tested the code by booting a KVM guest with idle=poll, on a system with NOHZ_FULL enabled on most CPUs, and a VCPU thread bound to a CPU. In a 10 second interval, rcuos/N threads on other CPUs got woken up several times, while the rcuos thread on the CPU running the bound and alwasy running VCPU thread never got woken up once. Thanks to Christian Borntraeger and Paul McKenney for reviewing the first version of this patch series, and helping optimize patch 4/5. I gave it a quick run on s390/kvm and everything still seem to be running fine. A also I like the idea of this patch set. We have seen several cases were the fact that we are in guest context a full tick for cpu bound guests (10ms on s390) caused significant latencies for host synchronize-rcu heavy workload - e.g. getting rid of macvtap devices on guest shutdown, adding hundreds of irq routes for many guest devices s390 has no context tracking infrastructure yet (no nohz_full), but this series looks like that the current case (nohz_idle) still works. With this in place, having hohz==full on s390 now even makes more sense, as KVM hosts with cpu bound guests should have get much quicker rcu response times when most host CPUs are in an extended quiescant state. Sure, if you need any help for context tracking, don't hesitate to ask, it can be a bit tricky to implement sometimes. Perhaps x86 isn't the best example because it does quite some weird dances to minimize fast path overhead. ARM is perhaps clearer. Christian -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v2 0/5] rcu,nohz,kvm: use RCU extended quiescent state when running KVM guest
When running a KVM guest on a system with NOHZ_FULL enabled, and the KVM guest running with idle=poll mode, we still get wakeups of the rcuos/N threads. This problem has already been solved for user space by telling the RCU subsystem that the CPU is in an extended quiescent state while running user space code. This patch series extends that code a little bit to make it usable to track KVM guest space, too. I tested the code by booting a KVM guest with idle=poll, on a system with NOHZ_FULL enabled on most CPUs, and a VCPU thread bound to a CPU. In a 10 second interval, rcuos/N threads on other CPUs got woken up several times, while the rcuos thread on the CPU running the bound and alwasy running VCPU thread never got woken up once. Thanks to Christian Borntraeger and Paul McKenney for reviewing the first version of this patch series, and helping optimize patch 4/5. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
