Disk IO throttling for VM guests?
Is there any possibilities to limit disk IO for virtualization guest on FreeBSD? I would like to know, if it is possible to limit IOps for jails, or Bhyve guest, or VirtualBox quests. There are ways to limit CPU or RAM for them, but CPU and RAM are really huge these days. On the other hand, HDDs are still very IO limited and if one guest runs disk IO hungy task, then all other guest are affected / slow. I read about plugable GEOM scheduler few years ago (GEOM_SCHED), but it seems that it is dead project and there is no module for it to allow some scheduling according to PID, JID or something like this. So do we have anything like this for Jails or Bhyve? http://wiki.qemu.org/Features/DiskIOLimits http://wiki.smartos.org/display/DOC/Tuning+the+IO+Throttle Miroslav Lachman ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: Disk IO throttling for VM guests?
On Thu, May 8, 2014 at 5:57 PM, Miroslav Lachman 000.f...@quip.cz wrote: Is there any possibilities to limit disk IO for virtualization guest on FreeBSD? I would like to know, if it is possible to limit IOps for jails, or Bhyve guest, or VirtualBox quests. There are ways to limit CPU or RAM for them, but CPU and RAM are really huge these days. On the other hand, HDDs are still very IO limited and if one guest runs disk IO hungy task, then all other guest are affected / slow. I read about plugable GEOM scheduler few years ago (GEOM_SCHED), but it seems that it is dead project and there is no module for it to allow some scheduling according to PID, JID or something like this. So do we have anything like this for Jails or Bhyve? http://wiki.qemu.org/Features/DiskIOLimits http://wiki.smartos.org/display/DOC/Tuning+the+IO+Throttle Miroslav Lachman Well, there is rctl. I haven't tried it yet though. Best regards Andreas ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: consistent VM hang during reboot
On Wednesday, May 07, 2014 7:15:43 pm John Nielsen wrote:
I am trying to solve a problem with amd64 FreeBSD virtual machines running on
a Linux+KVM hypervisor. To be honest I'm not sure if the problem is in
FreeBSD or
the hypervisor, but I'm trying to rule out the OS first.
The _second_ time FreeBSD boots in a virtual machine with more than one core,
the boot hangs just before the kernel would normally print e.g. SMP: AP CPU
#1
Launched! (The last line on the console is usbus0: 12Mbps Full Speed USB
v1.0, but the problem persists even without USB). The VM will boot fine a
first time,
but running either shutdown -r now OR reboot will lead to a hung second
boot. Stopping and starting the host qemu-kvm process is the only way to
continue.
The problem seems to be triggered by something in the SMP portion of
cpu_reset() (from sys/amd64/amd64/vm_machdep.c). If I hit the virtual reset
button the next
boot is fine. If I have 'kern.smp.disabled=1' set for the initial boot then
subsequent boots are fine (but I can only use one CPU core, of course).
However, if I
boot normally the first time then set 'kern.smp.disabled=1' for the second
(re)boot, the problem is triggered. Apparently something in the shutdown code
is
poisoning the well for the next boot.
The problem is present in FreeBSD 8.4, 9.2, 10.0 and 11-CURRENT as of
yesterday.
This (heavy-handed and wrong) patch (to HEAD) lets me avoid the issue:
--- sys/amd64/amd64/vm_machdep.c.orig 2014-05-07 13:19:07.400981580 -0600
+++ sys/amd64/amd64/vm_machdep.c 2014-05-07 17:02:52.416783795 -0600
@@ -593,7 +593,7 @@
void
cpu_reset()
{
-#ifdef SMP
+#if 0
cpuset_t map;
u_int cnt;
I've tried skipping or disabling smaller chunks of code within the #if block
but haven't found a consistent winner yet.
I'm hoping the list will have suggestions on how I can further narrow down
the problem, or theories on what might be going on.
Can you try forcing the reboot to occur on the BSP (via 'cpuset -l 0 reboot')
or a non-BSP ('cpuset -l 1 reboot') to see if that has any effect? It might
not, but if it does it would help narrow down the code to consider.
--
John Baldwin
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
freebsd-virtualization-unsubscr...@freebsd.org
Re: consistent VM hang during reboot
On May 8, 2014, at 11:03 AM, John Baldwin j...@freebsd.org wrote:
On Wednesday, May 07, 2014 7:15:43 pm John Nielsen wrote:
I am trying to solve a problem with amd64 FreeBSD virtual machines running
on a Linux+KVM hypervisor. To be honest I'm not sure if the problem is in
FreeBSD or
the hypervisor, but I'm trying to rule out the OS first.
The _second_ time FreeBSD boots in a virtual machine with more than one
core, the boot hangs just before the kernel would normally print e.g. SMP:
AP CPU #1
Launched! (The last line on the console is usbus0: 12Mbps Full Speed USB
v1.0, but the problem persists even without USB). The VM will boot fine a
first time,
but running either shutdown -r now OR reboot will lead to a hung second
boot. Stopping and starting the host qemu-kvm process is the only way to
continue.
The problem seems to be triggered by something in the SMP portion of
cpu_reset() (from sys/amd64/amd64/vm_machdep.c). If I hit the virtual
reset button the next
boot is fine. If I have 'kern.smp.disabled=1' set for the initial boot then
subsequent boots are fine (but I can only use one CPU core, of course).
However, if I
boot normally the first time then set 'kern.smp.disabled=1' for the second
(re)boot, the problem is triggered. Apparently something in the shutdown code
is
poisoning the well for the next boot.
The problem is present in FreeBSD 8.4, 9.2, 10.0 and 11-CURRENT as of
yesterday.
This (heavy-handed and wrong) patch (to HEAD) lets me avoid the issue:
--- sys/amd64/amd64/vm_machdep.c.orig2014-05-07 13:19:07.400981580
-0600
+++ sys/amd64/amd64/vm_machdep.c 2014-05-07 17:02:52.416783795 -0600
@@ -593,7 +593,7 @@
void
cpu_reset()
{
-#ifdef SMP
+#if 0
cpuset_t map;
u_int cnt;
I've tried skipping or disabling smaller chunks of code within the #if block
but haven't found a consistent winner yet.
I'm hoping the list will have suggestions on how I can further narrow down
the problem, or theories on what might be going on.
Can you try forcing the reboot to occur on the BSP (via 'cpuset -l 0 reboot')
or a non-BSP ('cpuset -l 1 reboot') to see if that has any effect? It might
not, but if it does it would help narrow down the code to consider.
Hello jhb, thanks for responding.
I tried your suggestion but unfortunately it does not make any difference. The
reboot hangs regardless of which CPU I assign the command to.
Any other suggestions?
JN
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
freebsd-virtualization-unsubscr...@freebsd.org
RE: consistent VM hang during reboot
When I was doing some early work on some of the Octeon multi-core chips, I
encountered something similar. If I remember correctly, there was an issue in
the shutdown sequence that did not properly halt the cores and set up the
start jump vector. So the first core would start, and when it tried to start
the next ones it would hang waiting for the ACK that they were running (since
they didn't have a start vector and hence never started). I know MIPS, not AMD,
so I can't say what the equivalent would be, but I'm sure there is one. Check
that part, setting up the early state.
If Juli and/or Adrian are reading this: do you remember anything about that,
something like 2 years ago?
Andrew L. Duane
ATT Technical Lead
JNCIA - JUNOS
m +1 603.770.7088
o+1 408.933.6944 (2-6944)
skype: andrewlduane
adu...@juniper.net
-Original Message-
From: owner-freebsd-hack...@freebsd.org
[mailto:owner-freebsd-hack...@freebsd.org] On Behalf Of John Nielsen
Sent: Thursday, May 08, 2014 1:56 PM
To: John Baldwin
Cc: freebsd-hack...@freebsd.org; freebsd-virtualization@freebsd.org
Subject: Re: consistent VM hang during reboot
On May 8, 2014, at 11:03 AM, John Baldwin j...@freebsd.org wrote:
On Wednesday, May 07, 2014 7:15:43 pm John Nielsen wrote:
I am trying to solve a problem with amd64 FreeBSD virtual machines running
on a Linux+KVM hypervisor. To be honest I'm not sure if the problem is in
FreeBSD or
the hypervisor, but I'm trying to rule out the OS first.
The _second_ time FreeBSD boots in a virtual machine with more than one
core, the boot hangs just before the kernel would normally print e.g. SMP:
AP CPU #1
Launched! (The last line on the console is usbus0: 12Mbps Full Speed USB
v1.0, but the problem persists even without USB). The VM will boot fine a
first time,
but running either shutdown -r now OR reboot will lead to a hung second
boot. Stopping and starting the host qemu-kvm process is the only way to
continue.
The problem seems to be triggered by something in the SMP portion of
cpu_reset() (from sys/amd64/amd64/vm_machdep.c). If I hit the virtual
reset button the next
boot is fine. If I have 'kern.smp.disabled=1' set for the initial boot then
subsequent boots are fine (but I can only use one CPU core, of course).
However, if I
boot normally the first time then set 'kern.smp.disabled=1' for the second
(re)boot, the problem is triggered. Apparently something in the shutdown code
is
poisoning the well for the next boot.
The problem is present in FreeBSD 8.4, 9.2, 10.0 and 11-CURRENT as of
yesterday.
This (heavy-handed and wrong) patch (to HEAD) lets me avoid the issue:
--- sys/amd64/amd64/vm_machdep.c.orig2014-05-07 13:19:07.400981580
-0600
+++ sys/amd64/amd64/vm_machdep.c 2014-05-07 17:02:52.416783795 -0600
@@ -593,7 +593,7 @@
void
cpu_reset()
{
-#ifdef SMP
+#if 0
cpuset_t map;
u_int cnt;
I've tried skipping or disabling smaller chunks of code within the #if block
but haven't found a consistent winner yet.
I'm hoping the list will have suggestions on how I can further narrow down
the problem, or theories on what might be going on.
Can you try forcing the reboot to occur on the BSP (via 'cpuset -l 0 reboot')
or a non-BSP ('cpuset -l 1 reboot') to see if that has any effect? It might
not, but if it does it would help narrow down the code to consider.
Hello jhb, thanks for responding.
I tried your suggestion but unfortunately it does not make any difference. The
reboot hangs regardless of which CPU I assign the command to.
Any other suggestions?
JN
___
freebsd-hack...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
freebsd-virtualization-unsubscr...@freebsd.org
Re: Disk IO throttling for VM guests?
Andreas Nilsson wrote: On Thu, May 8, 2014 at 5:57 PM, Miroslav Lachman 000.f...@quip.cz mailto:000.f...@quip.cz wrote: Is there any possibilities to limit disk IO for virtualization guest on FreeBSD? I would like to know, if it is possible to limit IOps for jails, or Bhyve guest, or VirtualBox quests. There are ways to limit CPU or RAM for them, but CPU and RAM are really huge these days. On the other hand, HDDs are still very IO limited and if one guest runs disk IO hungy task, then all other guest are affected / slow. I read about plugable GEOM scheduler few years ago (GEOM_SCHED), but it seems that it is dead project and there is no module for it to allow some scheduling according to PID, JID or something like this. So do we have anything like this for Jails or Bhyve? http://wiki.qemu.org/Features/ DiskIOLimits http://wiki.smartos.org/ display/DOC/Tuning+the+IO+ Throttle Miroslav Lachman Well, there is rctl. I haven't tried it yet though. Best regards Andreas As far as I know, it is just another way to limit CPU, memory, swap, SysV semaphores, but no way to limit disk iops or bandwidth. Miroslav Lachman ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
