[virtio-dev] Re: [Qemu-devel] [PATCH v7 0/5] virtio-balloon: free page hint reporting support
On Tue, May 29, 2018 at 11:00:21PM +0800, Hailiang Zhang wrote:
> On 2018/4/24 14:13, Wei Wang wrote:
> > This is the deivce part implementation to add a new feature,
> > VIRTIO_BALLOON_F_FREE_PAGE_HINT to the virtio-balloon device. The device
> > receives the guest free page hints from the driver and clears the
> > corresponding bits in the dirty bitmap, so that those free pages are
> > not transferred by the migration thread to the destination.
> >
> > - Test Environment
> > Host: Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz
> > Guest: 8G RAM, 4 vCPU
> > Migration setup: migrate_set_speed 100G, migrate_set_downtime 2 second
> >
> > - Test Results
> > - Idle Guest Live Migration Time (results are averaged over 10 runs):
> > - Optimization v.s. Legacy = 271ms vs 1769ms --> ~86% reduction
> > - Guest with Linux Compilation Workload (make bzImage -j4):
> > - Live Migration Time (average)
> >Optimization v.s. Legacy = 1265ms v.s. 2634ms --> ~51% reduction
> > - Linux Compilation Time
> >Optimization v.s. Legacy = 4min56s v.s. 5min3s
> >--> no obvious difference
> >
> > - Source Code
> > - QEMU: https://github.com/wei-w-wang/qemu-free-page-lm.git
> > - Linux: https://github.com/wei-w-wang/linux-free-page-lm.git
> >
> > ChangeLog:
> > v6->v7:
> >virtio-balloon/virtio_balloo_poll_free_page_hints:
> >- add virtio_notify() at the end to notify the driver that
> > the optimization is done, which indicates that the entries
> > have all been put back to the vq and ready to detach them.
> > v5->v6:
> >virtio-balloon: use iothread to get free page hint
> > v4->v5:
> > 1) migration:
> > - bitmap_clear_dirty: update the dirty bitmap and dirty page
> >count under the bitmap mutex as what other functions are doing;
> > - qemu_guest_free_page_hint:
> > - add comments for this function;
> > - check the !block case;
> > - check "offset > block->used_length" before proceed;
> > - assign used_len inside the for{} body;
> > - update the dirty bitmap and dirty page counter under the
> >bitmap mutex;
> > - ram_state_reset:
> > - rs->free_page_support: && with use "migrate_postcopy"
> >instead of migration_in_postcopy;
> > - clear the ram_bulk_stage flag if free_page_support is true;
> > 2) balloon:
> > - add the usage documentation of balloon_free_page_start and
> > balloon_free_page_stop in code;
> > - the optimization thread is named "balloon_fpo" to meet the
> > requirement of "less than 14 characters";
> > - virtio_balloon_poll_free_page_hints:
> > - run on condition when runstate_is_running() is true;
> > - add a qemu spin lock to synchronize accesses to the free
> > page reporting related fields shared among the migration
> > thread and the optimization thread;
> >- virtio_balloon_free_page_start: just return if
> > runstate_is_running is false;
> >- virtio_balloon_free_page_stop: access to the free page
> > reporting related fields under a qemu spin lock;
> >- virtio_balloon_device_unrealize/reset: call
> > virtio_balloon_free_page_stop is the free page hint feature is
> > used;
> >- virtio_balloon_set_status: call irtio_balloon_free_page_stop
> > in case the guest is stopped by qmp when the optimization is
> > running;
> > v3->v4:
> > 1) bitmap: add a new API to count 1s starting from an offset of a
> > bitmap
> > 2) migration:
> > - qemu_guest_free_page_hint: calculate
> >ram_state->migration_dirty_pages by counting how many bits of
> >free pages are truely cleared. If some of the bits were
> >already 0, they shouldn't be deducted by
> >ram_state->migration_dirty_pages. This wasn't needed for
> >previous versions since we optimized bulk stage only,
> >where all bits are guaranteed to be set. It's needed now
> >because we extened the usage of this optimizaton to all stages
> >except the last stop stage. From 2nd stage onward, there
> >are possibilities that some bits of free pages are already 0.
> > 3) virtio-balloon:
> > - virtio_balloon_free_page_report_status: introduce a new status,
> > FREE_PAGE_REPORT_S_EXIT. This status indicates that the
> > optimization thread has exited. FREE_PAGE_REPORT_S_STOP means
> > the reporting is stopped, but the optimization thread still
> > needs
> > to be joined by the migration thread.
> > v2->v3:
> > 1) virtio-balloon
> >
[virtio-dev] Re: [PATCH v7 4/5] virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_HINT
On Tue, Apr 24, 2018 at 02:13:47PM +0800, Wei Wang wrote:
> The new feature enables the virtio-balloon device to receive hints of
> guest free pages from the free page vq.
>
> balloon_free_page_start - start guest free page hint reporting.
> balloon_free_page_stop - stop guest free page hint reporting.
>
> Note: balloon will report pages which were free at the time
> of this call. As the reporting happens asynchronously, dirty bit logging
> must be enabled before this call is made. Guest reporting must be
> disabled before the migration dirty bitmap is synchronized.
>
> TODO:
> - handle the case when page poisoning is in use
>
> Signed-off-by: Wei Wang
> Signed-off-by: Liang Li
> CC: Michael S. Tsirkin
> CC: Dr. David Alan Gilbert
> CC: Juan Quintela
> ---
> balloon.c | 58 +-
> hw/virtio/virtio-balloon.c | 241
> ++--
> include/hw/virtio/virtio-balloon.h | 27 ++-
> include/standard-headers/linux/virtio_balloon.h | 7 +
> include/sysemu/balloon.h| 15 +-
> 5 files changed, 319 insertions(+), 29 deletions(-)
>
> diff --git a/balloon.c b/balloon.c
> index 6bf0a96..87a0410 100644
> --- a/balloon.c
> +++ b/balloon.c
> @@ -36,6 +36,9 @@
>
> static QEMUBalloonEvent *balloon_event_fn;
> static QEMUBalloonStatus *balloon_stat_fn;
> +static QEMUBalloonFreePageSupport *balloon_free_page_support_fn;
> +static QEMUBalloonFreePageStart *balloon_free_page_start_fn;
> +static QEMUBalloonFreePageStop *balloon_free_page_stop_fn;
> static void *balloon_opaque;
> static bool balloon_inhibited;
>
> @@ -64,19 +67,51 @@ static bool have_balloon(Error **errp)
> return true;
> }
>
> -int qemu_add_balloon_handler(QEMUBalloonEvent *event_func,
> - QEMUBalloonStatus *stat_func, void *opaque)
> +bool balloon_free_page_support(void)
> {
> -if (balloon_event_fn || balloon_stat_fn || balloon_opaque) {
> -/* We're already registered one balloon handler. How many can
> - * a guest really have?
> - */
> -return -1;
> +return balloon_free_page_support_fn &&
> + balloon_free_page_support_fn(balloon_opaque);
> +}
> +
> +/*
> + * Balloon will report pages which were free at the time of this call. As the
> + * reporting happens asynchronously, dirty bit logging must be enabled before
> + * this call is made.
> + */
> +void balloon_free_page_start(void)
> +{
> +balloon_free_page_start_fn(balloon_opaque);
> +}
Please create notifier support, not a single global.
> +
> +/*
> + * Guest reporting must be disabled before the migration dirty bitmap is
> + * synchronized.
> + */
> +void balloon_free_page_stop(void)
> +{
> +balloon_free_page_stop_fn(balloon_opaque);
> +}
> +
> +void qemu_add_balloon_handler(QEMUBalloonEvent *event_fn,
> + QEMUBalloonStatus *stat_fn,
> + QEMUBalloonFreePageSupport
> *free_page_support_fn,
> + QEMUBalloonFreePageStart *free_page_start_fn,
> + QEMUBalloonFreePageStop *free_page_stop_fn,
> + void *opaque)
> +{
> +if (balloon_event_fn || balloon_stat_fn || balloon_free_page_support_fn
> ||
> +balloon_free_page_start_fn || balloon_free_page_stop_fn ||
> +balloon_opaque) {
> +/* We already registered one balloon handler. */
> +return;
> }
> -balloon_event_fn = event_func;
> -balloon_stat_fn = stat_func;
> +
> +balloon_event_fn = event_fn;
> +balloon_stat_fn = stat_fn;
> +balloon_free_page_support_fn = free_page_support_fn;
> +balloon_free_page_start_fn = free_page_start_fn;
> +balloon_free_page_stop_fn = free_page_stop_fn;
> balloon_opaque = opaque;
> -return 0;
> }
>
> void qemu_remove_balloon_handler(void *opaque)
> @@ -86,6 +121,9 @@ void qemu_remove_balloon_handler(void *opaque)
> }
> balloon_event_fn = NULL;
> balloon_stat_fn = NULL;
> +balloon_free_page_support_fn = NULL;
> +balloon_free_page_start_fn = NULL;
> +balloon_free_page_stop_fn = NULL;
> balloon_opaque = NULL;
> }
>
> diff --git a/hw/virtio/virtio-balloon.c b/hw/virtio/virtio-balloon.c
> index f456cea..13bf0db 100644
> --- a/hw/virtio/virtio-balloon.c
> +++ b/hw/virtio/virtio-balloon.c
> @@ -31,6 +31,7 @@
>
> #include "hw/virtio/virtio-bus.h"
> #include "hw/virtio/virtio-access.h"
> +#include "migration/misc.h"
>
> #define BALLOON_PAGE_SIZE (1 << VIRTIO_BALLOON_PFN_SHIFT)
>
> @@ -308,6 +309,125 @@ out:
> }
> }
>
> +static void virtio_balloon_poll_free_page_hints(void *opaque)
> +{
> +VirtQueueElement *elem;
> +VirtIOBalloon *dev = opaque;
> +VirtIODevice *vdev = VIRTIO_DEVICE(dev);
> +VirtQueue *vq = dev->free_page_vq;
> +uint32_t id;
> +size_t size;
> +
> +while (1) {
> +qemu_mutex_lock(>free_page_lock);
> +
[virtio-dev] Re: [PATCH v4] content: support SR-IOV
On Fri, 25 May 2018 00:49:51 +0800 Tiwei Bie wrote: > Allocate a feature bit for virtio devices which support SR-IOV. > > Suggested-by: Michael S. Tsirkin > Signed-off-by: Tiwei Bie > Fixes: https://github.com/oasis-tcs/virtio-spec/issues/11 > --- > More details can be found from this thread: > https://patchwork.kernel.org/patch/10285541/ > > v3 -> v4: > - Limit the feature to PCI devices with SR-IOV cap (MST); > > v2 -> v3: > - Improve the wording (Cornelia); > > v1 -> v2: > - s/Reserve/Allocate/ (MST); > - Add a Fixes tag (MST); > - Be more explicit in driver requirement (MST); > - Remove the "device MAY fail" description (MST); > - Rebase on IO_BARRIER patch; > > RFC -> v1: > - Mention PCI in the description (Cornelia); > > content.tex | 21 +++-- > 1 file changed, 19 insertions(+), 2 deletions(-) Acked-by: Cornelia Huck - To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org
Re: [virtio-dev] [PATCH v4] content: support SR-IOV
On Fri, May 25, 2018 at 12:49:51AM +0800, Tiwei Bie wrote:
> Allocate a feature bit for virtio devices which support SR-IOV.
>
> Suggested-by: Michael S. Tsirkin
> Signed-off-by: Tiwei Bie
> Fixes: https://github.com/oasis-tcs/virtio-spec/issues/11
Please start the voting for this proposal. Thanks!
Best regards,
Tiwei Bie
> ---
> More details can be found from this thread:
> https://patchwork.kernel.org/patch/10285541/
>
> v3 -> v4:
> - Limit the feature to PCI devices with SR-IOV cap (MST);
>
> v2 -> v3:
> - Improve the wording (Cornelia);
>
> v1 -> v2:
> - s/Reserve/Allocate/ (MST);
> - Add a Fixes tag (MST);
> - Be more explicit in driver requirement (MST);
> - Remove the "device MAY fail" description (MST);
> - Rebase on IO_BARRIER patch;
>
> RFC -> v1:
> - Mention PCI in the description (Cornelia);
>
> content.tex | 21 +++--
> 1 file changed, 19 insertions(+), 2 deletions(-)
>
> diff --git a/content.tex b/content.tex
> index c4b3b5c..beec6c1 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -95,10 +95,10 @@ Feature bits are allocated as follows:
> \begin{description}
> \item[0 to 23] Feature bits for the specific device type
>
> -\item[24 to 36] Feature bits reserved for extensions to the queue and
> +\item[24 to 37] Feature bits reserved for extensions to the queue and
>feature negotiation mechanisms
>
> -\item[37 and above] Feature bits reserved for future extensions.
> +\item[38 and above] Feature bits reserved for future extensions.
> \end{description}
>
> \begin{note}
> @@ -5365,6 +5365,9 @@ Descriptors} and \ref{sec:Packed Virtqueues / Indirect
> Flag: Scatter-Gather Supp
>better performance. This feature indicates whether
>a stronger form of barrier suitable for hardware
>devices is necessary.
> + \item[VIRTIO_F_SR_IOV(37)] This feature indicates that
> + the device supports Single Root I/O Virtualization.
> + Currently only PCI devices support this feature.
> \end{description}
>
> \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
> @@ -5384,6 +5387,16 @@ A driver SHOULD accept VIRTIO_F_IO_BARRIER if it is
> offered.
> If VIRTIO_F_IO_BARRIER has been negotiated, a driver MUST use
> the barriers suitable for hardware devices.
>
> +If VIRTIO_F_SR_IOV has been negotiated, a driver can enable
> +virtual functions through the device's PCI SR-IOV capability
> +structure. A driver MUST NOT negotiate VIRTIO_F_SR_IOV if
> +the device does not have a PCI SR-IOV capability structure
> +or is not a PCI device. A driver MUST negotiate
> +VIRTIO_F_SR_IOV and complete the feature negotiation
> +(including setting the DRIVER_OK \field{status} bit) before
> +enabling virtual functions through the device's PCI SR-IOV
> +capability structure.
> +
> \devicenormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
>
> A device MUST offer VIRTIO_F_VERSION_1. A device MAY fail to operate further
> @@ -5400,6 +5413,10 @@ buffers in the same order in which they have been
> available.
> A device MAY fail to operate further if VIRTIO_F_IO_BARRIER
> is not accepted.
>
> +A device SHOULD offer VIRTIO_F_SR_IOV if it is a PCI device
> +and presents a PCI SR-IOV capability structure, otherwise
> +it MUST NOT offer VIRTIO_F_SR_IOV.
> +
> \section{Legacy Interface: Reserved Feature Bits}\label{sec:Reserved Feature
> Bits / Legacy Interface: Reserved Feature Bits}
>
> Transitional devices MAY offer the following:
> --
> 2.17.0
>
>
> -
> To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org
>
-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org
Re: [virtio-dev] [PATCH v4] content: support SR-IOV
On Fri, May 25, 2018 at 12:49:51AM +0800, Tiwei Bie wrote:
> Allocate a feature bit for virtio devices which support SR-IOV.
>
> Suggested-by: Michael S. Tsirkin
> Signed-off-by: Tiwei Bie
> Fixes: https://github.com/oasis-tcs/virtio-spec/issues/11
Acked-by: Michael S. Tsirkin
> ---
> More details can be found from this thread:
> https://patchwork.kernel.org/patch/10285541/
>
> v3 -> v4:
> - Limit the feature to PCI devices with SR-IOV cap (MST);
>
> v2 -> v3:
> - Improve the wording (Cornelia);
>
> v1 -> v2:
> - s/Reserve/Allocate/ (MST);
> - Add a Fixes tag (MST);
> - Be more explicit in driver requirement (MST);
> - Remove the "device MAY fail" description (MST);
> - Rebase on IO_BARRIER patch;
>
> RFC -> v1:
> - Mention PCI in the description (Cornelia);
>
> content.tex | 21 +++--
> 1 file changed, 19 insertions(+), 2 deletions(-)
>
> diff --git a/content.tex b/content.tex
> index c4b3b5c..beec6c1 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -95,10 +95,10 @@ Feature bits are allocated as follows:
> \begin{description}
> \item[0 to 23] Feature bits for the specific device type
>
> -\item[24 to 36] Feature bits reserved for extensions to the queue and
> +\item[24 to 37] Feature bits reserved for extensions to the queue and
>feature negotiation mechanisms
>
> -\item[37 and above] Feature bits reserved for future extensions.
> +\item[38 and above] Feature bits reserved for future extensions.
> \end{description}
>
> \begin{note}
> @@ -5365,6 +5365,9 @@ Descriptors} and \ref{sec:Packed Virtqueues / Indirect
> Flag: Scatter-Gather Supp
>better performance. This feature indicates whether
>a stronger form of barrier suitable for hardware
>devices is necessary.
> + \item[VIRTIO_F_SR_IOV(37)] This feature indicates that
> + the device supports Single Root I/O Virtualization.
> + Currently only PCI devices support this feature.
> \end{description}
>
> \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
> @@ -5384,6 +5387,16 @@ A driver SHOULD accept VIRTIO_F_IO_BARRIER if it is
> offered.
> If VIRTIO_F_IO_BARRIER has been negotiated, a driver MUST use
> the barriers suitable for hardware devices.
>
> +If VIRTIO_F_SR_IOV has been negotiated, a driver can enable
> +virtual functions through the device's PCI SR-IOV capability
> +structure. A driver MUST NOT negotiate VIRTIO_F_SR_IOV if
> +the device does not have a PCI SR-IOV capability structure
> +or is not a PCI device. A driver MUST negotiate
> +VIRTIO_F_SR_IOV and complete the feature negotiation
> +(including setting the DRIVER_OK \field{status} bit) before
> +enabling virtual functions through the device's PCI SR-IOV
> +capability structure.
> +
> \devicenormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
>
> A device MUST offer VIRTIO_F_VERSION_1. A device MAY fail to operate further
> @@ -5400,6 +5413,10 @@ buffers in the same order in which they have been
> available.
> A device MAY fail to operate further if VIRTIO_F_IO_BARRIER
> is not accepted.
>
> +A device SHOULD offer VIRTIO_F_SR_IOV if it is a PCI device
> +and presents a PCI SR-IOV capability structure, otherwise
> +it MUST NOT offer VIRTIO_F_SR_IOV.
> +
> \section{Legacy Interface: Reserved Feature Bits}\label{sec:Reserved Feature
> Bits / Legacy Interface: Reserved Feature Bits}
>
> Transitional devices MAY offer the following:
> --
> 2.17.0
>
>
> -
> To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org
-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org
