ping... for entire v3 version of the patchset. http://patchwork.ozlabs.org/project/qemu-devel/list/?submitter=68462
- Jitendra On Wed, May 18, 2016 at 4:50 PM, Jitendra Kolhe <jitendra.ko...@hpe.com> wrote: > While measuring live migration performance for qemu/kvm guest, it was observed > that the qemu doesn’t maintain any intelligence for the guest ram pages > released > by the guest balloon driver and treat such pages as any other > normal guest ram pages. This has direct impact on overall migration time for > the guest which has released (ballooned out) memory to the host. > > In case of large systems, where we can configure large guests with 1TB and > with considerable amount of memory released by balloon driver to the host, > the migration time gets worse. > > The solution proposed below is local to qemu (and does not require any > modification to Linux kernel or any guest driver). We have verified the fix > for large guests =1TB on HPE Superdome X (which can support up to 240 cores > and 12TB of memory). > > During live migration, as part of first iteration in ram_save_iterate() -> > ram_find_and_save_block () will try to migrate ram pages even if they are > released by vitrio-balloon driver (balloon inflate). Although these pages > which are returned to the host by virtio-balloon driver are zero pages, > the migration algorithm will still end up scanning the entire page > ram_find_and_save_block() -> ram_save_page()/ram_save_compressed_page() -> > save_zero_page() -> is_zero_range(). We also end-up sending header information > over network for these pages during migration. This adds to the total > migration time. > > The solution creates a balloon bitmap ramblock as a part of virtio-balloon > device initialization. The bits in the balloon bitmap represent a guest ram > page of size 1UL << VIRTIO_BALLOON_PFN_SHIFT or 4K. If TARGET_PAGE_BITS <= > VIRTIO_BALLOON_PFN_SHIFT, ram_addr offset for the dirty page which is used by > dirty page bitmap during migration is checked against the balloon bitmap as > is, if the bit is set in the balloon bitmap, the corresponding ram page will > be > excluded from scanning and sending header information during migration. In > case > TARGET_PAGE_BITS > VIRTIO_BALLOON_PFN_SHIFT for a given dirty page ram_addr, > all sub-pages of 1UL << VIRTIO_BALLOON_PFN_SHIFT size should be ballooned out > to avoid zero page scan and sending header information. > > The bitmap represents entire guest ram memory till max configured memory. > Guest ram pages claimed by the virtio-balloon driver will be represented by 1 > in the bitmap. Since the bitmap is maintained as a ramblock, it’s migrated to > target as part migration’s ram iterative and ram complete phase. So that > substituent migrations from the target can continue to use optimization. > > A new migration capability called skip-balloon is introduced. The user can > disable the capability in cases where user does not expect much benefit or in > case the migration is from an older version. > > During live migration setup the optimization can be set to disabled state if > . no virtio-balloon device is initialized. > . skip-balloon migration capability is disabled. > . If the guest virtio-balloon driver has not set > VIRTIO_BALLOON_F_MUST_TELL_HOST > flag. Which means the guest may start using a ram pages freed by guest > balloon > driver, even before the host/qemu is aware of it. In such case, the > optimization is disabled so that the ram pages that are being used by the > guest will continue to be scanned and migrated. > > Balloon bitmap ramblock size is set to zero if the optimization is disabled, > to avoid overhead of migrating the bitmap. If the bitmap is not migrated to > the target, the destination starts with a fresh bitmap and tracks the > ballooning operation thereafter. > > Jitendra Kolhe (4): > balloon: maintain bitmap for pages released by guest balloon driver. > balloon: add balloon bitmap migration capability and setup bitmap > migration status. > balloon: reset balloon bitmap ramblock size on source and target. > migration: skip scanning and migrating ram pages released by > virtio-balloon driver. > > Changed in v2: > - Resolved compilation issue for qemu-user binaries in exec.c > - Localize balloon bitmap test to save_zero_page(). > - Updated version string for newly added migration capability to 2.7. > - Made minor modifications to patch commit text. > > Changed in v3: > - Add balloon bitmap to RAMBlock. > - Resolve bitmap offset calculation by translating host addr back to a > RAMBlock and ram_addr > - Add balloon bitmap support for case if TARGET_PAGE_BITS > > VIRTIO_BALLOON_PFN_SHIFT. > - Remove dependency of skip-balloon migration capability on postcopy > migration. > - Disable optimization if the guest balloon driver does not support > VIRTIO_BALLOON_F_MUST_TELL_HOST feature. > - Split single patch into 4 small patches. > > balloon.c | 196 > ++++++++++++++++++++++++++++++++++++- > exec.c | 6 ++ > hw/virtio/virtio-balloon.c | 42 +++++++- > include/hw/virtio/virtio-balloon.h | 1 + > include/migration/migration.h | 1 + > include/sysemu/balloon.h | 13 ++- > migration/migration.c | 9 ++ > migration/ram.c | 9 +- > qapi-schema.json | 5 +- > 9 files changed, 276 insertions(+), 6 deletions(-) > > -- > 1.8.3.1 > >