[openstack-dev] [nova] Live migration performance tests on 100 compute nodes

[Koniszewski, Pawel]

Hello everyone,

We made a research to see how live migration performance varies between 
different configurations, especially we aimed to test tunneled vs non-tunneled 
live migrations. To test live migration we simulated a case of 0-day patching 
of 100 compute nodes (including reboot) with workloads that are close to the 
real world workloads. All the results were published [1] along with environment 
configuration and how we built test framework. Hope you find this useful.

[1] https://01.org/openstack/blogs/pkoniszewski/2016/ossc-zero-day-patching

Kind Regards,
Pawel Koniszewski

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Live migrations: Post-Copy and Auto-Converge features research

[Koniszewski, Pawel]

Adding operators list, as it might be interesting for them. Please note that 
some of tested options are not available in nova, i.e., XBZRLE and 
postcopy-after-precopy. Regarding postcopy, nova automatically decides whether 
live migration should be switched to postcopy mode (only if 
live_migration_permit_post_copy is set to True, by default it is set to False), 
postcopy-after-precopy is an internal virsh command which is not exposed 
through the libvirt API.

Kind Regards,
Pawel Koniszewski

From: Vlad Nykytiuk [mailto:vnykyt...@mirantis.com]
Sent: Monday, November 7, 2016 9:35 PM
To: openstack-dev@lists.openstack.org
Subject: [openstack-dev] [nova] Live migrations: Post-Copy and Auto-Converge 
features research

Hi,

As you may know, currently QEMU supports several features that help live 
migrations to operate more predictively. These features are: auto-converge and 
post-copy.
I made a research on performance characteristics of these two features, you can 
find it by the following link:
https://rk4n.github.io/2016/08/10/qemu-post-copy-and-auto-converge-features/

Hope you find it helpful.

Thanks
—
Vlad

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][kolla] Live migration, VNC / SPICE address?

[Koniszewski, Pawel]

Dave,

Thanks for pointing this out, looks like it is a regression in nova introduced 
during Newton cycle. At some point we moved checks of graphic and serial 
consoles to the live migration pre-checks on source node, but we never moved 
piece of code that populates this data while executing pre-checks on 
destination node.

I proposed a fix, you can take a look whether this fixes an issue that you 
observed - https://review.openstack.org/#/c/362756/. If yes I will update 
appropriate unit tests.
The bug is tracked here https://bugs.launchpad.net/nova/+bug/1618392

Kind Regards,
Pawel Koniszewski

From: Dave Walker [mailto:em...@daviey.com]
Sent: Tuesday, August 30, 2016 10:11 AM
To: OpenStack Development Mailing List 
Subject: [openstack-dev] [nova][kolla] Live migration, VNC / SPICE address?

Hi,

In Kolla, we are having an issue with Nova's VNC / SPICE address and live 
migration.  Currently, we are declaring the IP address for vncserver_listen on 
each node (via ansible).  However, when a live migration is performed, it fails 
due to this address not being available.

The hack is to switch the vncserver_listen to be 0.0.0.0, but this is horribly 
insecure and breaks the network isolation that kolla supports.

Looking at the relevant code, this looks like it should be functional via 
VIR_DOMAIN_XML_MIGRATABLE, but it doesn't seem to be working.

Could someone from Nova help us determine the cause?  We are tracking this as 
bug 1596724

https://github.com/openstack/nova/blob/04cef3b5d03be3db7efab6896de867fc2cbbd03a/nova/virt/libvirt/driver.py#L5393

https://github.com/openstack/nova/blob/04cef3b5d03be3db7efab6896de867fc2cbbd03a/nova/virt/libvirt/driver.py#L5754

Thanks

--
Kind Regards,
Dave Walker
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] Removal of live_migration_flag and block_migration_flag config options

[Koniszewski, Pawel]

In Mitaka development cycle 'live_migration_flag' and 'block_migration_flag' 
have been marked as deprecated for removal. I'm working on a patch [1] to 
remove both of them and want to ask what we should do with 
live_migration_tunnelled logic.

The default configuration of both flags contain VIR_MIGRATE_TUNNELLED option. 
It is there to avoid the need to configure the network to allow direct 
communication between hypervisors. However, tradeoff is that it slows down all 
migrations by up to 80% due to increased number of memory copies and 
single-threaded encryption mechanism in Libvirt. By 80% here I mean that 
transfer between source and destination node is around 2Gb/s on a 10Gb network. 
I believe that this is a configuration issue and people deploying OpenStack are 
not aware that live migrations with this flag will not work. I'm not sure that 
this is something we wanted to achieve. AFAIK most operators are turning it OFF 
in order to make live migration usable.

Going to a new flag that is there to keep possibility to turn tunneling on - 
Live_migration_tunnelled [2] which is a tri-state boolean - None, False, True:

* True - means that live migrations will be tunneled through libvirt.
* False - no tunneling, native hypervisor transport.
* None - nova will choose default based on, e.g., the availability of native 
encryption support in the hypervisor. (Default value)

Right now we don't have any logic implemented for None value which is a default 
value. So the question here is should I implement logic so that if 
live_migration_tunnelled=None it will still use VIR_MIGRATE_TUNNELLED if native 
encryption is not available? Given the impact of this flag I'm not sure that we 
really want to keep it there. Another option is to change default value of 
live_migration_tunnelled to be True. In both cases we will again end up with 
slower LM and people complaining that LM does not work at all in OpenStack.

Thoughts?

[1] https://review.openstack.org/#/c/334860/
[2] 
https://github.com/openstack/nova/blob/be59c19c969acf6b25b0711f0ebfb26aaed0a171/nova/conf/libvirt.py#L107

Kind Regards,
Pawel Koniszewski

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][libvirt] Deprecating the live_migration_flag and block_migration_flag config options

[Koniszewski, Pawel]

There is another fix proposed by Eli Qiao long time ago:

https://review.openstack.org/#/c/310707

Basically it blocks block live migration with BDMs and tunneling during 
pre-checks.

From: Timofei Durakov [mailto:tdura...@mirantis.com]
Sent: Tuesday, June 7, 2016 9:04 AM
To: OpenStack Development Mailing List (not for usage questions) 

Subject: Re: [openstack-dev] [nova][libvirt] Deprecating the 
live_migration_flag and block_migration_flag config options

I've submitted one more patch with potential fix:

https://review.openstack.org/#/c/326258/

Timofey

On Mon, Jun 6, 2016 at 11:58 PM, Timofei Durakov 
> wrote:
On Mon, Jun 6, 2016 at 11:26 PM, Matt Riedemann 
> wrote:
On 6/6/2016 12:15 PM, Matt Riedemann wrote:
On 1/8/2016 12:28 PM, Mark McLoughlin wrote:
On Fri, 2016-01-08 at 14:11 +, Daniel P. Berrange wrote:
On Thu, Jan 07, 2016 at 09:07:00PM +, Mark McLoughlin wrote:
On Thu, 2016-01-07 at 12:23 +0100, Sahid Orentino Ferdjaoui
wrote:
On Mon, Jan 04, 2016 at 09:12:06PM +, Mark McLoughlin
wrote:
Hi

commit 8ecf93e[1] got me thinking - the live_migration_flag
config option unnecessarily allows operators choose arbitrary
behavior of the migrateToURI() libvirt call, to the extent
that we allow the operator to configure a behavior that can
result in data loss[1].

I see that danpb recently said something similar:

https://review.openstack.org/171098

"Honestly, I wish we'd just kill off  'live_migration_flag'
and 'block_migration_flag' as config options. We really
should not be exposing low level libvirt API flags as admin
tunable settings.

Nova should really be in charge of picking the correct set of
flags for the current libvirt version, and the operation it
needs to perform. We might need to add other more sensible
config options in their place [..]"

Nova should really handle internal flags and this serie is
running in the right way.
...

4) Add a new config option for tunneled versus native:

[libvirt] live_migration_tunneled = true

This enables the use of the VIR_MIGRATE_TUNNELLED flag. We
have historically defaulted to tunneled mode because it
requires the least configuration and is currently the only
way to have a secure migration channel.

danpb's quote above continues with:

"perhaps a "live_migration_secure_channel" to indicate that
migration must use encryption, which would imply use of
TUNNELLED flag"

So we need to discuss whether the config option should
express the choice of tunneled vs native, or whether it
should express another choice which implies tunneled vs
native.

https://review.openstack.org/263434

We probably have to consider that operator does not know much
about internal libvirt flags, so options we are exposing for
him should reflect benefice of using them. I commented on your
review we should at least explain benefice of using this option
whatever the name is.

As predicted, plenty of discussion on this point in the review
:)

You're right that we don't give the operator any guidance in the
help message about how to choose true or false for this:

Whether to use tunneled migration, where migration data is
transported over the libvirtd connection. If True, we use the
VIR_MIGRATE_TUNNELLED migration flag

libvirt's own docs on this are here:

https://libvirt.org/migration.html#transport

which emphasizes:

- the data copies involved in tunneling - the extra configuration
steps required for native - the encryption support you get when
tunneling

The discussions I've seen on this topic wrt Nova have revolved
around:

- that tunneling allows for an encrypted transport[1] - that
qemu's NBD based drive-mirror block migration isn't supported
using tunneled mode, and that danpb is working on fixing this
limitation in libvirt - "selective" block migration[2] won't work
with the fallback qemu block migration support, and so won't
currently work in tunneled mode

I'm not working on fixing it, but IIRC some other dev had proposed
patches.

So, the advise to operators would be:

- You may want to choose tunneled=False for improved block
migration capabilities, but this limitation will go away in
future. - You may want to choose tunneled=False if you wish to
trade and encrypted transport for a (potentially negligible)
performance improvement.

Does that make sense?

As for how to name the option, and as I said in the review, I
think it makes sense to be straightforward here and make it
clearly about choosing to disable libvirt's tunneled transport.

If we name it any other way, I think our explanation for
operators will immediately jump to explaining (a) that it
influences the TUNNELLED flag, and (b) the differences between
the tunneled and native transports. So, if we're going to have to
talk about tunneled versus native, why obscure that detail?

Ultimately we need to recognise that libvirt's tunnelled mode was
added as a 

[openstack-dev] [nova] Migration progress

[Koniszewski, Pawel]

Hello everyone,

On the yesterday's live migration meeting we had concerns that interval of
writing migration progress to the database is too short.

Information about migration progress will be stored in the database and
exposed through the API (/servers//migrations/). In current
proposition [1] migration progress will be updated every 2 seconds. It
basically means that every 2 seconds a call through RPC will go from compute
to conductor to write migration data to the database. In case of parallel
live migrations each migration will report progress by itself.

Isn't 2 seconds interval too short for updates if the information is exposed
through the API and it requires RPC and DB call to actually save it in the
DB?

Our default configuration allows only for 1 concurrent live migration [2],
but it might vary between different deployments and use cases as it is
configurable. Someone might want to trigger 10 (or even more) parallel live
migrations and each might take even a day to finish in case of block
migration. Also if deployment is big enough rabbitmq might be fully-loaded.
I'm not sure whether updating each migration every 2 seconds makes sense in
this case. On the other hand it might be hard to observe fast enough that
migration is stuck if we increase this interval...

What's worth mentioning is that during Nova Midcycle we had discussion about
refactoring live migration flow. Proposition was to get rid of
compute->compute communication during live migration and make conductor
conduct whole process. Nikola proposed that in such case nova-compute should
be stateful and store migration status on compute node [3]. We might be able
to use state stored on compute node and get rid of RPC-DB queries every 2
seconds.

Thoughts?

Kind Regards,
Pawel Koniszewski

[1] https://review.openstack.org/#/c/258813
[2]
http://docs.openstack.org/liberty/config-reference/content/list-of-compute-c
onfig-options.html
[3] https://etherpad.openstack.org/p/mitaka-nova-midcycle


smime.p7s
Description: S/MIME cryptographic signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [RFC] how to enable xbzrle compress for live migration

[Koniszewski, Pawel]

2015-11-30 14:45 GMT+08:00 Koniszewski, Pawel <pawel.koniszew...@intel.com>:
> -Original Message-
> From: Murray, Paul (HP Cloud) [mailto:pmur...@hpe.com]
> Sent: Friday, November 27, 2015 4:29 PM
> To: Daniel P. Berrange; Carlton, Paul (Cloud Services)
> Cc: 少合冯; OpenStack Development Mailing List (not for usage questions);
> John Garbutt; Koniszewski, Pawel; Jin, Yuntong; Feng, Shaohe; Qiao, Liyong
> Subject: RE: [nova] [RFC] how to enable xbzrle compress for live migration
>
>
>
> > -Original Message-
> > From: Daniel P. Berrange [mailto:berra...@redhat.com]
> > Sent: 26 November 2015 17:58
> > To: Carlton, Paul (Cloud Services)
> > Cc: 少合冯; OpenStack Development Mailing List (not for usage
> questions);
> > John Garbutt; pawel.koniszew...@intel.com; yuntong@intel.com;
> > shaohe.f...@intel.com; Murray, Paul (HP Cloud); liyong.q...@intel.com
> > Subject: Re: [nova] [RFC] how to enable xbzrle compress for live
> > migration
> >
> > On Thu, Nov 26, 2015 at 05:49:50PM +, Paul Carlton wrote:
> > > Seems to me the prevailing view is that we should get live migration
> > > to figure out the best setting for itself where possible.  There was
> > > discussion of being able have a default policy setting that will
> > > allow the operator to define balance between speed of migration and
> > > impact on the instance.  This could be a global default for the
> > > cloud with overriding defaults per aggregate, image, tenant and
> > > instance as well as the ability to vary the setting during the
migration
> operation.
> > >
> > > Seems to me that items like compression should be set in
> > > configuration files based on what works best given the cloud
operator's
> environment?
> >
> > Merely turning on use of compression is the "easy" bit - there needs
> > to be a way to deal with compression cache size allocation, which
> > needs to have some smarts in Nova, as there's no usable "one size fits
> > all" value for the compression cache size. If we did want to hardcode
> > a compression cache size, you'd have to pick set it as a scaling factor
against
> the guest RAM size.
> > This is going to be very heavy on memory usage, so there needs careful
> > design work to solve the problem of migration compression triggering
> > host OOM scenarios, particularly since we can have multiple concurrent
> > migrations.
> >
>
>
> Use cases for live migration generally fall into two types:
>
> 1. I need to empty the host (host maintenance/reboot)
>
> 2. I generally want to balance load on the cloud
>
> The first case is by far the most common need right now and in that case
the
> node gets progressively more empty as VMs are moved off. So the resources
> available for caching etc. grow as the process goes on.
>I'd rather say that these resources might shrink. You need to turn off one
compute node, stack more VMs on remaining compute nodes and you need to
allocate cache on both sides, source and destination.

>why do we need on destination? 

XBZRLE sends only a delta over network and it works in two phases:
compressing and decompressing. During compression the original page and
updated page are XORed together and resulting information is passed over to
the RLE algorithm - the output is the delta page which is sent over network
to destination host. During decompression run length decodes each pair of
symbol-counter and the original page is XORed with the result from the run
length decoding - the output is the updated page. It means that it needs to
allocate cache on source and destination node.

> The second case is less likely to be urgent from the operators point of
view,
> so doing things more slowly may not be a problem.
>
> So looking at how much resource is available at the start of a migration
and
> deciding then what to do on a per VM basis is probably not a bad idea.
> Especially if we can differentiate between the two cases.


smime.p7s
Description: S/MIME cryptographic signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [RFC] how to enable xbzrle compress for live migration

[Koniszewski, Pawel]

> -Original Message-
> From: Murray, Paul (HP Cloud) [mailto:pmur...@hpe.com]
> Sent: Friday, November 27, 2015 4:29 PM
> To: Daniel P. Berrange; Carlton, Paul (Cloud Services)
> Cc: 少合冯; OpenStack Development Mailing List (not for usage questions);
> John Garbutt; Koniszewski, Pawel; Jin, Yuntong; Feng, Shaohe; Qiao, Liyong
> Subject: RE: [nova] [RFC] how to enable xbzrle compress for live migration
>
>
>
> > -Original Message-
> > From: Daniel P. Berrange [mailto:berra...@redhat.com]
> > Sent: 26 November 2015 17:58
> > To: Carlton, Paul (Cloud Services)
> > Cc: 少合冯; OpenStack Development Mailing List (not for usage
> questions);
> > John Garbutt; pawel.koniszew...@intel.com; yuntong@intel.com;
> > shaohe.f...@intel.com; Murray, Paul (HP Cloud); liyong.q...@intel.com
> > Subject: Re: [nova] [RFC] how to enable xbzrle compress for live
> > migration
> >
> > On Thu, Nov 26, 2015 at 05:49:50PM +, Paul Carlton wrote:
> > > Seems to me the prevailing view is that we should get live migration
> > > to figure out the best setting for itself where possible.  There was
> > > discussion of being able have a default policy setting that will
> > > allow the operator to define balance between speed of migration and
> > > impact on the instance.  This could be a global default for the
> > > cloud with overriding defaults per aggregate, image, tenant and
> > > instance as well as the ability to vary the setting during the
migration
> operation.
> > >
> > > Seems to me that items like compression should be set in
> > > configuration files based on what works best given the cloud
operator's
> environment?
> >
> > Merely turning on use of compression is the "easy" bit - there needs
> > to be a way to deal with compression cache size allocation, which
> > needs to have some smarts in Nova, as there's no usable "one size fits
> > all" value for the compression cache size. If we did want to hardcode
> > a compression cache size, you'd have to pick set it as a scaling factor
against
> the guest RAM size.
> > This is going to be very heavy on memory usage, so there needs careful
> > design work to solve the problem of migration compression triggering
> > host OOM scenarios, particularly since we can have multiple concurrent
> > migrations.
> >
>
>
> Use cases for live migration generally fall into two types:
>
> 1. I need to empty the host (host maintenance/reboot)
>
> 2. I generally want to balance load on the cloud
>
> The first case is by far the most common need right now and in that case
the
> node gets progressively more empty as VMs are moved off. So the resources
> available for caching etc. grow as the process goes on.
>I'd rather say that these resources might shrink. You need to turn off one
compute node, stack more VMs on remaining compute nodes and you need to
allocate cache on both sides, source and destination.

>why do we need on destination?
>XBZRLE sends only a delta over network and it works in two phases:
>compressing and decompressing. During compression the original page and
>updated page are XORed together and resulting information is passed over to
>the RLE algorithm - the output is the delta page which is sent over network
>to destination host. During decompression run length decodes each pair of
>symbol-counter and the original page is XORed with the result from the run
>length decoding - the output is the updated page. It means that it needs to
>allocate cache on source and destination node.

>But I think the RAM on the destination is the  original  page . Just 
>decompression 
>with the delta.

>It does not need extra cache.

Ah, got it. Misunderstood conception - destination host is updated in every 
iteration so no need to store a cache there.

Then we need something smart in nova to decide whether xbzrle can be turned on. 
 

> The second case is less likely to be urgent from the operators point of
view,
> so doing things more slowly may not be a problem.
>
> So looking at how much resource is available at the start of a migration
and
> deciding then what to do on a per VM basis is probably not a bad idea.
> Especially if we can differentiate between the two cases.



smime.p7s
Description: S/MIME cryptographic signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [RFC] how to enable xbzrle compress for live migration

[Koniszewski, Pawel]

> -Original Message-
> From: Murray, Paul (HP Cloud) [mailto:pmur...@hpe.com]
> Sent: Friday, November 27, 2015 4:29 PM
> To: Daniel P. Berrange; Carlton, Paul (Cloud Services)
> Cc: 少合冯; OpenStack Development Mailing List (not for usage questions);
> John Garbutt; Koniszewski, Pawel; Jin, Yuntong; Feng, Shaohe; Qiao, Liyong
> Subject: RE: [nova] [RFC] how to enable xbzrle compress for live migration
> 
> 
> 
> > -Original Message-
> > From: Daniel P. Berrange [mailto:berra...@redhat.com]
> > Sent: 26 November 2015 17:58
> > To: Carlton, Paul (Cloud Services)
> > Cc: 少合冯; OpenStack Development Mailing List (not for usage
> questions);
> > John Garbutt; pawel.koniszew...@intel.com; yuntong@intel.com;
> > shaohe.f...@intel.com; Murray, Paul (HP Cloud); liyong.q...@intel.com
> > Subject: Re: [nova] [RFC] how to enable xbzrle compress for live
> > migration
> >
> > On Thu, Nov 26, 2015 at 05:49:50PM +, Paul Carlton wrote:
> > > Seems to me the prevailing view is that we should get live migration
> > > to figure out the best setting for itself where possible.  There was
> > > discussion of being able have a default policy setting that will
> > > allow the operator to define balance between speed of migration and
> > > impact on the instance.  This could be a global default for the
> > > cloud with overriding defaults per aggregate, image, tenant and
> > > instance as well as the ability to vary the setting during the migration
> operation.
> > >
> > > Seems to me that items like compression should be set in
> > > configuration files based on what works best given the cloud operator's
> environment?
> >
> > Merely turning on use of compression is the "easy" bit - there needs
> > to be a way to deal with compression cache size allocation, which
> > needs to have some smarts in Nova, as there's no usable "one size fits
> > all" value for the compression cache size. If we did want to hardcode
> > a compression cache size, you'd have to pick set it as a scaling factor 
> > against
> the guest RAM size.
> > This is going to be very heavy on memory usage, so there needs careful
> > design work to solve the problem of migration compression triggering
> > host OOM scenarios, particularly since we can have multiple concurrent
> > migrations.
> >
> 
> 
> Use cases for live migration generally fall into two types:
> 
> 1. I need to empty the host (host maintenance/reboot)
> 
> 2. I generally want to balance load on the cloud
> 
> The first case is by far the most common need right now and in that case the
> node gets progressively more empty as VMs are moved off. So the resources
> available for caching etc. grow as the process goes on.

I'd rather say that these resources might shrink. You need to turn off one 
compute node, stack more VMs on remaining compute nodes and you need to 
allocate cache on both sides, source and destination.

> The second case is less likely to be urgent from the operators point of view,
> so doing things more slowly may not be a problem.
> 
> So looking at how much resource is available at the start of a migration and
> deciding then what to do on a per VM basis is probably not a bad idea.
> Especially if we can differentiate between the two cases.
> 
> 
> > Regards,
> > Daniel
> > --
> > |: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/
> :|
> > |: http://libvirt.org  -o- http://virt-manager.org 
> > :|
> > |: http://autobuild.org   -o- http://search.cpan.org/~danberr/ 
> > :|
> > |: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc 
> > :|



smime.p7s
Description: S/MIME cryptographic signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [RFC] how to enable xbzrle compress for live migration

[Koniszewski, Pawel]

> -Original Message-
> From: Daniel P. Berrange [mailto:berra...@redhat.com]
> Sent: Friday, November 27, 2015 12:50 PM
> To: 少合冯
> Cc: Feng, Shaohe; OpenStack Development Mailing List (not for usage
> questions); Xiao, Guangrong; Ding, Jian-feng; Dong, Eddie; Wang, Yong Y; Jin,
> Yuntong
> Subject: Re: [openstack-dev] [nova] [RFC] how to enable xbzrle compress for
> live migration
> 
> On Fri, Nov 27, 2015 at 07:37:50PM +0800, 少合冯 wrote:
> > 2015-11-27 2:19 GMT+08:00 Daniel P. Berrange :
> >
> > > On Thu, Nov 26, 2015 at 05:39:04PM +, Daniel P. Berrange wrote:
> > > > On Thu, Nov 26, 2015 at 11:55:31PM +0800, 少合冯 wrote:
> > > > > 3.  dynamically choose when to activate xbzrle compress for live
> > > migration.
> > > > >  This is the best.
> > > > >  xbzrle really wants to be used if the network is not able
> > > > > to keep
> > > up
> > > > > with the dirtying rate of the guest RAM.
> > > > >  But how do I check the coming migration fit this situation？
> > > >
> > > > FWIW, if we decide we want compression support in Nova, I think
> > > > that having the Nova libvirt driver dynamically decide when to use
> > > > it is the only viable approach. Unfortunately the way the QEMU
> > > > support is implemented makes it very hard to use, as QEMU forces
> > > > you to decide to use it upfront, at a time when you don't have any
> > > > useful information on which to make the decision :-(  To be useful
> > > > IMHO, we really need the ability to turn on compression on the fly
> > > > for an existing active migration process. ie, we'd start migration
> > > > off and let it run and only enable compression if we encounter
> problems with completion.
> > > > Sadly we can't do this with QEMU as it stands today :-(
> > > >
> > >
> > [Shaohe Feng]
> > Add more guys working on kernel/hypervisor in our loop.
> > Wonder whether there will be any good solutions to improve it in QEMU
> > in future.
> >
> >
> > > > Oh and of course we still need to address the issue of RAM usage
> > > > and communicating that need with the scheduler in order to avoid
> > > > OOM scenarios due to large compression cache.
> > > >
> > > > I tend to feel that the QEMU compression code is currently broken
> > > > by design and needs rework in QEMU before it can be pratically
> > > > used in an autonomous fashion :-(
> > >
> > > Actually thinking about it, there's not really any significant
> > > difference between Option 1 and Option 3. In both cases we want a
> > > nova.conf setting live_migration_compression=on|off to control
> > > whether we want to *permit* use  of compression.
> > >
> > > The only real difference between 1 & 3 is whether migration has
> > > compression enabled always, or whether we turn it on part way though
> > > migration.
> > >
> > > So although option 3 is our desired approach (which we can't
> > > actually implement due to QEMU limitations), option 1 could be made
> > > fairly similar if we start off with a very small compression cache
> > > size which would have the effect of more or less disabling
> > > compression initially.
> > >
> > > We already have logic in the code for dynamically increasing the max
> > > downtime value, which we could mirror here
> > >
> > > eg something like
> > >
> > >  live_migration_compression=on|off
> > >
> > >   - Whether to enable use of compression
> > >
> > >  live_migration_compression_cache_ratio=0.8
> > >
> > >   - The maximum size of the compression cache relative to
> > > the guest RAM size. Must be less than 1.0
> > >
> > >  live_migration_compression_cache_steps=10
> > >
> > >   - The number of steps to take to get from initial cache
> > > size to the maximum cache size
> > >
> > >  live_migration_compression_cache_delay=75
> > >
> > >   - The time delay in seconds between increases in cache
> > > size
> > >
> > >
> > > In the same way that we do with migration downtime, instead of
> > > increasing cache size linearly, we'd increase it in ever larger
> > > steps until we hit the maximum. So we'd start off fairly small a few
> > > MB, and monitoring the cache hit rates, we'd increase it
> > > periodically.  If the number of steps configured and time delay
> > > between steps are reasonably large, that would have the effect that
> > > most migrations would have a fairly small cache and would complete
> > > without needing much compression overhead.
> > >
> > > Doing this though, we still need a solution to the host OOM scenario
> > > problem. We can't simply check free RAM at start of migration and
> > > see if there's enough to spare for compression cache, as the
> > > schedular can spawn a new guest on the compute host at any time,
> > > pushing us into OOM. We really need some way to indicate that there
> > > is a (potentially very large) extra RAM overhead for the guest during
> migration.

What about CPU? We might end up with live migration that degrades performance 
of other VMs on source and/or destination node. AFAIK 

Re: [openstack-dev] [nova] [RFC] how to enable xbzrle compress for live migration

[Koniszewski, Pawel]

> -Original Message-
> From: Daniel P. Berrange [mailto:berra...@redhat.com]
> Sent: Friday, November 27, 2015 1:24 PM
> To: Koniszewski, Pawel
> Cc: OpenStack Development Mailing List (not for usage questions); ???; Feng,
> Shaohe; Xiao, Guangrong; Ding, Jian-feng; Dong, Eddie; Wang, Yong Y; Jin,
> Yuntong
> Subject: Re: [openstack-dev] [nova] [RFC] how to enable xbzrle compress for
> live migration
>
> On Fri, Nov 27, 2015 at 12:17:06PM +, Koniszewski, Pawel wrote:
> > > -Original Message-
> > > > > Doing this though, we still need a solution to the host OOM
> > > > > scenario problem. We can't simply check free RAM at start of
> > > > > migration and see if there's enough to spare for compression
> > > > > cache, as the schedular can spawn a new guest on the compute
> > > > > host at any time, pushing us into OOM. We really need some way
> > > > > to indicate that there is a (potentially very large) extra RAM
> > > > > overhead for the guest during
> > > migration.
> >
> > What about CPU? We might end up with live migration that degrades
> > performance of other VMs on source and/or destination node. AFAIK CPUs
> > are heavily oversubscribed in many cases and this does not help.
> > I'm not sure that this thing fits into Nova as it requires resource
> > monitoring.
>
> Nova already has the ability to set CPU usage tuning rules against each VM.
> Since the CPU overhead is attributed to the QEMU process, these existing
> tuning rules will apply. So there would only be impact on other VMs, if you
> do
> not have any CPU tuning rules set in Nova.

Not sure I understand it correctly, I assume that you are talking about CPU
pinning. Does it mean that compression/decompression runs as part of VM
threads?

If not then, well, it will require all VMs to be pinned on both hosts, source
and destination (and in the whole cluster because of static configuration...).
Also what about operating system performance? Will QEMU distinct OS processes
somehow and won't affect them?

Also, nova can reserve some memory for the host. Will QEMU also respect it?

I like all this stuff about XBZRLE, but in my understanding it is very
resource-sensitive feature.

Kind Regards,
Pawel Koniszewski


smime.p7s
Description: S/MIME cryptographic signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] live migration management approach

[Koniszewski, Pawel]

> -Original Message-
> From: Paul Carlton [mailto:paul.carlt...@hpe.com]
> Sent: Tuesday, November 10, 2015 4:51 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: [openstack-dev] [nova] live migration management approach
>
> All
>
> I inherited the task of producing an upstream spec for querying and aborting
> ongoing migrations from a co-worker and submitted a new spec
> https://review.openstack.org/#/c/228828//.  A related spec for pausing an
> instance being migrated to enable the migration to complete is also proposed
> https://review.openstack.org/#/c/229040/.
>
> However I am now wondering if building these capabilities around the
> migration object is the right way to go.  I'm now wondering if this is right
> way
> to approach this or whether it would be better to implement some additional
> operations on hte instance object?
>
> I'm become aware that details of the progress of a migration operation is
> already available to both cloud admins and instance owners via the server
> get
> details operation, which reports the progress of migration.
>
> Cancelling a migration could be implemented as a reset-task operation on an
> instance?

Reset-task operation would mean that we need to raise exceptions (bad request
or whatever else) for almost every vm task. How many of them are really
cancelable? Also I have a feeling that it is live migration related action so
current approach is ok with me.

> The proposal in https://review.openstack.org/#/c/229040/ relates to
> providing the facility to pause an instance that is being migrated so that
> the
> migration can complete (i.e. in the case where the instance is dirtying
> memory quicker than the migration processing can copy it).
> This could be implemented by simply allowing the existing pause operation
> on a migrating instance.  The existing implementation of live migration
> means
> that the instance will automatically resume again when the migration
> completes.  We could amend instance pause so that it issues a warning if a
> pause operation is performed on an instance that has a task state of
> migrating so the user is made aware that the pause will only be temporary.

My personal feeling is that we should not reuse the same operation for two
different purposes. IMHO pause operation should be as it is - operator pauses
an instance and it remains paused also after live migration completion. Such
case is valid from user perspective who might not be aware of ongoing live
migration. However, this might require some work in Libvirt, not sure we can
do it without changing/adding new Libvirt behavior.

Kind Regards,
Pawel Koniszewski


smime.p7s
Description: S/MIME cryptographic signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] live migration in Mitaka

[Koniszewski, Pawel]

> -Original Message-
> From: Mathieu Gagné [mailto:mga...@internap.com]
> Sent: Thursday, October 1, 2015 7:24 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [nova] live migration in Mitaka
> 
> On 2015-10-01 7:26 AM, Kashyap Chamarthy wrote:
> > On Wed, Sep 30, 2015 at 11:25:12AM +, Murray, Paul (HP Cloud) wrote:
> >>
> >>> Please respond to this post if you have an interest in this and what
> >>> you would like to see done.  Include anything you are already
> >>> getting on with so we get a clear picture.
> >>
> >> Thank you to those who replied to this thread. I have used the
> >> contents to start an etherpad page here:
> >>
> >> https://etherpad.openstack.org/p/mitaka-live-migration
> >
> > I added a couple of URLs for upstream libvirt work that allow for
> > selective block device migration, and the in-progress generic TLS
> > support work by Dan Berrange in upstream QEMU.
> >
> >> I have taken the liberty of listing those that responded to the
> >> thread and the authors of mentioned patches as interested people.
> >
> >> From the responses and looking at the specs up for review it looks
> >> like there are about five areas that could be addressed in Mitaka and
> >> several others that could come later. The first five are:
> >>
> >>
> >> - migrating instances with a mix of local disks and cinder volumes
> >
> > IIUC, this is possible with the selective block device migration work
> > merged in upstream libvirt:
> >
> > https://www.redhat.com/archives/libvir-list/2015-May/msg00955.html
> >
> 
> Can someone explain to me what is the actual "disk name" I have to pass in
> to libvirt? I couldn't find any documentation about how to use this
feature.

You have to pass device names from /dev/, e.g., if a VM has ephemeral disk
attached at /dev/vdb you need to pass in 'vdb'. Format expected by
migrate_disks is ",...".

Kind Regards,
Pawel Koniszewski


smime.p7s
Description: S/MIME cryptographic signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] live migration in Mitaka

[Koniszewski, Pawel]

> -Original Message-
> From: Murray, Paul (HP Cloud) [mailto:pmur...@hpe.com]
> Sent: Wednesday, September 30, 2015 1:25 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [nova] live migration in Mitaka
>
>
> > Please respond to this post if you have an interest in this and what you
> would like to see done.
> > Include anything you are already getting on with so we get a clear 
> > picture.
>
> Thank you to those who replied to this thread. I have used the contents to
> start an etherpad page here:
>
> https://etherpad.openstack.org/p/mitaka-live-migration
>
> I have taken the liberty of listing those that responded to the thread and

> the
> authors of mentioned patches as interested people.
>
> From the responses and looking at the specs up for review it looks like 
> there
> are about five areas that could be addressed in Mitaka and several others
> that could come later. The first five are:
>
> - migrating instances with a mix of local disks and cinder volumes

Preliminary patch is up for review [1], we need to switch it to libvirt's v3

migrate API.

> - pause instance during migration
> - cancel migration
> - migrate suspended instances

I'm not sure I understand this correctly. When user calls 'nova suspend' I 
thought that it actually "hibernates" VM and saves memory state to disk 
[2][3]. In such case there is nothing to "live" migrate - shouldn't 
cold-migration/resize solve this problem?

> - improve CI coverage
>
> Not all of these are covered by specs yet and all the existing specs need
> reviews. Please look at the etherpad and see if there is anything you
think 
> is
> missing.

Paul, thanks for taking care of this. I've added missing spec to force live 
migration to finish [4].

Hope we manage to discuss all these items in Tokyo.

[1] https://review.openstack.org/#/c/227278/
[2] 
https://github.com/openstack/nova/blob/e31d1e11bd42bcfbd7b2c3d732d184a367b75
d6f/nova/virt/libvirt/driver.py#L2311
[3] 
https://github.com/openstack/nova/blob/e31d1e11bd42bcfbd7b2c3d732d184a367b75
d6f/nova/virt/libvirt/guest.py#L308
[4] https://review.openstack.org/#/c/229040/

Kind Regards,
Pawel Koniszewski


smime.p7s
Description: S/MIME cryptographic signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] live migration in Mitaka

[Koniszewski, Pawel]

> -Original Message-
> From: Daniel P. Berrange [mailto:berra...@redhat.com]
> Sent: Friday, September 18, 2015 5:24 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [nova] live migration in Mitaka
>
> On Fri, Sep 18, 2015 at 11:53:05AM +, Murray, Paul (HP Cloud) wrote:
> > Hi All,
> >
> > There are various efforts going on around live migration at the moment:
> > fixing up CI, bug fixes, additions to cover more corner cases,
> > proposals for new operations
> >
> > Generally live migration could do with a little TLC (see: [1]), so I
> > am going to suggest we give some of that care in the next cycle.
> >
> > Please respond to this post if you have an interest in this and what
> > you would like to see done. Include anything you are already getting
> > on with so we get a clear picture. If there is enough interest I'll
> > put this together as a proposal for a work stream. Something along the
> > lines of "robustify live migration".
>
> We merged some robustness improvements for migration during Liberty.
> Specifically, with KVM we now track the progress of data transfer and if
it
> is
> not making forward progress during a set window of time, we will abort the
> migration. This ensures you don't get a migration that never ends. We also
> now have code which dynamically increases the max permitted downtime
> during switchover, to try and make it more likely to succeeed. We could do
> with getting feedback on how well the various tunable settings work in
> practie for real world deployments, to see if we need to change any
> defaults.
>
> There was a proposal to nova to allow the 'pause' operation to be invoked
> while migration was happening. This would turn a live migration into a
coma-
> migration, thereby ensuring it succeeds.
> I cna't remember if this merged or not, as i can't find the review
offhand,
> but
> its important to have this ASAP IMHO, as when evacuating VMs from a host
> admins need a knob to use to force successful evacuation, even at the cost
> of pausing the guest temporarily.

There are two different proposals - cancel on-going live migration and pause
VM during live migration. Both are very important. Right now there is no way

to interact with on-going live migration through Nova.

Specification for 'cancel on-going live migration' is up for review [1].
'Pause VM during live migration' (it might be something like
force-live-migration) depends on this change so I'm waiting with
specification
until 'cancel' spec is merged. I'll try to prepare it before summit so both
specs can be discussed in Tokyo.

> In libvirt upstream we now have the ability to filter what disks are
> migrated
> during block migration. We need to leverage that new feature to fix the
long
> standing problems of block migration when non-local images are attached -
> eg cinder volumes. We definitely want this in Mitaka.
>
> We should look at what we need to do to isolate the migration data network
> from the main management network. Currently we live migrate over
> whatever network is associated with the compute hosts primary Hostname /
> IP address. This is not neccessarily the fastest NIC on the host. We ought
> to
> be able to record an alternative hostname / IP address against each
compute
> host to indicate the desired migration interface.
>
> Libvirt/KVM have the ability to turn on compression for migration which
> again
> improves the chances of convergance & thus success.
> We would look at leveraging that.

It is merged in QEMU (version 2.4), however, it isn't merged in Libvirt
yet[2]
(1-9 patches from ShaoHe Feng). The simplest solution shouldn't require any
work in Nova, it's just another live migration flag. To extend this we will
probably need to add API call to nova, e.g, to change compression
ratio or to change number of compression threads.

However, this work is for O cycle (or even later) IMHO. The latest used QEMU
is 2.3 (in Ubuntu 15.10). Adoption of QEMU 2.4 and Libvirt with compression
will take some time, so we don't need to focus on it right now.

> QEMU has a crude "auto-converge" flag you can turn on, which limits guest
> CPU execution time, in an attempt to slow down data dirtying rate to again
> improve chance of successful convergance.
>
> I'm working on enhancements to QEMU itself to support TLS encryption for
> migration. This will enable openstack to have secure migration datastream,
> without having to tunnel via libvirtd. This is useful as tunneling via
> libvirtd
> doesn't work with block migration. It will also be much faster than
> tunnelling.
> This probably might be merged in QEMU before Mitaka cycle ends, but more
> likely it is Nxxx cycle

+++ Looking forward to see it!

> There is also work on post-copy migration in QEMU. Normally with live
> migration, the guest doesn't start executing on the target host until
> migration
> has transferred all data. There are many workloads where that doesn't
work,
> as the 

Re: [openstack-dev] [Nova]Devstack live migration.

[Koniszewski, Pawel]

As far as I know this calls migrate_server in conductor/manager.py



Kind Regards,

Pawel Koniszewski



From: kk bk [mailto:kkbr...@gmail.com]
Sent: Friday, April 24, 2015 12:19 AM
To: openstack-dev@lists.openstack.org
Subject: [openstack-dev] [Nova]Devstack live migration.



I am trying to execute a live migration of a VM from node-1 to node-2. As i 
drill thro' the code it ends up calling function migrate_server of 
nova/nova/conductor/rpcapi.py



This migrate_server ends up calling



cctxt.call(context, 'migrate_server', **kw)



I am trying to understand, what this call leads up to. Any pointers would be 
helpful



Thanks







smime.p7s
Description: S/MIME cryptographic signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Glance] Proposal to change Glance meeting time.

[Koniszewski, Pawel]

+1 for consistent time (I prefer 1400UTC)

 

From: Fei Long Wang [mailto:feil...@catalyst.net.nz] 
Sent: Sunday, March 15, 2015 9:00 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Glance] Proposal to change Glance meeting time.

 

+1 for consistent time

On 14/03/15 10:11, Nikhil Komawar wrote:

Here's what it looks like so far:-

 

1400UTC: 3 votes (all core reviewers)

1500UTC: 5 votes (one core reviewer)

Both: 4 votes (all core reviewers)

 

Let's wait another couple days to see if more people respond. 

 

I have a feeling that the weight is slightly tilted towards 1400UTC based on a 
few assumptions about the past activity of those participants, their cross 
project inputs, etc.

 

Thanks,
-Nikhil


  _  


From: Mikhail Fedosin  mailto:mfedo...@mirantis.com mfedo...@mirantis.com
Sent: Friday, March 13, 2015 3:07 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Glance] Proposal to change Glance meeting time. 

 

Both options are good, it's a little better at 1500 UTC.



+1 consistent time.



 

On Fri, Mar 13, 2015 at 9:23 PM, Steve Lewis steve.le...@rackspace.com 
mailto:steve.le...@rackspace.com  wrote:

+1 consistent time

+1 for 1500 UTC since that has come up.

On 09/03/15 09:07, Nikhil Komawar wrote:

So, the new proposal is:
Glance meetings [1] to be conducted
weekly on
Thursdays at 1400UTC [2] on
#openstack-meeting-4

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe 
http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe 
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

 






__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe 
mailto:openstack-dev-requ...@lists.openstack.org?subject:unsubscribe 
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev





-- 
Cheers  Best regards,
Fei Long Wang (王飞龙)
--
Senior Cloud Software Engineer
Tel: +64-48032246
Email: flw...@catalyst.net.nz mailto:flw...@catalyst.net.nz 
Catalyst IT Limited
Level 6, Catalyst House, 150 Willis Street, Wellington
-- 


smime.p7s
Description: S/MIME cryptographic signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev