Re: [openstack-dev] [Nova] Live Migration post feature freeze update
> -Original Message- > From: Sean Dague [mailto:s...@dague.net] > Sent: 04 March 2016 20:09 > To: openstack-dev@lists.openstack.org > Subject: Re: [openstack-dev] [Nova] Live Migration post feature freeze > update > > On 03/04/2016 02:54 PM, Matt Riedemann wrote: > > > > > > On 3/4/2016 10:34 AM, Murray, Paul (HP Cloud) wrote: > >> Hi All, > >> - snip 8<-- > > > > The gate-tempest-dsvm-multinode-full job which runs live migration > > tests on nova patches has been non-voting for awhile now. There are at > > least two known tracked bugs so we can keep an eye on failure rates. > > > > 1. Volume based live migration aborted unexpectedly: > > > > http://status.openstack.org/elastic-recheck/index.html#1524898 > > > > 2. Libvirt live block migration migration stalls > > > > http://status.openstack.org/elastic-recheck/index.html#1539271 > > > > Those are actually the top two failures in the check queue. > > > > The job is bouncing between 25% and ~80% failure rates: > > > > http://tinyurl.com/gvt5h56 > > > > At one point that job was relatively stable, it had to have been > > because it was voting. > > It's never been voting. Or at least not with the number of live migration > scenarios. The grenade multinode job is the only one that's hit the threshold > of voting and kept it. It does not run live migration though. > > -Sean > Thanks Matt, we'll discuss in the meeting tomorrow and see what we can do about it. Paul __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Live Migration post feature freeze update
On 03/04/2016 02:54 PM, Matt Riedemann wrote: > > > On 3/4/2016 10:34 AM, Murray, Paul (HP Cloud) wrote: >> Hi All, >> >> Now that we have passed the feature freeze I thought it was worth giving >> a quick update >> >> on where we are with the live migration priority. >> >> The following is a list of work items that have been merged in this >> cycle ( for the live migration >> >> sub-team’s working page see >> https://etherpad.openstack.org/p/mitaka-live-migration ). There >> >> is also a number of merged and on-going bug fixes that are not listed >> here. >> >> _Progress reporting_ >> >> Provide progress reporting information for on-going live migrations. >> >> ·https://blueprints.launchpad.net/nova/+spec/live-migration-progress-report >> >> >> * >> https://review.openstack.org/#/q/topic:bp/live-migration-progress-report >> >> __ >> >> _Force complete_ >> >> Force an on-going live migration to complete by pausing the virtual >> machine for the >> >> duration of the migration. >> >> ·https://blueprints.launchpad.net/nova/+spec/pause-vm-during-live-migration >> >> >> ·https://review.openstack.org/#/q/topic:bp/pause-vm-during-live-migration >> >> __ >> >> _Cancel_ >> >> Cancel an on-going live migration. >> >> ·https://blueprints.launchpad.net/nova/+spec/abort-live-migration >> >> * https://review.openstack.org/#/q/topic:bp/abort-live-migration >> >> __ >> >> _Block live migration with attached volumes_ >> >> Enable live migration of VMs with a combination of local and shared >> storage. >> >> ·https://blueprints.launchpad.net/nova/+spec/block-live-migrate-with-attached-volumes >> >> >> >> ·https://review.openstack.org/#/c/227278 >> >> __ >> >> _Split networking_ >> >> Send live migration traffic over a specified network. >> >> ·https://blueprints.launchpad.net/nova/+spec/split-network-plane-for-live-migration >> >> >> >> ·https://review.openstack.org/#/q/topic:bp/split-network-plane-for-live-migration >> >> >> >> __ >> >> _Make live migration api friendly_ >> >> Remove –disk_over_commit flag and add –block_migration=auto (let nova >> determine >> >> how to migrate the disks) >> >> ·https://blueprints.launchpad.net/nova/+spec/making-live-migration-api-friendly >> >> >> >> * >> https://review.openstack.org/#/q/topic:bp/making-live-migration-api-friendly >> >> >> __ >> >> _Use request spec_ >> >> Add scheduling to live migration and evacuate using original request >> spec (includes all >> >> original scheduling properties) >> >> ·https://blueprints.launchpad.net/nova/+spec/check-destination-on-migrations >> >> >> ·https://review.openstack.org/#/c/277800/ >> >> ·https://review.openstack.org/#/c/273104/ >> >> _Deprecate migration flags_ >> >> Replace the combination of migration configuration flags with a single >> tunneled flag. >> >> ·(no blueprint) >> >> ·https://review.openstack.org/#/q/project:openstack/nova+branch:master+topic:deprecate-migration-flags-config >> >> >> __ >> >> _Objectify live migrate data_ >> >> Use the migrate object instead of a dictionary in migration code. >> >> ·https://blueprints.launchpad.net/nova/+spec/objectify-live-migrate-data >> >> ·https://review.openstack.org/#/q/project:openstack/nova+branch:master+topic:bp/objectify-live-migrate-data >> >> >> >> Next steps… >> >> Now we have passed the feature freeze we will be turning attention to >> the following >> >> three tasks: >> >> 1.Documenting the new features >> >> 2.Adding expanding the CI coverage >> >> 3.Fixing bugs >> >> The CI job gate-tempest-dsvm-multinode-live-migration was added to the >> experimental >> >> queue earlier In the cycle. We now need to add tests to this job to >> increase coverage. If >> >> you have any suggestions for CI improvements please contribute them on >> this page: >> >> https://etherpad.openstack.org/p/nova-live-migration-CI-ideas >> >> If you can contributed to live migration bug fixing you can look for >> things to do here: >> >> https://bugs.launchpad.net/nova/+bugs?field.tag=live-migration >> >> For priority reviews see the live migration section here: >> >> https://etherpad.openstack.org/p/mitaka-nova-priorities-tracking >> >> The live migration sub-team has an IRC meeting on Tuesdays at 14:00 >> UTC on >> >> #openstack-meeting-3: >> >> https://wiki.openstack.org/wiki/Meetings/NovaLiveMigration >> >> Best regards, >> >> Paul >> >> Paul Murray >> >> Technical Lead, HPE Cloud >> >> Hewlett Packard Enterprise >> >> +44 117 316 2527 >> >> >> >> __ >> >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > The gate-tempest-dsvm-multinode-full job which runs live migration tests > on nova patches has been non-voting for awhile now. There are at least > two known tracked bugs so we can keep an eye on failure rates. > > 1. Volume based live migration
Re: [openstack-dev] [Nova] Live Migration post feature freeze update
On 3/4/2016 10:34 AM, Murray, Paul (HP Cloud) wrote: Hi All, Now that we have passed the feature freeze I thought it was worth giving a quick update on where we are with the live migration priority. The following is a list of work items that have been merged in this cycle ( for the live migration sub-team’s working page see https://etherpad.openstack.org/p/mitaka-live-migration ). There is also a number of merged and on-going bug fixes that are not listed here. _Progress reporting_ Provide progress reporting information for on-going live migrations. ·https://blueprints.launchpad.net/nova/+spec/live-migration-progress-report * https://review.openstack.org/#/q/topic:bp/live-migration-progress-report __ _Force complete_ Force an on-going live migration to complete by pausing the virtual machine for the duration of the migration. ·https://blueprints.launchpad.net/nova/+spec/pause-vm-during-live-migration ·https://review.openstack.org/#/q/topic:bp/pause-vm-during-live-migration __ _Cancel_ Cancel an on-going live migration. ·https://blueprints.launchpad.net/nova/+spec/abort-live-migration * https://review.openstack.org/#/q/topic:bp/abort-live-migration __ _Block live migration with attached volumes_ Enable live migration of VMs with a combination of local and shared storage. ·https://blueprints.launchpad.net/nova/+spec/block-live-migrate-with-attached-volumes ·https://review.openstack.org/#/c/227278 __ _Split networking_ Send live migration traffic over a specified network. ·https://blueprints.launchpad.net/nova/+spec/split-network-plane-for-live-migration ·https://review.openstack.org/#/q/topic:bp/split-network-plane-for-live-migration __ _Make live migration api friendly_ Remove –disk_over_commit flag and add –block_migration=auto (let nova determine how to migrate the disks) ·https://blueprints.launchpad.net/nova/+spec/making-live-migration-api-friendly * https://review.openstack.org/#/q/topic:bp/making-live-migration-api-friendly __ _Use request spec_ Add scheduling to live migration and evacuate using original request spec (includes all original scheduling properties) ·https://blueprints.launchpad.net/nova/+spec/check-destination-on-migrations ·https://review.openstack.org/#/c/277800/ ·https://review.openstack.org/#/c/273104/ _Deprecate migration flags_ Replace the combination of migration configuration flags with a single tunneled flag. ·(no blueprint) ·https://review.openstack.org/#/q/project:openstack/nova+branch:master+topic:deprecate-migration-flags-config __ _Objectify live migrate data_ Use the migrate object instead of a dictionary in migration code. ·https://blueprints.launchpad.net/nova/+spec/objectify-live-migrate-data ·https://review.openstack.org/#/q/project:openstack/nova+branch:master+topic:bp/objectify-live-migrate-data Next steps… Now we have passed the feature freeze we will be turning attention to the following three tasks: 1.Documenting the new features 2.Adding expanding the CI coverage 3.Fixing bugs The CI job gate-tempest-dsvm-multinode-live-migration was added to the experimental queue earlier In the cycle. We now need to add tests to this job to increase coverage. If you have any suggestions for CI improvements please contribute them on this page: https://etherpad.openstack.org/p/nova-live-migration-CI-ideas If you can contributed to live migration bug fixing you can look for things to do here: https://bugs.launchpad.net/nova/+bugs?field.tag=live-migration For priority reviews see the live migration section here: https://etherpad.openstack.org/p/mitaka-nova-priorities-tracking The live migration sub-team has an IRC meeting on Tuesdays at 14:00 UTC on #openstack-meeting-3: https://wiki.openstack.org/wiki/Meetings/NovaLiveMigration Best regards, Paul Paul Murray Technical Lead, HPE Cloud Hewlett Packard Enterprise +44 117 316 2527 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev The gate-tempest-dsvm-multinode-full job which runs live migration tests on nova patches has been non-voting for awhile now. There are at least two known tracked bugs so we can keep an eye on failure rates. 1. Volume based live migration aborted unexpectedly: http://status.openstack.org/elastic-recheck/index.html#1524898 2. Libvirt live block migration migration stalls http://status.openstack.org/elastic-recheck/index.html#1539271 Those are actually the top two failures in the check queue. The job is bouncing between 25% and ~80% failure rates: http://tinyurl.com/gvt5h56 At one point that job was relatively stable, it had to have been because it was voting. So I'm not sure what's going on, but those should probably be the top priority bugs for live migration. The problem,
[openstack-dev] [Nova] Live Migration post feature freeze update
Hi All, Now that we have passed the feature freeze I thought it was worth giving a quick update on where we are with the live migration priority. The following is a list of work items that have been merged in this cycle ( for the live migration sub-team's working page see https://etherpad.openstack.org/p/mitaka-live-migration ). There is also a number of merged and on-going bug fixes that are not listed here. Progress reporting Provide progress reporting information for on-going live migrations. * https://blueprints.launchpad.net/nova/+spec/live-migration-progress-report * https://review.openstack.org/#/q/topic:bp/live-migration-progress-report Force complete Force an on-going live migration to complete by pausing the virtual machine for the duration of the migration. * https://blueprints.launchpad.net/nova/+spec/pause-vm-during-live-migration * https://review.openstack.org/#/q/topic:bp/pause-vm-during-live-migration Cancel Cancel an on-going live migration. * https://blueprints.launchpad.net/nova/+spec/abort-live-migration * https://review.openstack.org/#/q/topic:bp/abort-live-migration Block live migration with attached volumes Enable live migration of VMs with a combination of local and shared storage. * https://blueprints.launchpad.net/nova/+spec/block-live-migrate-with-attached-volumes * https://review.openstack.org/#/c/227278 Split networking Send live migration traffic over a specified network. * https://blueprints.launchpad.net/nova/+spec/split-network-plane-for-live-migration * https://review.openstack.org/#/q/topic:bp/split-network-plane-for-live-migration Make live migration api friendly Remove -disk_over_commit flag and add -block_migration=auto (let nova determine how to migrate the disks) * https://blueprints.launchpad.net/nova/+spec/making-live-migration-api-friendly * https://review.openstack.org/#/q/topic:bp/making-live-migration-api-friendly Use request spec Add scheduling to live migration and evacuate using original request spec (includes all original scheduling properties) * https://blueprints.launchpad.net/nova/+spec/check-destination-on-migrations * https://review.openstack.org/#/c/277800/ * https://review.openstack.org/#/c/273104/ Deprecate migration flags Replace the combination of migration configuration flags with a single tunneled flag. * (no blueprint) * https://review.openstack.org/#/q/project:openstack/nova+branch:master+topic:deprecate-migration-flags-config Objectify live migrate data Use the migrate object instead of a dictionary in migration code. * https://blueprints.launchpad.net/nova/+spec/objectify-live-migrate-data * https://review.openstack.org/#/q/project:openstack/nova+branch:master+topic:bp/objectify-live-migrate-data Next steps... Now we have passed the feature freeze we will be turning attention to the following three tasks: 1. Documenting the new features 2. Adding expanding the CI coverage 3. Fixing bugs The CI job gate-tempest-dsvm-multinode-live-migration was added to the experimental queue earlier In the cycle. We now need to add tests to this job to increase coverage. If you have any suggestions for CI improvements please contribute them on this page: https://etherpad.openstack.org/p/nova-live-migration-CI-ideas If you can contributed to live migration bug fixing you can look for things to do here: https://bugs.launchpad.net/nova/+bugs?field.tag=live-migration For priority reviews see the live migration section here: https://etherpad.openstack.org/p/mitaka-nova-priorities-tracking The live migration sub-team has an IRC meeting on Tuesdays at 14:00 UTC on #openstack-meeting-3: https://wiki.openstack.org/wiki/Meetings/NovaLiveMigration Best regards, Paul Paul Murray Technical Lead, HPE Cloud Hewlett Packard Enterprise +44 117 316 2527 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
