[Yahoo-eng-team] [Bug 1535918] Re: instance.host not updated on evacuation

2017-10-30 Thread James Page 
nova (2:13.1.4-0ubuntu4.1~cloud0) trusty-mitaka; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 nova (2:13.1.4-0ubuntu4.1) xenial; urgency=medium
 .
   * d/nova.conf: Add connection strings to default config for sqlite. This
 enables daemons to start by default and fixes failing autopkgtests.
   * d/tests/nova-daemons: Update test to be resilient to timing failures.


** Changed in: cloud-archive/mitaka
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1535918

Title:
  instance.host not updated on evacuation

Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive mitaka series:
  Fix Released
Status in OpenStack Compute (nova):
  Fix Released
Status in nova-powervm:
  Fix Released
Status in nova package in Ubuntu:
  Fix Released
Status in nova source package in Xenial:
  Fix Released
Status in nova source package in Zesty:
  Fix Released
Status in nova source package in Artful:
  Fix Released

Bug description:
  [Impact]

  I created several VM instances and checked they are all ACTIVE state after 
creating vm.
  Right after checking them, shutdown nova-compute on their host(to test in 
this case).
  Then, I tried to evacuate them to the other host. But it is failed with ERROR 
state.
  I did some test and analysis.
  I found two commits below are related.(Please refer to [Others] section)
  In this context, migration_context is DB field to pass information when 
migration or evacuation.

  for [1], This gets host info from migration_context. if
  migration_context is abnormal or empty, migration would be fail.
  actually, with only this patch, migration_context is empty. so [2] is
  needed. I touched self.client.prepare part in rpcapi.py from original
  patch which is replaced on newer version. because it is related newer
  functionality, I remained mitaka's function call for this issue.

  for [2], This moves recreation check code to former if condition. and it 
calls rebuild_claim to create migration_context when recreate state not only 
scheduled. I adjusted test code which are pop up from backport process and 
seems to be needed. Someone want to backport or cherrypick code related to 
this, they could find it is already exist.
  Only one patch of them didn’t fix this issue as test said.

  [Test case]

  In below env,

  http://pastebin.ubuntu.com/25337153/

  Network configuration is important in this case, because I tested different 
configuration. but couldn't reproduce it.
  reproduction test script ( based on juju )

  http://pastebin.ubuntu.com/25360805/

  [Regression Potential]

  Existing ACTIVE instances or newly creating instances are not affected
  by this code because these commits are only called when doing
  migration or evacuation. If there are ACTIVE instances and instances
  with ERROR state caused by this issue in one host, upgrading to have
  this fix will not affect any existing instances. After upgrading to
  have this fix and trying to evacuate problematic instance again, ERROR
  state should be fixed to ACTIVE. I tested this scenario on simple env,
  but still need to be considered possibility in complex, crowded
  environment.

  [Others]

  In test, I should patch two commits, one from
  https://bugs.launchpad.net/nova/+bug/1686041

  Related Patches.
  [1] 
https://github.com/openstack/nova/commit/a5b920a197c70d2ae08a1e1335d979857f923b4f
  [2] 
https://github.com/openstack/nova/commit/0f2d87416eff1e96c0fbf0f4b08bf6b6b22246d5
 ( backported to newton from below original)
  - 
https://github.com/openstack/nova/commit/a2b0824aca5cb4a2ae579f625327c51ed0414d35
 (
  original)

  [Original description]

  I'm working on the nova-powervm driver for Mitaka and trying to add
  support for evacuation.

  The problem I'm hitting is that instance.host is not updated when the
  compute driver is called to spawn the instance on the destination
  host.  It is still set to the source host.  It's not until after the
  spawn completes that the compute manager updates instance.host to
  reflect the destination host.

  The nova-powervm driver uses instance events callback mechanism during
  plug VIF to determine when Neutron has finished provisioning the
  network.  The instance events code sends the event to instance.host
  and hence is sending the event to the source host (which is down).
  This causes the spawn to fail and also causes weirdness when the
  source host gets the events when it's powered back up.

  To temporarily work around the problem, I hacked in setting
  instance.host = CONF.host; instance.save() in the compute driver but
  that's not a good solution.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1535918/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to :

[Yahoo-eng-team] [Bug 1535918] Re: instance.host not updated on evacuation

2017-09-04 Thread Launchpad Bug Tracker 
This bug was fixed in the package nova - 2:13.1.4-0ubuntu3

---
nova (2:13.1.4-0ubuntu3) xenial; urgency=medium

  * Fix evacuation error when nova-compute is down just
after VM is started.

- d/p/make-sure-to-rebuild-claim-on-recreate.patch
  (backported from newton 0f2d874, upstream a2b0824)

- d/p/Send-events-to-all-relevant-hosts-if-migrating.patch (LP: #1535918)
  (backported from a5b920)

 -- Seyeong Kim   Fri, 04 Aug 2017 04:46:40
+0900

** Changed in: nova (Ubuntu Xenial)
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1535918

Title:
  instance.host not updated on evacuation

Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive mitaka series:
  Fix Committed
Status in OpenStack Compute (nova):
  Fix Released
Status in nova-powervm:
  Fix Released
Status in nova package in Ubuntu:
  Fix Released
Status in nova source package in Xenial:
  Fix Released
Status in nova source package in Zesty:
  Fix Released
Status in nova source package in Artful:
  Fix Released

Bug description:
  [Impact]

  I created several VM instances and checked they are all ACTIVE state after 
creating vm.
  Right after checking them, shutdown nova-compute on their host(to test in 
this case).
  Then, I tried to evacuate them to the other host. But it is failed with ERROR 
state.
  I did some test and analysis.
  I found two commits below are related.(Please refer to [Others] section)
  In this context, migration_context is DB field to pass information when 
migration or evacuation.

  for [1], This gets host info from migration_context. if
  migration_context is abnormal or empty, migration would be fail.
  actually, with only this patch, migration_context is empty. so [2] is
  needed. I touched self.client.prepare part in rpcapi.py from original
  patch which is replaced on newer version. because it is related newer
  functionality, I remained mitaka's function call for this issue.

  for [2], This moves recreation check code to former if condition. and it 
calls rebuild_claim to create migration_context when recreate state not only 
scheduled. I adjusted test code which are pop up from backport process and 
seems to be needed. Someone want to backport or cherrypick code related to 
this, they could find it is already exist.
  Only one patch of them didn’t fix this issue as test said.

  [Test case]

  In below env,

  http://pastebin.ubuntu.com/25337153/

  Network configuration is important in this case, because I tested different 
configuration. but couldn't reproduce it.
  reproduction test script ( based on juju )

  http://pastebin.ubuntu.com/25360805/

  [Regression Potential]

  Existing ACTIVE instances or newly creating instances are not affected
  by this code because these commits are only called when doing
  migration or evacuation. If there are ACTIVE instances and instances
  with ERROR state caused by this issue in one host, upgrading to have
  this fix will not affect any existing instances. After upgrading to
  have this fix and trying to evacuate problematic instance again, ERROR
  state should be fixed to ACTIVE. I tested this scenario on simple env,
  but still need to be considered possibility in complex, crowded
  environment.

  [Others]

  In test, I should patch two commits, one from
  https://bugs.launchpad.net/nova/+bug/1686041

  Related Patches.
  [1] 
https://github.com/openstack/nova/commit/a5b920a197c70d2ae08a1e1335d979857f923b4f
  [2] 
https://github.com/openstack/nova/commit/0f2d87416eff1e96c0fbf0f4b08bf6b6b22246d5
 ( backported to newton from below original)
  - 
https://github.com/openstack/nova/commit/a2b0824aca5cb4a2ae579f625327c51ed0414d35
 (
  original)

  [Original description]

  I'm working on the nova-powervm driver for Mitaka and trying to add
  support for evacuation.

  The problem I'm hitting is that instance.host is not updated when the
  compute driver is called to spawn the instance on the destination
  host.  It is still set to the source host.  It's not until after the
  spawn completes that the compute manager updates instance.host to
  reflect the destination host.

  The nova-powervm driver uses instance events callback mechanism during
  plug VIF to determine when Neutron has finished provisioning the
  network.  The instance events code sends the event to instance.host
  and hence is sending the event to the source host (which is down).
  This causes the spawn to fail and also causes weirdness when the
  source host gets the events when it's powered back up.

  To temporarily work around the problem, I hacked in setting
  instance.host = CONF.host; instance.save() in the compute driver but
  that's not a good solution.

To manage notifications about this bug go to:

[Yahoo-eng-team] [Bug 1535918] Re: instance.host not updated on evacuation

2017-08-28 Thread Corey Bryant 
** Also affects: cloud-archive/mitaka
   Importance: Undecided
   Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1535918

Title:
  instance.host not updated on evacuation

Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive mitaka series:
  New
Status in OpenStack Compute (nova):
  Fix Released
Status in nova-powervm:
  Fix Released
Status in nova package in Ubuntu:
  Fix Released
Status in nova source package in Xenial:
  Fix Committed
Status in nova source package in Zesty:
  Fix Released
Status in nova source package in Artful:
  Fix Released

Bug description:
  [Impact]

  I created several VM instances and checked they are all ACTIVE state after 
creating vm.
  Right after checking them, shutdown nova-compute on their host(to test in 
this case).
  Then, I tried to evacuate them to the other host. But it is failed with ERROR 
state.
  I did some test and analysis.
  I found two commits below are related.(Please refer to [Others] section)
  In this context, migration_context is DB field to pass information when 
migration or evacuation.

  for [1], This gets host info from migration_context. if
  migration_context is abnormal or empty, migration would be fail.
  actually, with only this patch, migration_context is empty. so [2] is
  needed. I touched self.client.prepare part in rpcapi.py from original
  patch which is replaced on newer version. because it is related newer
  functionality, I remained mitaka's function call for this issue.

  for [2], This moves recreation check code to former if condition. and it 
calls rebuild_claim to create migration_context when recreate state not only 
scheduled. I adjusted test code which are pop up from backport process and 
seems to be needed. Someone want to backport or cherrypick code related to 
this, they could find it is already exist.
  Only one patch of them didn’t fix this issue as test said.

  [Test case]

  In below env,

  http://pastebin.ubuntu.com/25337153/

  Network configuration is important in this case, because I tested different 
configuration. but couldn't reproduce it.
  reproduction test script ( based on juju )

  http://pastebin.ubuntu.com/25360805/

  [Regression Potential]

  Existing ACTIVE instances or newly creating instances are not affected
  by this code because these commits are only called when doing
  migration or evacuation. If there are ACTIVE instances and instances
  with ERROR state caused by this issue in one host, upgrading to have
  this fix will not affect any existing instances. After upgrading to
  have this fix and trying to evacuate problematic instance again, ERROR
  state should be fixed to ACTIVE. I tested this scenario on simple env,
  but still need to be considered possibility in complex, crowded
  environment.

  [Others]

  In test, I should patch two commits, one from
  https://bugs.launchpad.net/nova/+bug/1686041

  Related Patches.
  [1] 
https://github.com/openstack/nova/commit/a5b920a197c70d2ae08a1e1335d979857f923b4f
  [2] 
https://github.com/openstack/nova/commit/0f2d87416eff1e96c0fbf0f4b08bf6b6b22246d5
 ( backported to newton from below original)
  - 
https://github.com/openstack/nova/commit/a2b0824aca5cb4a2ae579f625327c51ed0414d35
 (
  original)

  [Original description]

  I'm working on the nova-powervm driver for Mitaka and trying to add
  support for evacuation.

  The problem I'm hitting is that instance.host is not updated when the
  compute driver is called to spawn the instance on the destination
  host.  It is still set to the source host.  It's not until after the
  spawn completes that the compute manager updates instance.host to
  reflect the destination host.

  The nova-powervm driver uses instance events callback mechanism during
  plug VIF to determine when Neutron has finished provisioning the
  network.  The instance events code sends the event to instance.host
  and hence is sending the event to the source host (which is down).
  This causes the spawn to fail and also causes weirdness when the
  source host gets the events when it's powered back up.

  To temporarily work around the problem, I hacked in setting
  instance.host = CONF.host; instance.save() in the compute driver but
  that's not a good solution.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1535918/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 1535918] Re: instance.host not updated on evacuation

2017-08-21 Thread Eric Desrochers 
** Also affects: nova (Ubuntu Artful)
   Importance: Undecided
   Status: New

** Also affects: nova (Ubuntu Zesty)
   Importance: Undecided
   Status: New

** Changed in: nova (Ubuntu Artful)
   Status: New => Fix Released

** Changed in: nova (Ubuntu Zesty)
   Status: New => Fix Released

** Changed in: nova (Ubuntu Xenial)
   Status: New => In Progress

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1535918

Title:
  instance.host not updated on evacuation

Status in Ubuntu Cloud Archive:
  Fix Released
Status in OpenStack Compute (nova):
  Fix Released
Status in nova-powervm:
  Fix Released
Status in nova package in Ubuntu:
  Fix Released
Status in nova source package in Xenial:
  In Progress
Status in nova source package in Zesty:
  Fix Released
Status in nova source package in Artful:
  Fix Released

Bug description:
  [Impact]

  I created several VM instances and checked they are all ACTIVE state after 
creating vm.
  Right after checking them, shutdown nova-compute on their host(to test in 
this case).
  Then, I tried to evacuate them to the other host. But it is failed with ERROR 
state.
  I did some test and analysis.
  I found two commits below are related.(Please refer to [Others] section)
  In this context, migration_context is DB field to pass information when 
migration or evacuation.

  for [1], This gets host info from migration_context. if
  migration_context is abnormal or empty, migration would be fail.
  actually, with only this patch, migration_context is empty. so [2] is
  needed. I touched self.client.prepare part in rpcapi.py from original
  patch which is replaced on newer version. because it is related newer
  functionality, I remained mitaka's function call for this issue.

  for [2], This moves recreation check code to former if condition. and it 
calls rebuild_claim to create migration_context when recreate state not only 
scheduled. I adjusted test code which are pop up from backport process and 
seems to be needed. Someone want to backport or cherrypick code related to 
this, they could find it is already exist.
  Only one patch of them didn’t fix this issue as test said.

  [Test case]

  In below env,

  http://pastebin.ubuntu.com/25337153/

  Network configuration is important in this case, because I tested different 
configuration. but couldn't reproduce it.
  reproduction test script ( based on juju )

  http://pastebin.ubuntu.com/25360805/

  [Regression Potential]

  Existing ACTIVE VMs or Newly creating VMs are not affected by this
  code because these commits are only called when doing migration or
  evacuation. If there are ACTIVE VMs and VMs got ERROR state caused by
  this issue in one host, and after upgrading pkg, All VMs should not be
  affected anything by this upgrading. After trying to evacuate
  problematic VM again, ERROR state should be fixed to ACTIVE. I tested
  this scenario on simple env, but still need to be considered
  possibility in complex, crowded environment.

  [Others]

  In test, I should patch two commits, one from
  https://bugs.launchpad.net/nova/+bug/1686041

  Related Patches.
  [1] 
https://github.com/openstack/nova/commit/a5b920a197c70d2ae08a1e1335d979857f923b4f
  [2] 
https://github.com/openstack/nova/commit/0f2d87416eff1e96c0fbf0f4b08bf6b6b22246d5
 ( backported to newton from below original)
  - 
https://github.com/openstack/nova/commit/a2b0824aca5cb4a2ae579f625327c51ed0414d35
 ( 
  original)

  [Original description]

  I'm working on the nova-powervm driver for Mitaka and trying to add
  support for evacuation.

  The problem I'm hitting is that instance.host is not updated when the
  compute driver is called to spawn the instance on the destination
  host.  It is still set to the source host.  It's not until after the
  spawn completes that the compute manager updates instance.host to
  reflect the destination host.

  The nova-powervm driver uses instance events callback mechanism during
  plug VIF to determine when Neutron has finished provisioning the
  network.  The instance events code sends the event to instance.host
  and hence is sending the event to the source host (which is down).
  This causes the spawn to fail and also causes weirdness when the
  source host gets the events when it's powered back up.

  To temporarily work around the problem, I hacked in setting
  instance.host = CONF.host; instance.save() in the compute driver but
  that's not a good solution.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1535918/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 1535918] Re: instance.host not updated on evacuation

2017-08-21 Thread Eric Desrochers 
** Also affects: nova (Ubuntu Xenial)
   Importance: Undecided
   Status: New

** Changed in: nova (Ubuntu Xenial)
 Assignee: (unassigned) => Seyeong Kim (xtrusia)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1535918

Title:
  instance.host not updated on evacuation

Status in Ubuntu Cloud Archive:
  Fix Released
Status in OpenStack Compute (nova):
  Fix Released
Status in nova-powervm:
  Fix Released
Status in nova package in Ubuntu:
  New
Status in nova source package in Xenial:
  New

Bug description:
  [Impact]

  I created several VM instances and checked they are all ACTIVE state after 
creating vm.
  Right after checking them, shutdown nova-compute on their host(to test in 
this case).
  Then, I tried to evacuate them to the other host. But it is failed with ERROR 
state.
  I did some test and analysis.
  I found two commits below are related.(Please refer to [Others] section)
  In this context, migration_context is DB field to pass information when 
migration or evacuation.

  for [1], This gets host info from migration_context. if
  migration_context is abnormal or empty, migration would be fail.
  actually, with only this patch, migration_context is empty. so [2] is
  needed. I touched self.client.prepare part in rpcapi.py from original
  patch which is replaced on newer version. because it is related newer
  functionality, I remained mitaka's function call for this issue.

  for [2], This moves recreation check code to former if condition. and it 
calls rebuild_claim to create migration_context when recreate state not only 
scheduled. I adjusted test code which are pop up from backport process and 
seems to be needed. Someone want to backport or cherrypick code related to 
this, they could find it is already exist.
  Only one patch of them didn’t fix this issue as test said.

  [Test case]

  In below env,

  http://pastebin.ubuntu.com/25337153/

  Network configuration is important in this case, because I tested different 
configuration. but couldn't reproduce it.
  reproduction test script ( based on juju )

  http://pastebin.ubuntu.com/25360805/

  [Regression Potential]

  Existing ACTIVE VMs or Newly creating VMs are not affected by this
  code because these commits are only called when doing migration or
  evacuation. If there are ACTIVE VMs and VMs got ERROR state caused by
  this issue in one host, and after upgrading pkg, All VMs should not be
  affected anything by this upgrading. After trying to evacuate
  problematic VM again, ERROR state should be fixed to ACTIVE. I tested
  this scenario on simple env, but still need to be considered
  possibility in complex, crowded environment.

  [Others]

  In test, I should patch two commits, one from
  https://bugs.launchpad.net/nova/+bug/1686041

  Related Patches.
  [1] 
https://github.com/openstack/nova/commit/a5b920a197c70d2ae08a1e1335d979857f923b4f
  [2] 
https://github.com/openstack/nova/commit/0f2d87416eff1e96c0fbf0f4b08bf6b6b22246d5
 ( backported to newton from below original)
  - 
https://github.com/openstack/nova/commit/a2b0824aca5cb4a2ae579f625327c51ed0414d35
 ( 
  original)

  [Original description]

  I'm working on the nova-powervm driver for Mitaka and trying to add
  support for evacuation.

  The problem I'm hitting is that instance.host is not updated when the
  compute driver is called to spawn the instance on the destination
  host.  It is still set to the source host.  It's not until after the
  spawn completes that the compute manager updates instance.host to
  reflect the destination host.

  The nova-powervm driver uses instance events callback mechanism during
  plug VIF to determine when Neutron has finished provisioning the
  network.  The instance events code sends the event to instance.host
  and hence is sending the event to the source host (which is down).
  This causes the spawn to fail and also causes weirdness when the
  source host gets the events when it's powered back up.

  To temporarily work around the problem, I hacked in setting
  instance.host = CONF.host; instance.save() in the compute driver but
  that's not a good solution.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1535918/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 1535918] Re: instance.host not updated on evacuation

2017-08-21 Thread Edward Hope-Morley 
** Changed in: cloud-archive
   Status: New => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1535918

Title:
  instance.host not updated on evacuation

Status in Ubuntu Cloud Archive:
  Fix Released
Status in OpenStack Compute (nova):
  Fix Released
Status in nova-powervm:
  Fix Released
Status in nova package in Ubuntu:
  New

Bug description:
  [Impact]

  Affected to Xenial Mitaka, UCA Mitaka

  just after creating vm and state ACTIVE,

  When evacuating it, it is failed with ERROR state.

  [Test case]

  In below env,
  http://pastebin.ubuntu.com/25337153/

  Network configuration is important in this case, because I tested
  different configuration. but couldn't reproduce it.

  reproduction test script ( based on juju )

  http://pastebin.ubuntu.com/25360805/

  [Regression Potential]

  this is about evacuation, Could be issue on evacuation.
  especially recreating vm

  [Others]

  In test, I should patch two commits, one from

  https://bugs.launchpad.net/nova/+bug/1686041

  Related Patches.
  
https://github.com/openstack/nova/commit/a5b920a197c70d2ae08a1e1335d979857f923b4f
  
https://github.com/openstack/nova/commit/a2b0824aca5cb4a2ae579f625327c51ed0414d35

  [Original description]

  I'm working on the nova-powervm driver for Mitaka and trying to add
  support for evacuation.

  The problem I'm hitting is that instance.host is not updated when the
  compute driver is called to spawn the instance on the destination
  host.  It is still set to the source host.  It's not until after the
  spawn completes that the compute manager updates instance.host to
  reflect the destination host.

  The nova-powervm driver uses instance events callback mechanism during
  plug VIF to determine when Neutron has finished provisioning the
  network.  The instance events code sends the event to instance.host
  and hence is sending the event to the source host (which is down).
  This causes the spawn to fail and also causes weirdness when the
  source host gets the events when it's powered back up.

  To temporarily work around the problem, I hacked in setting
  instance.host = CONF.host; instance.save() in the compute driver but
  that's not a good solution.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1535918/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 1535918] Re: instance.host not updated on evacuation

2017-08-17 Thread Seyeong Kim 
** Also affects: cloud-archive
   Importance: Undecided
   Status: New

** Also affects: nova (Ubuntu)
   Importance: Undecided
   Status: New

** Description changed:

  [Impact]
  
- Affected to Xenial, UCA Mitaka
+ Affected to Xenial Mitaka, UCA Mitaka
  
  just after creating vm and state ACTIVE,
  
  When evacuating it, it is failed with ERROR state.
  
  [Test case]
  
- In below env, 
+ In below env,
  http://pastebin.ubuntu.com/25337153/
  
  Network configuration is important in this case, because I tested
  different configuration. but couldn't reproduce it.
  
  ##in progress##
  
  making detail script
  
  [Regression Potential]
  
  this is about evacuation, Could be issue on evacuation.
  especially recreating vm
  
  [Others]
  
  Related Patches.
  
https://github.com/openstack/nova/commit/a5b920a197c70d2ae08a1e1335d979857f923b4f
  
https://github.com/openstack/nova/commit/a2b0824aca5cb4a2ae579f625327c51ed0414d35
- 
  
  Original description
  
  I'm working on the nova-powervm driver for Mitaka and trying to add
  support for evacuation.
  
  The problem I'm hitting is that instance.host is not updated when the
  compute driver is called to spawn the instance on the destination host.
  It is still set to the source host.  It's not until after the spawn
  completes that the compute manager updates instance.host to reflect the
  destination host.
  
  The nova-powervm driver uses instance events callback mechanism during
  plug VIF to determine when Neutron has finished provisioning the
  network.  The instance events code sends the event to instance.host and
  hence is sending the event to the source host (which is down).  This
  causes the spawn to fail and also causes weirdness when the source host
  gets the events when it's powered back up.
  
  To temporarily work around the problem, I hacked in setting
  instance.host = CONF.host; instance.save() in the compute driver but
  that's not a good solution.

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1535918

Title:
  instance.host not updated on evacuation

Status in Ubuntu Cloud Archive:
  New
Status in OpenStack Compute (nova):
  Fix Released
Status in nova-powervm:
  Fix Released
Status in nova package in Ubuntu:
  New

Bug description:
  [Impact]

  Affected to Xenial Mitaka, UCA Mitaka

  just after creating vm and state ACTIVE,

  When evacuating it, it is failed with ERROR state.

  [Test case]

  In below env,
  http://pastebin.ubuntu.com/25337153/

  Network configuration is important in this case, because I tested
  different configuration. but couldn't reproduce it.

  ##in progress##

  making detail script

  [Regression Potential]

  this is about evacuation, Could be issue on evacuation.
  especially recreating vm

  [Others]

  Related Patches.
  
https://github.com/openstack/nova/commit/a5b920a197c70d2ae08a1e1335d979857f923b4f
  
https://github.com/openstack/nova/commit/a2b0824aca5cb4a2ae579f625327c51ed0414d35

  Original description

  I'm working on the nova-powervm driver for Mitaka and trying to add
  support for evacuation.

  The problem I'm hitting is that instance.host is not updated when the
  compute driver is called to spawn the instance on the destination
  host.  It is still set to the source host.  It's not until after the
  spawn completes that the compute manager updates instance.host to
  reflect the destination host.

  The nova-powervm driver uses instance events callback mechanism during
  plug VIF to determine when Neutron has finished provisioning the
  network.  The instance events code sends the event to instance.host
  and hence is sending the event to the source host (which is down).
  This causes the spawn to fail and also causes weirdness when the
  source host gets the events when it's powered back up.

  To temporarily work around the problem, I hacked in setting
  instance.host = CONF.host; instance.save() in the compute driver but
  that's not a good solution.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1535918/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 1535918] Re: instance.host not updated on evacuation

2016-11-01 Thread OpenStack Infra 
Reviewed:  https://review.openstack.org/371048
Committed: 
https://git.openstack.org/cgit/openstack/nova/commit/?id=a5b920a197c70d2ae08a1e1335d979857f923b4f
Submitter: Jenkins
Branch:master

commit a5b920a197c70d2ae08a1e1335d979857f923b4f
Author: Artom Lifshitz 
Date:   Wed Oct 5 14:37:03 2016 -0400

Send events to all relevant hosts if migrating

Previously, external events were sent to the instance object's host
field. This patch fixes the external event dispatching to check for
migration. If an instance is being migrated, the source and
destination compute are added to the set of hosts to which the event
is sent.

Change-Id: If00736ab36df4a5a3be4f02b0a550e4bcae77b1b
Closes-bug: 1535918
Closes-bug: 1624052


** Changed in: nova
   Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1535918

Title:
  instance.host not updated on evacuation

Status in OpenStack Compute (nova):
  Fix Released
Status in nova-powervm:
  Fix Released

Bug description:
  I'm working on the nova-powervm driver for Mitaka and trying to add
  support for evacuation.

  The problem I'm hitting is that instance.host is not updated when the
  compute driver is called to spawn the instance on the destination
  host.  It is still set to the source host.  It's not until after the
  spawn completes that the compute manager updates instance.host to
  reflect the destination host.

  The nova-powervm driver uses instance events callback mechanism during
  plug VIF to determine when Neutron has finished provisioning the
  network.  The instance events code sends the event to instance.host
  and hence is sending the event to the source host (which is down).
  This causes the spawn to fail and also causes weirdness when the
  source host gets the events when it's powered back up.

  To temporarily work around the problem, I hacked in setting
  instance.host = CONF.host; instance.save() in the compute driver but
  that's not a good solution.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1535918/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 1535918] Re: instance.host not updated on evacuation

2016-05-15 Thread Taylor Peoples 
I am able to reproduce this same issue on a multinode devstack running
libvirt.

On the source host, the last call to
nova/network/base_api.py::update_instance_cache_with_nw_info for a
specific instance before the source host crashes has the nw_info passed
in as a VIF object with the "active" attribute set to False. This is
because the VM has just been deployed and the network was just created.
In other words, the last time the instance's InstanceInfoCache's
network_info attribute was updated before the source host went down, the
VIF was considered not active. In some environments, especially when
doing concurrent deploys, it may take a while for the InstanceInfoCache
to update the network_info to show as active.

What this boils down to is that Nova's InstanceInfoCache can potentially
have a stale network_info active state. This causes the rebuild flow
(which is the same as the spawn flow) to potentially end up waiting for
the network-vif-plugged event, which will never come because it was sent
to the source host instead of the destination. This results in the
rebuild to fail because the VIF plugging times out.

Steps:

1) Deploy VM(s) to host A
2) Take host A down (e.g., kill it's nova api and nova compute processes) once 
VM(s) from (1) are finished deploying
3) Try to evacuate VM(s) from host A to host B
4) Evacuation will potentially time out based on explanation above. It is much 
easier to reproduce if you do step (2) as soon as possible after the VM(s) 
finish deploying

stack@controller:~$ glance image-list
+--+-+
| ID   | Name|
+--+-+
| f91197db-16b5-44b2-beb4-72a9e57041c2 | cirros-0.3.4-x86_64-uec |
| 1348de9b-501d-426c-8cb5-e65381208085 | cirros-0.3.4-x86_64-uec-kernel  |
| 790ebadb-bc5b-48be-b1f0-95a9214a11ae | cirros-0.3.4-x86_64-uec-ramdisk |
+--+-+
stack@controller:~$
stack@controller:~$ neutron net-list
+--+-+--+
| id   | name| subnets  
|
+--+-+--+
| 4ba74a3e-e7a8-4ca4-9de5-8a1d9e1042b8 | public  | 
c9210289-4895-481b-946a-b406ba5889b4 2001:db8::/64   |
|  | | 
9a044095-ab4d-4767-817e-02d81cbe90ef 172.24.4.0/24   |
| d7faf346-1a26-41a0-bb62-b08808f6ba13 | private | 
f45ab890-a0d6-48c1-906e-9c8f81659d65 fdfd:f0f5:a83a::/64 |
|  | | 
0e85f797-0270-49e9-9600-6f21b9cf47d0 10.254.1.0/24   |
+--+-+--+
stack@controller:~$
stack@controller:~$ nova boot tdp-test-vm --flavor 1 --availability-zone 
nova:hostA --block-device 
id=f91197db-16b5-44b2-beb4-72a9e57041c2,source=image,dest=volume,size=1,bootindex=0
 --nic net-id=4ba74a3e-e7a8-4ca4-9de5-8a1d9e1042b8 --min-count 5 --poll
+--+-+
| Property | Value  
 |
+--+-+
| OS-DCF:diskConfig| MANUAL 
 |
| OS-EXT-AZ:availability_zone  | nova   
 |
| OS-EXT-SRV-ATTR:host | -  
 |
| OS-EXT-SRV-ATTR:hostname | tdp-test-vm-1  
 |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | -  
 |
| OS-EXT-SRV-ATTR:instance_name| instance-0021  
 |
| OS-EXT-SRV-ATTR:kernel_id| 1348de9b-501d-426c-8cb5-e65381208085   
 |
| OS-EXT-SRV-ATTR:launch_index | 0  
 |
| OS-EXT-SRV-ATTR:ramdisk_id   | 790ebadb-bc5b-48be-b1f0-95a9214a11ae   
 |
| OS-EXT-SRV-ATTR:reservation_id   | r-erf2jgt0 
 |
| OS-EXT-SRV-ATTR:root_device_name | -  
 |
| OS-EXT-SRV-ATTR:user_data| -  
 |
| OS-EXT-STS:power_state   | 0  
 |
| OS-EXT-STS:task_state| scheduling 
 |
| OS-EXT-STS:vm_state  | building   
 |
| OS-SRV-USG:launched_at   | -

[Yahoo-eng-team] [Bug 1535918] Re: instance.host not updated on evacuation

2016-01-28 Thread Drew Thorstensen 
The issue with the PowerVM driver is actually in neutron.  I set up a
libvirt environment, and the difference is that the PowerVM VIF is for
some reason in a BUILD state, where as it is ACTIVE in libvirt.

If the PowerVM VIF was in an ACTIVE state, this wouldn't occur, and no
neutron events would need to be waited for.

I'll investigate what's going on with the port state for networking-
powervm.  The state up is being sent...so this requires some
verification.


It is true that the nova instance.host isn't updated until after the spawn in 
nova.  That could be investigated...but this is the root reason why PowerVM is 
seeing different behavior than Libvirt.

** Project changed: nova => networking-powervm

** Changed in: networking-powervm
 Assignee: Wen Zhi Yu (yuywz) => Drew Thorstensen (thorst)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1535918

Title:
  instance.host not updated on evacuation

Status in networking-powervm:
  In Progress

Bug description:
  I'm working on the nova-powervm driver for Mitaka and trying to add
  support for evacuation.

  The problem I'm hitting is that instance.host is not updated when the
  compute driver is called to spawn the instance on the destination
  host.  It is still set to the source host.  It's not until after the
  spawn completes that the compute manager updates instance.host to
  reflect the destination host.

  The nova-powervm driver uses instance events callback mechanism during
  plug VIF to determine when Neutron has finished provisioning the
  network.  The instance events code sends the event to instance.host
  and hence is sending the event to the source host (which is down).
  This causes the spawn to fail and also causes weirdness when the
  source host gets the events when it's powered back up.

  To temporarily work around the problem, I hacked in setting
  instance.host = CONF.host; instance.save() in the compute driver but
  that's not a good solution.

To manage notifications about this bug go to:
https://bugs.launchpad.net/networking-powervm/+bug/1535918/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp