[Yahoo-eng-team] [Bug 1638381] [NEW] Live Migration failure: 'unicode' object has no attribute '_o'

[Einar Næss Jensen]

Public bug reported:

nova live-migration a7ff526d-0cab-4da7-92c2-49d8e89d1d27
dev02-compute-01 --block-migrate


gives this in nova log:
2016-11-01 21:48:49.242 14425 ERROR nova.virt.libvirt.driver 
[req-750740d4-c988-4d91-b690-78fca9ac20b4 08e955c2ffdb49b1abeb5b5c4f8fad03 
805b50c70a744113b7d87dd248111275 - - -] [instance: 
a7ff526d-0cab-4da7-92c2-49d8e89d1d27] Live Migration failure: 'unicode' object 
has no attribute '_o'
2016-11-01 21:48:49.288 14425 ERROR nova.virt.libvirt.driver 
[req-750740d4-c988-4d91-b690-78fca9ac20b4 08e955c2ffdb49b1abeb5b5c4f8fad03 
805b50c70a744113b7d87dd248111275 - - -] [instance: 
a7ff526d-0cab-4da7-92c2-49d8e89d1d27] Migration operation has aborted


and instance is not live-migrated.

regular migrating works:


root@dev02-controller-01:~# nova migrate a7ff526d-0cab-4da7-92c2-49d8e89d1d27 
root@dev02-controller-01:~# nova list --all
WARNING: Option "--all_tenants" is deprecated; use "--all-tenants"; this option 
will be removed in novaclient 3.3.0.
+--+--+--+-+--+-++
| ID   | Name | Tenant ID   
 | Status  | Task State   | Power State | Networks  
 |
+--+--+--+-+--+-++
| a7ff526d-0cab-4da7-92c2-49d8e89d1d27 | Centos7-provider-01  | 
975ce89aac384a44be29269198817d00 | RESIZE  | resize_migrating | Running | 
provider=10.100.16.53  |
| edbf68ea-fa00-4493-813e-214fa87599b1 | centos7-selfserved   | 
975ce89aac384a44be29269198817d00 | ACTIVE  | -| Running | 
selfservice=172.16.1.5 |
| d303140e-447f-4ec6-9226-fa041a777f3c | democirr | 
975ce89aac384a44be29269198817d00 | SHUTOFF | -| Shutdown| 
provider=10.100.16.57  |
| 93aeb4c8-46b9-44f2-b32f-152305e3e747 | frontend | 
8e37b03ab46f4cc8bfc21f1659f2613b | SHUTOFF | -| Shutdown| 
provider=10.100.16.55  |
| 24104cff-d89d-4ecf-a53a-786e68a5d1e4 | fsdfs| 
975ce89aac384a44be29269198817d00 | ACTIVE  | -| Running | 
provider=10.100.16.63  |
| 9538b44e-2c92-40cc-a37e-032af7dd3cf7 | provider1| 
975ce89aac384a44be29269198817d00 | SHUTOFF | -| Shutdown| 
provider=10.100.16.52  |
| 2744b560-7105-4cb8-ada0-0dfefa92fdf4 | selfservice-instance | 
975ce89aac384a44be29269198817d00 | ACTIVE  | -| Running | 
selfservice=172.16.1.3 |
+--+--+--+-+--+-++
root@dev02-controller-01:~# nova list --all
WARNING: Option "--all_tenants" is deprecated; use "--all-tenants"; this option 
will be removed in novaclient 3.3.0.
+--+--+--+---++-++
| ID   | Name | Tenant ID   
 | Status| Task State | Power State | Networks  
 |
+--+--+--+---++-++
| a7ff526d-0cab-4da7-92c2-49d8e89d1d27 | Centos7-provider-01  | 
975ce89aac384a44be29269198817d00 | VERIFY_RESIZE | -  | Running | 
provider=10.100.16.53  |
| edbf68ea-fa00-4493-813e-214fa87599b1 | centos7-selfserved   | 
975ce89aac384a44be29269198817d00 | ACTIVE| -  | Running | 
selfservice=172.16.1.5 |
| d303140e-447f-4ec6-9226-fa041a777f3c | democirr | 
975ce89aac384a44be29269198817d00 | SHUTOFF   | -  | Shutdown| 
provider=10.100.16.57  |
| 93aeb4c8-46b9-44f2-b32f-152305e3e747 | frontend | 
8e37b03ab46f4cc8bfc21f1659f2613b | SHUTOFF   | -  | Shutdown| 
provider=10.100.16.55  |
| 24104cff-d89d-4ecf-a53a-786e68a5d1e4 | fsdfs| 
975ce89aac384a44be29269198817d00 | ACTIVE| -  | Running | 
provider=10.100.16.63  |
| 9538b44e-2c92-40cc-a37e-032af7dd3cf7 | provider1| 
975ce89aac384a44be29269198817d00 | SHUTOFF   | -  | Shutdown| 
provider=10.100.16.52  |
| 2744b560-7105-4cb8-ada0-0dfefa92fdf4 | selfservice-instance | 
975ce89aac384a44be29269198817d00 | ACTIVE| -  | Running | 
selfservice=172.16.1.3 |
+--+--+--+---++-++
root@dev02-controller-01:~# nova resize-confirm 
a7ff526d-0cab-4da7-92c2-49d8e89d1d27


root@dev02-controller-01:~# nova --debug 

[Yahoo-eng-team] [Bug 1599508] [NEW] nova api: Unexpected API Error

[Einar Næss Jensen]

Public bug reported:

Unexpected API Error

root@dev02-controller-01:~# openstack security group rule list
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and 
attach the Nova API log if possible.
 (HTTP 500) 
(Request-ID: req-1487f9be-27ac-46ab-8273-4b9f41ac2a72)
root@dev02-controller-01:~# dpkg -l | grep nova
ii  nova-api 2:13.0.0-0ubuntu5 
all  OpenStack Compute - API frontend
ii  nova-cert2:13.0.0-0ubuntu5 
all  OpenStack Compute - certificate management
ii  nova-common  2:13.0.0-0ubuntu5 
all  OpenStack Compute - common files
ii  nova-conductor   2:13.0.0-0ubuntu5 
all  OpenStack Compute - conductor service
ii  nova-consoleauth 2:13.0.0-0ubuntu5 
all  OpenStack Compute - Console Authenticator
ii  nova-novncproxy  2:13.0.0-0ubuntu5 
all  OpenStack Compute - NoVNC proxy
ii  nova-scheduler   2:13.0.0-0ubuntu5 
all  OpenStack Compute - virtual machine scheduler
ii  python-nova  2:13.0.0-0ubuntu5 
all  OpenStack Compute Python libraries
ii  python-novaclient2:3.3.1-2 
all  client library for OpenStack Compute API - Python 2.7
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and 
attach the Nova API log if possible.
 (HTTP 500) 
(Request-ID: req-1487f9be-27ac-46ab-8273-4b9f41ac2a72)


from nova.api.log:2016-07-06 12:24:46.620 27574 INFO nova.metadata.wsgi.server 
[req-9bb23067-987f-4ebd-8e22-c9ced9027be7 - - - - -] (27574) wsgi starting up 
on http://0.0.0.0:8775
2016-07-06 12:24:46.644 27583 INFO nova.metadata.wsgi.server 
[req-0fe2629d-d880-40a6-a602-985862f01533 - - - - -] (27583) wsgi starting up 
on http://0.0.0.0:8775
2016-07-06 12:24:46.687 27582 INFO nova.metadata.wsgi.server 
[req-e7453ea3-e468-4c49-9bf5-08d14addb9d9 - - - - -] (27582) wsgi starting up 
on http://0.0.0.0:8775
2016-07-06 12:53:10.685 27483 ERROR nova.api.openstack.extensions 
[req-1487f9be-27ac-46ab-8273-4b9f41ac2a72 9be590e88269400aa82fcfdbfef3cde6 
c3d120c542704aa698f2c123a8814d42 - - -] Unexpected exception in API method
2016-07-06 12:53:10.685 27483 ERROR nova.api.openstack.extensions Traceback 
(most recent call last):
2016-07-06 12:53:10.685 27483 ERROR nova.api.openstack.extensions   File 
"/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, 
in wrapped
2016-07-06 12:53:10.685 27483 ERROR nova.api.openstack.extensions return 
f(*args, **kwargs)
2016-07-06 12:53:10.685 27483 ERROR nova.api.openstack.extensions   File 
"/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/security_groups.py",
 line 165, in index
2016-07-06 12:53:10.685 27483 ERROR nova.api.openstack.extensions 
search_opts=search_opts)
2016-07-06 12:53:10.685 27483 ERROR nova.api.openstack.extensions   File 
"/usr/lib/python2.7/dist-packages/nova/network/security_group/neutron_driver.py",
 line 189, in list
2016-07-06 12:53:10.685 27483 ERROR nova.api.openstack.extensions 
security_groups = neutron.list_security_groups(**params).get(
2016-07-06 12:53:10.685 27483 ERROR nova.api.openstack.extensions   File 
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 97, in 
with_params
2016-07-06 12:53:10.685 27483 ERROR nova.api.openstack.extensions ret = 
self.function(instance, *args, **kwargs)
2016-07-06 12:53:10.685 27483 ERROR nova.api.openstack.extensions   File 
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 865, in 
list_security_groups
2016-07-06 12:53:10.685 27483 ERROR nova.api.openstack.extensions 
retrieve_all, **_params)
2016-07-06 12:53:10.685 27483 ERROR nova.api.openstack.extensions   File 
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 373, in 
list
2016-07-06 12:53:10.685 27483 ERROR nova.api.openstack.extensions for r in 
self._pagination(collection, path, **params):
2016-07-06 12:53:10.685 27483 ERROR nova.api.openstack.extensions   File 
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 388, in 
_pagination
2016-07-06 12:53:10.685 27483 ERROR nova.api.openstack.extensions res = 
self.get(path, params=params)
2016-07-06 12:53:10.685 27483 ERROR nova.api.openstack.extensions   File 
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 358, in 
get
2016-07-06 12:53:10.685 27483 ERROR nova.api.openstack.extensions 
headers=headers, params=params)
2016-07-06 12:53:10.685 27483 ERROR nova.api.openstack.extensions   File 
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 335, in 
retry_request
2016-07-06 12:53:10.685 27483 ERROR nova.api.openstack.extensions