** Changed in: nova
Status: Invalid => Confirmed
** Changed in: nova
Status: Confirmed => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1499312
Title:
[kilo]Some instances are in error status after 20 instances launched
concurrently due to neutron scheduling.
Status in OpenStack Compute (nova):
Fix Released
Status in nova-docker:
New
Bug description:
[Summary]
Launch 20 instances concurrently, found 5 instances are in ERROR status.
[Topo]
Both controller and compute nodes are running on centos7.0 img:
3.10.0-229.11.1.el7.x86_64
ovs_version: "2.3.1"
[Reproduceable or not]
Can be reproduced easily.
[Recreate Steps]
1. Launch 20 instances concurrently:
[root@quasarin-1 ~(keystone_admin)]$ nova --os-tenant-name herman-tenant-1
boot --flavor 1 --image dbeaeaf5-b996-48ee-8272-dada151cf34e --nic
net-id=d5be47c2-f54e-49b9-9feb-d89cc6b86c56 --availability-zone
scaling:quasarin-1 in1-20 --max 20
2. check instances state, 5 instances are in error status:
[root@quasarin-1 ~(keystone_admin)]$ nova --os-tenant-name herman-tenant-1
list
/usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90:
InsecurePlatformWarning: A true SSLContext object is not available. This
prevents urllib3 from configuring SSL appropriately and may cause certain SSL
connections to fail. For more information, see
https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:251:
SecurityWarning: Certificate has no `subjectAltName`, falling back to check for
a `commonName` for now. This feature is being removed by major browsers and
deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for
details.)
SecurityWarning
+--------------------------------------+-----------+--------+------------+-------------+----------------------------------+
| ID | Name | Status | Task State |
Power State | Networks |
+--------------------------------------+-----------+--------+------------+-------------+----------------------------------+
| 5f048fb4-fd48-4b13-8ac3-8d3b371c14f5 | in1-20-1 | ACTIVE | - |
Running | herman-tenant-1-net-1=100.1.1.2 |
| 0dffe66b-8299-4e2d-ac3e-e9e6a7c0c430 | in1-20-10 | ACTIVE | - |
Running | herman-tenant-1-net-1=100.1.1.9 |
| eef2edd3-2fce-4a2d-a89a-17c06b744a4c | in1-20-11 | ACTIVE | - |
Running | herman-tenant-1-net-1=100.1.1.12 |
| 003584f6-ff38-49d5-9f43-11e965cf3897 | in1-20-12 | ACTIVE | - |
Running | herman-tenant-1-net-1=100.1.1.13 |
| bf41557b-466c-423f-a36f-f25dd1453981 | in1-20-13 | ERROR | spawning |
NOSTATE | |
| a4412824-4b37-4c7e-87ba-ac2d5e38cd37 | in1-20-14 | ACTIVE | - |
Running | herman-tenant-1-net-1=100.1.1.15 |
| 623439fd-90d0-45a5-86ef-b88fed1c2b5b | in1-20-15 | ACTIVE | - |
Running | herman-tenant-1-net-1=100.1.1.16 |
| a12bbd01-f4db-430a-9818-37e6fee18796 | in1-20-16 | ERROR | spawning |
NOSTATE | |
| ad3f1752-7fba-48fd-aceb-f0ff7face845 | in1-20-17 | ERROR | spawning |
NOSTATE | |
| f31378f2-1c4c-42e9-92bc-85de500afbd2 | in1-20-18 | ACTIVE | - |
Running | herman-tenant-1-net-1=100.1.1.19 |
| 44ec8355-12a3-4d8c-8754-61de3047e71e | in1-20-19 | ACTIVE | - |
Running | herman-tenant-1-net-1=100.1.1.20 |
| 5b826315-9711-4051-aa6b-0bb8262074d6 | in1-20-2 | ERROR | spawning |
NOSTATE | |
| a518dcb4-338b-41ef-9740-f3ad7da39d61 | in1-20-20 | ACTIVE | - |
Running | herman-tenant-1-net-1=100.1.1.21 |
| a418d237-7ce2-4880-8791-7b5e958be1ab | in1-20-3 | ACTIVE | - |
Running | herman-tenant-1-net-1=100.1.1.4 |
| e3a4cc36-dd90-4030-b296-2c2a6964aae0 | in1-20-4 | ACTIVE | - |
Running | herman-tenant-1-net-1=100.1.1.11 |
| e65bc507-6be5-4929-80c0-c0f37ea1eefd | in1-20-5 | ERROR | spawning |
NOSTATE | |
| e3e21145-6e5f-4263-855a-2a95bf35d05d | in1-20-6 | ACTIVE | - |
Running | herman-tenant-1-net-1=100.1.1.6 |
| 0f1abea6-7d4f-4acd-b17a-298b82a3c34b | in1-20-7 | ACTIVE | - |
Running | herman-tenant-1-net-1=100.1.1.8 |
| 7f3a22b2-70d6-4dc3-95d4-168fbd0707fc | in1-20-8 | ACTIVE | - |
Running | herman-tenant-1-net-1=100.1.1.7 |
| 470fbabe-fcce-4a1d-8a8c-065134b1c977 | in1-20-9 | ACTIVE | - |
Running | herman-tenant-1-net-1=100.1.1.10 |
+--------------------------------------+-----------+--------+------------+-------------+----------------------------------+
[logs]
[root@quasarin-1 ~(keystone_admin)]$ nova --os-tenant-name herman-tenant-1
list
/usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90:
InsecurePlatformWarning: A true SSLContext object is not available. This
prevents urllib3 from configuring SSL appropriately and may cause certain SSL
connections to fail. For more information, see
https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:251:
SecurityWarning: Certificate has no `subjectAltName`, falling back to check for
a `commonName` for now. This feature is being removed by major browsers and
deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for
details.)
SecurityWarning
+--------------------------------------+-----------+--------+------------+-------------+----------------------------------+
| ID | Name | Status | Task State |
Power State | Networks |
+--------------------------------------+-----------+--------+------------+-------------+----------------------------------+
| 5f048fb4-fd48-4b13-8ac3-8d3b371c14f5 | in1-20-1 | ACTIVE | - |
Running | herman-tenant-1-net-1=100.1.1.2 |
| 0dffe66b-8299-4e2d-ac3e-e9e6a7c0c430 | in1-20-10 | ACTIVE | - |
Running | herman-tenant-1-net-1=100.1.1.9 |
| eef2edd3-2fce-4a2d-a89a-17c06b744a4c | in1-20-11 | ACTIVE | - |
Running | herman-tenant-1-net-1=100.1.1.12 |
| 003584f6-ff38-49d5-9f43-11e965cf3897 | in1-20-12 | ACTIVE | - |
Running | herman-tenant-1-net-1=100.1.1.13 |
| bf41557b-466c-423f-a36f-f25dd1453981 | in1-20-13 | ERROR | spawning |
NOSTATE | |
| a4412824-4b37-4c7e-87ba-ac2d5e38cd37 | in1-20-14 | ACTIVE | - |
Running | herman-tenant-1-net-1=100.1.1.15 |
| 623439fd-90d0-45a5-86ef-b88fed1c2b5b | in1-20-15 | ACTIVE | - |
Running | herman-tenant-1-net-1=100.1.1.16 |
| a12bbd01-f4db-430a-9818-37e6fee18796 | in1-20-16 | ERROR | spawning |
NOSTATE | |
| ad3f1752-7fba-48fd-aceb-f0ff7face845 | in1-20-17 | ERROR | spawning |
NOSTATE | |
| f31378f2-1c4c-42e9-92bc-85de500afbd2 | in1-20-18 | ACTIVE | - |
Running | herman-tenant-1-net-1=100.1.1.19 |
| 44ec8355-12a3-4d8c-8754-61de3047e71e | in1-20-19 | ACTIVE | - |
Running | herman-tenant-1-net-1=100.1.1.20 |
| 5b826315-9711-4051-aa6b-0bb8262074d6 | in1-20-2 | ERROR | spawning |
NOSTATE | |
| a518dcb4-338b-41ef-9740-f3ad7da39d61 | in1-20-20 | ACTIVE | - |
Running | herman-tenant-1-net-1=100.1.1.21 |
| a418d237-7ce2-4880-8791-7b5e958be1ab | in1-20-3 | ACTIVE | - |
Running | herman-tenant-1-net-1=100.1.1.4 |
| e3a4cc36-dd90-4030-b296-2c2a6964aae0 | in1-20-4 | ACTIVE | - |
Running | herman-tenant-1-net-1=100.1.1.11 |
| e65bc507-6be5-4929-80c0-c0f37ea1eefd | in1-20-5 | ERROR | spawning |
NOSTATE | |
| e3e21145-6e5f-4263-855a-2a95bf35d05d | in1-20-6 | ACTIVE | - |
Running | herman-tenant-1-net-1=100.1.1.6 |
| 0f1abea6-7d4f-4acd-b17a-298b82a3c34b | in1-20-7 | ACTIVE | - |
Running | herman-tenant-1-net-1=100.1.1.8 |
| 7f3a22b2-70d6-4dc3-95d4-168fbd0707fc | in1-20-8 | ACTIVE | - |
Running | herman-tenant-1-net-1=100.1.1.7 |
| 470fbabe-fcce-4a1d-8a8c-065134b1c977 | in1-20-9 | ACTIVE | - |
Running | herman-tenant-1-net-1=100.1.1.10 |
+--------------------------------------+-----------+--------+------------+-------------+----------------------------------+
[root@quasarin-1 ~(keystone_admin)]$ nova show
bf41557b-466c-423f-a36f-f25dd1453981
/usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90:
InsecurePlatformWarning: A true SSLContext object is not available. This
prevents urllib3 from configuring SSL appropriately and may cause certain SSL
connections to fail. For more information, see
https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:251:
SecurityWarning: Certificate has no `subjectAltName`, falling back to check for
a `commonName` for now. This feature is being removed by major browsers and
deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for
details.)
SecurityWarning
+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value
|
+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL
|
| OS-EXT-AZ:availability_zone | scaling
|
| OS-EXT-SRV-ATTR:host | quasarin-1
|
| OS-EXT-SRV-ATTR:hypervisor_hostname | quasarin-1.node.os
|
| OS-EXT-SRV-ATTR:instance_name | instance-000002da
|
| OS-EXT-STS:power_state | 0
|
| OS-EXT-STS:task_state | spawning
|
| OS-EXT-STS:vm_state | error
|
| OS-SRV-USG:launched_at | -
|
| OS-SRV-USG:terminated_at | -
|
| accessIPv4 |
|
| accessIPv6 |
|
| config_drive |
|
| created | 2015-09-07T02:46:19Z
|
| fault | {"message": "Build of instance
bf41557b-466c-423f-a36f-f25dd1453981 aborted: Failed to allocate the
network(s), not rescheduling.", "code": 500, "details": " File
\"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 2235, in
_do_build_and_run_instance |
| | filter_properties)
|
| | File
\"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 2368, in
_build_and_run_instance
|
| | reason=msg)
|
| | ", "created":
"2015-09-07T02:52:08Z"}
|
| flavor | m1.tiny (1)
|
| hostId |
47e713ab57ba5e9d11ddbec87bfe89f48987bf3761592656d5149dfb
|
| id | bf41557b-466c-423f-a36f-f25dd1453981
|
| image | cirros image
(dbeaeaf5-b996-48ee-8272-dada151cf34e)
|
| key_name | -
|
| metadata | {}
|
| name | in1-20-13
|
| os-extended-volumes:volumes_attached | []
|
| status | ERROR
|
| tenant_id | 3729e963b79d421ab2274f32a50b7918
|
| updated | 2015-09-07T02:52:09Z
|
| user_id | 97f21887fa8e400cac95208acfb00c24
|
+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[root@quasarin-1 ~(keystone_admin)]$
Compute nova log:
2015-09-06 20:04:39.504 10862 ERROR nova.compute.manager
[req-c69a4bec-93e9-4948-8b66-e89e05963a79 97f21887fa8e400cac95208acfb00c24
89480271287c49a49c1d26f467289497 - - -] [instance:
2151425f-c0bb-4226-8dc0-fef4285d176f] Instance failed to spawn
2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance:
2151425f-c0bb-4226-8dc0-fef4285d176f] Traceback (most recent call last):
2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance:
2151425f-c0bb-4226-8dc0-fef4285d176f] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2461, in
_build_resources
2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance:
2151425f-c0bb-4226-8dc0-fef4285d176f] yield resources
2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance:
2151425f-c0bb-4226-8dc0-fef4285d176f] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2333, in
_build_and_run_instance
2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance:
2151425f-c0bb-4226-8dc0-fef4285d176f] block_device_info=block_device_info)
2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance:
2151425f-c0bb-4226-8dc0-fef4285d176f] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2385, in
spawn
2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance:
2151425f-c0bb-4226-8dc0-fef4285d176f] block_device_info=block_device_info)
2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance:
2151425f-c0bb-4226-8dc0-fef4285d176f] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4424, in
_create_domain_and_network
2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance:
2151425f-c0bb-4226-8dc0-fef4285d176f] raise
exception.VirtualInterfaceCreateException()
2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance:
2151425f-c0bb-4226-8dc0-fef4285d176f] VirtualInterfaceCreateException: Virtual
Interface creation failed
2015-09-06 20:04:39.504 10862 TRACE nova.compute.manager [instance:
2151425f-c0bb-4226-8dc0-fef4285d176f]
2015-09-06 20:04:39.789 10862 ERROR nova.compute.manager
[req-c69a4bec-93e9-4948-8b66-e89e05963a79 97f21887fa8e400cac95208acfb00c24
89480271287c49a49c1d26f467289497 - - -] [instance:
2151425f-c0bb-4226-8dc0-fef4285d176f] Failed to allocate network(s)
2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance:
2151425f-c0bb-4226-8dc0-fef4285d176f] Traceback (most recent call last):
2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance:
2151425f-c0bb-4226-8dc0-fef4285d176f] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2333, in
_build_and_run_instance
2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance:
2151425f-c0bb-4226-8dc0-fef4285d176f] block_device_info=block_device_info)
2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance:
2151425f-c0bb-4226-8dc0-fef4285d176f] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2385, in
spawn
2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance:
2151425f-c0bb-4226-8dc0-fef4285d176f] block_device_info=block_device_info)
2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance:
2151425f-c0bb-4226-8dc0-fef4285d176f] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4424, in
_create_domain_and_network
2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance:
2151425f-c0bb-4226-8dc0-fef4285d176f] raise
exception.VirtualInterfaceCreateException()
2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance:
2151425f-c0bb-4226-8dc0-fef4285d176f] VirtualInterfaceCreateException: Virtual
Interface creation failed
2015-09-06 20:04:39.789 10862 TRACE nova.compute.manager [instance:
2151425f-c0bb-4226-8dc0-fef4285d176f]
2015-09-06 20:04:39.815 10862 ERROR nova.compute.manager
[req-c69a4bec-93e9-4948-8b66-e89e05963a79 97f21887fa8e400cac95208acfb00c24
89480271287c49a49c1d26f467289497 - - -] [instance:
2151425f-c0bb-4226-8dc0-fef4285d176f] Build of instance
2151425f-c0bb-4226-8dc0-fef4285d176f aborted: Failed to allocate the
network(s), not rescheduling.
2015-09-06 20:04:39.815 10862 TRACE nova.compute.manager [instance:
2151425f-c0bb-4226-8dc0-fef4285d176f] Traceback (most recent call last):
2015-09-06 20:04:39.815 10862 TRACE nova.compute.manager [instance:
2151425f-c0bb-4226-8dc0-fef4285d176f] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2235, in
_do_build_and_run_instance
2015-09-06 20:04:39.815 10862 TRACE nova.compute.manager [instance:
2151425f-c0bb-4226-8dc0-fef4285d176f] filter_properties)
2015-09-06 20:04:39.815 10862 TRACE nova.compute.manager [instance:
2151425f-c0bb-4226-8dc0-fef4285d176f] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2368, in
_build_and_run_instance
2015-09-06 20:04:39.815 10862 TRACE nova.compute.manager [instance:
2151425f-c0bb-4226-8dc0-fef4285d176f] reason=msg)
2015-09-06 20:04:39.815 10862 TRACE nova.compute.manager [instance:
2151425f-c0bb-4226-8dc0-fef4285d176f] BuildAbortException: Build of instance
2151425f-c0bb-4226-8dc0-fef4285d176f aborted: Failed to allocate the
network(s), not rescheduling.
2015-09-06 20:04:39.815 10862 TRACE nova.compute.manager [instance:
2151425f-c0bb-4226-8dc0-fef4285d176f]
2015-09-06 20:04:40.325 10862 ERROR nova.network.neutronv2.api
[req-c69a4bec-93e9-4948-8b66-e89e05963a79 97f21887fa8e400cac95208acfb00c24
89480271287c49a49c1d26f467289497 - - -] Unable to clear device ID for port
'None'
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api Traceback
(most recent call last):
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api File
"/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 366, in
_unbind_ports
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api
port_client.update_port(port_id, port_req_body)
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api File
"/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 102, in
with_params
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api ret =
self.function(instance, *args, **kwargs)
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api File
"/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 549, in
update_port
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api return
self.put(self.port_path % (port), body=body)
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api File
"/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 302, in
put
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api
headers=headers, params=params)
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api File
"/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 270, in
retry_request
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api
headers=headers, params=params)
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api File
"/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 211, in
do_request
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api
self._handle_fault_response(status_code, replybody)
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api File
"/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 185, in
_handle_fault_response
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api
exception_handler_v20(status_code, des_error_body)
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api File
"/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 83, in
exception_handler_v20
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api
message=message)
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api
NeutronClientException: 404 Not Found
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api The resource
could not be found.
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api
2015-09-06 20:04:40.325 10862 TRACE nova.network.neutronv2.api
2015-09-06 20:13:55.084 10862 WARNING nova.compute.manager
[req-dd8f958e-e606-412d-9a18-19ccfd1ffee8 - - - - -] While synchronizing
instance power states, found 4 instances in the database and 3 instances on the
hypervisor.
2015-09-06 20:23:58.084 10862 WARNING nova.compute.manager
[req-dd8f958e-e606-412d-9a18-19ccfd1ffee8 - - - - -] While synchronizing
instance power states, found 4 instances in the database and 3 instances on the
hypervisor.
2015-09-06 20:34:01.293 10862 WARNING nova.virt.libvirt.imagecache
[req-dd8f958e-e606-412d-9a18-19ccfd1ffee8 - - - - -] Unknown base file:
/var/lib/nova/instances/_base/66c80db57c54e077e4edc7f728f4ed0a8b43785f
2015-09-06 20:34:01.293 10862 WARNING nova.virt.libvirt.imagecache
[req-dd8f958e-e606-412d-9a18-19ccfd1ffee8 - - - - -] Unknown base file:
/var/lib/nova/instances/_base/786b79d8d23613ac3129cb9b6bc1e5ef01ff824d
2015-09-06 20:34:14.082 10862 WARNING nova.compute.manager
[req-dd8f958e-e606-412d-9a18-19ccfd1ffee8 - - - - -] While synchronizing
instance power states, found 4 instances in the database and 3 instances on the
hypervisor.
2015-09-06 20:44:19.109 10862 INFO oslo_messaging._drivers.impl_rabbit [-]
Connecting to AMQP server on openstack-rabbitmq.service.os:5672
2015-09-06 20:44:19.191 10862 INFO oslo_messaging._drivers.impl_rabbit [-]
Connected to AMQP server on openstack-rabbitmq.service.os:5672
2015-09-06 20:54:28.101 10862 INFO oslo_messaging._drivers.impl_rabbit [-]
Connecting to AMQP server on openstack-rabbitmq.service.os:5672
2015-09-06 20:54:28.150 10862 INFO oslo_messaging._drivers.impl_rabbit [-]
Connected to AMQP server on openstack-rabbitmq.service.os:5672
2015-09-06 21:16:00.291 10862 WARNING nova.virt.libvirt.imagecache
[req-dd8f958e-e606-412d-9a18-19ccfd1ffee8 - - - - -] Unknown base file:
/var/lib/nova/instances/_base/66c80db57c54e077e4edc7f728f4ed0a8b43785f
2015-09-06 21:16:00.292 10862 WARNING nova.virt.libvirt.imagecache
[req-dd8f958e-e606-412d-9a18-19ccfd1ffee8 - - - - -] Unknown base file:
/var/lib/nova/instances/_base/786b79d8d23613ac3129cb9b6bc1e5ef01ff824d
2015-09-06 21:18:53.025 10862 WARNING nova.virt.libvirt.driver
[req-6f0706bf-196e-463a-a38d-866945415326 97f21887fa8e400cac95208acfb00c24
99784719d0a04f4fa1f2333d984e7c24 - - -] Timeout waiting for vif plugging
callback for instance 66a809c3-4cbd-4d30-bfc4-41316b82cd2a
2015-09-06 21:18:53.667 10862 ERROR nova.compute.manager
[req-6f0706bf-196e-463a-a38d-866945415326 97f21887fa8e400cac95208acfb00c24
99784719d0a04f4fa1f2333d984e7c24 - - -] [instance:
66a809c3-4cbd-4d30-bfc4-41316b82cd2a] Instance failed to spawn
2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance:
66a809c3-4cbd-4d30-bfc4-41316b82cd2a] Traceback (most recent call last):
2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance:
66a809c3-4cbd-4d30-bfc4-41316b82cd2a] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2461, in
_build_resources
2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance:
66a809c3-4cbd-4d30-bfc4-41316b82cd2a] yield resources
2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance:
66a809c3-4cbd-4d30-bfc4-41316b82cd2a] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2333, in
_build_and_run_instance
2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance:
66a809c3-4cbd-4d30-bfc4-41316b82cd2a] block_device_info=block_device_info)
2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance:
66a809c3-4cbd-4d30-bfc4-41316b82cd2a] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2385, in
spawn
2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance:
66a809c3-4cbd-4d30-bfc4-41316b82cd2a] block_device_info=block_device_info)
2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance:
66a809c3-4cbd-4d30-bfc4-41316b82cd2a] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4424, in
_create_domain_and_network
2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance:
66a809c3-4cbd-4d30-bfc4-41316b82cd2a] raise
exception.VirtualInterfaceCreateException()
2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance:
66a809c3-4cbd-4d30-bfc4-41316b82cd2a] VirtualInterfaceCreateException: Virtual
Interface creation failed
2015-09-06 21:18:53.667 10862 TRACE nova.compute.manager [instance:
66a809c3-4cbd-4d30-bfc4-41316b82cd2a]
2015-09-06 21:18:53.970 10862 ERROR nova.compute.manager
[req-6f0706bf-196e-463a-a38d-866945415326 97f21887fa8e400cac95208acfb00c24
99784719d0a04f4fa1f2333d984e7c24 - - -] [instance:
66a809c3-4cbd-4d30-bfc4-41316b82cd2a] Failed to allocate network(s)
2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance:
66a809c3-4cbd-4d30-bfc4-41316b82cd2a] Traceback (most recent call last):
2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance:
66a809c3-4cbd-4d30-bfc4-41316b82cd2a] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2333, in
_build_and_run_instance
2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance:
66a809c3-4cbd-4d30-bfc4-41316b82cd2a] block_device_info=block_device_info)
2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance:
66a809c3-4cbd-4d30-bfc4-41316b82cd2a] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2385, in
spawn
2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance:
66a809c3-4cbd-4d30-bfc4-41316b82cd2a] block_device_info=block_device_info)
2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance:
66a809c3-4cbd-4d30-bfc4-41316b82cd2a] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4424, in
_create_domain_and_network
2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance:
66a809c3-4cbd-4d30-bfc4-41316b82cd2a] raise
exception.VirtualInterfaceCreateException()
2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance:
66a809c3-4cbd-4d30-bfc4-41316b82cd2a] VirtualInterfaceCreateException: Virtual
Interface creation failed
2015-09-06 21:18:53.970 10862 TRACE nova.compute.manager [instance:
66a809c3-4cbd-4d30-bfc4-41316b82cd2a]
2015-09-06 21:18:53.993 10862 ERROR nova.compute.manager
[req-6f0706bf-196e-463a-a38d-866945415326 97f21887fa8e400cac95208acfb00c24
99784719d0a04f4fa1f2333d984e7c24 - - -] [instance:
66a809c3-4cbd-4d30-bfc4-41316b82cd2a] Build of instance
66a809c3-4cbd-4d30-bfc4-41316b82cd2a aborted: Failed to allocate the
network(s), not rescheduling.
2015-09-06 21:18:53.993 10862 TRACE nova.compute.manager [instance:
66a809c3-4cbd-4d30-bfc4-41316b82cd2a] Traceback (most recent call last):
2015-09-06 21:18:53.993 10862 TRACE nova.compute.manager [instance:
66a809c3-4cbd-4d30-bfc4-41316b82cd2a] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2235, in
_do_build_and_run_instance
2015-09-06 21:18:53.993 10862 TRACE nova.compute.manager [instance:
66a809c3-4cbd-4d30-bfc4-41316b82cd2a] filter_properties)
2015-09-06 21:18:53.993 10862 TRACE nova.compute.manager [instance:
66a809c3-4cbd-4d30-bfc4-41316b82cd2a] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2368, in
_build_and_run_instance
2015-09-06 21:18:53.993 10862 TRACE nova.compute.manager [instance:
66a809c3-4cbd-4d30-bfc4-41316b82cd2a] reason=msg)
2015-09-06 21:18:53.993 10862 TRACE nova.compute.manager [instance:
66a809c3-4cbd-4d30-bfc4-41316b82cd2a] BuildAbortException: Build of instance
66a809c3-4cbd-4d30-bfc4-41316b82cd2a aborted: Failed to allocate the
network(s), not rescheduling.
2015-09-06 21:18:53.993 10862 TRACE nova.compute.manager [instance:
66a809c3-4cbd-4d30-bfc4-41316b82cd2a]
2015-09-06 21:18:55.114 10862 ERROR nova.network.neutronv2.api
[req-6f0706bf-196e-463a-a38d-866945415326 97f21887fa8e400cac95208acfb00c24
99784719d0a04f4fa1f2333d984e7c24 - - -] Unable to clear device ID for port
'None'
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api Traceback
(most recent call last):
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api File
"/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 366, in
_unbind_ports
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api
port_client.update_port(port_id, port_req_body)
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api File
"/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 102, in
with_params
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api ret =
self.function(instance, *args, **kwargs)
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api File
"/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 549, in
update_port
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api return
self.put(self.port_path % (port), body=body)
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api File
"/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 302, in
put
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api
headers=headers, params=params)
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api File
"/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 270, in
retry_request
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api
headers=headers, params=params)
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api File
"/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 211, in
do_request
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api
self._handle_fault_response(status_code, replybody)
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api File
"/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 185, in
_handle_fault_response
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api
exception_handler_v20(status_code, des_error_body)
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api File
"/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 83, in
exception_handler_v20
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api
message=message)
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api
NeutronClientException: 404 Not Found
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api The resource
could not be found.
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api
2015-09-06 21:18:55.114 10862 TRACE nova.network.neutronv2.api
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1499312/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
