[Yahoo-eng-team] [Bug 1627183] [NEW] Cannot access storage file '/var/lib/nova/mnt/5fd17b88c08c2759966c40234dfd10c0/volume-67bbce20-b476-466d-b20a-8337957af780' (as uid:107, gid:107): No such file or directory", u'code': 500

[Alex Ortiz]

Public bug reported:

Ran tempest, hit this error. I can't get to the system right now to get
additional info. If i can, i will add more info.

19:42:37
19:42:37 ==============================
19:42:37 Failed 1 tests - output below:
19:42:37 ==============================
19:42:37
19:42:37 
tempest.api.compute.servers.test_server_rescue_negative.ServerRescueNegativeTestJSON.test_rescued_vm_detach_volume[id-f56e465b-fe10-48bf-b75d-646cda3a8bc9,negative,volume]
19:42:37 
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
19:42:37
19:42:37 Captured traceback:
19:42:37 ~~~~~~~~~~~~~~~~~~~
19:42:37     Traceback (most recent call last):
19:42:37       File 
"/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/api/compute/servers/test_server_rescue_negative.py",
 line 80, in _unrescue
19:42:37         server_id, 'ACTIVE')
19:42:37       File 
"/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/common/waiters.py",
 line 75, in wait_for_server_status
19:42:37         server_id=server_id)
19:42:37     tempest.exceptions.BuildErrorException: Server 
04e4aa2a-13ed-44ca-a275-bbb703c12493 failed to build and is in ERROR status
19:42:37     Details: {u'message': u"Cannot access storage file 
'/var/lib/nova/mnt/5fd17b88c08c2759966c40234dfd10c0/volume-67bbce20-b476-466d-b20a-8337957af780'
 (as uid:107, gid:107): No such file or directory", u'code': 500, u'created': 
u'2016-09-23T19:39:46Z'}
19:42:37
19:42:37
19:42:37 Captured pythonlogging:
19:42:37 ~~~~~~~~~~~~~~~~~~~~~~~
19:42:37     2016-09-23 19:39:36,692 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 POST 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes
 0.411s
19:42:37     2016-09-23 19:39:36,692 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: {"volume": {"display_name": 
"tempest-ServerRescueNegativeTestJSON_volume-1225043131", "size": 1}}
19:42:37         Response - Headers: {'status': '200', 'content-length': '360', 
'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes',
 'x-compute-request-id': 'req-beb5ecea-c1dc-467e-9381-6c43be677b2f', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780',
 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:36 
GMT', 'content-type': 'application/json'}
19:42:37             Body: {"volume": {"status": "creating", 
"displayDescription": null, "availabilityZone": "nova", "displayName": 
"tempest-ServerRescueNegativeTestJSON_volume-1225043131", "attachments": [{}], 
"volumeType": "sl_nfs_SL01SEV316090_1", "snapshotId": null, "metadata": {}, 
"id": "67bbce20-b476-466d-b20a-8337957af780", "createdAt": 
"2016-09-23T19:39:36.489134", "size": 1}}
19:42:37     2016-09-23 19:39:36,817 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780
 0.122s
19:42:37     2016-09-23 19:39:36,817 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: None
19:42:37         Response - Headers: {'status': '200', 'content-length': '360', 
'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780',
 'x-compute-request-id': 'req-fa69239e-7fe6-4b8e-b3e9-786eb994a9f6', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 
Sep 2016 19:39:36 GMT', 'content-type': 'application/json'}
19:42:37             Body: {"volume": {"status": "creating", 
"displayDescription": null, "availabilityZone": "nova", "displayName": 
"tempest-ServerRescueNegativeTestJSON_volume-1225043131", "attachments": [{}], 
"volumeType": "sl_nfs_SL01SEV316090_1", "snapshotId": null, "metadata": {}, 
"id": "67bbce20-b476-466d-b20a-8337957af780", "createdAt": 
"2016-09-23T19:39:36.000000", "size": 1}}
19:42:37     2016-09-23 19:39:37,937 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780
 0.116s
19:42:37     2016-09-23 19:39:37,938 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: None
19:42:37         Response - Headers: {'status': '200', 'content-length': '361', 
'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780',
 'x-compute-request-id': 'req-b1ec1378-2e49-48cc-aaac-6f5d7218624f', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 
Sep 2016 19:39:37 GMT', 'content-type': 'application/json'}
19:42:37             Body: {"volume": {"status": "available", 
"displayDescription": null, "availabilityZone": "nova", "displayName": 
"tempest-ServerRescueNegativeTestJSON_volume-1225043131", "attachments": [{}], 
"volumeType": "sl_nfs_SL01SEV316090_1", "snapshotId": null, "metadata": {}, 
"id": "67bbce20-b476-466d-b20a-8337957af780", "createdAt": 
"2016-09-23T19:39:36.000000", "size": 1}}
19:42:37     2016-09-23 19:39:38,329 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 POST 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/os-volume_attachments
 0.388s
19:42:37     2016-09-23 19:39:38,330 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: {"volumeAttachment": {"device": "/dev/vdb", 
"volumeId": "67bbce20-b476-466d-b20a-8337957af780"}}
19:42:37         Response - Headers: {'status': '200', 'content-length': '194', 
'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/os-volume_attachments',
 'x-compute-request-id': 'req-7f079443-51fb-4f41-8797-0e542d6e2995', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 
Sep 2016 19:39:38 GMT', 'content-type': 'application/json'}
19:42:37             Body: {"volumeAttachment": {"device": "/dev/vdb", 
"serverId": "04e4aa2a-13ed-44ca-a275-bbb703c12493", "id": 
"67bbce20-b476-466d-b20a-8337957af780", "volumeId": 
"67bbce20-b476-466d-b20a-8337957af780"}}
19:42:37     2016-09-23 19:39:38,465 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780
 0.134s
19:42:37     2016-09-23 19:39:38,466 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: None
19:42:37         Response - Headers: {'status': '200', 'content-length': '361', 
'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780',
 'x-compute-request-id': 'req-586a4fca-99cf-4c35-9c7c-80b23ecf7faf', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 
Sep 2016 19:39:38 GMT', 'content-type': 'application/json'}
19:42:37             Body: {"volume": {"status": "attaching", 
"displayDescription": null, "availabilityZone": "nova", "displayName": 
"tempest-ServerRescueNegativeTestJSON_volume-1225043131", "attachments": [{}], 
"volumeType": "sl_nfs_SL01SEV316090_1", "snapshotId": null, "metadata": {}, 
"id": "67bbce20-b476-466d-b20a-8337957af780", "createdAt": 
"2016-09-23T19:39:36.000000", "size": 1}}
19:42:37     2016-09-23 19:39:39,595 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780
 0.125s
19:42:37     2016-09-23 19:39:39,596 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: None
19:42:37         Response - Headers: {'status': '200', 'content-length': '570', 
'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780',
 'x-compute-request-id': 'req-17bbc784-e95d-437c-9d20-f0a528a0762f', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 
Sep 2016 19:39:39 GMT', 'content-type': 'application/json'}
19:42:37             Body: {"volume": {"status": "in-use", 
"displayDescription": null, "availabilityZone": "nova", "displayName": 
"tempest-ServerRescueNegativeTestJSON_volume-1225043131", "attachments": 
[{"device": "/dev/vdb", "serverId": "04e4aa2a-13ed-44ca-a275-bbb703c12493", 
"id": "67bbce20-b476-466d-b20a-8337957af780", "volumeId": 
"67bbce20-b476-466d-b20a-8337957af780"}], "volumeType": 
"sl_nfs_SL01SEV316090_1", "snapshotId": null, "metadata": {"readonly": "False", 
"attached_mode": "rw"}, "id": "67bbce20-b476-466d-b20a-8337957af780", 
"createdAt": "2016-09-23T19:39:36.000000", "size": 1}}
19:42:37     2016-09-23 19:39:39,833 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 POST 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/action
 0.234s
19:42:37     2016-09-23 19:39:39,833 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: {"rescue": {"adminPass": "X2!oCgz87DGWc8g"}}
19:42:37         Response - Headers: {'status': '200', 'content-length': '32', 
'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/action',
 'x-compute-request-id': 'req-e93333a1-c339-4137-8363-1ec692e93594', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 
Sep 2016 19:39:39 GMT', 'content-type': 'application/json'}
19:42:37             Body: {"adminPass": "X2!oCgz87DGWc8g"}
19:42:37     2016-09-23 19:39:40,069 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493
 0.234s
19:42:37     2016-09-23 19:39:40,069 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: None
19:42:37         Response - Headers: {'status': '200', 'content-length': 
'1787', 'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493',
 'x-compute-request-id': 'req-98335fbb-f98d-44d2-b02a-314a6687e20e', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 
Sep 2016 19:39:40 GMT', 'content-type': 'application/json'}
19:42:37             Body: {"server": {"OS-EXT-STS:task_state": "rescuing", 
"addresses": {"tempest-ServerRescueNegativeTestJSON-1526350074-network": 
[{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:5b:05:34", "version": 4, "addr": 
"10.100.0.5", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";,
 "rel": "self"}, {"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";,
 "rel": "bookmark"}], "image": {"id": "6e947667-ca70-45d1-966f-7aa2920a8228", 
"links": [{"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/images/6e947667-ca70-45d1-966f-7aa2920a8228";,
 "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "active", 
"OS-SRV-USG:launched_at": "2016-09-23T19:39:31.000000", "flavor": {"id": "1", 
"links": [{"href": "https://uat-dal09-compute-253.
 open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/flavors/1", "rel": 
"bookmark"}]}, "id": "04e4aa2a-13ed-44ca-a275-bbb703c12493", "security_groups": 
[{"name": "default"}], "user_id": "a0f08771b3654f0e9b1492c32e507022", 
"OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": 
0, "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", 
"metadata": {}, "status": "ACTIVE", "updated": "2016-09-23T19:39:40Z", 
"hostId": "df76544eefef439a0e210b1ea7f4046c939f3c0b72bd58871b5a0917", 
"OS-SRV-USG:terminated_at": null, "key_name": null, "name": 
"tempest.common.compute-instance-859751003", "created": "2016-09-23T19:39:09Z", 
"tenant_id": "015383e7edf74544afac10dbcea21a76", 
"os-extended-volumes:volumes_attached": [{"id": 
"67bbce20-b476-466d-b20a-8337957af780"}], "config_drive": ""}}
19:42:37     2016-09-23 19:39:41,306 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493
 0.228s
19:42:37     2016-09-23 19:39:41,306 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: None
19:42:37         Response - Headers: {'status': '200', 'content-length': 
'1787', 'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493',
 'x-compute-request-id': 'req-fe829b17-4da3-44de-bbb9-6839c6fb4beb', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 
Sep 2016 19:39:41 GMT', 'content-type': 'application/json'}
19:42:37             Body: {"server": {"OS-EXT-STS:task_state": "rescuing", 
"addresses": {"tempest-ServerRescueNegativeTestJSON-1526350074-network": 
[{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:5b:05:34", "version": 4, "addr": 
"10.100.0.5", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";,
 "rel": "self"}, {"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";,
 "rel": "bookmark"}], "image": {"id": "6e947667-ca70-45d1-966f-7aa2920a8228", 
"links": [{"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/images/6e947667-ca70-45d1-966f-7aa2920a8228";,
 "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "active", 
"OS-SRV-USG:launched_at": "2016-09-23T19:39:31.000000", "flavor": {"id": "1", 
"links": [{"href": "https://uat-dal09-compute-253.
 open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/flavors/1", "rel": 
"bookmark"}]}, "id": "04e4aa2a-13ed-44ca-a275-bbb703c12493", "security_groups": 
[{"name": "default"}], "user_id": "a0f08771b3654f0e9b1492c32e507022", 
"OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": 
0, "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", 
"metadata": {}, "status": "ACTIVE", "updated": "2016-09-23T19:39:40Z", 
"hostId": "df76544eefef439a0e210b1ea7f4046c939f3c0b72bd58871b5a0917", 
"OS-SRV-USG:terminated_at": null, "key_name": null, "name": 
"tempest.common.compute-instance-859751003", "created": "2016-09-23T19:39:09Z", 
"tenant_id": "015383e7edf74544afac10dbcea21a76", 
"os-extended-volumes:volumes_attached": [{"id": 
"67bbce20-b476-466d-b20a-8337957af780"}], "config_drive": ""}}
19:42:37     2016-09-23 19:39:42,541 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493
 0.227s
19:42:37     2016-09-23 19:39:42,542 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: None
19:42:37         Response - Headers: {'status': '200', 'content-length': 
'1787', 'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493',
 'x-compute-request-id': 'req-06e65562-36ac-440b-9b38-45eb5322fa43', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 
Sep 2016 19:39:42 GMT', 'content-type': 'application/json'}
19:42:37             Body: {"server": {"OS-EXT-STS:task_state": "rescuing", 
"addresses": {"tempest-ServerRescueNegativeTestJSON-1526350074-network": 
[{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:5b:05:34", "version": 4, "addr": 
"10.100.0.5", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";,
 "rel": "self"}, {"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";,
 "rel": "bookmark"}], "image": {"id": "6e947667-ca70-45d1-966f-7aa2920a8228", 
"links": [{"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/images/6e947667-ca70-45d1-966f-7aa2920a8228";,
 "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "active", 
"OS-SRV-USG:launched_at": "2016-09-23T19:39:31.000000", "flavor": {"id": "1", 
"links": [{"href": "https://uat-dal09-compute-253.
 open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/flavors/1", "rel": 
"bookmark"}]}, "id": "04e4aa2a-13ed-44ca-a275-bbb703c12493", "security_groups": 
[{"name": "default"}], "user_id": "a0f08771b3654f0e9b1492c32e507022", 
"OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": 
0, "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", 
"metadata": {}, "status": "ACTIVE", "updated": "2016-09-23T19:39:40Z", 
"hostId": "df76544eefef439a0e210b1ea7f4046c939f3c0b72bd58871b5a0917", 
"OS-SRV-USG:terminated_at": null, "key_name": null, "name": 
"tempest.common.compute-instance-859751003", "created": "2016-09-23T19:39:09Z", 
"tenant_id": "015383e7edf74544afac10dbcea21a76", 
"os-extended-volumes:volumes_attached": [{"id": 
"67bbce20-b476-466d-b20a-8337957af780"}], "config_drive": ""}}
19:42:37     2016-09-23 19:39:43,799 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493
 0.243s
19:42:37     2016-09-23 19:39:43,799 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: None
19:42:37         Response - Headers: {'status': '200', 'content-length': 
'1787', 'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493',
 'x-compute-request-id': 'req-c693fccb-0153-4737-8192-d9869122659a', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 
Sep 2016 19:39:43 GMT', 'content-type': 'application/json'}
19:42:37             Body: {"server": {"OS-EXT-STS:task_state": "rescuing", 
"addresses": {"tempest-ServerRescueNegativeTestJSON-1526350074-network": 
[{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:5b:05:34", "version": 4, "addr": 
"10.100.0.5", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";,
 "rel": "self"}, {"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";,
 "rel": "bookmark"}], "image": {"id": "6e947667-ca70-45d1-966f-7aa2920a8228", 
"links": [{"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/images/6e947667-ca70-45d1-966f-7aa2920a8228";,
 "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "active", 
"OS-SRV-USG:launched_at": "2016-09-23T19:39:31.000000", "flavor": {"id": "1", 
"links": [{"href": "https://uat-dal09-compute-253.
 open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/flavors/1", "rel": 
"bookmark"}]}, "id": "04e4aa2a-13ed-44ca-a275-bbb703c12493", "security_groups": 
[{"name": "default"}], "user_id": "a0f08771b3654f0e9b1492c32e507022", 
"OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": 
0, "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", 
"metadata": {}, "status": "ACTIVE", "updated": "2016-09-23T19:39:40Z", 
"hostId": "df76544eefef439a0e210b1ea7f4046c939f3c0b72bd58871b5a0917", 
"OS-SRV-USG:terminated_at": null, "key_name": null, "name": 
"tempest.common.compute-instance-859751003", "created": "2016-09-23T19:39:09Z", 
"tenant_id": "015383e7edf74544afac10dbcea21a76", 
"os-extended-volumes:volumes_attached": [{"id": 
"67bbce20-b476-466d-b20a-8337957af780"}], "config_drive": ""}}
19:42:37     2016-09-23 19:39:45,058 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493
 0.251s
19:42:37     2016-09-23 19:39:45,058 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: None
19:42:37         Response - Headers: {'status': '200', 'content-length': 
'1767', 'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493',
 'x-compute-request-id': 'req-be1c970e-b954-4d73-9b75-23820d0aff25', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 
Sep 2016 19:39:45 GMT', 'content-type': 'application/json'}
19:42:37             Body: {"server": {"OS-EXT-STS:task_state": null, 
"addresses": {"tempest-ServerRescueNegativeTestJSON-1526350074-network": 
[{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:5b:05:34", "version": 4, "addr": 
"10.100.0.5", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";,
 "rel": "self"}, {"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";,
 "rel": "bookmark"}], "image": {"id": "6e947667-ca70-45d1-966f-7aa2920a8228", 
"links": [{"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/images/6e947667-ca70-45d1-966f-7aa2920a8228";,
 "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "rescued", 
"OS-SRV-USG:launched_at": "2016-09-23T19:39:44.000000", "flavor": {"id": "1", 
"links": [{"href": "https://uat-dal09-compute-253.open.
 softlayer.com:8774/015383e7edf74544afac10dbcea21a76/flavors/1", "rel": 
"bookmark"}]}, "id": "04e4aa2a-13ed-44ca-a275-bbb703c12493", "security_groups": 
[{"name": "default"}], "user_id": "a0f08771b3654f0e9b1492c32e507022", 
"OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", 
"OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "metadata": 
{}, "status": "RESCUE", "updated": "2016-09-23T19:39:44Z", "hostId": 
"df76544eefef439a0e210b1ea7f4046c939f3c0b72bd58871b5a0917", 
"OS-SRV-USG:terminated_at": null, "key_name": null, "name": 
"tempest.common.compute-instance-859751003", "created": "2016-09-23T19:39:09Z", 
"tenant_id": "015383e7edf74544afac10dbcea21a76", 
"os-extended-volumes:volumes_attached": [{"id": 
"67bbce20-b476-466d-b20a-8337957af780"}], "config_drive": ""}}
19:42:37     2016-09-23 19:39:45,064 18008 INFO     [tempest.common.waiters] 
State transition "ACTIVE/rescuing" ==> "RESCUE/None" after 5 second wait
19:42:37     2016-09-23 19:39:45,254 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 409 DELETE 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/os-volume_attachments/67bbce20-b476-466d-b20a-8337957af780
 0.188s
19:42:37     2016-09-23 19:39:45,254 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: None
19:42:37         Response - Headers: {'status': '409', 'content-length': '152', 
'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/os-volume_attachments/67bbce20-b476-466d-b20a-8337957af780',
 'x-compute-request-id': 'req-e572880b-7dcd-457d-a06a-ed004351adaf', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 
Sep 2016 19:39:45 GMT', 'content-type': 'application/json; charset=UTF-8'}
19:42:37             Body: {"conflictingRequest": {"message": "Cannot 
'detach_volume' instance 04e4aa2a-13ed-44ca-a275-bbb703c12493 while it is in 
vm_state rescued", "code": 409}}
19:42:37     2016-09-23 19:39:45,396 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:_run_cleanups): 202 POST 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/action
 0.140s
19:42:37     2016-09-23 19:39:45,396 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: {"unrescue": {}}
19:42:37         Response - Headers: {'status': '202', 'content-length': '0', 
'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/action',
 'x-compute-request-id': 'req-7010ff1d-7d7c-4052-a2d5-3cddb6ddee4c', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 
Sep 2016 19:39:45 GMT', 'content-type': 'application/json'}
19:42:37             Body:
19:42:37     2016-09-23 19:39:45,671 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:_run_cleanups): 200 GET 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493
 0.274s
19:42:37     2016-09-23 19:39:45,672 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: None
19:42:37         Response - Headers: {'status': '200', 'content-length': 
'1775', 'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493',
 'x-compute-request-id': 'req-8069154d-767b-4d75-ab32-83c8d07ae745', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 
Sep 2016 19:39:45 GMT', 'content-type': 'application/json'}
19:42:37             Body: {"server": {"OS-EXT-STS:task_state": "unrescuing", 
"addresses": {"tempest-ServerRescueNegativeTestJSON-1526350074-network": 
[{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:5b:05:34", "version": 4, "addr": 
"10.100.0.5", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";,
 "rel": "self"}, {"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";,
 "rel": "bookmark"}], "image": {"id": "6e947667-ca70-45d1-966f-7aa2920a8228", 
"links": [{"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/images/6e947667-ca70-45d1-966f-7aa2920a8228";,
 "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "rescued", 
"OS-SRV-USG:launched_at": "2016-09-23T19:39:44.000000", "flavor": {"id": "1", 
"links": [{"href": "https://uat-dal09-compute-2
 53.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/flavors/1", "rel": 
"bookmark"}]}, "id": "04e4aa2a-13ed-44ca-a275-bbb703c12493", "security_groups": 
[{"name": "default"}], "user_id": "a0f08771b3654f0e9b1492c32e507022", 
"OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", 
"OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "metadata": 
{}, "status": "RESCUE", "updated": "2016-09-23T19:39:45Z", "hostId": 
"df76544eefef439a0e210b1ea7f4046c939f3c0b72bd58871b5a0917", 
"OS-SRV-USG:terminated_at": null, "key_name": null, "name": 
"tempest.common.compute-instance-859751003", "created": "2016-09-23T19:39:09Z", 
"tenant_id": "015383e7edf74544afac10dbcea21a76", 
"os-extended-volumes:volumes_attached": [{"id": 
"67bbce20-b476-466d-b20a-8337957af780"}], "config_drive": ""}}
19:42:37     2016-09-23 19:39:46,921 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:_run_cleanups): 200 GET 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493
 0.240s
19:42:37     2016-09-23 19:39:46,921 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: None
19:42:37         Response - Headers: {'status': '200', 'content-length': 
'2010', 'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493',
 'x-compute-request-id': 'req-99919910-98c2-40bb-90fd-0828ead5f1bf', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 
Sep 2016 19:39:46 GMT', 'content-type': 'application/json'}
19:42:37             Body: {"server": {"OS-EXT-STS:task_state": null, 
"addresses": {"tempest-ServerRescueNegativeTestJSON-1526350074-network": 
[{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:5b:05:34", "version": 4, "addr": 
"10.100.0.5", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";,
 "rel": "self"}, {"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";,
 "rel": "bookmark"}], "image": {"id": "6e947667-ca70-45d1-966f-7aa2920a8228", 
"links": [{"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/images/6e947667-ca70-45d1-966f-7aa2920a8228";,
 "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "error", 
"OS-SRV-USG:launched_at": "2016-09-23T19:39:44.000000", "flavor": {"id": "1", 
"links": [{"href": "https://uat-dal09-compute-253.open.so
 ftlayer.com:8774/015383e7edf74544afac10dbcea21a76/flavors/1", "rel": 
"bookmark"}]}, "id": "04e4aa2a-13ed-44ca-a275-bbb703c12493", "security_groups": 
[{"name": "default"}], "user_id": "a0f08771b3654f0e9b1492c32e507022", 
"OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", 
"OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "metadata": 
{}, "status": "ERROR", "updated": "2016-09-23T19:39:46Z", "hostId": 
"df76544eefef439a0e210b1ea7f4046c939f3c0b72bd58871b5a0917", 
"OS-SRV-USG:terminated_at": null, "key_name": null, "name": 
"tempest.common.compute-instance-859751003", "created": "2016-09-23T19:39:09Z", 
"tenant_id": "015383e7edf74544afac10dbcea21a76", 
"os-extended-volumes:volumes_attached": [{"id": 
"67bbce20-b476-466d-b20a-8337957af780"}], "fault": {"message": "Cannot access 
storage file 
'/var/lib/nova/mnt/5fd17b88c08c2759966c40234dfd10c0/volume-67bbce20-b476-466d-b20a-8337957af780'
 (as uid:107, gid:107): No such file or directory", "code": 500, "created": "20
 16-09-23T19:39:46Z"}, "config_drive": ""}}
19:42:37     2016-09-23 19:39:46,928 18008 INFO     [tempest.common.waiters] 
State transition "RESCUE/unrescuing" ==> "ERROR/None" after 1 second wait
19:42:37     2016-09-23 19:39:47,151 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:_run_cleanups): 409 DELETE 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/os-volume_attachments/67bbce20-b476-466d-b20a-8337957af780
 0.222s
19:42:37     2016-09-23 19:39:47,152 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: None
19:42:37         Response - Headers: {'status': '409', 'content-length': '150', 
'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/os-volume_attachments/67bbce20-b476-466d-b20a-8337957af780',
 'x-compute-request-id': 'req-2244f3e2-8d8c-4792-8aef-959638c81197', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 
Sep 2016 19:39:47 GMT', 'content-type': 'application/json; charset=UTF-8'}
19:42:37             Body: {"conflictingRequest": {"message": "Cannot 
'detach_volume' instance 04e4aa2a-13ed-44ca-a275-bbb703c12493 while it is in 
vm_state error", "code": 409}}
19:42:37     2016-09-23 19:39:47,265 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:_run_cleanups): 500 DELETE 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780
 0.111s
19:42:37     2016-09-23 19:39:47,266 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
19:42:37             Body: None
19:42:37         Response - Headers: {'status': '500', 'content-length': '201', 
'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780',
 'x-compute-request-id': 'req-b19a6519-ddf4-406c-b004-a544b8f4aeeb', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 
Sep 2016 19:39:47 GMT', 'content-type': 'application/json; charset=UTF-8'}
19:42:37             Body: {"computeFault": {"message": "Unexpected API Error. 
Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API 
log if possible.\n<class 'nova.exception.InvalidInput'>", "code": 500}}
19:42:37
19:42:37
19:42:37 Captured traceback-2:
19:42:37 ~~~~~~~~~~~~~~~~~~~~~
19:42:37     Traceback (most recent call last):
19:42:37       File 
"/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/api/compute/base.py",
 line 346, in delete_volume
19:42:37         cls._delete_volume(cls.volumes_extensions_client, volume_id)
19:42:37       File 
"/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/api/compute/base.py",
 line 277, in _delete_volume
19:42:37         volumes_client.delete_volume(volume_id)
19:42:37       File 
"/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/services/compute/volumes_client.py",
 line 75, in delete_volume
19:42:37         resp, body = self.delete("os-volumes/%s" % volume_id)
19:42:37       File 
"/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/common/rest_client.py",
 line 307, in delete
19:42:37         return self.request('DELETE', url, extra_headers, headers, 
body)
19:42:37       File 
"/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/services/compute/base_compute_client.py",
 line 48, in request
19:42:37         method, url, extra_headers, headers, body, chunked)
19:42:37       File 
"/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/common/rest_client.py",
 line 665, in request
19:42:37         resp, resp_body)
19:42:37       File 
"/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/common/rest_client.py",
 line 829, in _error_checker
19:42:37         message=message)
19:42:37     tempest.lib.exceptions.ServerFault: Got server fault
19:42:37     Details: Unexpected API Error. Please report this at 
http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
19:42:37     <class 'nova.exception.InvalidInput'>
19:42:37
19:42:37
19:42:37 Captured traceback-1:
19:42:37 ~~~~~~~~~~~~~~~~~~~~~
19:42:37     Traceback (most recent call last):
19:42:37       File 
"/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/api/compute/servers/test_server_rescue_negative.py",
 line 73, in _detach
19:42:37         self.servers_client.detach_volume(server_id, volume_id)
19:42:37       File 
"/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/services/compute/servers_client.py",
 line 404, in detach_volume
19:42:37         (server_id, volume_id))
19:42:37       File 
"/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/common/rest_client.py",
 line 307, in delete
19:42:37         return self.request('DELETE', url, extra_headers, headers, 
body)
19:42:37       File 
"/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/services/compute/base_compute_client.py",
 line 48, in request
19:42:37         method, url, extra_headers, headers, body, chunked)
19:42:37       File 
"/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/common/rest_client.py",
 line 665, in request
19:42:37         resp, resp_body)
19:42:37       File 
"/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/common/rest_client.py",
 line 778, in _error_checker
19:42:37         raise exceptions.Conflict(resp_body, resp=resp)
19:42:37     tempest.lib.exceptions.Conflict: An object with that identifier 
already exists
19:42:37     Details: {u'message': u"Cannot 'detach_volume' instance 
04e4aa2a-13ed-44ca-a275-bbb703c12493 while it is in vm_state error", u'code': 
409}
19:42:37

** Affects: nova
     Importance: Undecided
         Status: Confirmed


** Tags: libvirt nfs volume

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1627183

Title:
  Cannot access storage file
  '/var/lib/nova/mnt/5fd17b88c08c2759966c40234dfd10c0/volume-
  67bbce20-b476-466d-b20a-8337957af780' (as uid:107, gid:107): No such
  file or directory", u'code': 500

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  Ran tempest, hit this error. I can't get to the system right now to
  get additional info. If i can, i will add more info.

  19:42:37
  19:42:37 ==============================
  19:42:37 Failed 1 tests - output below:
  19:42:37 ==============================
  19:42:37
  19:42:37 
tempest.api.compute.servers.test_server_rescue_negative.ServerRescueNegativeTestJSON.test_rescued_vm_detach_volume[id-f56e465b-fe10-48bf-b75d-646cda3a8bc9,negative,volume]
  19:42:37 
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  19:42:37
  19:42:37 Captured traceback:
  19:42:37 ~~~~~~~~~~~~~~~~~~~
  19:42:37     Traceback (most recent call last):
  19:42:37       File 
"/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/api/compute/servers/test_server_rescue_negative.py",
 line 80, in _unrescue
  19:42:37         server_id, 'ACTIVE')
  19:42:37       File 
"/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/common/waiters.py",
 line 75, in wait_for_server_status
  19:42:37         server_id=server_id)
  19:42:37     tempest.exceptions.BuildErrorException: Server 
04e4aa2a-13ed-44ca-a275-bbb703c12493 failed to build and is in ERROR status
  19:42:37     Details: {u'message': u"Cannot access storage file 
'/var/lib/nova/mnt/5fd17b88c08c2759966c40234dfd10c0/volume-67bbce20-b476-466d-b20a-8337957af780'
 (as uid:107, gid:107): No such file or directory", u'code': 500, u'created': 
u'2016-09-23T19:39:46Z'}
  19:42:37
  19:42:37
  19:42:37 Captured pythonlogging:
  19:42:37 ~~~~~~~~~~~~~~~~~~~~~~~
  19:42:37     2016-09-23 19:39:36,692 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 POST 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes
 0.411s
  19:42:37     2016-09-23 19:39:36,692 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: {"volume": {"display_name": 
"tempest-ServerRescueNegativeTestJSON_volume-1225043131", "size": 1}}
  19:42:37         Response - Headers: {'status': '200', 'content-length': 
'360', 'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes',
 'x-compute-request-id': 'req-beb5ecea-c1dc-467e-9381-6c43be677b2f', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780',
 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 Sep 2016 19:39:36 
GMT', 'content-type': 'application/json'}
  19:42:37             Body: {"volume": {"status": "creating", 
"displayDescription": null, "availabilityZone": "nova", "displayName": 
"tempest-ServerRescueNegativeTestJSON_volume-1225043131", "attachments": [{}], 
"volumeType": "sl_nfs_SL01SEV316090_1", "snapshotId": null, "metadata": {}, 
"id": "67bbce20-b476-466d-b20a-8337957af780", "createdAt": 
"2016-09-23T19:39:36.489134", "size": 1}}
  19:42:37     2016-09-23 19:39:36,817 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780
 0.122s
  19:42:37     2016-09-23 19:39:36,817 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: None
  19:42:37         Response - Headers: {'status': '200', 'content-length': 
'360', 'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780',
 'x-compute-request-id': 'req-fa69239e-7fe6-4b8e-b3e9-786eb994a9f6', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 
Sep 2016 19:39:36 GMT', 'content-type': 'application/json'}
  19:42:37             Body: {"volume": {"status": "creating", 
"displayDescription": null, "availabilityZone": "nova", "displayName": 
"tempest-ServerRescueNegativeTestJSON_volume-1225043131", "attachments": [{}], 
"volumeType": "sl_nfs_SL01SEV316090_1", "snapshotId": null, "metadata": {}, 
"id": "67bbce20-b476-466d-b20a-8337957af780", "createdAt": 
"2016-09-23T19:39:36.000000", "size": 1}}
  19:42:37     2016-09-23 19:39:37,937 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780
 0.116s
  19:42:37     2016-09-23 19:39:37,938 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: None
  19:42:37         Response - Headers: {'status': '200', 'content-length': 
'361', 'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780',
 'x-compute-request-id': 'req-b1ec1378-2e49-48cc-aaac-6f5d7218624f', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 
Sep 2016 19:39:37 GMT', 'content-type': 'application/json'}
  19:42:37             Body: {"volume": {"status": "available", 
"displayDescription": null, "availabilityZone": "nova", "displayName": 
"tempest-ServerRescueNegativeTestJSON_volume-1225043131", "attachments": [{}], 
"volumeType": "sl_nfs_SL01SEV316090_1", "snapshotId": null, "metadata": {}, 
"id": "67bbce20-b476-466d-b20a-8337957af780", "createdAt": 
"2016-09-23T19:39:36.000000", "size": 1}}
  19:42:37     2016-09-23 19:39:38,329 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 POST 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/os-volume_attachments
 0.388s
  19:42:37     2016-09-23 19:39:38,330 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: {"volumeAttachment": {"device": "/dev/vdb", 
"volumeId": "67bbce20-b476-466d-b20a-8337957af780"}}
  19:42:37         Response - Headers: {'status': '200', 'content-length': 
'194', 'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/os-volume_attachments',
 'x-compute-request-id': 'req-7f079443-51fb-4f41-8797-0e542d6e2995', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 
Sep 2016 19:39:38 GMT', 'content-type': 'application/json'}
  19:42:37             Body: {"volumeAttachment": {"device": "/dev/vdb", 
"serverId": "04e4aa2a-13ed-44ca-a275-bbb703c12493", "id": 
"67bbce20-b476-466d-b20a-8337957af780", "volumeId": 
"67bbce20-b476-466d-b20a-8337957af780"}}
  19:42:37     2016-09-23 19:39:38,465 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780
 0.134s
  19:42:37     2016-09-23 19:39:38,466 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: None
  19:42:37         Response - Headers: {'status': '200', 'content-length': 
'361', 'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780',
 'x-compute-request-id': 'req-586a4fca-99cf-4c35-9c7c-80b23ecf7faf', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 
Sep 2016 19:39:38 GMT', 'content-type': 'application/json'}
  19:42:37             Body: {"volume": {"status": "attaching", 
"displayDescription": null, "availabilityZone": "nova", "displayName": 
"tempest-ServerRescueNegativeTestJSON_volume-1225043131", "attachments": [{}], 
"volumeType": "sl_nfs_SL01SEV316090_1", "snapshotId": null, "metadata": {}, 
"id": "67bbce20-b476-466d-b20a-8337957af780", "createdAt": 
"2016-09-23T19:39:36.000000", "size": 1}}
  19:42:37     2016-09-23 19:39:39,595 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780
 0.125s
  19:42:37     2016-09-23 19:39:39,596 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: None
  19:42:37         Response - Headers: {'status': '200', 'content-length': 
'570', 'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780',
 'x-compute-request-id': 'req-17bbc784-e95d-437c-9d20-f0a528a0762f', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 
Sep 2016 19:39:39 GMT', 'content-type': 'application/json'}
  19:42:37             Body: {"volume": {"status": "in-use", 
"displayDescription": null, "availabilityZone": "nova", "displayName": 
"tempest-ServerRescueNegativeTestJSON_volume-1225043131", "attachments": 
[{"device": "/dev/vdb", "serverId": "04e4aa2a-13ed-44ca-a275-bbb703c12493", 
"id": "67bbce20-b476-466d-b20a-8337957af780", "volumeId": 
"67bbce20-b476-466d-b20a-8337957af780"}], "volumeType": 
"sl_nfs_SL01SEV316090_1", "snapshotId": null, "metadata": {"readonly": "False", 
"attached_mode": "rw"}, "id": "67bbce20-b476-466d-b20a-8337957af780", 
"createdAt": "2016-09-23T19:39:36.000000", "size": 1}}
  19:42:37     2016-09-23 19:39:39,833 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 POST 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/action
 0.234s
  19:42:37     2016-09-23 19:39:39,833 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: {"rescue": {"adminPass": "X2!oCgz87DGWc8g"}}
  19:42:37         Response - Headers: {'status': '200', 'content-length': 
'32', 'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/action',
 'x-compute-request-id': 'req-e93333a1-c339-4137-8363-1ec692e93594', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 
Sep 2016 19:39:39 GMT', 'content-type': 'application/json'}
  19:42:37             Body: {"adminPass": "X2!oCgz87DGWc8g"}
  19:42:37     2016-09-23 19:39:40,069 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493
 0.234s
  19:42:37     2016-09-23 19:39:40,069 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: None
  19:42:37         Response - Headers: {'status': '200', 'content-length': 
'1787', 'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493',
 'x-compute-request-id': 'req-98335fbb-f98d-44d2-b02a-314a6687e20e', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 
Sep 2016 19:39:40 GMT', 'content-type': 'application/json'}
  19:42:37             Body: {"server": {"OS-EXT-STS:task_state": "rescuing", 
"addresses": {"tempest-ServerRescueNegativeTestJSON-1526350074-network": 
[{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:5b:05:34", "version": 4, "addr": 
"10.100.0.5", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";,
 "rel": "self"}, {"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";,
 "rel": "bookmark"}], "image": {"id": "6e947667-ca70-45d1-966f-7aa2920a8228", 
"links": [{"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/images/6e947667-ca70-45d1-966f-7aa2920a8228";,
 "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "active", 
"OS-SRV-USG:launched_at": "2016-09-23T19:39:31.000000", "flavor": {"id": "1", 
"links": [{"href": "https://uat-dal09-compute-25
 3.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/flavors/1", "rel": 
"bookmark"}]}, "id": "04e4aa2a-13ed-44ca-a275-bbb703c12493", "security_groups": 
[{"name": "default"}], "user_id": "a0f08771b3654f0e9b1492c32e507022", 
"OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": 
0, "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", 
"metadata": {}, "status": "ACTIVE", "updated": "2016-09-23T19:39:40Z", 
"hostId": "df76544eefef439a0e210b1ea7f4046c939f3c0b72bd58871b5a0917", 
"OS-SRV-USG:terminated_at": null, "key_name": null, "name": 
"tempest.common.compute-instance-859751003", "created": "2016-09-23T19:39:09Z", 
"tenant_id": "015383e7edf74544afac10dbcea21a76", 
"os-extended-volumes:volumes_attached": [{"id": 
"67bbce20-b476-466d-b20a-8337957af780"}], "config_drive": ""}}
  19:42:37     2016-09-23 19:39:41,306 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493
 0.228s
  19:42:37     2016-09-23 19:39:41,306 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: None
  19:42:37         Response - Headers: {'status': '200', 'content-length': 
'1787', 'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493',
 'x-compute-request-id': 'req-fe829b17-4da3-44de-bbb9-6839c6fb4beb', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 
Sep 2016 19:39:41 GMT', 'content-type': 'application/json'}
  19:42:37             Body: {"server": {"OS-EXT-STS:task_state": "rescuing", 
"addresses": {"tempest-ServerRescueNegativeTestJSON-1526350074-network": 
[{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:5b:05:34", "version": 4, "addr": 
"10.100.0.5", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";,
 "rel": "self"}, {"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";,
 "rel": "bookmark"}], "image": {"id": "6e947667-ca70-45d1-966f-7aa2920a8228", 
"links": [{"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/images/6e947667-ca70-45d1-966f-7aa2920a8228";,
 "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "active", 
"OS-SRV-USG:launched_at": "2016-09-23T19:39:31.000000", "flavor": {"id": "1", 
"links": [{"href": "https://uat-dal09-compute-25
 3.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/flavors/1", "rel": 
"bookmark"}]}, "id": "04e4aa2a-13ed-44ca-a275-bbb703c12493", "security_groups": 
[{"name": "default"}], "user_id": "a0f08771b3654f0e9b1492c32e507022", 
"OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": 
0, "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", 
"metadata": {}, "status": "ACTIVE", "updated": "2016-09-23T19:39:40Z", 
"hostId": "df76544eefef439a0e210b1ea7f4046c939f3c0b72bd58871b5a0917", 
"OS-SRV-USG:terminated_at": null, "key_name": null, "name": 
"tempest.common.compute-instance-859751003", "created": "2016-09-23T19:39:09Z", 
"tenant_id": "015383e7edf74544afac10dbcea21a76", 
"os-extended-volumes:volumes_attached": [{"id": 
"67bbce20-b476-466d-b20a-8337957af780"}], "config_drive": ""}}
  19:42:37     2016-09-23 19:39:42,541 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493
 0.227s
  19:42:37     2016-09-23 19:39:42,542 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: None
  19:42:37         Response - Headers: {'status': '200', 'content-length': 
'1787', 'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493',
 'x-compute-request-id': 'req-06e65562-36ac-440b-9b38-45eb5322fa43', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 
Sep 2016 19:39:42 GMT', 'content-type': 'application/json'}
  19:42:37             Body: {"server": {"OS-EXT-STS:task_state": "rescuing", 
"addresses": {"tempest-ServerRescueNegativeTestJSON-1526350074-network": 
[{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:5b:05:34", "version": 4, "addr": 
"10.100.0.5", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";,
 "rel": "self"}, {"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";,
 "rel": "bookmark"}], "image": {"id": "6e947667-ca70-45d1-966f-7aa2920a8228", 
"links": [{"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/images/6e947667-ca70-45d1-966f-7aa2920a8228";,
 "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "active", 
"OS-SRV-USG:launched_at": "2016-09-23T19:39:31.000000", "flavor": {"id": "1", 
"links": [{"href": "https://uat-dal09-compute-25
 3.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/flavors/1", "rel": 
"bookmark"}]}, "id": "04e4aa2a-13ed-44ca-a275-bbb703c12493", "security_groups": 
[{"name": "default"}], "user_id": "a0f08771b3654f0e9b1492c32e507022", 
"OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": 
0, "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", 
"metadata": {}, "status": "ACTIVE", "updated": "2016-09-23T19:39:40Z", 
"hostId": "df76544eefef439a0e210b1ea7f4046c939f3c0b72bd58871b5a0917", 
"OS-SRV-USG:terminated_at": null, "key_name": null, "name": 
"tempest.common.compute-instance-859751003", "created": "2016-09-23T19:39:09Z", 
"tenant_id": "015383e7edf74544afac10dbcea21a76", 
"os-extended-volumes:volumes_attached": [{"id": 
"67bbce20-b476-466d-b20a-8337957af780"}], "config_drive": ""}}
  19:42:37     2016-09-23 19:39:43,799 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493
 0.243s
  19:42:37     2016-09-23 19:39:43,799 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: None
  19:42:37         Response - Headers: {'status': '200', 'content-length': 
'1787', 'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493',
 'x-compute-request-id': 'req-c693fccb-0153-4737-8192-d9869122659a', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 
Sep 2016 19:39:43 GMT', 'content-type': 'application/json'}
  19:42:37             Body: {"server": {"OS-EXT-STS:task_state": "rescuing", 
"addresses": {"tempest-ServerRescueNegativeTestJSON-1526350074-network": 
[{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:5b:05:34", "version": 4, "addr": 
"10.100.0.5", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";,
 "rel": "self"}, {"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";,
 "rel": "bookmark"}], "image": {"id": "6e947667-ca70-45d1-966f-7aa2920a8228", 
"links": [{"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/images/6e947667-ca70-45d1-966f-7aa2920a8228";,
 "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "active", 
"OS-SRV-USG:launched_at": "2016-09-23T19:39:31.000000", "flavor": {"id": "1", 
"links": [{"href": "https://uat-dal09-compute-25
 3.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/flavors/1", "rel": 
"bookmark"}]}, "id": "04e4aa2a-13ed-44ca-a275-bbb703c12493", "security_groups": 
[{"name": "default"}], "user_id": "a0f08771b3654f0e9b1492c32e507022", 
"OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": 
0, "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", 
"metadata": {}, "status": "ACTIVE", "updated": "2016-09-23T19:39:40Z", 
"hostId": "df76544eefef439a0e210b1ea7f4046c939f3c0b72bd58871b5a0917", 
"OS-SRV-USG:terminated_at": null, "key_name": null, "name": 
"tempest.common.compute-instance-859751003", "created": "2016-09-23T19:39:09Z", 
"tenant_id": "015383e7edf74544afac10dbcea21a76", 
"os-extended-volumes:volumes_attached": [{"id": 
"67bbce20-b476-466d-b20a-8337957af780"}], "config_drive": ""}}
  19:42:37     2016-09-23 19:39:45,058 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 200 GET 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493
 0.251s
  19:42:37     2016-09-23 19:39:45,058 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: None
  19:42:37         Response - Headers: {'status': '200', 'content-length': 
'1767', 'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493',
 'x-compute-request-id': 'req-be1c970e-b954-4d73-9b75-23820d0aff25', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 
Sep 2016 19:39:45 GMT', 'content-type': 'application/json'}
  19:42:37             Body: {"server": {"OS-EXT-STS:task_state": null, 
"addresses": {"tempest-ServerRescueNegativeTestJSON-1526350074-network": 
[{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:5b:05:34", "version": 4, "addr": 
"10.100.0.5", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";,
 "rel": "self"}, {"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";,
 "rel": "bookmark"}], "image": {"id": "6e947667-ca70-45d1-966f-7aa2920a8228", 
"links": [{"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/images/6e947667-ca70-45d1-966f-7aa2920a8228";,
 "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "rescued", 
"OS-SRV-USG:launched_at": "2016-09-23T19:39:44.000000", "flavor": {"id": "1", 
"links": [{"href": "https://uat-dal09-compute-253.ope
 n.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/flavors/1", "rel": 
"bookmark"}]}, "id": "04e4aa2a-13ed-44ca-a275-bbb703c12493", "security_groups": 
[{"name": "default"}], "user_id": "a0f08771b3654f0e9b1492c32e507022", 
"OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", 
"OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "metadata": 
{}, "status": "RESCUE", "updated": "2016-09-23T19:39:44Z", "hostId": 
"df76544eefef439a0e210b1ea7f4046c939f3c0b72bd58871b5a0917", 
"OS-SRV-USG:terminated_at": null, "key_name": null, "name": 
"tempest.common.compute-instance-859751003", "created": "2016-09-23T19:39:09Z", 
"tenant_id": "015383e7edf74544afac10dbcea21a76", 
"os-extended-volumes:volumes_attached": [{"id": 
"67bbce20-b476-466d-b20a-8337957af780"}], "config_drive": ""}}
  19:42:37     2016-09-23 19:39:45,064 18008 INFO     [tempest.common.waiters] 
State transition "ACTIVE/rescuing" ==> "RESCUE/None" after 5 second wait
  19:42:37     2016-09-23 19:39:45,254 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:test_rescued_vm_detach_volume): 409 DELETE 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/os-volume_attachments/67bbce20-b476-466d-b20a-8337957af780
 0.188s
  19:42:37     2016-09-23 19:39:45,254 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: None
  19:42:37         Response - Headers: {'status': '409', 'content-length': 
'152', 'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/os-volume_attachments/67bbce20-b476-466d-b20a-8337957af780',
 'x-compute-request-id': 'req-e572880b-7dcd-457d-a06a-ed004351adaf', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 
Sep 2016 19:39:45 GMT', 'content-type': 'application/json; charset=UTF-8'}
  19:42:37             Body: {"conflictingRequest": {"message": "Cannot 
'detach_volume' instance 04e4aa2a-13ed-44ca-a275-bbb703c12493 while it is in 
vm_state rescued", "code": 409}}
  19:42:37     2016-09-23 19:39:45,396 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:_run_cleanups): 202 POST 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/action
 0.140s
  19:42:37     2016-09-23 19:39:45,396 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: {"unrescue": {}}
  19:42:37         Response - Headers: {'status': '202', 'content-length': '0', 
'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/action',
 'x-compute-request-id': 'req-7010ff1d-7d7c-4052-a2d5-3cddb6ddee4c', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 
Sep 2016 19:39:45 GMT', 'content-type': 'application/json'}
  19:42:37             Body:
  19:42:37     2016-09-23 19:39:45,671 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:_run_cleanups): 200 GET 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493
 0.274s
  19:42:37     2016-09-23 19:39:45,672 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: None
  19:42:37         Response - Headers: {'status': '200', 'content-length': 
'1775', 'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493',
 'x-compute-request-id': 'req-8069154d-767b-4d75-ab32-83c8d07ae745', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 
Sep 2016 19:39:45 GMT', 'content-type': 'application/json'}
  19:42:37             Body: {"server": {"OS-EXT-STS:task_state": "unrescuing", 
"addresses": {"tempest-ServerRescueNegativeTestJSON-1526350074-network": 
[{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:5b:05:34", "version": 4, "addr": 
"10.100.0.5", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";,
 "rel": "self"}, {"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";,
 "rel": "bookmark"}], "image": {"id": "6e947667-ca70-45d1-966f-7aa2920a8228", 
"links": [{"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/images/6e947667-ca70-45d1-966f-7aa2920a8228";,
 "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "rescued", 
"OS-SRV-USG:launched_at": "2016-09-23T19:39:44.000000", "flavor": {"id": "1", 
"links": [{"href": "https://uat-dal09-compute
 -253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/flavors/1", 
"rel": "bookmark"}]}, "id": "04e4aa2a-13ed-44ca-a275-bbb703c12493", 
"security_groups": [{"name": "default"}], "user_id": 
"a0f08771b3654f0e9b1492c32e507022", "OS-DCF:diskConfig": "MANUAL", 
"accessIPv4": "", "accessIPv6": "", "OS-EXT-STS:power_state": 1, 
"OS-EXT-AZ:availability_zone": "nova", "metadata": {}, "status": "RESCUE", 
"updated": "2016-09-23T19:39:45Z", "hostId": 
"df76544eefef439a0e210b1ea7f4046c939f3c0b72bd58871b5a0917", 
"OS-SRV-USG:terminated_at": null, "key_name": null, "name": 
"tempest.common.compute-instance-859751003", "created": "2016-09-23T19:39:09Z", 
"tenant_id": "015383e7edf74544afac10dbcea21a76", 
"os-extended-volumes:volumes_attached": [{"id": 
"67bbce20-b476-466d-b20a-8337957af780"}], "config_drive": ""}}
  19:42:37     2016-09-23 19:39:46,921 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:_run_cleanups): 200 GET 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493
 0.240s
  19:42:37     2016-09-23 19:39:46,921 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: None
  19:42:37         Response - Headers: {'status': '200', 'content-length': 
'2010', 'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493',
 'x-compute-request-id': 'req-99919910-98c2-40bb-90fd-0828ead5f1bf', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 
Sep 2016 19:39:46 GMT', 'content-type': 'application/json'}
  19:42:37             Body: {"server": {"OS-EXT-STS:task_state": null, 
"addresses": {"tempest-ServerRescueNegativeTestJSON-1526350074-network": 
[{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:5b:05:34", "version": 4, "addr": 
"10.100.0.5", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";,
 "rel": "self"}, {"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493";,
 "rel": "bookmark"}], "image": {"id": "6e947667-ca70-45d1-966f-7aa2920a8228", 
"links": [{"href": 
"https://uat-dal09-compute-253.open.softlayer.com:8774/015383e7edf74544afac10dbcea21a76/images/6e947667-ca70-45d1-966f-7aa2920a8228";,
 "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "error", 
"OS-SRV-USG:launched_at": "2016-09-23T19:39:44.000000", "flavor": {"id": "1", 
"links": [{"href": "https://uat-dal09-compute-253.open.
 softlayer.com:8774/015383e7edf74544afac10dbcea21a76/flavors/1", "rel": 
"bookmark"}]}, "id": "04e4aa2a-13ed-44ca-a275-bbb703c12493", "security_groups": 
[{"name": "default"}], "user_id": "a0f08771b3654f0e9b1492c32e507022", 
"OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", 
"OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "metadata": 
{}, "status": "ERROR", "updated": "2016-09-23T19:39:46Z", "hostId": 
"df76544eefef439a0e210b1ea7f4046c939f3c0b72bd58871b5a0917", 
"OS-SRV-USG:terminated_at": null, "key_name": null, "name": 
"tempest.common.compute-instance-859751003", "created": "2016-09-23T19:39:09Z", 
"tenant_id": "015383e7edf74544afac10dbcea21a76", 
"os-extended-volumes:volumes_attached": [{"id": 
"67bbce20-b476-466d-b20a-8337957af780"}], "fault": {"message": "Cannot access 
storage file 
'/var/lib/nova/mnt/5fd17b88c08c2759966c40234dfd10c0/volume-67bbce20-b476-466d-b20a-8337957af780'
 (as uid:107, gid:107): No such file or directory", "code": 500, "created": "
 2016-09-23T19:39:46Z"}, "config_drive": ""}}
  19:42:37     2016-09-23 19:39:46,928 18008 INFO     [tempest.common.waiters] 
State transition "RESCUE/unrescuing" ==> "ERROR/None" after 1 second wait
  19:42:37     2016-09-23 19:39:47,151 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:_run_cleanups): 409 DELETE 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/os-volume_attachments/67bbce20-b476-466d-b20a-8337957af780
 0.222s
  19:42:37     2016-09-23 19:39:47,152 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: None
  19:42:37         Response - Headers: {'status': '409', 'content-length': 
'150', 'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/servers/04e4aa2a-13ed-44ca-a275-bbb703c12493/os-volume_attachments/67bbce20-b476-466d-b20a-8337957af780',
 'x-compute-request-id': 'req-2244f3e2-8d8c-4792-8aef-959638c81197', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 
Sep 2016 19:39:47 GMT', 'content-type': 'application/json; charset=UTF-8'}
  19:42:37             Body: {"conflictingRequest": {"message": "Cannot 
'detach_volume' instance 04e4aa2a-13ed-44ca-a275-bbb703c12493 while it is in 
vm_state error", "code": 409}}
  19:42:37     2016-09-23 19:39:47,265 18008 INFO     
[tempest.lib.common.rest_client] Request 
(ServerRescueNegativeTestJSON:_run_cleanups): 500 DELETE 
https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780
 0.111s
  19:42:37     2016-09-23 19:39:47,266 18008 DEBUG    
[tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 
'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
  19:42:37             Body: None
  19:42:37         Response - Headers: {'status': '500', 'content-length': 
'201', 'content-location': 
'https://uat-dal09-compute-253.open.softlayer.com:8774/v2.1/015383e7edf74544afac10dbcea21a76/os-volumes/67bbce20-b476-466d-b20a-8337957af780',
 'x-compute-request-id': 'req-b19a6519-ddf4-406c-b004-a544b8f4aeeb', 'vary': 
'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 
'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Fri, 23 
Sep 2016 19:39:47 GMT', 'content-type': 'application/json; charset=UTF-8'}
  19:42:37             Body: {"computeFault": {"message": "Unexpected API 
Error. Please report this at http://bugs.launchpad.net/nova/ and attach the 
Nova API log if possible.\n<class 'nova.exception.InvalidInput'>", "code": 500}}
  19:42:37
  19:42:37
  19:42:37 Captured traceback-2:
  19:42:37 ~~~~~~~~~~~~~~~~~~~~~
  19:42:37     Traceback (most recent call last):
  19:42:37       File 
"/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/api/compute/base.py",
 line 346, in delete_volume
  19:42:37         cls._delete_volume(cls.volumes_extensions_client, volume_id)
  19:42:37       File 
"/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/api/compute/base.py",
 line 277, in _delete_volume
  19:42:37         volumes_client.delete_volume(volume_id)
  19:42:37       File 
"/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/services/compute/volumes_client.py",
 line 75, in delete_volume
  19:42:37         resp, body = self.delete("os-volumes/%s" % volume_id)
  19:42:37       File 
"/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/common/rest_client.py",
 line 307, in delete
  19:42:37         return self.request('DELETE', url, extra_headers, headers, 
body)
  19:42:37       File 
"/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/services/compute/base_compute_client.py",
 line 48, in request
  19:42:37         method, url, extra_headers, headers, body, chunked)
  19:42:37       File 
"/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/common/rest_client.py",
 line 665, in request
  19:42:37         resp, resp_body)
  19:42:37       File 
"/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/common/rest_client.py",
 line 829, in _error_checker
  19:42:37         message=message)
  19:42:37     tempest.lib.exceptions.ServerFault: Got server fault
  19:42:37     Details: Unexpected API Error. Please report this at 
http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  19:42:37     <class 'nova.exception.InvalidInput'>
  19:42:37
  19:42:37
  19:42:37 Captured traceback-1:
  19:42:37 ~~~~~~~~~~~~~~~~~~~~~
  19:42:37     Traceback (most recent call last):
  19:42:37       File 
"/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/api/compute/servers/test_server_rescue_negative.py",
 line 73, in _detach
  19:42:37         self.servers_client.detach_volume(server_id, volume_id)
  19:42:37       File 
"/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/services/compute/servers_client.py",
 line 404, in detach_volume
  19:42:37         (server_id, volume_id))
  19:42:37       File 
"/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/common/rest_client.py",
 line 307, in delete
  19:42:37         return self.request('DELETE', url, extra_headers, headers, 
body)
  19:42:37       File 
"/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/services/compute/base_compute_client.py",
 line 48, in request
  19:42:37         method, url, extra_headers, headers, body, chunked)
  19:42:37       File 
"/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/common/rest_client.py",
 line 665, in request
  19:42:37         resp, resp_body)
  19:42:37       File 
"/opt/bbc/openstack-11.0-master/tempest/local/lib/python2.7/site-packages/tempest/lib/common/rest_client.py",
 line 778, in _error_checker
  19:42:37         raise exceptions.Conflict(resp_body, resp=resp)
  19:42:37     tempest.lib.exceptions.Conflict: An object with that identifier 
already exists
  19:42:37     Details: {u'message': u"Cannot 'detach_volume' instance 
04e4aa2a-13ed-44ca-a275-bbb703c12493 while it is in vm_state error", u'code': 
409}
  19:42:37

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1627183/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to     : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp