I'd suggest looking at how the devstack-plugin-ceph repo configures ceph and the openstack services:
https://github.com/openstack/devstack-plugin-ceph But I'm going to close this as invalid since it's a support request, not a bug per se, since we have ceph CI jobs running on xenial without issues. ** Changed in: nova Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1623199 Title: nova.virt.block_device: Driver failed to attach volume (RBD backed Cinder) Status in OpenStack Compute (nova): Invalid Bug description: Hello, I have working Mitaka deployment on Ubuntu 16.04. I am using Ceph RBD backed Nova Ephemeral storage, Cinder volumes, and Glance Images. Neutron is configured for provider network using linux bridge agent. Everything is working correctly, except I am unable to attach Cinder volume to Nova instance. I have googled several similar bugs, but all of them are old and give inconclusive solutions. Upon attempting nova volume attach, it gives a response, but logs show an internal error, and the attach is failed. arcuser@arccloud01:~$ nova --debug volume-attach fd3620de-6c48-4019-a0c6-d6bcc084f095 009579fd-52b7-46e3-8a51-c09bef28852d DEBUG (extension:157) found extension EntryPoint.parse('v2token = keystoneauth1.loading._plugins.identity.v2:Token') DEBUG (extension:157) found extension EntryPoint.parse('v3oauth1 = keystoneauth1.extras.oauth1._loading:V3OAuth1') DEBUG (extension:157) found extension EntryPoint.parse('admin_token = keystoneauth1.loading._plugins.admin_token:AdminToken') DEBUG (extension:157) found extension EntryPoint.parse('v3oidcauthcode = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAuthorizationCode') DEBUG (extension:157) found extension EntryPoint.parse('v2password = keystoneauth1.loading._plugins.identity.v2:Password') DEBUG (extension:157) found extension EntryPoint.parse('v3samlpassword = keystoneauth1.extras._saml2._loading:Saml2Password') DEBUG (extension:157) found extension EntryPoint.parse('v3password = keystoneauth1.loading._plugins.identity.v3:Password') DEBUG (extension:157) found extension EntryPoint.parse('v3oidcaccesstoken = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAccessToken') DEBUG (extension:157) found extension EntryPoint.parse('v3oidcpassword = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectPassword') DEBUG (extension:157) found extension EntryPoint.parse('v3kerberos = keystoneauth1.extras.kerberos._loading:Kerberos') DEBUG (extension:157) found extension EntryPoint.parse('token = keystoneauth1.loading._plugins.identity.generic:Token') DEBUG (extension:157) found extension EntryPoint.parse('v3oidcclientcredentials = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectClientCredentials') DEBUG (extension:157) found extension EntryPoint.parse('v3tokenlessauth = keystoneauth1.loading._plugins.identity.v3:TokenlessAuth') DEBUG (extension:157) found extension EntryPoint.parse('v3token = keystoneauth1.loading._plugins.identity.v3:Token') DEBUG (extension:157) found extension EntryPoint.parse('v3totp = keystoneauth1.loading._plugins.identity.v3:TOTP') DEBUG (extension:157) found extension EntryPoint.parse('password = keystoneauth1.loading._plugins.identity.generic:Password') DEBUG (extension:157) found extension EntryPoint.parse('v3fedkerb = keystoneauth1.extras.kerberos._loading:MappedKerberos') DEBUG (extension:157) found extension EntryPoint.parse('password-aodh-legacy = aodh.keystone_client:LegacyAodhKeystoneLoader') DEBUG (extension:157) found extension EntryPoint.parse('password-ceilometer-legacy = ceilometer.keystone_client:LegacyCeilometerKeystoneLoader') DEBUG (extension:157) found extension EntryPoint.parse('aodh-noauth = aodhclient.noauth:AodhNoAuthLoader') DEBUG (extension:157) found extension EntryPoint.parse('gnocchi-noauth = gnocchiclient.noauth:GnocchiNoAuthLoader') DEBUG (session:337) REQ: curl -g -i -X GET http://controller:35357/v3 -H "Accept: application/json" -H "User-Agent: novakeystoneauth1/2.12.1 python- requests/2.11.1 CPython/2.7.12" INFO (connectionpool:214) Starting new HTTP connection (1): controller DEBUG (connectionpool:401) "GET /v3 HTTP/1.1" 200 250 DEBUG (session:366) RESP: [200] Date: Tue, 13 Sep 2016 21:02:10 GMT Server: Apache/2.4.18 (Ubuntu) Vary: X-Auth-Token X-Distribution: Ubuntu x-openstack- request-id: req-06a57b71-c6f0-4119-99ad-c60396408a98 Content-Length: 250 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: application/json RESP BODY: {"version": {"status": "stable", "updated": "2016-04-04T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.6", "links": [{"href": "http://controller:35357/v3/";, "rel": "self"}]}} DEBUG (base:165) Making authentication request to http://controller:35357/v3/auth/tokens DEBUG (connectionpool:401) "POST /v3/auth/tokens HTTP/1.1" 201 8331 DEBUG (base:170) {"token": {"methods": ["password"], "roles": [{"id": "2911c6c7981d4572b777456c4c032236", "name": "admin"}], "expires_at": "2016-09- 15T03:02:10.883899Z", "project": {"domain": {"id": "c91a4fc8d1244e71b274882386c74138", "name": "default"}, "id": "47a164e5de59452987ee2fc215169e49", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://controller:8776/v1/47a164e5de59452987ee2fc215169e49";, "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "5a48e964052f454689322705b15e4034"}, {"url": "http://controller:8776/v1/47a164e5de59452987ee2fc215169e49";, "in terface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "5b615ad63eb54bfea46d1f5b13134458"}, {"url": " http://controller:8776/v1/47a164e5de59452987ee2fc215169e49";, "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "8a011aa63ad840ccbc4ea01fdc5dd68e"}], "type": "volume", "id": "06511bb4c8344e51a42f54e71261ae57", "name": "cinder"}, {"endpoints": [{"url": "http://controller:8000/v1";, "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "07084108ae964d40aeeb95e8d2b0fe5d"}, {"url": "http://controller:8000/v1";, "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "ce3dc9d8102449b09f5b4136dfcda2b3"}, {"url": "http://controller:8000/v1";, "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "d7a0a6c2309e4383be7f0193e5 e95635"}], "type": "cloudformation", "id": "463f66baba8a4e6abf4cd0b6aba16116", "name": "heat-cfn"}, {"endpoints": [{"url ": "http://controller:9696";, "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "8ff0e9839e fc4b30a69b32a3dd78aa60"}, {"url": "http://controller:9696";, "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "c534498289044e0ba3f1a36f5f0f3dda"}, {"url": "http://controller:9696";, "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "ffe0025ead9844869b9f2bc1db09e3c2"}], "type": "network", "id": "5311e966f4094aca80785a95b9f5f559", "name": "neutron"}, {"endpoints": [{"url": "http://controller:8042";, "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "39d65178607a4547b3e4efac634b64a3"}, {"url": "http://controller:8042";, "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "5e24d13c2d894fe18daf3ea88c1ee255"}, {"u rl": "http://controller:8042";, "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "94bfe518 311e4e7e9a0656bdd7b020c0"}], "type": "alarming", "id": "5428a916de284d87b5be6afbfcef7a34", "name": "aodh"}, {"endpoints" : [{"url": "http://controller:8774/v2.1/47a164e5de59452987ee2fc215169e49";, "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "069e435d2c144966b34cd398f155366c"}, {"url": "http://controller:8774/v2.1/47a164e5de59452987ee2fc215169e49";, "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "3b74a66620034c33bb39a8a85d3731eb"}, {"url": "http://controller:8774/v2.1/47a164e5de59452987ee2fc215169e49";, "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "6661ab10af08453ebabdc9442fde4b1c"}], "type": "compute", "id": "62fb930c ccc4406ea3ce88d239186577", "name": "nova"}, {"endpoints": [{"url": "http://controller:35357/v3";, "interface": "admin", " region": "RegionOne", "region_id": "RegionOne", "id": "606cb8c8ed474a528a33dbbccf8e39a3"}, {"url": "http://controller:5000/v3";, "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "aaa4f14f9e9c4ba9a65216e16f309808"}, {"url": "http://controller:5000/v3";, "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "b5a089ea7b0a451b8133ea9ffd6c65f0"}], "type": "identity", "id": "7bff48186ae541a68cb2022a4fffa98c", "name": "keystone"}, {"endpoints": [{"url": "http://controller:9511/v1";, "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "420b2746d2fa429485de7665a22d0b39"}, {"url": "http://controller:9511/v1";, "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "c9fc4628720d44eab31aa73cbc3b3e1b"}, {"url": "http://controller:9511/v1";, "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "dfc09427dca14c0c84140e8e81dbc39a"}], "type": "container-infra", "id": "a44dc4e12183403c9e3ab2d8d1f61cfe", "name": "magnum"}, {"endpoints": [{"url": "http://controller:8777";, "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "0ff4ea709333455b9afd827ef38a 6d54"}, {"url": "http://controller:8777";, "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "a5da7a65e1d142249a8cc0dcb643666e"}, {"url": "http://controller:8777";, "interface": "admin", "region": "RegionOne", "reg ion_id": "RegionOne", "id": "a917bd7b206b40949f6a1ddb2461d2c2"}], "type": "metering", "id": "ae2546cdac9249c1b1687817a28baffe", "name": "ceilometer"}, {"endpoints": [{"url": "http://controller:8004/v1/47a164e5de59452987ee2fc215169e49";, "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "1bb8460eca9040c98d8e7be0785e7810"}, {"url": "http://controller:8004/v1/47a164e5de59452987ee2fc215169e49";, "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "3c63f2c47eb349aa8a80e25600a44449"}, {"url": "http://controller:8004/v1/47a164e5de59452987ee2fc2151 69e49", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "5dae8f72428c4fd6a80fb0b2ba3400b3"}], "type": "orchestration", "id": "bd4e32b1f2d048958b7c482165d3e2aa", "name": "heat"}, {"endpoints": [{"url": "http://controller:8786/v2/47a164e5de59452987ee2fc215169e49";, "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "8dcf49e775064513ba517b5121b1badf"}, {"url": "http://controller:8786/v2/47a164e5de59452987ee2fc215169e49";, "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "95ecc90d77374770b9f1c98f94438761"}, {"url": "http://controller:8786/v2/47a164e5de59452987ee2fc215169e49";, "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "c51efa731b45475db9a2d2cb601916e8"}], "type": "sharev2", "id": "c4fa0da0002c4629b8222faa971c5a91", "name": "manilav2"}, {"endpoints": [{"url": "http://controller:8786/v1/47a164e5de59452987ee2fc215169e49";, "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "303efb49815d4a8c97436fb90cb717a9"}, {"url": "http://controller:8786/v1/47a164e5de59452987ee2fc215169e49";, "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "6715fede21464e2b9b63f8271ed964aa"}, {"url": "http://controller:8786/v1/47a164e5de59452987ee2fc215169e49";, "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "7ce9668fbdb149508da4219ab5dfd6b2"}], "type": "share", "id": "cda885af638643b3a88e5db325a891b2", "name": "manila"}, {"endpoints": [{"url": "http://controller:8776/v2/47a164e5de59452987ee2fc215169e49";, "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "1bca402c3b8245b6a5d3929a006563a7"}, {"url": "http://controller:8776/v2/47a164e5de59452987ee2fc215169e49";, "interface": "inter nal", "region": "RegionOne", "region_id": "RegionOne", "id": "37566cd0af7b49df8d24f433e27d9b1a"}, {"url": "http://controller:8776/v2/47a164e5de59452987ee2fc215169e49";, "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "b7d748e7ec254b44ae135f63c9793c82"}], "type": "volumev2", "id": "df06dd503bf74756b846f5847b27ed29", "name": "cinderv2"}, {"endpoints": [{"url": "http://192.168.92.128:9292";, "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "5fd33c23c79746c3af8f81d4d7847952"}, {"url": "http://192.168.92.128:9292";, "interface": "internal", "reg ion": "RegionOne", "region_id": "RegionOne", "id": "729788ec2d754fa08ebd409d305077ce"}, {"url": "http://192.168.92.128:9292";, "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "cd1c589b5a014ece93e7140706818e42"}], "type": "image", "id": "eb8cfdf9cef845e1adea9b1acf3dfcb3", "name": "glance"}], "user": {"domain": {"id": "c91a4fc8d1244e71b274882386c74138", "name": "default"}, "id": "996a74a0b9de4473b64cf1d94ff38c9f", "name": "admin"}, "audit_ids": ["P7uMGNjyRwWRN7upb4Itwg"], "issued_at": "2016-09-13T21:02:10.000000Z"}} DEBUG (session:337) REQ: curl -g -i -X GET http://controller:8774/v2.1/47a164e5de59452987ee2fc215169e49 -H "User-Agent:python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}136c85e0c0255243f320a5716c3b99f95d4d19db" INFO (connectionpool:214) Starting new HTTP connection (1): controller DEBUG (connectionpool:401) "GET /v2.1/47a164e5de59452987ee2fc215169e49 HTTP/1.1" 404 52 DEBUG (session:366) RESP: [404] Content-Length: 52 Content-Type: text/plain; charset=UTF-8 X-Compute-Request-Id: req-bb9a88ca-dab2-412a-8928-4350cfc323f2 Date: Tue, 13 Sep 2016 21:02:11 GMT Connection: keep-alive RESP BODY: 404 Not Found The resource could not be found. DEBUG (session:337) REQ: curl -g -i -X GET http://controller:8774/v2.1/ -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}136c85e0c0255243f320a5716c3b99f95d4d19db" DEBUG (connectionpool:401) "GET /v2.1/ HTTP/1.1" 200 384 DEBUG (session:366) RESP: [200] Content-Length: 384 Content-Type: application/json X-Openstack-Nova-Api-Version: 2.1 Vary: X-OpenStack-Nova-API-Version X- Compute-Request-Id: req-093d2854-7336-4238-a7b2-2f171c45528f Date: Tue, 13 Sep 2016 21:02:11 GMT Connection: keep-alive RESP BODY: {"version": {"status": "CURRENT", "updated": "2013-07-23T11:33:21Z", "links": [{"href": "http://controller:8774/v2.1/";, "rel": "self"}, {"href": "http://docs.openstack.org/";, "type": "text/html", "rel": "describedby"}], "min_version": "2.1", "version": "2.25", "media- types": [{"base": "application/json", "type": "application/vnd.openstack.compute+json;version=2.1"}], "id": "v2.1"}} DEBUG (extension:157) found extension EntryPoint.parse('v2token = keystoneauth1.loading._plugins.identity.v2:Token') DEBUG (extension:157) found extension EntryPoint.parse('v3oauth1 = keystoneauth1.extras.oauth1._loading:V3OAuth1') DEBUG (extension:157) found extension EntryPoint.parse('admin_token = keystoneauth1.loading._plugins.admin_token:AdminToken') DEBUG (extension:157) found extension EntryPoint.parse('v3oidcauthcode = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAuthorizationCode') DEBUG (extension:157) found extension EntryPoint.parse('v2password = keystoneauth1.loading._plugins.identity.v2:Password') DEBUG (extension:157) found extension EntryPoint.parse('v3samlpassword = keystoneauth1.extras._saml2._loading:Saml2Password') DEBUG (extension:157) found extension EntryPoint.parse('v3password = keystoneauth1.loading._plugins.identity.v3:Password') DEBUG (extension:157) found extension EntryPoint.parse('v3oidcaccesstoken = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAccessToken') DEBUG (extension:157) found extension EntryPoint.parse('v3oidcpassword = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectPassword') DEBUG (extension:157) found extension EntryPoint.parse('v3kerberos = keystoneauth1.extras.kerberos._loading:Kerberos') DEBUG (extension:157) found extension EntryPoint.parse('token = keystoneauth1.loading._plugins.identity.generic:Token') DEBUG (extension:157) found extension EntryPoint.parse('v3oidcclientcredentials = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectClientCredentials') DEBUG (extension:157) found extension EntryPoint.parse('v3tokenlessauth = keystoneauth1.loading._plugins.identity.v3:TokenlessAuth') DEBUG (extension:157) found extension EntryPoint.parse('v3token = keystoneauth1.loading._plugins.identity.v3:Token') DEBUG (extension:157) found extension EntryPoint.parse('v3totp = keystoneauth1.loading._plugins.identity.v3:TOTP') DEBUG (extension:157) found extension EntryPoint.parse('password = keystoneauth1.loading._plugins.identity.generic:Password') DEBUG (extension:157) found extension EntryPoint.parse('v3fedkerb = keystoneauth1.extras.kerberos._loading:MappedKerberos') DEBUG (extension:157) found extension EntryPoint.parse('password-aodh-legacy = aodh.keystone_client:LegacyAodhKeystoneLoader') DEBUG (extension:157) found extension EntryPoint.parse('password-ceilometer-legacy = ceilometer.keystone_client:LegacyCeilometerKeystoneLoader') DEBUG (extension:157) found extension EntryPoint.parse('aodh-noauth = aodhclient.noauth:AodhNoAuthLoader') DEBUG (extension:157) found extension EntryPoint.parse('gnocchi-noauth = gnocchiclient.noauth:GnocchiNoAuthLoader') DEBUG (session:337) REQ: curl -g -i -X GET http://controller:8774/v2.1/47a164e5de59452987ee2fc215169e49/servers/fd3620de-6c48-4019-a0c6-d6bcc084f095 -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X -OpenStack-Nova-API-Version: 2.25" -H "X-Auth-Token: {SHA1} 136c85e0c0255243f320a5716c3b99f95d4d19db" DEBUG (connectionpool:401) "GET /v2.1/47a164e5de59452987ee2fc215169e49/servers/fd3620de-6c48-4019-a0c6-d6bcc084f095 HTTP/1.1" 200 2193 DEBUG (session:366) RESP: [200] Content-Length: 2193 Content-Type: application/json X-Openstack-Nova-Api-Version: 2.25 Vary: X-OpenStack-Nova-API-Version X- Compute-Request-Id: req-7aed24e7-9c89-4c9d-bde4-21858a746ac5 Date: Tue, 13 Sep 2016 21:02:11 GMT Connection: keep-alive RESP BODY: {"server": {"status": "ACTIVE", "OS-EXT-SRV-ATTR:ramdisk_id": "", "updated": "2016-09-12T23:19:32Z", "hostId" : "fed0f3554a473289f1895016c33c4e9532c1ab4558e76ffb690e85c4", "OS-EXT- SRV-ATTR:host": "arccloud01", "addresses": {"External": [{"OS-EXT-IPS- MAC:mac_addr": "fa:16:3e:6b:e9:94", "version": 4, "addr": "10.155.93.102", "OS-EXT- IPS:type": "fixed"}], "Internal": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:84:07:e9", "version": 4, "addr": "192.168.93.87", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "http://controller:8774/v2.1/47a164e5de59452987ee2fc215169e49/servers /fd3620de-6c48-4019-a0c6-d6bcc084f095", "rel": "self"}, {"href": "http://controller:8774/47a164e5de59452987ee2fc215169e49/servers /fd3620de-6c48-4019-a0c6-d6bcc084f095", "rel": "bookmark"}], "key_name": "ARC_Key", "image": {"id": "6b99ed84-45a1-45e8-aec3-e16a34347254", "links": [{"href": "http://controller:8774/47a164e5de59452987ee2fc215169e49/images/6b99ed84-45a1-45e8-aec3- e16a34347254", "rel": "bookmark"}]}, "OS-EXT-SRV-ATTR:user_data": null, "OS-EXT-STS:task_state": null, "OS-EXT-STS:vm_state": "active", "OS-EXT-SRV- ATTR:instance_name": "instance-00000075", "OS-EXT-SRV- ATTR:root_device_name": "/dev/vda", "OS-SRV-USG:launched_at": "2016-09-12T23:19:32.000000", "locked": false, "flavor": {"id": "2", "links": [{"href": "http://controller:8774/47a164e5de59452987ee2fc215169e49/flavors/2";, "rel": "bookmark"}]}, "id": "fd3620de- 6c48-4019-a0c6-d6bcc084f095", "security_groups": [{"name": "default"}, {"name": "default"}], "description": "test for volume", "OS-EXT-SRV- ATTR:kernel_id": "", "host_status": "UP", "OS-EXT-AZ:availability_zone": "nova", "user_id": "996a74a0b9de4473b64cf1d94ff38c9f", "name": "test for volume", "OS-EXT-SRV- ATTR:launch_index": 0, "created": "2016-09-12T23:17:47Z", "tenant_id": "47a164e5de59452987ee2fc215169e49", "OS-DCF:diskConfig": "AUTO", "OS- EXT-SRV- ATTR:hypervisor_hostname": "arccloud01", "os-extended- volumes:volumes_attached": [], "accessIPv4": "", "accessIPv6": "", "OS-EXT-SRV-ATTR:reservation_id": "r-geqolgne", "OS-EXT-SRV-ATTR:hostname": "test-for-volume", "progress": 0, "OS-EXT-STS:power_state": 1, "config_drive": "", "OS- SRV-USG:terminated_at": null, "metadata": {}}} DEBUG (session:337) REQ: curl -g -i -X POST http://controller:8774/v2.1/47a164e5de59452987ee2fc215169e49/servers /fd3620de-6c48-4019-a0c6-d6bcc084f095/os- volume_attachments -H "User-Agent: python-novaclient" -H "Content- Type: application/json" -H "Accept: application/json" -H "X-OpenStack- Nova-API-Version: 2.25" -H "X-Auth-Token: {SHA1}136c85e0c0255243f320a5716c3b99f95d4d19db" -d '{"volumeAttachment": {"volumeId": "009579fd-52b7-46e3-8a51-c09bef28852d"}}' DEBUG (connectionpool:401) "POST /v2.1/47a164e5de59452987ee2fc215169e49/servers/fd3620de-6c48-4019-a0c6-d6bcc084f095/os-volume_attachments HTTP/1.1" 200 194 DEBUG (session:366) RESP: [200] Content-Length: 194 Content-Type: application/json X-Openstack-Nova-Api-Version: 2.25 Vary: X-OpenStack-Nova-API-Version X- Compute-Request-Id: req-42da6d93-e7dd-427f-bf73-14f94b925a77 Date: Tue, 13 Sep 2016 21:02:12 GMT Connection: keep-alive RESP BODY: {"volumeAttachment": {"device": "/dev/vdb", "serverId": "fd3620de-6c48-4019-a0c6-d6bcc084f095", "id": "009579fd-52b7-46e3-8a51-c09bef28852d", "volumeId": "009579fd-52b7-46e3-8a51-c09bef28852d"}} +----------+--------------------------------------+ | Property | Value | +----------+--------------------------------------+ | device | /dev/vdb | | id | 009579fd-52b7-46e3-8a51-c09bef28852d | | serverId | fd3620de-6c48-4019-a0c6-d6bcc084f095 | | volumeId | 009579fd-52b7-46e3-8a51-c09bef28852d | +----------+--------------------------------------+ From nova-compute.log: ERROR nova.virt.block_device [req-5c342ce8-fc35-4245-8bff-5731859257c3 996a74a0b9de4473b64cf1d94ff38c9f 47a164e5de59452987ee2fc215169e49 - - -] [instance: fd3620de-6c48-4019-a0c6-d6bcc084f095] Driver failed to attach volume 009579fd-52b7-46e3-8a51-c09bef28852d at /dev/vdb Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 279, in attach device_type=self['device_type'], encryption=encryption) File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1359, in attach_volume self._disconnect_volume(connection_info, disk_dev) File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ self.force_reraise() File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise six.reraise(self.type_, self.value, self.tb) File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1348, in attach_volume guest.attach_device(conf, persistent=True, live=live) File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 250, in attach_device self._domain.attachDeviceFlags(conf.to_xml(), flags=flags) File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit result = proxy_call(self._autowrap, f, *args, **kwargs) File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call rv = execute(f, *args, **kwargs) File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute six.reraise(c, e, tb) File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker rv = meth(*args, **kwargs) File "/usr/lib/python2.7/dist-packages/libvirt.py", line 560, in attachDeviceFlags if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self) libvirtError: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1' cat /etc/ceph/ceph.conf: [global] fsid = 6e647506-631a-457e-a52a-f21a3866a023 mon_initial_members = arccloud01, arccloud02, arccloud03 mon_host = 10.155.92.128,10.155.92.129,10.155.92.130 mon_pg_warn_max_per_osd = 400 mon_lease = 50 mon_lease_renew_interval = 30 mon_lease_ack_timeout = 100 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx public_network = 10.155.92.0/22 cluster_network = 192.168.92.0/22 [client.glanceimages] keyring = /etc/ceph/ceph.client.glanceimages.keyring [client.novapool] keyring = /etc/ceph/ceph.client.novapool.keyring [client.cindervolumes] keyring = /etc/ceph/ceph.client.cindervolumes.keyring [client.manila] client_mount_uid = 0 client_mount_gid = 0 log_file = /opt/stack/logs/ceph-client.manila.log admin_socket = /opt/stack/status/stack/ceph-$name.$pid.asok keyring = /etc/ceph/ceph.client.manila.keyring [mon.arccloud01] host = arccloud01 mon addr = 10.155.92.128:6789 [mon.arccloud02] host = arccloud02 mon addr = 10.155.92.129:6789 [mon.arccloud03] host = arccloud03 mon addr = 10.155.92.130:6789 [osd.2] host = arccloud01 public addr = 10.155.92.128 cluster addr = 192.168.92.128 [osd.1] host = arccloud02 public addr = 10.155.92.129 cluster addr = 192.168.92.129 [osd.0] host = arccloud03 public addr = 10.155.92.130 cluster addr = 192.168.92.130 rbd /etc/apparmor.d/abstractions/libvirt-qemu: # for rbd /etc/ceph/** r, /usr/lib/x86_64-linux-gnu/qemu/block-rbd.so rm, /usr/lib/x86_64-linux-gnu/qemu/block-curl.so rm, # for qemu-block-extra /usr/lib/@{multiarch}/qemu/*.so rm, cat /etc/cinder/cinder.conf: [DEFAULT] rootwrap_config = /etc/cinder/rootwrap.conf api_paste_confg = /etc/cinder/api-paste.ini iscsi_helper = tgtadm volume_name_template = volume-%s volume_group = cinder-volumes verbose = True auth_strategy = keystone state_path = /var/lib/cinder lock_path = /var/lock/cinder volumes_dir = /var/lib/cinder/volumes rpc_backend = rabbit my_ip = 10.155.92.128 enabled_backends = RBD-backend glance_api_servers = http://192.168.92.128:9292 [database] connection = mysql+pymysql://cinder:xxxx@controller/cinder [oslo_messaging_rabbit] rabbit_host = controller rabbit_userid = openstack rabbit_password = xxxx [oslo_messaging_notifications] driver = messagingv2 [keystone_authtoken] auth_uri = http://controller:5000 auth_url = http://controller:35357 memcached_servers = controller:11211 auth_type = password project_domain_name = default user_domain_name = default project_name = service username = cinder password = xxxx [oslo_concurrency] lock_path = /var/lib/cinder/tmp [RBD-backend] volume_backend_name=RBD-backend rbd_pool=cindervolumes rbd_user=cindervolumes rbd_secret_uuid=d845ac85-4517-4b0f-8a5f-a9c5daca5f8d volume_driver=cinder.volume.drivers.rbd.RBDDriver rbd_ceph_conf=/etc/ceph/ceph.conf cat /etc/nova/nova.conf [DEFAULT] dhcpbridge_flagfile=/etc/nova/nova.conf dhcpbridge=/usr/bin/nova-dhcpbridge # logdir=/var/log/nova state_path=/var/lib/nova lock_path=/var/lock/nova force_dhcp_release=True libvirt_use_virtio_for_bridges=True verbose=True ec2_private_dns_show_ip=True api_paste_config=/etc/nova/api-paste.ini enabled_apis=osapi_compute,metadata rpc_backend = rabbit auth_strategy = keystone my_ip = 10.155.92.128 use_neutron = True firewall_driver = nova.virt.firewall.NoopFirewallDriver # Missing from the original config: linuxnet_interface_driver = nova.network.linux_net.NeutronLinuxBridgeInterfaceDriver instance_usage_audit = True instance_usage_audit_period = hour notify_on_state_change = vm_and_task_state notification_driver = messagingv2 [api_database] connection = mysql+pymysql://nova:xxxx@controller/nova_api [database] connection = mysql+pymysql://nova:xxxx@controller/nova [oslo_messaging_rabbit] rabbit_host = controller rabbit_userid = openstack rabbit_password = xxxx [keystone_authtoken] auth_uri = http://controller:5000 auth_url = http://controller:35357 memcached_servers = controller:11211 auth_type = password project_domain_name = default user_domain_name = default project_name = service username = nova password = xxxx [vnc] enabled = True vncserver_listen = 0.0.0.0 vncserver_proxyclient_address = $my_ip novncproxy_base_url = http://10.155.92.128:6080/vnc_auto.html [glance] api_servers = http://192.168.92.128:9292 [oslo_concurrency] lock_path = /var/lib/nova/tmp [libvirt] images_rbd_pool=novapool images_type=rbd rbd_secret_uuid=efd948be-8153-4775-8631-7c9e7b3f4aa0 rbd_user=novapool [neutron] url = http://controller:9696 auth_url = http://controller:35357 auth_type = password project_domain_name = default user_domain_name = default region_name = RegionOne project_name = service username = neutron password = xxxx service_metadata_proxy = True metadata_proxy_shared_secret = efd948be-8153-4775-8631-7c9e7b3f4aa0 [cinder] os_region_name = RegionOne root@arccloud01:~# dpkg -l | grep nova ii nova-api 2:13.1.0-0ubuntu1 all OpenStack Compute - API frontend ii nova-common 2:13.1.0-0ubuntu1 all OpenStack Compute - common files ii nova-compute 2:13.1.0-0ubuntu1 all OpenStack Compute - compute node base ii nova-compute-kvm 2:13.1.0-0ubuntu1 all OpenStack Compute - compute node (KVM) ii nova-compute-libvirt 2:13.1.0-0ubuntu1 all OpenStack Compute - compute node libvirt support ii nova-conductor 2:13.1.0-0ubuntu1 all OpenStack Compute - conductor service ii nova-consoleauth 2:13.1.0-0ubuntu1 all OpenStack Compute - Console Authenticator ii nova-novncproxy 2:13.1.0-0ubuntu1 all OpenStack Compute - NoVNC proxy ii nova-scheduler 2:13.1.0-0ubuntu1 all OpenStack Compute - virtual machine scheduler ii python-nova 2:13.1.0-0ubuntu1 all OpenStack Compute Python libraries ii python-novaclient 2:3.3.1-2 all client library for OpenStack Compute API - Python 2.7 root@arccloud01:~# dpkg -l | grep cinder ii cinder-api 2:8.0.0-0ubuntu1 all Cinder storage service - API server ii cinder-common 2:8.0.0-0ubuntu1 all Cinder storage service - common files ii cinder-scheduler 2:8.0.0-0ubuntu1 all Cinder storage service - Scheduler server ii cinder-volume 2:8.0.0-0ubuntu1 all Cinder storage service - Volume server ii python-cinder 2:8.0.0-0ubuntu1 all Cinder Python libraries ii python-cinderclient 1:1.6.0-2 all Python bindings to the OpenStack Volume API - Python 2.x root@arccloud01:~# dpkg -l | grep ceph ii ceph 10.2.2-0ubuntu0.16.04.2 amd64 distributed storage and file system ii ceph-common 10.2.2-0ubuntu0.16.04.2 amd64 common utilities to mount and interact with a ceph storage cluster ii ceph-deploy 1.5.32-0ubuntu1 all Deployment and configuration of Ceph. ii ceph-fuse 10.2.2-0ubuntu0.16.04.2 amd64 FUSE-based client for the Ceph distributed file system ii ceph-mds 10.2.2-0ubuntu0.16.04.2 amd64 metadata server for the ceph distributed file system ii libcephfs1 10.2.2-0ubuntu0.16.04.2 amd64 Ceph distributed file system client library ii python-cephfs 10.2.2-0ubuntu0.16.04.2 amd64 Python libraries for the Ceph libcephfs library root@arccloud01:~# uname -a Linux arccloud01 4.4.0-36-generic #55-Ubuntu SMP Thu Aug 11 18:01:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux root@arccloud01:~# dpkg -l | grep neutron ii neutron-common 2:8.1.2-0ubuntu1 all Neutron is a virtual network service for Openstack - common ii neutron-dhcp-agent 2:8.1.2-0ubuntu1 all Neutron is a virtual network service for Openstack - DHCP agent ii neutron-l3-agent 2:8.1.2-0ubuntu1 all Neutron is a virtual network service for Openstack - l3 agent ii neutron-linuxbridge-agent 2:8.1.2-0ubuntu1 all Neutron is a virtual network service for Openstack - linuxbridge agent ii neutron-metadata-agent 2:8.1.2-0ubuntu1 all Neutron is a virtual network service for Openstack - metadata agent ii neutron-plugin-ml2 2:8.1.2-0ubuntu1 all Neutron is a virtual network service for Openstack - ML2 plugin ii neutron-server 2:8.1.2-0ubuntu1 all Neutron is a virtual network service for Openstack - server ii python-neutron 2:8.1.2-0ubuntu1 all Neutron is a virtual network service for Openstack - Python library ii python-neutron-fwaas 1:8.0.0-0ubuntu1 all Firewall-as-a-Service driver for OpenStack Neutron ii python-neutron-lib 0.0.2-2 all Neutron shared routines and utilities - Python 2.7 ii python-neutronclient 1:4.1.1-2 all client API library for Neutron - Python 2.7 root@arccloud01:~# dpkg -l | grep libvirt ii libsys-virt-perl 1.2.16-1ubuntu2 amd64 Perl module providing an extension for the libvirt library ii libvirt-bin 1.3.1-1ubuntu10.1 amd64 programs for the libvirt library ii libvirt0:amd64 1.3.1-1ubuntu10.1 amd64 library for interfacing with different virtualization systems ii nova-compute-libvirt 2:13.1.0-0ubuntu1 all OpenStack Compute - compute node libvirt support ii python-libvirt 1.3.1-1ubuntu1 amd64 libvirt Python bindings root@arccloud01:~# dpkg -l | grep qemu ii ipxe-qemu 1.0.0+git-20150424.a25a16d-1ubuntu1 all PXE boot firmware - ROM images for qemu ii qemu-block-extra:amd64 1:2.5+dfsg-5ubuntu10.4 amd64 extra block backend modules for qemu-system and qemu-utils ii qemu-kvm 1:2.5+dfsg-5ubuntu10.4 amd64 QEMU Full virtualization ii qemu-system-common 1:2.5+dfsg-5ubuntu10.4 amd64 QEMU full system emulation binaries (common files) ii qemu-system-x86 1:2.5+dfsg-5ubuntu10.4 amd64 QEMU full system emulation binaries (x86) ii qemu-utils 1:2.5+dfsg-5ubuntu10.4 amd64 QEMU utilities To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1623199/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
