It seems you have a proper [neutron] and [placement] section in your nova.conf but you don't have a [cinder] section. You need to configure the similar [cinder] section to allow nova to talk to cinder. I think this is the root case of the error you see:
ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'cinderclient.exceptions.Unauthorized'> (HTTP 500) I'm setting this bug to Invalid now. Please fix you config and try again. If you still see the same issue then set the bug status back to New and add your new config to the report. ** Changed in: nova Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1878481 Title: server add volume fails Status in OpenStack Compute (nova): Invalid Bug description: Description =========== <class 'cinderclient.exceptions.Unauthorized'> (HTTP 500) It seems to start by not finding the instance in nova. Environment =========== Environment: (Rocky) openstack-nova-common-18.3.0-1.el7.noarch openstack-nova-conductor-18.3.0-1.el7.noarch openstack-nova-api-18.3.0-1.el7.noarch openstack-nova-scheduler-18.3.0-1.el7.noarch openstack-nova-console-18.3.0-1.el7.noarch openstack-nova-placement-api-18.3.0-1.el7.noarch python2-novaclient-11.0.1-1.el7.noarch openstack-nova-novncproxy-18.3.0-1.el7.noarch python-nova-18.3.0-1.el7.noarch Hypervisor: Linux KVM Storate: CEPH: 14.2.8 / RBD volumes Network: Neutron (OpenVSwitch) Steps to reproduce ================== I did: openstack --debug server add volume c331814d-b758-460e-9972-bc1e987b933d 0fc5bec7-9364-458c-a809-f38389890a60 I also did: openstack --debug server add volume allalal peter-test-001 I received: ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'cinderclient.exceptions.Unauthorized'> (HTTP 500) (Request-ID: req-fb493f88-61c5-4ebd-829b-305b31aaa010) Expected result =============== Associate the volume and exit cleanly. Actual result ============= Nova fails to find existing instance and fails the attempt to add the volume. All other server commands for the instance work as expected (stop, start, migrate, etc.) Logs & Configs ============== /etc/nova/nova.conf [DEFAULT] cpu_allocation_ratio = 16.0 debug = true enabled_apis = osapi_compute,metadata metadata_proxy_shared_secret = <secret> metadata_workers = 4 my_ip = 10.176.0.42 ram_allocation_ratio = 1.0 scheduler_host_subset_size = 2 transport_url = rabbit://openstack:<passwd>@mq-a01.xxx.cloud,openstack:<passwd>@mq-a02.xxx.cloud,openstack:<passwd>@mq-a03.xxx.cloud cinder_catalog_info=volumev3:cinderv3:publicURL [api] auth_strategy = keystone [api_database] connection = mysql+pymysql://atx_nova_api:<passwd>@db-a00.xxx.cloud/atx_nova_api [filter_scheduler] enabled_filters = AvailabilityZoneFilter,ComputeCapabilitiesFilter,ComputeFilter,ImagePropertiesFilter,RamFilter,RetryFilter,ServerGroupAffinityFilter,ServerGroupAntiAffinityFilter host_subset_size = 4 io_ops_weight_multiplier = 10.0 max_instances_per_host = 100 ram_weight_multiplier = 6.0 soft_affinity_weight_multiplier = 10.0 soft_anti_affinity_weight_multiplier = 10.0 weight_classes = nova.scheduler.weights.all_weighers [database] connection = mysql+pymysql://atx_nova:<passwd>@db-a00.xxx.cloud/atx_nova [glance] api_servers = https://glance-a00.xxx.cloud [keystone_authtoken] auth_type = password www_authenticate_uri = https://keystone-a00.xxx.cloud auth_url = https://keystone-a00.xxx.cloud memcached_servers = memcached-a01.xxx.cloud:11211,memcached-a02.xxx.cloud:11211,memcached-a03.xxx.cloud:11211 password = <passwd> project_domain_name = default project_name = service user_domain_name = default username = atx_nova service_token_roles_required = true [neutron] auth_type = password auth_url = https://keystone-a00.xxx.cloud metadata_proxy_shared_secret = <secret> password = <passwd> project_domain_name = Default project_name = service region_name = atx service_metadata_proxy = true #uses keystoneauth1: url = https://neutron-a00.xxx.cloud user_domain_name = Default username = atx_neutron [oslo_concurrency] lock_path = /var/lib/nova/tmp [placement] auth_type = password auth_url = https://keystone-a00.xxx.cloud os_region_name = atx password = <passwd> project_domain_name = Default project_name = service user_domain_name = Default username = atx_placement [scheduler] discover_hosts_in_cells_interval = 300 [oslo_notifications_group] driver = messaging topics = notifications [cache] backend=oslo_cache.memcache_pool enable = true [oslo_messaging_rabbit] amqp_durable_queues = true rabbit_ha_queues = true rabbit_retry_backoff = 2 rabbit_retry_interval = 1 [peisch@jump ~]$ openstack volume list +--------------------------------------+----------------+-----------+------+-------------+ | ID | Name | Status | Size | Attached to | +--------------------------------------+----------------+-----------+------+-------------+ | ... | ... | available | 10 | | | 0fc5bec7-9364-458c-a809-f38389890a60 | peter-test-001 | available | 10 | | | ... | ... | available | 10 | | +--------------------------------------+----------------+-----------+------+-------------+ [peisch@hopslam ~]$ openstack server list +--------------------------------------+-----------+--------+---------------------+----------+--------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-----------+--------+---------------------+----------+--------+ | ... | ... | ACTIVE | admin=10.1.2.5 | Centos77 | 2x2x20 | | c331814d-b758-460e-9972-bc1e987b933d | allalal | ACTIVE | admin=10.1.2.7 | Centos77 | 2x2x20 | +--------------------------------------+-----------+--------+---------------------+----------+--------+ [peisch@jump ~]$ Openstack Debug Log: (starting after keystone auth) Starting new HTTPS connection (1): nova-a00.xxx.cloud https://nova-a00.xxx.cloud:443 "GET /v2.1/servers/c331814d-b758-460e-9972-bc1e987b933d HTTP/1.1" 200 1646 RESP: [200] Content-Length: 1646 Content-Type: application/json Date: Wed, 13 May 2020 19:39:35 GMT Openstack-Api-Version: compute 2.1 Vary: OpenStack-API-Version, X-OpenStack-Nova-API-Version X-Compute-Request-Id: req-a98e058c-27df-471a-9553-8f9f52dab060 X-Openstack-Nova-Api-Version: 2.1 X-Openstack-Request-Id: req-a98e058c-27df-471a-9553-8f9f52dab060 RESP BODY: {"server": {"OS-EXT-STS:task_state": null, "addresses": {"admin": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:a3:6a:5c", "version": 4, "addr": "10.176.128.7", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "http://nova-a00.xxx.cloud/v2.1/servers/c331814d-b758-460e-9972-bc1e987b933d";, "rel": "self"}, {"href": "http://nova-a00.xxx.cloud/servers/c331814d-b758-460e-9972-bc1e987b933d";, "rel": "bookmark"}], "image": {"id": "41bb501b-907e-4e0f-a9fe-072a5d3fbe58", "links": [{"href": "http://nova-a00.xxx.cloud/images/41bb501b-907e-4e0f-a9fe-072a5d3fbe58";, "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "active", "OS-EXT-SRV-ATTR:instance_name": "instance-00000601", "OS-SRV-USG:launched_at": "2020-04-27T20:44:42.000000", "flavor": {"id": "d15ae531-2911-4c5a-aeb5-d62081def56d", "links": [{"href": "http://nova-a00.xxx.cloud/flavors/d15ae531-2911-4c5a-aeb5-d62081def56d";, "rel": "bookmark"}]}, "id": "c331814d-b758-460e-9972-bc1e987b933d", "security_groups": [{"name": "default"}], "user_id": "497a19c5da034431b6c01d5d8f523968", "OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "status": "ACTIVE", "updated": "2020-04-27T20:45:57Z", "hostId": "d98666c745c89cc71a5fa952c75899cc9bd67808a24a117dc5054f5b", "OS-EXT-SRV-ATTR:host": "compute-a19.xxx.cloud", "OS-SRV-USG:terminated_at": null, "key_name": null, "OS-EXT-SRV-ATTR:hypervisor_hostname": "compute-a19.xxx.cloud", "name": "allalal", "created": "2020-04-24T14:51:33Z", "tenant_id": "1134ba6f01ef43f49961aa6bbe49f5bc", "os-extended-volumes:volumes_attached": [], "metadata": {}}} GET call to compute for https://nova-a00.xxx.cloud/v2.1/servers/c331814d-b758-460e-9972-bc1e987b933d used request id req-a98e058c-27df-471a-9553-8f9f52dab060 REQ: curl -g -i -X GET https://cinder-a00.xxx.cloud/v3/1134ba6f01ef43f49961aa6bbe49f5bc/volumes/0fc5bec7-9364-458c-a809-f38389890a60 -H "Accept: application/json" -H "User-Agent: python-cinderclient" -H "X-Auth-Token: {SHA1}99c93352a2ccbc3dd1f8df8997f0469a95f9f367" Starting new HTTPS connection (1): cinder-a00.xxx.cloud https://cinder-a00.xxx.cloud:443 "GET /v3/1134ba6f01ef43f49961aa6bbe49f5bc/volumes/0fc5bec7-9364-458c-a809-f38389890a60 HTTP/1.1" 200 1055 RESP: [200] Content-Length: 1055 Content-Type: application/json Date: Wed, 13 May 2020 19:39:35 GMT Openstack-Api-Version: volume 3.0 Vary: OpenStack-API-Version X-Compute-Request-Id: req-c192e90e-711d-498e-806d-1eeded367306 X-Openstack-Request-Id: req-c192e90e-711d-498e-806d-1eeded367306 RESP BODY: {"volume": {"migration_status": null, "attachments": [], "links": [{"href": "https://cinder-a00.xxx.cloud/v3/1134ba6f01ef43f49961aa6bbe49f5bc/volumes/0fc5bec7-9364-458c-a809-f38389890a60";, "rel": "self"}, {"href": "https://cinder-a00.xxx.cloud/1134ba6f01ef43f49961aa6bbe49f5bc/volumes/0fc5bec7-9364-458c-a809-f38389890a60";, "rel": "bookmark"}], "availability_zone": "nova", "os-vol-host-attr:host": "cinder-a03.xxx.cloud@sizeone#sizeone", "encrypted": false, "updated_at": "2020-04-24T14:52:25.000000", "replication_status": null, "snapshot_id": null, "id": "0fc5bec7-9364-458c-a809-f38389890a60", "size": 10, "user_id": "497a19c5da034431b6c01d5d8f523968", "os-vol-tenant-attr:tenant_id": "1134ba6f01ef43f49961aa6bbe49f5bc", "os-vol-mig-status-attr:migstat": null, "metadata": {}, "status": "available", "description": null, "multiattach": false, "source_volid": null, "consistencygroup_id": null, "os-vol-mig-status-attr:name_id": null, "name": "peter-test-001", "bootable": "false", "created_at": "2020-04-24T14:52:24.000000", "volume_type": "sizeone"}} GET call to volumev3 for https://cinder-a00.xxx.cloud/v3/1134ba6f01ef43f49961aa6bbe49f5bc/volumes/0fc5bec7-9364-458c-a809-f38389890a60 used request id req-c192e90e-711d-498e-806d-1eeded367306 REQ: curl -g -i -X POST https://nova-a00.xxx.cloud/v2.1/servers/c331814d-b758-460e-9972-bc1e987b933d/os-volume_attachments -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA1}99c93352a2ccbc3dd1f8df8997f0469a95f9f367" -d '{"volumeAttachment": {"volumeId": "0fc5bec7-9364-458c-a809-f38389890a60"}}' https://nova-a00.xxx.cloud:443 "POST /v2.1/servers/c331814d-b758-460e-9972-bc1e987b933d/os-volume_attachments HTTP/1.1" 500 210 RESP: [500] Content-Length: 210 Content-Type: application/json; charset=UTF-8 Date: Wed, 13 May 2020 19:39:36 GMT Openstack-Api-Version: compute 2.1 Vary: OpenStack-API-Version, X-OpenStack-Nova-API-Version X-Compute-Request-Id: req-fb493f88-61c5-4ebd-829b-305b31aaa010 X-Openstack-Nova-Api-Version: 2.1 X-Openstack-Request-Id: req-fb493f88-61c5-4ebd-829b-305b31aaa010 RESP BODY: {"computeFault": {"message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<class 'cinderclient.exceptions.Unauthorized'>", "code": 500}} POST call to compute for https://nova-a00.xxx.cloud/v2.1/servers/c331814d-b758-460e-9972-bc1e987b933d/os-volume_attachments used request id req-fb493f88-61c5-4ebd-829b-305b31aaa010 Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'cinderclient.exceptions.Unauthorized'> (HTTP 500) (Request-ID: req-fb493f88-61c5-4ebd-829b-305b31aaa010) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cliff/app.py", line 402, in run_subcommand result = cmd.run(parsed_args) File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run return super(Command, self).run(parsed_args) File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run return_code = self.take_action(parsed_args) or 0 File "/usr/lib/python2.7/site-packages/openstackclient/compute/v2/server.py", line 464, in take_action parsed_args.device, File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 393, in substitution return methods[-1].func(obj, *args, **kwargs) File "/usr/lib/python2.7/site-packages/novaclient/v2/volumes.py", line 56, in create_server_volume body, "volumeAttachment") File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 361, in _create resp, body = self.api.client.post(url, body=body) File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 334, in post return self.request(url, 'POST', **kwargs) File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 83, in request raise exceptions.from_response(resp, body, url, method) ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'cinderclient.exceptions.Unauthorized'> (HTTP 500) (Request-ID: req-fb493f88-61c5-4ebd-829b-305b31aaa010) clean_up AddServerVolume: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'cinderclient.exceptions.Unauthorized'> (HTTP 500) (Request-ID: req-fb493f88-61c5-4ebd-829b-305b31aaa010) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 135, in run ret_val = super(OpenStackShell, self).run(argv) File "/usr/lib/python2.7/site-packages/cliff/app.py", line 281, in run result = self.run_subcommand(remainder) File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 175, in run_subcommand ret_value = super(OpenStackShell, self).run_subcommand(argv) File "/usr/lib/python2.7/site-packages/cliff/app.py", line 402, in run_subcommand result = cmd.run(parsed_args) File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run return super(Command, self).run(parsed_args) File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run return_code = self.take_action(parsed_args) or 0 File "/usr/lib/python2.7/site-packages/openstackclient/compute/v2/server.py", line 464, in take_action parsed_args.device, File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 393, in substitution return methods[-1].func(obj, *args, **kwargs) File "/usr/lib/python2.7/site-packages/novaclient/v2/volumes.py", line 56, in create_server_volume body, "volumeAttachment") File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 361, in _create resp, body = self.api.client.post(url, body=body) File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 334, in post return self.request(url, 'POST', **kwargs) File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 83, in request raise exceptions.from_response(resp, body, url, method) ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'cinderclient.exceptions.Unauthorized'> (HTTP 500) (Request-ID: req-fb493f88-61c5-4ebd-829b-305b31aaa010) END return value: 1 ==> /var/log/nova/nova-api.log <== 2020-05-13 13:59:06.159 14830 DEBUG nova.api.openstack.wsgi [req-1299524b-1fb1-46b1-82de-9d3ced5c694f c351f9a7674a4a0cbe63ca633f66f606 1134ba6f01ef43f49961aa6bbe49f5bc - default default] Action: 'create', calling method: <bound method VolumeAttachmentController.create of <nova.api.openstack.compute.volumes.VolumeAttachmentController object at 0x7f5113d97050>>, body: {"volumeAttachment": {"volumeId": "0fc5bec7-9364-458c-a809-f38389890a60"}} _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:615 2020-05-13 13:59:06.163 14830 DEBUG nova.compute.api [req-1299524b-1fb1-46b1-82de-9d3ced5c694f c351f9a7674a4a0cbe63ca633f66f606 1134ba6f01ef43f49961aa6bbe49f5bc - default default] [instance: c331814d-b758-460e-9972-bc1e987b933d] Fetching instance by UUID get /usr/lib/python2.7/site-packages/nova/compute/api.py:2421 2020-05-13 13:59:06.171 14830 DEBUG oslo_concurrency.lockutils [req-1299524b-1fb1-46b1-82de-9d3ced5c694f c351f9a7674a4a0cbe63ca633f66f606 1134ba6f01ef43f49961aa6bbe49f5bc - default default] Lock "3bc099c6-6bc5-4d39-9259-3c2fca30c75e" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:273 2020-05-13 13:59:06.171 14830 DEBUG oslo_concurrency.lockutils [req-1299524b-1fb1-46b1-82de-9d3ced5c694f c351f9a7674a4a0cbe63ca633f66f606 1134ba6f01ef43f49961aa6bbe49f5bc - default default] Lock "3bc099c6-6bc5-4d39-9259-3c2fca30c75e" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.001s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285 2020-05-13 13:59:07.223 14830 INFO nova.api.openstack.wsgi [req-1299524b-1fb1-46b1-82de-9d3ced5c694f c351f9a7674a4a0cbe63ca633f66f606 1134ba6f01ef43f49961aa6bbe49f5bc - default default] HTTP exception thrown: Volume 0fc5bec7-9364-458c-a809-f38389890a60 could not be found. 2020-05-13 13:59:07.223 14830 DEBUG nova.api.openstack.wsgi [req-1299524b-1fb1-46b1-82de-9d3ced5c694f c351f9a7674a4a0cbe63ca633f66f606 1134ba6f01ef43f49961aa6bbe49f5bc - default default] Returning 404 to user: Volume 0fc5bec7-9364-458c-a809-f38389890a60 could not be found. __call__ /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:1077 2020-05-13 13:59:07.225 14830 INFO nova.osapi_compute.wsgi.server [req-1299524b-1fb1-46b1-82de-9d3ced5c694f c351f9a7674a4a0cbe63ca633f66f606 1134ba6f01ef43f49961aa6bbe49f5bc - default default] 172.22.0.66,10.176.0.102 "POST /v2.1/servers/c331814d-b758-460e-9972-bc1e987b933d/os-volume_attachments HTTP/1.1" status: 404 len: 506 time: 1.0708430 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1878481/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
