Hi OpenStack Community, I am having some issues with key management in a multinode devstack (from master branch 27th July '18) environment where Barbican is the configured key_manager. I have followed setup instructions from the following pages:
* https://docs.openstack.org/barbican/latest/contributor/devstack.html (manual configuration) * https://docs.openstack.org/cinder/latest/configuration/block-storage/volume-encryption.html So far: * Unencrypted block volumes can be attached to instances on any compute node * Instances with unencrypted volumes can also be live migrated to other compute node * Encrypted bootable volumes created successfully * Instances can be launched using these encrypted volumes when the instance is spawned on demo_machine1 (controller & compute node) * Instances cannot be launched using encrypted volumes when the instance is spawned on demo_machine2 or demo_machine3 (compute only), the same failure can be seen in nova logs from both compute nodes: Jul 30 14:35:18 demo_machine2 nova-compute[25686]: DEBUG cinderclient.v3.client [None req-3c977faa-a64c-4536-82c8-d1dbaf856b99 admin admin] GET call to cinderv3 for http://10.0.0.63/volume/v3/3f22a0262a7b4832a08c24ac0295cbd9/volumes/296148bf-edb8-4c9f-88c2-44464907f7e7/encryption used request id req-71fa7f20-c0bc-46c3-9f07-5866344d31a1 {{(pid=25686) request /usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py:844}} Jul 30 14:35:18 demo_machine2 nova-compute[25686]: DEBUG os_brick.encryptors [None req-3c977faa-a64c-4536-82c8-d1dbaf856b99 admin admin] Using volume encryption metadata '{u'cipher': u'aes-xts-plain64', u'encryption_key_id': u'da7ee21c-67ff-4d74-95a0-18ee6c25d85a', u'provider': u'luks', u'key_size': 256, u'control_location': u'front-end'}' for connection: {'status': u'attaching', 'detached_at': u'', u'volume_id': u'296148bf-edb8-4c9f-88c2-44464907f7e7', 'attach_mode': u'null', 'driver_volume_type': u'iscsi', 'instance': u'e0dc6eac-09bb-4232-bea7-7b8b161cfa31', 'attached_at': u'2018-07-30T13:35:17.000000', 'serial': u'296148bf-edb8-4c9f-88c2-44464907f7e7', 'data': {'device_path': '/dev/disk/by-id/scsi-SEMC_SYMMETRIX_900049_wy000', u'target_discovered': True, u'encrypted': True, u'qos_specs': None, u'target_iqn': u'iqn.1992-04.com.emc:600009700bcbb7112504018f00000000', u'target_portal': u'192.168.0.60:3260', u'volume_id': u'296148bf-edb8-4c9f-88c2-44464907f7e7', u'target_lun': 1, u'access_mode': u'rw'}} {{(pid=25686) get_encryption_metadata /usr/local/lib/python2.7/dist-packages/os_brick/encryptors/__init__.py:125}} Jul 30 14:35:18 demo_machine2 nova-compute[25686]: WARNING keystoneauth.identity.generic.base [None req-3c977faa-a64c-4536-82c8-d1dbaf856b99 admin admin] Failed to discover available identity versions when contacting http://localhost/identity/v3. Attempting to parse version from URL.: NotFound: Not Found (HTTP 404) Jul 30 14:35:18 demo_machine2 nova-compute[25686]: ERROR castellan.key_manager.barbican_key_manager [None req-3c977faa-a64c-4536-82c8-d1dbaf856b99 admin admin] Error creating Barbican client: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Not Found (HTTP 404): DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Not Found (HTTP 404) All instance of Nova have [key_manager] configured as follows: [key_manager] backend = barbican auth_url = http://10.0.0.63/identity/ ### Tried with and without the below config options, same result # auth_type = password # password = devstack # username = barbican Any assistance here would be greatly appreciated, I have spent a lot of time looking for some additional information for the use of Barbican in multinode devstack environments or with live migration but there is nothing out there, everything is for all-in-one environments and I'm not having any issues when everything is on one node. I am wondering if at this point there is something I am missing in terms of services in a multinode devstack environment, qualification of barbican in a multinode environment is outside of the recommended test config but following the docs it looks very straight forward. Some information on the three nodes in my environment are below, if there is any other information I can provide let me know, thanks for the help! Node & Service Breakdown Node 1 (Controller & Compute) stack@demo_machine1:~$ openstack service list +----------------------------------+-------------+----------------+ | ID | Name | Type | +----------------------------------+-------------+----------------+ | 43a1334c755c4c81969565097cc9c30c | cinder | volume | | 52a8927c09154e33900f24c7c95a9f8b | cinderv2 | volumev2 | | 5427a9dff3b6477197062e1747843c4d | nova_legacy | compute_legacy | | 5b319b6d50634661998fdd8dc70a85e3 | nova | compute | | 5ffbb2e9f7c84c9e9601ab7aba0cf5e1 | placement | placement | | 787fd29afe2f41b0bb44f9c301fd22c5 | cinderv3 | volumev3 | | 96813e167b8842aba9d8b94fad67904f | neutron | network | | 993e615a03cc49e3be94840c0b82636b | swift | object-store | | b3834468ffc44f30b792459611f5f4e9 | cinder | block-storage | | cab9ff9e175f4566a1865ea35a377d0d | barbican | key-manager | | d12f710b815442fb970c22087b6e8f4f | glance | image | | eb80de21e42b4e978985db979b175f79 | keystone | identity | +----------------------------------+-------------+----------------+ stack@demo_machine1:~$ openstack endpoint list +----------------------------------+-----------+--------------+----------------+---------+-----------+-------------------------------------------------+ | ID | Region | Service Name | Service Type | Enabled | Interface | URL | +----------------------------------+-----------+--------------+----------------+---------+-----------+-------------------------------------------------+ | 00b276609956454d8d80dd0dde0df231 | RegionOne | cinder | volume | True | public | http://10.0.0.63/volume/v1/$(project_id)s | | 18e5d431143d47ed980ee0ffbf0d03d7 | RegionOne | barbican | key-manager | True | public | http://10.0.0.63/key-manager | | 20cfe0a80cc94b6eb8ea8e6784839198 | RegionOne | barbican | key-manager | True | internal | http://10.0.0.63/key-manager | | 3a740b472e7349f19d0cf110c1792122 | RegionOne | cinderv3 | volumev3 | True | public | http://10.0.0.63/volume/v3/$(project_id)s | | 4d957921fe894abba296331869f82f7f | RegionOne | cinderv2 | volumev2 | True | public | http://10.0.0.63/volume/v2/$(project_id)s | | 4df258794fde476ab82502c682848e58 | RegionOne | swift | object-store | True | admin | http://10.0.0.63:8080 | | 719eabec7cb94580af9f928278589878 | RegionOne | keystone | identity | True | public | http://10.0.0.63/identity | | 792f4c99085f4b008643b08aff463759 | RegionOne | keystone | identity | True | admin | http://10.0.0.63/identity | | 9e8c27c6e22f4a70865bfcdd815ed3c0 | RegionOne | cinder | block-storage | True | public | http://10.0.0.63/volume/v3/$(project_id)s | | a271f19f29d443a0b5545626584389d7 | RegionOne | glance | image | True | public | http://10.0.0.63/image | | a975403a2ff149bb88ce2d2227d17a80 | RegionOne | nova | compute | True | public | http://10.0.0.63/compute/v2.1 | | b65b46e83b4547588eb694d63cb5cdd5 | RegionOne | swift | object-store | True | public | http://10.0.0.63:8080/v1/AUTH_$(project_id)s | | bfd1f91ba18b4bc0bc83586ee358a73c | RegionOne | placement | placement | True | public | http://10.0.0.63/placement | | d38a11dcfe824fe28f70b45422277d26 | RegionOne | nova_legacy | compute_legacy | True | public | http://10.0.0.63/compute/v2/$(project_id)s | | ea9139e670e84ff39d1c052347a04695 | RegionOne | neutron | network | True | public | http://10.0.0.63:9696/ | +----------------------------------+-----------+--------------+----------------+---------+-----------+-------------------------------------------------+ stack@demo_machine1:~$ openstack secret store +---------------+---------------------------------------------------------------------------------+ | Field | Value | +---------------+---------------------------------------------------------------------------------+ | Secret href | http://10.0.0.63/key-manager/v1/secrets/72a3955b-a494-4352-b1f6-ae3f322e5656 | | Name | None | | Created | 2018-07-30T12:58:33+00:00 | | Status | ACTIVE | | Content types | None | | Algorithm | aes | | Bit length | 256 | | Secret type | opaque | | Mode | cbc | | Expiration | None | +---------------+---------------------------------------------------------------------------------+ Node 2 & 3 (Compute Only) Services: stack@demo_machine2:~$ sudo systemctl list-unit-files | grep devstack@* devstack@n-api-meta.service<mailto:devstack@n-api-meta.service> enabled devstack@n-cpu.service<mailto:devstack@n-cpu.service> enabled devstack@q-agt.service<mailto:devstack@q-agt.service> enabled stack@demo_machine3:~$ sudo systemctl list-unit-files | grep devstack@* devstack@n-api-meta.service<mailto:devstack@n-api-meta.service> enabled devstack@n-cpu.service<mailto:devstack@n-cpu.service> enabled devstack@q-agt.service<mailto:devstack@q-agt.service> enabled ******************************************************************** Michael McAleer Software Engineer 1, Core Technologies Dell EMC | Enterprise Storage Division Phone: +353 21 428 1729 michael.mcal...@dell.com<mailto:michael.mcal...@dell.com> Ireland COE, Ovens, Co. Cork, Ireland
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
