Public bug reported: Hello, I've bumped into issue with Nova: After upgrading Neutron to Ocata via openstack-ansible, I was unable to start any of the servers.
openstack server list |grep a6200dff-2010-44a2-92ca-3af303ec3577 | a6200dff-2010-44a2-92ca-3af303ec3577 | Win10VDI-3-ephemeral | SHUTOFF | internal=10.0.0.17| openstack server start a6200dff-2010-44a2-92ca-3af303ec3577 openstack server list |grep a6200dff-2010-44a2-92ca-3af303ec3577 | a6200dff-2010-44a2-92ca-3af303ec3577 | Win10VDI-3-ephemeral | SHUTOFF | internal=10.0.0.17| Server is still in SHUTOFF state. nova.log contains the flowing entries: http://paste.ubuntu.com/24465172/ After digging more into problem, I've found, that https://github.com/openstack/nova/blob/stable/ocata/nova/virt/libvirt/volume/net.py#L65 is defined as None. This made me read _set_auth_config_rbd description, so I've reconfigured my cinder nodes and added `rbd_secret_uuid` line (this line was not provided in my Newton cinder configuration, but everything seemed to work fine back then). After reconfiguration, I've managed to create and start new machines, but machines, which were created in Newton sill failed. After some more debugging it came to me, that old machines still had no 'secret_uuid' in their connection_info dictionary: {u'driver_volume_type': u'rbd', u'connector': {u'wwpns': [u'21000024ff54b106', u'21000024ff54b107'], u'wwnns': [u'20000024ff54b106', u'20000024ff54b107'], u'ip': u'172.31.105.203', u'initiator': u'iqn.1993-08.org.debian:01:b7f25b95aff0', u'platform': u'x86_64', u'host': u'ostack-ibm3', u'do_local_attach': False, u'os_type': u'linux2', u'multipath': False}, u'serial': u'87c5d950-c212 -4e9c-9d13-6a583fed9e62', u'data': {u'secret_type': u'ceph', u'name': u 'cinder-volumes/volume-87c5d950-c212-4e9c-9d13-6a583fed9e62', u'encrypted': False, u'cluster_name': u'ceph', u'secret_uuid': None, u'qos_specs': None, u'hosts': [u'172.31.104.1', u'172.31.104.2', u'172.31.104.3'], u'volume_id': u'87c5d950-c212-4e9c-9d13-6a583fed9e62', u'auth_enabled': True, u'access_mode': u'rw', u'auth_username': u'ostack', u'ports': [u'6789', u'6789', u'6789']}} It seems, that Ocata requires secret_uuid to be provided in database. Machines form Newton release did not contain that entry: Newton : | {"driver_volume_type": "rbd", "connector": {"wwpns": ["21000024ff54b106", "21000024ff54b107"], "wwnns": ["20000024ff54b106", "20000024ff54b107"], "ip": "172.31.105.203", "initiator": "iqn.1993-08.org.debian:01:b7f25b95aff0", "platform": "x86_64", "host": "ostack-ibm3", "do_local_attach": false, "os_type": "linux2", "multipath": false}, "serial": "87326e73-1e9a-4589-9140-985950e93068", "data": {"secret_type": "ceph", "name": "cinder-volumes/volume-87326e73-1e9a-4589-9140-985950e93068", "encrypted": false, "cluster_name": "ceph", "secret_uuid": null, "qos_specs": null, "hosts": ["172.31.104.1", "172.31.104.2", "172.31.104.3"], "volume_id": "87326e73-1e9a-4589-9140-985950e93068", "auth_enabled": true, "access_mode": "rw", "auth_username": "ostack", "ports": ["6789", "6789", "6789"]}} Ocata: | | {"driver_volume_type": "rbd", "connector": {"wwpns": ["21000024ff54b106", "21000024ff54b107"], "wwnns": ["20000024ff54b106", "20000024ff54b107"], "ip": "172.31.105.203", "initiator": "iqn.1993-08.org.debian:01:b7f25b95aff0", "platform": "x86_64", "host": "ostack-ibm3", "do_local_attach": false, "os_type": "linux2", "multipath": false}, "serial": "565ba4ab-2f8a-40fd-ae82-9346414ceb15", "data": {"secret_type": "ceph", "name": "cinder-volumes/volume-565ba4ab-2f8a-40fd-ae82-9346414ceb15", "encrypted": false, "cluster_name": "ceph", "secret_uuid": "a11833c5-9403-4423-8a26-111222333444", "qos_specs": null, "hosts": ["172.31.104.1", "172.31.104.2", "172.31.104.3"], "volume_id": "565ba4ab-2f8a-40fd-ae82-9346414ceb15", "auth_enabled": true, "access_mode": "rw", "auth_username": "ostack", "ports": ["6789", "6789", "6789"]}} So, is there any way to refresh database to fix this, or should I refresh block_device_mapping table by hand? I'm not sure if this is a bug, perhaps there is need to put some warning about this in upgrade documentation? Thank you. ** Affects: nova Importance: Undecided Status: New ** Summary changed: - Newton > Ocata upgrace. secret_uuid empty + Newton > Ocata upgrade. secret_uuid empty -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1687581 Title: Newton > Ocata upgrade. secret_uuid empty Status in OpenStack Compute (nova): New Bug description: Hello, I've bumped into issue with Nova: After upgrading Neutron to Ocata via openstack-ansible, I was unable to start any of the servers. openstack server list |grep a6200dff-2010-44a2-92ca-3af303ec3577 | a6200dff-2010-44a2-92ca-3af303ec3577 | Win10VDI-3-ephemeral | SHUTOFF | internal=10.0.0.17| openstack server start a6200dff-2010-44a2-92ca-3af303ec3577 openstack server list |grep a6200dff-2010-44a2-92ca-3af303ec3577 | a6200dff-2010-44a2-92ca-3af303ec3577 | Win10VDI-3-ephemeral | SHUTOFF | internal=10.0.0.17| Server is still in SHUTOFF state. nova.log contains the flowing entries: http://paste.ubuntu.com/24465172/ After digging more into problem, I've found, that https://github.com/openstack/nova/blob/stable/ocata/nova/virt/libvirt/volume/net.py#L65 is defined as None. This made me read _set_auth_config_rbd description, so I've reconfigured my cinder nodes and added `rbd_secret_uuid` line (this line was not provided in my Newton cinder configuration, but everything seemed to work fine back then). After reconfiguration, I've managed to create and start new machines, but machines, which were created in Newton sill failed. After some more debugging it came to me, that old machines still had no 'secret_uuid' in their connection_info dictionary: {u'driver_volume_type': u'rbd', u'connector': {u'wwpns': [u'21000024ff54b106', u'21000024ff54b107'], u'wwnns': [u'20000024ff54b106', u'20000024ff54b107'], u'ip': u'172.31.105.203', u'initiator': u'iqn.1993-08.org.debian:01:b7f25b95aff0', u'platform': u'x86_64', u'host': u'ostack-ibm3', u'do_local_attach': False, u'os_type': u'linux2', u'multipath': False}, u'serial': u'87c5d950-c212-4e9c-9d13-6a583fed9e62', u'data': {u'secret_type': u'ceph', u'name': u'cinder-volumes/volume-87c5d950-c212-4e9c- 9d13-6a583fed9e62', u'encrypted': False, u'cluster_name': u'ceph', u'secret_uuid': None, u'qos_specs': None, u'hosts': [u'172.31.104.1', u'172.31.104.2', u'172.31.104.3'], u'volume_id': u'87c5d950-c212-4e9c- 9d13-6a583fed9e62', u'auth_enabled': True, u'access_mode': u'rw', u'auth_username': u'ostack', u'ports': [u'6789', u'6789', u'6789']}} It seems, that Ocata requires secret_uuid to be provided in database. Machines form Newton release did not contain that entry: Newton : | {"driver_volume_type": "rbd", "connector": {"wwpns": ["21000024ff54b106", "21000024ff54b107"], "wwnns": ["20000024ff54b106", "20000024ff54b107"], "ip": "172.31.105.203", "initiator": "iqn.1993-08.org.debian:01:b7f25b95aff0", "platform": "x86_64", "host": "ostack-ibm3", "do_local_attach": false, "os_type": "linux2", "multipath": false}, "serial": "87326e73-1e9a-4589-9140-985950e93068", "data": {"secret_type": "ceph", "name": "cinder-volumes/volume-87326e73-1e9a-4589-9140-985950e93068", "encrypted": false, "cluster_name": "ceph", "secret_uuid": null, "qos_specs": null, "hosts": ["172.31.104.1", "172.31.104.2", "172.31.104.3"], "volume_id": "87326e73-1e9a-4589-9140-985950e93068", "auth_enabled": true, "access_mode": "rw", "auth_username": "ostack", "ports": ["6789", "6789", "6789"]}} Ocata: | | {"driver_volume_type": "rbd", "connector": {"wwpns": ["21000024ff54b106", "21000024ff54b107"], "wwnns": ["20000024ff54b106", "20000024ff54b107"], "ip": "172.31.105.203", "initiator": "iqn.1993-08.org.debian:01:b7f25b95aff0", "platform": "x86_64", "host": "ostack-ibm3", "do_local_attach": false, "os_type": "linux2", "multipath": false}, "serial": "565ba4ab-2f8a-40fd-ae82-9346414ceb15", "data": {"secret_type": "ceph", "name": "cinder-volumes/volume-565ba4ab-2f8a-40fd-ae82-9346414ceb15", "encrypted": false, "cluster_name": "ceph", "secret_uuid": "a11833c5-9403-4423-8a26-111222333444", "qos_specs": null, "hosts": ["172.31.104.1", "172.31.104.2", "172.31.104.3"], "volume_id": "565ba4ab-2f8a-40fd-ae82-9346414ceb15", "auth_enabled": true, "access_mode": "rw", "auth_username": "ostack", "ports": ["6789", "6789", "6789"]}} So, is there any way to refresh database to fix this, or should I refresh block_device_mapping table by hand? I'm not sure if this is a bug, perhaps there is need to put some warning about this in upgrade documentation? Thank you. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1687581/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
