Public bug reported:
This was a brand new healthy VM. On Reboot, I noticed it and all other
VM's would not come up and as i run kolla ansible in containers, i
assumed it to be a race condition corrupting it. I created all new VM's
(that is fine) they started up and even shut down a few times ok, i
tried via SDK and also CLI and then then wont start again.... (no
shutdown involved) so this must be a nova or iscsi issue. Thoughts?
==> /var/log/kolla/nova/nova-compute.log <==
2024-04-19 00:12:23.073 8 INFO nova.virt.libvirt.driver [-] [instance:
712454ec-3ab3-4648-97c5-ede2edce5c9c] Instance destroyed successfully.
2024-04-19 00:12:23.087 8 INFO os_vif [None
req-038ecf7f-d511-4833-948c-0236b81820bf 34df1169571b4955a0ff9527fda70c98
e95c8c84b62e416daf03b91a9648962b - - default default] Successfully unplugged
vif
VIFBridge(active=False,address=fa:16:3e:cc:30:e0,bridge_name='qbrc93e0089-32',has_traffic_filtering=True,id=c93e0089-3236-4271-bd61-90f2cd47afdd,network=Network(53acdf7a-617f-4ab5-a62e-0cb96936d619),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapc93e0089-32')
2024-04-19 00:12:23.090 8 WARNING os_brick.initiator.connectors.base [None
req-038ecf7f-d511-4833-948c-0236b81820bf 34df1169571b4955a0ff9527fda70c98
e95c8c84b62e416daf03b91a9648962b - - default default] Service needs to call
os_brick.setup() before connecting volumes, if it doesn't it will break on the
next release
2024-04-19 00:12:23.140 8 WARNING os_brick.exception [None
req-038ecf7f-d511-4833-948c-0236b81820bf 34df1169571b4955a0ff9527fda70c98
e95c8c84b62e416daf03b91a9648962b - - default default] Disconnect from
192.168.5.1:3260
iqn.2010-10.org.openstack:volume-3324028d-086a-473f-9bc1-4dcac228e5e6 failed:
oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while
running command.
2024-04-19 00:12:23.140 8 WARNING os_brick.initiator.connectors.iscsi [None
req-038ecf7f-d511-4833-948c-0236b81820bf 34df1169571b4955a0ff9527fda70c98
e95c8c84b62e416daf03b91a9648962b - - default default] There were errors
removing set(), leftovers may remain in the system
==> /var/log/kolla/nova/nova-compute.log <==
2024-04-19 00:12:23.352 8 INFO nova.compute.manager [None
req-038ecf7f-d511-4833-948c-0236b81820bf 34df1169571b4955a0ff9527fda70c98
e95c8c84b62e416daf03b91a9648962b - - default default] [instance:
712454ec-3ab3-4648-97c5-ede2edce5c9c] Successfully reverted task state from
powering-on on failure for instance.
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server [None
req-038ecf7f-d511-4833-948c-0236b81820bf 34df1169571b4955a0ff9527fda70c98
e95c8c84b62e416daf03b91a9648962b - - default default] Exception during message
handling: os_brick.exception.ExceptionChainer:
Chained Exception #1
Traceback (most recent call last):
File
"/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py",
line 1177, in _disconnect_connection
self._disconnect_from_iscsi_portal(props)
File
"/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py",
line 1159, in _disconnect_from_iscsi_portal
self._run_iscsiadm(connection_properties, ("--logout",),
File
"/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py",
line 1025, in _run_iscsiadm
(out, err) = self._execute('iscsiadm', '-m', 'node', '-T',
File
"/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/executor.py", line
55, in _execute
result = self.__execute(*args, **kwargs)
File
"/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/privileged/rootwrap.py",
line 172, in execute
return execute_root(*cmd, **kwargs)
File
"/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_privsep/priv_context.py",
line 271, in _wrap
return self.channel.remote_call(name, args, kwargs,
File
"/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_privsep/daemon.py", line
215, in remote_call
raise exc_type(*result[2])
oslo_concurrency.processutils.ProcessExecutionError: Unexpected error
while running command.
Command: iscsiadm -m node -T
iqn.2010-10.org.openstack:volume-3324028d-086a-473f-9bc1-4dcac228e5e6 -p
192.168.5.1:3260 --logout
Exit code: 32
Stdout: 'Logging out of session [sid: 1, target:
iqn.2010-10.org.openstack:volume-3324028d-086a-473f-9bc1-4dcac228e5e6, portal:
192.168.5.1,3260]\n'
Stderr: 'iscsiadm: Could not logout of [sid: 1, target:
iqn.2010-10.org.openstack:volume-3324028d-086a-473f-9bc1-4dcac228e5e6, portal:
192.168.5.1,3260].\niscsiadm: initiator reported error (32 - target likely not
connected)\niscsiadm: Could not logout of all requested sessions\n'
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server Traceback (most
recent call last):
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/server.py",
line 165, in _process_incoming
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server res =
self.dispatcher.dispatch(message)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py",
line 309, in dispatch
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server return
self._do_dispatch(endpoint, method, ctxt, args)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py",
line 229, in _do_dispatch
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server result =
func(ctxt, **new_args)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/nova/exception_wrapper.py",
line 65, in wrapped
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server with
excutils.save_and_reraise_exception():
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line
227, in __exit__
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
self.force_reraise()
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line
200, in force_reraise
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server raise self.value
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/nova/exception_wrapper.py",
line 63, in wrapped
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server return f(self,
context, *args, **kw)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py",
line 166, in decorated_function
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server with
excutils.save_and_reraise_exception():
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line
227, in __exit__
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
self.force_reraise()
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line
200, in force_reraise
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server raise self.value
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py",
line 157, in decorated_function
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/utils.py", line
1439, in decorated_function
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py",
line 213, in decorated_function
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server with
excutils.save_and_reraise_exception():
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line
227, in __exit__
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
self.force_reraise()
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line
200, in force_reraise
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server raise self.value
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py",
line 203, in decorated_function
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py",
line 3424, in start_instance
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
self._power_on(context, instance)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py",
line 3392, in _power_on
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
self.driver.power_on(context, instance,
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py",
line 4136, in power_on
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
self._hard_reboot(context, instance, network_info, block_device_info,
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py",
line 3955, in _hard_reboot
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
self.destroy(context, instance, network_info, destroy_disks=False,
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py",
line 1565, in destroy
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
self.cleanup(context, instance, network_info, block_device_info,
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py",
line 1635, in cleanup
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server return
self._cleanup(
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py",
line 1690, in _cleanup
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server with
excutils.save_and_reraise_exception() as ctxt:
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line
227, in __exit__
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
self.force_reraise()
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line
200, in force_reraise
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server raise self.value
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py",
line 1686, in _cleanup
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
self._disconnect_volume(
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py",
line 2017, in _disconnect_volume
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
vol_driver.disconnect_volume(
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/volume/iscsi.py",
line 74, in disconnect_volume
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
self.connector.disconnect_volume(
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/utils.py", line 173,
in trace_logging_wrapper
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server return f(*args,
**kwargs)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/base.py",
line 78, in inner
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server return f(*args,
**kwargs)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/utils.py", line 405,
in change_encrypted
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server res =
func(**call_args)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py",
line 891, in disconnect_volume
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server return
self._cleanup_connection(connection_properties, force=force,
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py",
line 969, in _cleanup_connection
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server raise exc #
type: ignore
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
os_brick.exception.ExceptionChainer:
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server Chained Exception #1
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server Traceback (most
recent call last):
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py",
line 1177, in _disconnect_connection
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
self._disconnect_from_iscsi_portal(props)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py",
line 1159, in _disconnect_from_iscsi_portal
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
self._run_iscsiadm(connection_properties, ("--logout",),
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py",
line 1025, in _run_iscsiadm
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server (out, err)
= self._execute('iscsiadm', '-m', 'node', '-T',
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/executor.py", line
55, in _execute
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server result =
self.__execute(*args, **kwargs)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/privileged/rootwrap.py",
line 172, in execute
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server return
execute_root(*cmd, **kwargs)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_privsep/priv_context.py",
line 271, in _wrap
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server return
self.channel.remote_call(name, args, kwargs,
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_privsep/daemon.py", line
215, in remote_call
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server raise
exc_type(*result[2])
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while
running command.
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server Command:
iscsiadm -m node -T
iqn.2010-10.org.openstack:volume-3324028d-086a-473f-9bc1-4dcac228e5e6 -p
192.168.5.1:3260 --logout
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server Exit code: 32
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server Stdout:
'Logging out of session [sid: 1, target:
iqn.2010-10.org.openstack:volume-3324028d-086a-473f-9bc1-4dcac228e5e6, portal:
192.168.5.1,3260]\n'
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server Stderr:
'iscsiadm: Could not logout of [sid: 1, target:
iqn.2010-10.org.openstack:volume-3324028d-086a-473f-9bc1-4dcac228e5e6, portal:
192.168.5.1,3260].\niscsiadm: initiator reported error (32 - target likely not
connected)\niscsiadm: Could not logout of all requested sessions\n'
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2062459
Title:
VM's Won't Start Again After CLI/SDK Driven Shutdown - Exception
during message handling: os_brick.exception.ExceptionChainer
Status in OpenStack Compute (nova):
New
Bug description:
This was a brand new healthy VM. On Reboot, I noticed it and all other
VM's would not come up and as i run kolla ansible in containers, i
assumed it to be a race condition corrupting it. I created all new
VM's (that is fine) they started up and even shut down a few times ok,
i tried via SDK and also CLI and then then wont start again.... (no
shutdown involved) so this must be a nova or iscsi issue. Thoughts?
==> /var/log/kolla/nova/nova-compute.log <==
2024-04-19 00:12:23.073 8 INFO nova.virt.libvirt.driver [-] [instance:
712454ec-3ab3-4648-97c5-ede2edce5c9c] Instance destroyed successfully.
2024-04-19 00:12:23.087 8 INFO os_vif [None
req-038ecf7f-d511-4833-948c-0236b81820bf 34df1169571b4955a0ff9527fda70c98
e95c8c84b62e416daf03b91a9648962b - - default default] Successfully unplugged
vif
VIFBridge(active=False,address=fa:16:3e:cc:30:e0,bridge_name='qbrc93e0089-32',has_traffic_filtering=True,id=c93e0089-3236-4271-bd61-90f2cd47afdd,network=Network(53acdf7a-617f-4ab5-a62e-0cb96936d619),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapc93e0089-32')
2024-04-19 00:12:23.090 8 WARNING os_brick.initiator.connectors.base [None
req-038ecf7f-d511-4833-948c-0236b81820bf 34df1169571b4955a0ff9527fda70c98
e95c8c84b62e416daf03b91a9648962b - - default default] Service needs to call
os_brick.setup() before connecting volumes, if it doesn't it will break on the
next release
2024-04-19 00:12:23.140 8 WARNING os_brick.exception [None
req-038ecf7f-d511-4833-948c-0236b81820bf 34df1169571b4955a0ff9527fda70c98
e95c8c84b62e416daf03b91a9648962b - - default default] Disconnect from
192.168.5.1:3260
iqn.2010-10.org.openstack:volume-3324028d-086a-473f-9bc1-4dcac228e5e6 failed:
oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while
running command.
2024-04-19 00:12:23.140 8 WARNING os_brick.initiator.connectors.iscsi [None
req-038ecf7f-d511-4833-948c-0236b81820bf 34df1169571b4955a0ff9527fda70c98
e95c8c84b62e416daf03b91a9648962b - - default default] There were errors
removing set(), leftovers may remain in the system
==> /var/log/kolla/nova/nova-compute.log <==
2024-04-19 00:12:23.352 8 INFO nova.compute.manager [None
req-038ecf7f-d511-4833-948c-0236b81820bf 34df1169571b4955a0ff9527fda70c98
e95c8c84b62e416daf03b91a9648962b - - default default] [instance:
712454ec-3ab3-4648-97c5-ede2edce5c9c] Successfully reverted task state from
powering-on on failure for instance.
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server [None
req-038ecf7f-d511-4833-948c-0236b81820bf 34df1169571b4955a0ff9527fda70c98
e95c8c84b62e416daf03b91a9648962b - - default default] Exception during message
handling: os_brick.exception.ExceptionChainer:
Chained Exception #1
Traceback (most recent call last):
File
"/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py",
line 1177, in _disconnect_connection
self._disconnect_from_iscsi_portal(props)
File
"/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py",
line 1159, in _disconnect_from_iscsi_portal
self._run_iscsiadm(connection_properties, ("--logout",),
File
"/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py",
line 1025, in _run_iscsiadm
(out, err) = self._execute('iscsiadm', '-m', 'node', '-T',
File
"/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/executor.py", line
55, in _execute
result = self.__execute(*args, **kwargs)
File
"/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/privileged/rootwrap.py",
line 172, in execute
return execute_root(*cmd, **kwargs)
File
"/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_privsep/priv_context.py",
line 271, in _wrap
return self.channel.remote_call(name, args, kwargs,
File
"/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_privsep/daemon.py", line
215, in remote_call
raise exc_type(*result[2])
oslo_concurrency.processutils.ProcessExecutionError: Unexpected error
while running command.
Command: iscsiadm -m node -T
iqn.2010-10.org.openstack:volume-3324028d-086a-473f-9bc1-4dcac228e5e6 -p
192.168.5.1:3260 --logout
Exit code: 32
Stdout: 'Logging out of session [sid: 1, target:
iqn.2010-10.org.openstack:volume-3324028d-086a-473f-9bc1-4dcac228e5e6, portal:
192.168.5.1,3260]\n'
Stderr: 'iscsiadm: Could not logout of [sid: 1, target:
iqn.2010-10.org.openstack:volume-3324028d-086a-473f-9bc1-4dcac228e5e6, portal:
192.168.5.1,3260].\niscsiadm: initiator reported error (32 - target likely not
connected)\niscsiadm: Could not logout of all requested sessions\n'
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server Traceback (most
recent call last):
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/server.py",
line 165, in _process_incoming
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server res =
self.dispatcher.dispatch(message)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py",
line 309, in dispatch
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server return
self._do_dispatch(endpoint, method, ctxt, args)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py",
line 229, in _do_dispatch
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server result =
func(ctxt, **new_args)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/nova/exception_wrapper.py",
line 65, in wrapped
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server with
excutils.save_and_reraise_exception():
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line
227, in __exit__
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
self.force_reraise()
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line
200, in force_reraise
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server raise self.value
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/nova/exception_wrapper.py",
line 63, in wrapped
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server return f(self,
context, *args, **kw)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py",
line 166, in decorated_function
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server with
excutils.save_and_reraise_exception():
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line
227, in __exit__
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
self.force_reraise()
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line
200, in force_reraise
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server raise self.value
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py",
line 157, in decorated_function
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/utils.py", line
1439, in decorated_function
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py",
line 213, in decorated_function
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server with
excutils.save_and_reraise_exception():
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line
227, in __exit__
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
self.force_reraise()
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line
200, in force_reraise
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server raise self.value
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py",
line 203, in decorated_function
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py",
line 3424, in start_instance
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
self._power_on(context, instance)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py",
line 3392, in _power_on
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
self.driver.power_on(context, instance,
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py",
line 4136, in power_on
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
self._hard_reboot(context, instance, network_info, block_device_info,
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py",
line 3955, in _hard_reboot
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
self.destroy(context, instance, network_info, destroy_disks=False,
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py",
line 1565, in destroy
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
self.cleanup(context, instance, network_info, block_device_info,
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py",
line 1635, in cleanup
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server return
self._cleanup(
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py",
line 1690, in _cleanup
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server with
excutils.save_and_reraise_exception() as ctxt:
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line
227, in __exit__
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
self.force_reraise()
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line
200, in force_reraise
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server raise self.value
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py",
line 1686, in _cleanup
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
self._disconnect_volume(
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py",
line 2017, in _disconnect_volume
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
vol_driver.disconnect_volume(
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/volume/iscsi.py",
line 74, in disconnect_volume
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
self.connector.disconnect_volume(
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/utils.py", line 173,
in trace_logging_wrapper
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server return f(*args,
**kwargs)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/base.py",
line 78, in inner
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server return f(*args,
**kwargs)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/utils.py", line 405,
in change_encrypted
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server res =
func(**call_args)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py",
line 891, in disconnect_volume
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server return
self._cleanup_connection(connection_properties, force=force,
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py",
line 969, in _cleanup_connection
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server raise exc #
type: ignore
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
os_brick.exception.ExceptionChainer:
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server Chained Exception #1
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server Traceback
(most recent call last):
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py",
line 1177, in _disconnect_connection
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
self._disconnect_from_iscsi_portal(props)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py",
line 1159, in _disconnect_from_iscsi_portal
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
self._run_iscsiadm(connection_properties, ("--logout",),
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py",
line 1025, in _run_iscsiadm
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server (out,
err) = self._execute('iscsiadm', '-m', 'node', '-T',
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/executor.py", line
55, in _execute
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server result =
self.__execute(*args, **kwargs)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/privileged/rootwrap.py",
line 172, in execute
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server return
execute_root(*cmd, **kwargs)
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_privsep/priv_context.py",
line 271, in _wrap
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server return
self.channel.remote_call(name, args, kwargs,
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server File
"/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_privsep/daemon.py", line
215, in remote_call
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server raise
exc_type(*result[2])
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while
running command.
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server Command:
iscsiadm -m node -T
iqn.2010-10.org.openstack:volume-3324028d-086a-473f-9bc1-4dcac228e5e6 -p
192.168.5.1:3260 --logout
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server Exit code: 32
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server Stdout:
'Logging out of session [sid: 1, target:
iqn.2010-10.org.openstack:volume-3324028d-086a-473f-9bc1-4dcac228e5e6, portal:
192.168.5.1,3260]\n'
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server Stderr:
'iscsiadm: Could not logout of [sid: 1, target:
iqn.2010-10.org.openstack:volume-3324028d-086a-473f-9bc1-4dcac228e5e6, portal:
192.168.5.1,3260].\niscsiadm: initiator reported error (32 - target likely not
connected)\niscsiadm: Could not logout of all requested sessions\n'
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
2024-04-19 00:12:23.355 8 ERROR oslo_messaging.rpc.server
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2062459/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
