[Yahoo-eng-team] [Bug 1530214] Re: Tempest failures due to iSCSI DB failure on compute node
*** This bug is a duplicate of bug 1324670 *** https://bugs.launchpad.net/bugs/1324670 Looks like this is an old one we thought was fixed on the brick side. Removing Nova and marking as a duplicate of the original bug: 1324670 ** No longer affects: nova ** This bug has been marked a duplicate of bug 1324670 'iscsiadm ... -o delete' fails occasionally on bulk deployments -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1530214 Title: Tempest failures due to iSCSI DB failure on compute node Status in os-brick: New Bug description: Noticed a couple of these today in the SolidFire CI system. These are initiator side errors in Nova. Excerpt from log is below, but additional logs can also be viewed here: http://54.164.167.86/solidfire-ci-logs/refs-changes-67-244867-12/logs/screen-n-cpu.log.txt 2015-12-30 20:53:50.829 ERROR nova.virt.block_device [req-0f1602ac-0604-4051-8d65-81ede0976dc5 tempest-DeleteServersTestJSON-1052804341 tempest-DeleteServersTestJSON-308638552] [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] Driver failed to attach volume 23049317-f6c2-40b2-9306-0b838848f911 at /dev/vdb 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] Traceback (most recent call last): 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] File "/opt/stack/nova/nova/virt/block_device.py", line 288, in attach 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] device_type=self['device_type'], encryption=encryption) 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1114, in attach_volume 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] self._connect_volume(connection_info, disk_info) 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1064, in _connect_volume 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] vol_driver.connect_volume(connection_info, disk_info) 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] File "/opt/stack/nova/nova/virt/libvirt/volume/iscsi.py", line 84, in connect_volume 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] device_info = self.connector.connect_volume(connection_info['data']) 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] return f(*args, **kwargs) 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] File "/opt/stack/os-brick/os_brick/initiator/connector.py", line 763, in connect_volume 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] connection_properties) 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] File "/opt/stack/os-brick/os_brick/initiator/connector.py", line 591, in _get_potential_volume_paths 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] if self._connect_to_iscsi_portal(props): 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] File "/opt/stack/os-brick/os_brick/initiator/connector.py", line 1044, in _connect_to_iscsi_portal 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] self._run_iscsiadm(connection_properties, ()) 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] File "/opt/stack/os-brick/os_brick/initiator/connector.py", line 948, in _run_iscsiadm 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] delay_on_retry=delay_on_retry) 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 312, in execute 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c]
[Yahoo-eng-team] [Bug 1530214] Re: Tempest failures due to iSCSI DB failure on compute node
** Also affects: os-brick Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1530214 Title: Tempest failures due to iSCSI DB failure on compute node Status in OpenStack Compute (nova): New Status in os-brick: New Bug description: Noticed a couple of these today in the SolidFire CI system. These are initiator side errors in Nova. Excerpt from log is below, but additional logs can also be viewed here: http://54.164.167.86/solidfire-ci-logs/refs-changes-67-244867-12/logs/screen-n-cpu.log.txt 2015-12-30 20:53:50.829 ERROR nova.virt.block_device [req-0f1602ac-0604-4051-8d65-81ede0976dc5 tempest-DeleteServersTestJSON-1052804341 tempest-DeleteServersTestJSON-308638552] [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] Driver failed to attach volume 23049317-f6c2-40b2-9306-0b838848f911 at /dev/vdb 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] Traceback (most recent call last): 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] File "/opt/stack/nova/nova/virt/block_device.py", line 288, in attach 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] device_type=self['device_type'], encryption=encryption) 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1114, in attach_volume 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] self._connect_volume(connection_info, disk_info) 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1064, in _connect_volume 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] vol_driver.connect_volume(connection_info, disk_info) 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] File "/opt/stack/nova/nova/virt/libvirt/volume/iscsi.py", line 84, in connect_volume 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] device_info = self.connector.connect_volume(connection_info['data']) 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] return f(*args, **kwargs) 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] File "/opt/stack/os-brick/os_brick/initiator/connector.py", line 763, in connect_volume 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] connection_properties) 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] File "/opt/stack/os-brick/os_brick/initiator/connector.py", line 591, in _get_potential_volume_paths 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] if self._connect_to_iscsi_portal(props): 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] File "/opt/stack/os-brick/os_brick/initiator/connector.py", line 1044, in _connect_to_iscsi_portal 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] self._run_iscsiadm(connection_properties, ()) 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] File "/opt/stack/os-brick/os_brick/initiator/connector.py", line 948, in _run_iscsiadm 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] delay_on_retry=delay_on_retry) 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 312, in execute 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] cmd=sanitized_cmd) 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] ProcessExecutionError: Unexpected error while running command. 2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d15263