Public bug reported:
There seem to be 17 occurrences of this error in the last 7 days
[starting 3DEC2015]:
message: "libvirtError: cannot read header" AND tags:
"screen-n-cpu.txt"
And, 195 occurrences of the below error message string when looked at
logstash, :
message: "session 2 login rejected: Initiator failed authentication
with target" AND tags: "syslog"
From:
http://logs.openstack.org/80/251980/1/check/gate-grenade-dsvm-multinode/5716e36/logs/old/screen-n-cpu.txt.gz#_2015-12-02_10_57_35_812
[. . .]
2015-12-02 10:57:35.812 ERROR nova.compute.manager
[req-b5b4691f-da88-4abd-bffd-a72d52242dae
tempest-TestVolumeBootPattern-440462397
tempest-TestVolumeBootPattern-1264952841] [instance:
4d176a3a-2678-4207-99c6-f84894a23778] Instance failed to spawn
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] Traceback (most recent call last):
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] File
"/opt/stack/old/nova/nova/compute/manager.py", line 2155, in _build_resources
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] yield resources
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] File
"/opt/stack/old/nova/nova/compute/manager.py", line 2009, in
_build_and_run_instance
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] block_device_info=block_device_info)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] File
"/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 2444, in spawn
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] block_device_info=block_device_info)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] File
"/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 4519, in
_create_domain_and_network
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] xml, pause=pause, power_on=power_on)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] File
"/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 4449, in _create_domain
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] guest.launch(pause=pause)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] File
"/opt/stack/old/nova/nova/virt/libvirt/guest.py", line 141, in launch
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] self._encoded_xml, errors='ignore')
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in
__exit__
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] six.reraise(self.type_, self.value,
self.tb)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] File
"/opt/stack/old/nova/nova/virt/libvirt/guest.py", line 136, in launch
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] return
self._domain.createWithFlags(flags)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] File
"/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] result = proxy_call(self._autowrap,
f, *args, **kwargs)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] File
"/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in
proxy_call
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] rv = execute(f, *args, **kwargs)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] File
"/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] six.reraise(c, e, tb)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] File
"/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] rv = meth(*args, **kwargs)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] File
"/usr/local/lib/python2.7/dist-packages/libvirt.py", line 900, in
createWithFlags
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] if ret == -1: raise libvirtError
('virDomainCreateWithFlags() failed', dom=self)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] libvirtError: cannot read header
'/dev/disk/by-path/ip-149.202.189.35:3260-iscsi-iqn.2010-10.org.openstack:volume-690bf429-192d-40f1-97bf-f558a8106208-lun-1':
Input/output error
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778]
[. . .]
And, there are errors from 'iscsid' from syslog:
http://logs.openstack.org/80/251980/1/check/gate-grenade-dsvm-
multinode/5716e36/logs/syslog.txt.gz#_Dec__2_10_55_36
[. . .]
Dec 2 10:55:36 ubuntu iscsid: Kernel reported iSCSI connection 2:0 error (1020
- ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)
Dec 2 10:55:36 ubuntu iscsid: Login failed to authenticate with target
iqn.2010-10.org.openstack:volume-6fb73a19-26f9-4c76-95e0-ccfa57489593
Dec 2 10:55:36 ubuntu iscsid: session 1 login rejected: Initiator failed
authentication with target
Dec 2 10:55:36 ubuntu object-auditor: Object audit (ALL) "forever" mode
completed: 3.39s. Total quarantined: 0, Total errors: 0, Total files/sec: 7.66,
Total bytes/sec: 9986657.72, Auditing time: 3.39, Rate: 1.00
Dec 2 10:55:36 ubuntu kernel: [ 2566.962683] type=1400
audit(1449053736.838:29): apparmor="STATUS" operation="profile_load"
profile="unconfined" name="libvirt-4ac13a03-c8c8-4c20-8cd1-2f2354131a67"
pid=1807 comm="apparmor_parser"
Dec 2 10:55:38 ubuntu kernel: [ 2568.159143] connection2:0: detected conn
error (1020)
[. . .]
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1522488
Title:
'TestBootVolumePattern' fails in 'gate-grenade-dsvm-multinode' job
with "libvirtError: cannot read header '/dev/disk/by-
path/ip-[...]-lun-1'"
Status in OpenStack Compute (nova):
New
Bug description:
There seem to be 17 occurrences of this error in the last 7 days
[starting 3DEC2015]:
message: "libvirtError: cannot read header" AND tags:
"screen-n-cpu.txt"
And, 195 occurrences of the below error message string when looked at
logstash, :
message: "session 2 login rejected: Initiator failed
authentication with target" AND tags: "syslog"
From:
http://logs.openstack.org/80/251980/1/check/gate-grenade-dsvm-multinode/5716e36/logs/old/screen-n-cpu.txt.gz#_2015-12-02_10_57_35_812
[. . .]
2015-12-02 10:57:35.812 ERROR nova.compute.manager
[req-b5b4691f-da88-4abd-bffd-a72d52242dae
tempest-TestVolumeBootPattern-440462397
tempest-TestVolumeBootPattern-1264952841] [instance:
4d176a3a-2678-4207-99c6-f84894a23778] Instance failed to spawn
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] Traceback (most recent call last):
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] File
"/opt/stack/old/nova/nova/compute/manager.py", line 2155, in _build_resources
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] yield resources
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] File
"/opt/stack/old/nova/nova/compute/manager.py", line 2009, in
_build_and_run_instance
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] block_device_info=block_device_info)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] File
"/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 2444, in spawn
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] block_device_info=block_device_info)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] File
"/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 4519, in
_create_domain_and_network
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] xml, pause=pause, power_on=power_on)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] File
"/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 4449, in _create_domain
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] guest.launch(pause=pause)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] File
"/opt/stack/old/nova/nova/virt/libvirt/guest.py", line 141, in launch
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] self._encoded_xml, errors='ignore')
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in
__exit__
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] six.reraise(self.type_, self.value,
self.tb)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] File
"/opt/stack/old/nova/nova/virt/libvirt/guest.py", line 136, in launch
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] return
self._domain.createWithFlags(flags)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] File
"/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] result = proxy_call(self._autowrap,
f, *args, **kwargs)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] File
"/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in
proxy_call
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] rv = execute(f, *args, **kwargs)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] File
"/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] six.reraise(c, e, tb)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] File
"/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] rv = meth(*args, **kwargs)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] File
"/usr/local/lib/python2.7/dist-packages/libvirt.py", line 900, in
createWithFlags
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] if ret == -1: raise libvirtError
('virDomainCreateWithFlags() failed', dom=self)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778] libvirtError: cannot read header
'/dev/disk/by-path/ip-149.202.189.35:3260-iscsi-iqn.2010-10.org.openstack:volume-690bf429-192d-40f1-97bf-f558a8106208-lun-1':
Input/output error
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance:
4d176a3a-2678-4207-99c6-f84894a23778]
[. . .]
And, there are errors from 'iscsid' from syslog:
http://logs.openstack.org/80/251980/1/check/gate-grenade-dsvm-
multinode/5716e36/logs/syslog.txt.gz#_Dec__2_10_55_36
[. . .]
Dec 2 10:55:36 ubuntu iscsid: Kernel reported iSCSI connection 2:0 error
(1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)
Dec 2 10:55:36 ubuntu iscsid: Login failed to authenticate with target
iqn.2010-10.org.openstack:volume-6fb73a19-26f9-4c76-95e0-ccfa57489593
Dec 2 10:55:36 ubuntu iscsid: session 1 login rejected: Initiator failed
authentication with target
Dec 2 10:55:36 ubuntu object-auditor: Object audit (ALL) "forever" mode
completed: 3.39s. Total quarantined: 0, Total errors: 0, Total files/sec: 7.66,
Total bytes/sec: 9986657.72, Auditing time: 3.39, Rate: 1.00
Dec 2 10:55:36 ubuntu kernel: [ 2566.962683] type=1400
audit(1449053736.838:29): apparmor="STATUS" operation="profile_load"
profile="unconfined" name="libvirt-4ac13a03-c8c8-4c20-8cd1-2f2354131a67"
pid=1807 comm="apparmor_parser"
Dec 2 10:55:38 ubuntu kernel: [ 2568.159143] connection2:0: detected conn
error (1020)
[. . .]
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1522488/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
