Verification done on {mantic,jammy,focal}-proposed.
The detailed verification steps as per comment #3
are attached. This is the summary with key steps.
With the packages from -proposed, when (re)starting
{cinder-volume,tgt}.service,
the cinder-volume service is started _after_ the tgt service, and
the volumes remain with 'in-use' status (no longer 'error' status).
Reproducer (45-second startup delay in tgt.service)
---
FILE=/etc/systemd/system/tgt.service.d/start-delay.conf
mkdir -p $(dirname $FILE)
cat <<EOF > $FILE
[Service]
ExecStartPre=$(which sleep) 45
EOF
systemctl daemon-reload
# systemctl show tgt.service | grep ^ExecStartPre=
ExecStartPre={ path=/usr/bin/sleep ; argv[]=/usr/bin/sleep 45 ;
ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ;
status=0/0 }
# date; systemctl restart cinder-volume.service tgt.service;
date
Before / packages from -updates:
---
... Notice that cinder-volume.service is Started _BEFORE_ tgt.service
... And the 'tgtadm' error _IS_ seen in the logs.
... And volumes are listed with 'error' Status.
mantic:
root@cinder-mantic:~# dpkg -l | grep cinder-volume
ii cinder-volume 2:23.0.0-0ubuntu1.1
all Cinder storage service - Volume server
root@cinder-mantic:~# date; systemctl restart cinder-volume.service
tgt.service; date
Tue Apr 30 15:54:43 UTC 2024
Tue Apr 30 15:55:29 UTC 2024
root@cinder-mantic:~# journalctl -b -u cinder-volume.service -u
tgt.service | grep Start | tail -3
Apr 30 15:54:43 cinder-mantic systemd[1]: Started cinder-volume.service
- OpenStack Cinder Volume.
Apr 30 15:54:44 cinder-mantic systemd[1]: Starting tgt.service -
(i)SCSI target daemon...
Apr 30 15:55:29 cinder-mantic systemd[1]: Started tgt.service - (i)SCSI
target daemon.
root@cinder-mantic:~# grep 'ERROR cinder.volume.manager'
/var/log/cinder/cinder-volume.log | grep tgtadm
2024-04-30 15:54:54.603 13692 ERROR cinder.volume.manager (out,
err) = cinder.privsep.targets.tgt.tgtadm_show()
2024-04-30 15:54:54.603 13692 ERROR cinder.volume.manager Command:
tgtadm --lld iscsi --op show --mode target
2024-04-30 15:54:54.603 13692 ERROR cinder.volume.manager Stderr:
'tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is not
connected\n'
root@cinder-mantic:~# cinder list
+--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+
| ID | Status | Name | Size |
Consumes Quota | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+
| 45b39ae0-2552-4703-a664-ae8de77a23a1 | error | test-volume | 1 |
True | __DEFAULT__ | false | |
+--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+
jammy:
root@cinder-jammy:~# dpkg -l | grep cinder-volume
ii cinder-volume 2:20.3.1-0ubuntu1.1
all Cinder storage service - Volume server
root@cinder-jammy:~# date; systemctl restart cinder-volume.service
tgt.service; date
Tue Apr 30 15:54:45 UTC 2024
Tue Apr 30 15:55:30 UTC 2024
root@cinder-jammy:~# journalctl -b -u cinder-volume.service -u
tgt.service | grep Start | tail -3
Apr 30 15:54:45 cinder-jammy systemd[1]: Started OpenStack Cinder
Volume.
Apr 30 15:54:45 cinder-jammy systemd[1]: Starting (i)SCSI target
daemon...
Apr 30 15:55:30 cinder-jammy systemd[1]: Started (i)SCSI target daemon.
root@cinder-jammy:~# grep 'ERROR cinder.volume.manager'
/var/log/cinder/cinder-volume.log | grep tgtadm
2024-04-30 15:54:55.918 14576 ERROR cinder.volume.manager (out,
err) = cinder.privsep.targets.tgt.tgtadm_show()
2024-04-30 15:54:55.918 14576 ERROR cinder.volume.manager Command:
tgtadm --lld iscsi --op show --mode target
2024-04-30 15:54:55.918 14576 ERROR cinder.volume.manager Stderr:
'tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is not
connected\n'
root@cinder-jammy:~# cinder list
+--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+
| ID | Status | Name | Size |
Consumes Quota | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+
| 88024eca-203c-4f8f-9ff0-4de704041a2f | error | test-volume | 1 |
True | __DEFAULT__ | false | |
+--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+
focal:
root@cinder-focal:~# dpkg -l | grep cinder-volume
ii cinder-volume 2:16.4.2-0ubuntu2.4
all Cinder storage service - Volume server
root@cinder-focal:~# date; systemctl restart cinder-volume.service
tgt.service; date
Tue Apr 30 15:54:46 UTC 2024
Tue Apr 30 15:55:31 UTC 2024
root@cinder-focal:~# journalctl -b -u cinder-volume.service -u
tgt.service | grep Start | tail -3
Apr 30 15:54:46 cinder-focal systemd[1]: Started OpenStack Cinder
Volume.
Apr 30 15:54:46 cinder-focal systemd[1]: Starting (i)SCSI target
daemon...
Apr 30 15:55:31 cinder-focal systemd[1]: Started (i)SCSI target daemon.
root@cinder-focal:~# grep 'ERROR cinder.volume.manager'
/var/log/cinder/cinder-volume.log | grep tgtadm
2024-04-30 15:54:55.830 15834 ERROR cinder.volume.manager (out,
err) = cinder.privsep.targets.tgt.tgtadm_show()
2024-04-30 15:54:55.830 15834 ERROR cinder.volume.manager Command:
tgtadm --lld iscsi --op show --mode target
2024-04-30 15:54:55.830 15834 ERROR cinder.volume.manager Stderr:
'tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is not
connected\n'
root@cinder-focal:~# cinder list
+--------------------------------------+--------+-------------+------+-------------+----------+-------------+
| ID | Status | Name | Size |
Volume Type | Bootable | Attached to |
+--------------------------------------+--------+-------------+------+-------------+----------+-------------+
| 5fa41f17-afaf-4a56-8494-8c9246531523 | error | test-volume | 1 |
__DEFAULT__ | false | |
+--------------------------------------+--------+-------------+------+-------------+----------+-------------+
After / packages from -proposed
... Notice that cinder-volume.service is Started *AFTER* tgt.service now
... And now the 'tgtadm' error is *NOT* seen in the logs.
... And the volumes are listed with Status 'in-use' (no longer 'error')
mantic:
root@cinder-mantic:~# apt-cache policy cinder-volume
cinder-volume:
Installed: 2:23.0.0-0ubuntu1.2
Candidate: 2:23.0.0-0ubuntu1.2
Version table:
*** 2:23.0.0-0ubuntu1.2 500
400 http://archive.ubuntu.com/ubuntu mantic-proposed/main amd64
Packages
...
root@cinder-mantic:~# dpkg -l | grep cinder-volume
ii cinder-volume 2:23.0.0-0ubuntu1.2
all Cinder storage service - Volume server
root@cinder-mantic:~# date; systemctl restart cinder-volume.service
tgt.service; date
Tue Apr 30 16:34:03 UTC 2024
Tue Apr 30 16:34:49 UTC 2024
root@cinder-mantic:~# journalctl -b -u cinder-volume.service -u
tgt.service | grep Start | tail -3
Apr 30 16:34:03 cinder-mantic systemd[1]: Starting tgt.service -
(i)SCSI target daemon...
Apr 30 16:34:49 cinder-mantic systemd[1]: Started tgt.service - (i)SCSI
target daemon.
Apr 30 16:34:49 cinder-mantic systemd[1]: Started cinder-volume.service
- OpenStack Cinder Volume.
(error timestamps are from the previous/non-proposed run)
root@cinder-mantic:~# grep 'ERROR cinder.volume.manager'
/var/log/cinder/cinder-volume.log | grep tgtadm
2024-04-30 15:54:54.603 13692 ERROR cinder.volume.manager (out,
err) = cinder.privsep.targets.tgt.tgtadm_show()
2024-04-30 15:54:54.603 13692 ERROR cinder.volume.manager Command:
tgtadm --lld iscsi --op show --mode target
2024-04-30 15:54:54.603 13692 ERROR cinder.volume.manager Stderr:
'tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is not
connected\n'
root@cinder-mantic:~# cinder list
+--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+
| ID | Status | Name | Size |
Consumes Quota | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+
| 45b39ae0-2552-4703-a664-ae8de77a23a1 | in-use | test-volume | 1 |
True | __DEFAULT__ | false | |
+--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+
jammy:
root@cinder-jammy:~# apt-cache policy cinder-volume
cinder-volume:
Installed: 2:20.3.1-0ubuntu1.2
Candidate: 2:20.3.1-0ubuntu1.2
Version table:
*** 2:20.3.1-0ubuntu1.2 400
400 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64
Packages
...
root@cinder-jammy:~# dpkg -l | grep cinder-volume
ii cinder-volume 2:20.3.1-0ubuntu1.2
all Cinder storage service - Volume server
root@cinder-jammy:~# date; systemctl restart cinder-volume.service
tgt.service; date
Tue Apr 30 16:34:04 UTC 2024
Tue Apr 30 16:34:50 UTC 2024
root@cinder-jammy:~# journalctl -b -u cinder-volume.service -u
tgt.service | grep Start | tail -3
Apr 30 16:34:04 cinder-jammy systemd[1]: Starting (i)SCSI target
daemon...
Apr 30 16:34:50 cinder-jammy systemd[1]: Started (i)SCSI target daemon.
Apr 30 16:34:50 cinder-jammy systemd[1]: Started OpenStack Cinder
Volume.
(error timestamps are from the previous/non-proposed run)
root@cinder-jammy:~# grep 'ERROR cinder.volume.manager'
/var/log/cinder/cinder-volume.log | grep tgtadm
2024-04-30 15:54:55.918 14576 ERROR cinder.volume.manager (out,
err) = cinder.privsep.targets.tgt.tgtadm_show()
2024-04-30 15:54:55.918 14576 ERROR cinder.volume.manager Command:
tgtadm --lld iscsi --op show --mode target
2024-04-30 15:54:55.918 14576 ERROR cinder.volume.manager Stderr:
'tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is not
connected\n'
root@cinder-jammy:~# cinder list
+--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+
| ID | Status | Name | Size |
Consumes Quota | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+
| 88024eca-203c-4f8f-9ff0-4de704041a2f | in-use | test-volume | 1 |
True | __DEFAULT__ | false | |
+--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+
focal:
root@cinder-focal:~# apt-cache policy cinder-volume
cinder-volume:
Installed: 2:16.4.2-0ubuntu2.5
Candidate: 2:16.4.2-0ubuntu2.5
Version table:
*** 2:16.4.2-0ubuntu2.5 500
400 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64
Packages
...
root@cinder-focal:~# dpkg -l | grep cinder-volume
ii cinder-volume 2:16.4.2-0ubuntu2.5
all Cinder storage service - Volume server
root@cinder-focal:~# date; systemctl restart cinder-volume.service
tgt.service; date
Tue Apr 30 16:34:05 UTC 2024
Tue Apr 30 16:34:51 UTC 2024
root@cinder-focal:~# journalctl -b -u cinder-volume.service -u
tgt.service | grep Start | tail -3
Apr 30 16:34:05 cinder-focal systemd[1]: Starting (i)SCSI target
daemon...
Apr 30 16:34:51 cinder-focal systemd[1]: Started (i)SCSI target daemon.
Apr 30 16:34:51 cinder-focal systemd[1]: Started OpenStack Cinder
Volume.
(error timestamps are from the previous/non-proposed run)
root@cinder-focal:~# grep 'ERROR cinder.volume.manager'
/var/log/cinder/cinder-volume.log | grep tgtadm
2024-04-30 15:54:55.830 15834 ERROR cinder.volume.manager (out,
err) = cinder.privsep.targets.tgt.tgtadm_show()
2024-04-30 15:54:55.830 15834 ERROR cinder.volume.manager Command:
tgtadm --lld iscsi --op show --mode target
2024-04-30 15:54:55.830 15834 ERROR cinder.volume.manager Stderr:
'tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is not
connected\n'
root@cinder-focal:~# cinder list
+--------------------------------------+--------+-------------+------+-------------+----------+-------------+
| ID | Status | Name | Size |
Volume Type | Bootable | Attached to |
+--------------------------------------+--------+-------------+------+-------------+----------+-------------+
| 5fa41f17-afaf-4a56-8494-8c9246531523 | in-use | test-volume | 1 |
__DEFAULT__ | false | |
+--------------------------------------+--------+-------------+------+-------------+----------+-------------+
** Attachment added: "2024 04 30 proposed verification mantic jammy focal.txt"
https://bugs.launchpad.net/cloud-archive/ussuri/+bug/1987663/+attachment/5773339/+files/2024%2004%2030%20proposed%20verification%20mantic%20jammy%20focal.txt
** Tags removed: verification-needed verification-needed-focal
verification-needed-jammy verification-needed-mantic
** Tags added: verification-done verification-done-focal
verification-done-jammy verification-done-mantic
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1987663
Title:
cinder-volume: "Failed to re-export volume, setting to ERROR" with
"tgtadm: failed to send request hdr to tgt daemon, Transport endpoint
is not connected" on service startup
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1987663/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
