Hi,
Am 9/9/25 um 17:34 schrieb Yuri Weinstein:
* Dashboard: Added support for NVMe/TCP (gateway groups, multiple
namespaces),
I have just upgraded a test cluster and NVMEoF does not work any more.
The previous version running was quay.ceph.io/ceph-ci/ceph:squid-nvmeof and
NVMEoF was working there.
The gateways seem to work and subsystem and namespaces are still configured but
an initiator does not see namespaces.
# ceph orch ps --daemon_type nvmeof
NAME HOST PORTS STATUS
REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID
nvmeof.hosa.b236asbrce101.ccaujq b236asbrce101 *:5500,4420,8009,10008
running (13h) 36s ago 13h 128M - 1.5.7 2c3e7ef350b8
74672e87a945
nvmeof.hosa.b236asbrce102.gnbwsc b236asbrce102 *:5500,4420,8009,10008
running (13h) 36s ago 13h 120M - 1.5.7 2c3e7ef350b8
1929979ab7e1
nvmeof.hosa.b236asbrce103.bthewi b236asbrce103 *:5500,4420,8009,10008
running (13h) 36s ago 13h 122M - 1.5.7 2c3e7ef350b8
14885171d1a5
nvmeof.hosa.b236asbrce104.vcgzvg b236asbrce104 *:5500,4420,8009,10008
running (13h) 36s ago 13h 122M - 1.5.7 2c3e7ef350b8
7b00b0f3356a
# ceph nvmeof subsystem list
+---------------------------------+------------------+--------------------+---------------+-------+--------------+--------------+--------------+-------------------+
|Nqn |Serial Number |Model Number
|Namespace Count|Subtype|Max Namespaces|Has Dhchap Key|Allow Any Host|Created
Without Key|
+---------------------------------+------------------+--------------------+---------------+-------+--------------+--------------+--------------+-------------------+
|nqn.2025-08.com.example:HOSA.hosa|Ceph30844860597194|Ceph bdev Controller|2
|NVMe |128 |False |True |False
|
+---------------------------------+------------------+--------------------+---------------+-------+--------------+--------------+--------------+-------------------+
# ceph nvmeof namespace list nqn.2025-08.com.example:HOSA.hosa
+-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+---------------------------------+-----------+-------------------+---------+
|Bdev Name |Rbd Image|Rbd Pool|Lb Group|Rbd
Image Size|Block Size|R/W Ios/Sec|R/W Mbs/Sec|Read Mbs/Sec|Write Mbs/Sec|Auto
Visible|Hosts|Nsid|Uuid |Ns Subsystem Nqn
|Trash Image|Disable Auto Resize|Read Only|
+-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+---------------------------------+-----------+-------------------+---------+
|bdev_57f2c81a-0b07-406b-84aa-c62d66b5f10d|ns01 |nvmeof |5 |100GB
|512B |0 |0 |0 |0 |True
|[] |1
|57f2c81a-0b07-406b-84aa-c62d66b5f10d|nqn.2025-08.com.example:HOSA.hosa|False
|False |False |
|bdev_b4bf328a-8c67-4879-bf13-a799884d6dbe|ns02 |nvmeof |5 |200GB
|512B |0 |0 |0 |0 |True
|[] |2
|b4bf328a-8c67-4879-bf13-a799884d6dbe|nqn.2025-08.com.example:HOSA.hosa|False
|False |False |
+-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+---------------------------------+-----------+-------------------+---------+
# ceph nvmeof listener list nqn.2025-08.com.example:HOSA.hosa
+-------------+------+--------------+------+------+-------+
|Host Name |Trtype|Traddr |Secure|Adrfam|Trsvcid|
+-------------+------+--------------+------+------+-------+
|b236asbrce101|TCP |10.105.141.195|False |ipv4 |4420 |
|b236asbrce102|TCP |10.105.141.196|False |ipv4 |4420 |
|b236asbrce103|TCP |10.105.141.197|False |ipv4 |4420 |
|b236asbrce104|TCP |10.105.141.198|False |ipv4 |4420 |
+-------------+------+--------------+------+------+-------+
On the initiator host:
# nvme discover -t tcp -a 10.105.141.195 -s 8009 -v
warning: use hostid which does not match uuid in hostnqn
nvme0: nqn.2014-08.org.nvmexpress.discovery connected
Discovery Log Number of Records 4, Generation counter 6
=====Discovery Log Entry 0======
trtype: tcp
adrfam: ipv4
subtype: nvme subsystem
treq: not required
portid: 0
trsvcid: 4420
subnqn: nqn.2025-08.com.example:HOSA.hosa
traddr: 10.105.141.195
eflags: none
sectype: none
=====Discovery Log Entry 1======
trtype: tcp
adrfam: ipv4
subtype: nvme subsystem
treq: not required
portid: 1
trsvcid: 4420
subnqn: nqn.2025-08.com.example:HOSA.hosa
traddr: 10.105.141.196
eflags: none
sectype: none
=====Discovery Log Entry 2======
trtype: tcp
adrfam: ipv4
subtype: nvme subsystem
treq: not required
portid: 2
trsvcid: 4420
subnqn: nqn.2025-08.com.example:HOSA.hosa
traddr: 10.105.141.197
eflags: none
sectype: none
=====Discovery Log Entry 3======
trtype: tcp
adrfam: ipv4
subtype: nvme subsystem
treq: not required
portid: 3
trsvcid: 4420
subnqn: nqn.2025-08.com.example:HOSA.hosa
traddr: 10.105.141.198
eflags: none
sectype: none
nvme0: nqn.2014-08.org.nvmexpress.discovery disconnected
# nvme connect-all -t tcp -a 10.105.141.195 -s 8009 -v
warning: use hostid which does not match uuid in hostnqn
nvme0: nqn.2014-08.org.nvmexpress.discovery connected
nvme1: nqn.2025-08.com.example:HOSA.hosa connected
nvme2: nqn.2025-08.com.example:HOSA.hosa connected
nvme3: nqn.2025-08.com.example:HOSA.hosa connected
nvme4: nqn.2025-08.com.example:HOSA.hosa connected
nvme0: nqn.2014-08.org.nvmexpress.discovery disconnected
Looking good so far, but:
# nvme list
Node Generic SN Model
Namespace Usage Format
FW Rev
--------------------- --------------------- --------------------
---------------------------------------- ---------- --------------------------
---------------- --------
# nvme list -v
Subsystem Subsystem-NQN
Controllers
----------------
--------------------------------------------------------------------------------------------
----------------
nvme-subsys1 nqn.2025-08.com.example:HOSA.hosa
nvme1, nvme2, nvme3, nvme4
Device SN MN FR
TxPort Address Slot Subsystem Namespaces
-------- -------------------- ---------------------------------------- --------
------ -------------- ------ ------------ ----------------
nvme1 Ceph30844860597194 Ceph bdev Controller 25.05
tcp traddr=10.105.141.195,trsvcid=4420,src_addr=10.105.141.212
nvme-subsys1
nvme2 Ceph30844860597194 Ceph bdev Controller 25.05
tcp traddr=10.105.141.196,trsvcid=4420,src_addr=10.105.141.212
nvme-subsys1
nvme3 Ceph30844860597194 Ceph bdev Controller 25.05
tcp traddr=10.105.141.197,trsvcid=4420,src_addr=10.105.141.212
nvme-subsys1
nvme4 Ceph30844860597194 Ceph bdev Controller 25.05
tcp traddr=10.105.141.198,trsvcid=4420,src_addr=10.105.141.212
nvme-subsys1
Device Generic NSID Usage Format
Controllers
------------ ------------ ---------- --------------------------
---------------- ----------------
# nvme list-ns /dev/nvme1 -v -a
opcode : 06
flags : 00
rsvd1 : 0000
nsid : 00000000
cdw2 : 00000000
cdw3 : 00000000
data_len : 00001000
metadata_len : 00000000
addr : 616b6af40000
metadata : 0
cdw10 : 00000010
cdw11 : 00000000
cdw12 : 00000000
cdw13 : 00000000
cdw14 : 00000000
cdw15 : 00000000
timeout_ms : 00000000
result : 00000000
err : 2
latency : 569 us
NVMe status: Invalid Field in Command: A reserved coded value or an unsupported
value in a defined field(0x2)
How do I make it work again?
Regards
--
Robert Sander
Linux Consultant
Heinlein Consulting GmbH
Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: +49 30 405051 - 0
Fax: +49 30 405051 - 19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B
Geschäftsführer: Peer Heinlein - Sitz: Berlin
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
