Hi,
the last message you sent is normal for an OSD that hasn't reported
back its status yet.
I would check the ceph-volume.log and the cephadm.log, maybe an OSD
log as well if it tried to boot. If this is a test cluster, did you
properly wipe all disks before trying to deploy OSDs? For me 'cephadm
ceph-volume lvm zap --destroy /dev/sdx /dev/sdy /dev/sdz ...' (locally
on the host) has been working great for years now. You can zap them
with the orchestrator as well, but only one disk a time, so a for loop
would be useful.
There have been reports every now and then from users who tried to
deploy many disks per host, I don't have a link available right now.
And I haven't had the chance yet to deploy multiple hosts/many OSDs
with 19.2.3, so there might be a regression in ceph-volume.
Regards,
Eugen
Zitat von Gilles Mocellin <gilles.mocel...@nuagelibre.org>:
Le 2025-08-18 11:47, Gilles Mocellin a écrit :
Le 2025-08-18 11:30, Gilles Mocellin a écrit :
On Hi Cephers,
I'm building a new Squid cluster with cephadm on Ubuntu 24.04.
After expanding my cluster in the Dashboard (adding my 7 hosts),
I choose throughput_optimized proflie wich create a generic spec
for hybrid HDD/SSD :
service_type: osd
service_id: throughput_optimized
service_name: osd.throughput_optimized
placement:
host_pattern: '*'
spec:
data_devices:
rotational: 1
db_devices:
rotational: 0
encrypted: true
filter_logic: AND
objectstore: bluestore
The cluster is for a LAB environment, on each of the 7 nodes, I
have 17 HDD SAS 1.2TB drives and 1 SSD SAS Enterprise 400GB drive.
On my first try, only 28 OSD where created (out of the 119), the
others appeared as down, but they won't start, I didn't find
systemd units created on the hosts.
But, the VGs and LVs where created, there are 17 LVs on the SSD
for WAL/DB of the 17 HDD (yes, small : 29GB).
On my second try, it creates 72 OSDs, still it stops, and never
tries to continue, or re-create the down OSDs.
I didn't manage to find them, but it seems I saw some OSD creation
timeout in the logs...
What can I do to have my missing OSD created ?
Some additional information :
ceph -s
cluster:
id: 3ebf83bf-7927-11f0-9f3a-246e96bd90a4
health: HEALTH_OK
services:
mon: 5 daemons, quorum
fidcl-lyo1-sto-sds-lab-01,fidcl-lyo1-sto-sds-lab-02,fidcl-lyo1-sto-sds-lab-03,fidcl-lyo1-sto-sds-lab-04,fidcl-lyo1-sto-sds-lab-05 (age
3d)
mgr: fidcl-lyo1-sto-sds-lab-01.ovbjpb(active, since 3d),
standbys: fidcl-lyo1-sto-sds-lab-02.nqdhpl,
fidcl-lyo1-sto-sds-lab-03.cizytz
osd: 119 osds: 72 up (since 3d), 89 in (since 41m)
data:
pools: 1 pools, 1 pgs
objects: 2 objects, 769 KiB
usage: 1.5 TiB used, 79 TiB / 80 TiB avail
pgs: 1 active+clean
One of the "missing" (not fully created) OSD, is present, but not found :
ceph osd find 6
{
"osd": 6,
"addrs": {
"addrvec": []
},
"osd_fsid": "f8745284-8026-4713-8329-cd57cb6842f7",
"crush_location": {}
}
ceph osd info 6
osd.6 down out weight 0 up_from 0 up_thru 0 down_at 0
last_clean_interval [0,0) autoout,exists,new
f8745284-8026-4713-8329-cd57cb6842f7
The missing OSDs are not chown with the device list command :
ceph device ls | grep osd.6
is empty...
Also, the MGR reports that on all not fully created OSD :
Aug 18 10:09:41 fidcl-lyo1-sto-sds-lab-01 ceph-mgr[4186078]: mgr
get_metadata_python Requested missing service osd.99
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
