Hi List!
We have Squid 19.2.2. It is cephadm/docker based deployment (recently
upgraded from Pacific 16.2.15).
We are using 8 SAS drives for Block and 2 SSD drives for DB on every OSD
Host.
Problem:
One of the SAS Block drives failed on OSD Host and we need to replace it.
When our Ceph cluster was running on Pacific, we usually performed drive
replacement using this steps:
1) Edit -> MARK osd.xx as OUT [re-balancing starts, wait until it is
completed]
2) Edit -> MARK osd.xx as DOWN
3) Edit -> DELETE osd.xx [put check on "Preserve OSD ID" and "Yes, I am
sure"]
4) Edit -> DESTROY osd.xx
5) Edit -> PURGE osd.xx [re-balancing starts, wait until it is completed]
6) Set "noout" and "norebalance" flags. Put OSD Host in maintenance
mode. Shutdown OSD Host. Replace failed drive. Start OSD Host.
7) Wipe old DB Logical Volume (LV) [dd if=/dev/zero
of=/dev/ceph-xxx/osd-db-xxx bs=1M count=10 conv=fsync].
8) Wipe new Block disk. Destroy old DB LV. Wait for automatic discovery
and creation of new osd.xx instance.
In Pacific 16.2.15, after execution of PURGE command, Ceph just removed
old osd.xx instance from cluster without deletion/zapping of DB and
Block LVs.
Now in Squid 19.2.2 we see what Ceph behaves differently.
Execution of step 3 (Edit -> DELETE osd.xx) automatically executes
DESTROY and PURGE, and after that Ceph automatically performs zapping
and deletion of DB LV and Block LV!
And after that it's automatic discovery finds "clean" SAS disk + free
space on SSD drive and happily forms new osd.xx instance form failed
drive what we need to replace :)
Questions:
1) What is correct procedure how-to replace failed Block drive in Ceph
Squid 19.2.2?
2) Is it possible to disable Zapping?
3) Is it possible to temporary disable automatic discovery of new drives
for OSD service?
P.S.
Here is our Pacement Specification for OSD service:
[ceph: root@ceph-mon12 /]# ceph orch ls osd
osd.dashboard-admin-1633624229976 --export
service_type: osd
service_id: dashboard-admin-1633624229976
service_name: osd.dashboard-admin-1633624229976
placement:
host_pattern: '*'
spec:
data_devices:
rotational: true
db_devices:
rotational: false
db_slots: 4
filter_logic: AND
objectstore: bluestore
Logs from Ceph:
12/8/25 08:54 AM [INF] Cluster is now healthy
12/8/25 08:54 AM [INF] Health check cleared: PG_DEGRADED (was: Degraded
data redundancy: 11/121226847 objects degraded (0.000%), 1 pg degraded)
12/8/25 08:54 AM [WRN] Health check update: Degraded data redundancy:
11/121226847 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
12/8/25 08:54 AM [INF] Health check cleared: PG_AVAILABILITY (was:
Reduced data availability: 1 pg peering)
12/8/25 08:54 AM [WRN] Health check failed: Degraded data redundancy:
12/121226847 objects degraded (0.000%), 2 pgs degraded (PG_DEGRADED)
12/8/25 08:54 AM [WRN] Health check failed: Reduced data availability: 1
pg peering (PG_AVAILABILITY)
12/8/25 08:54 AM [INF] osd.33
[v2:10.10.10.105:6824/297218036,v1:10.10.10.105:6825/297218036] boot
12/8/25 08:54 AM [WRN] OSD bench result of 1909.305644 IOPS is not
within the threshold limit range of 50.000000 IOPS and 500.000000 IOPS
for osd.33. IOPS capacity is unchanged at 315.000000 IOPS. The
recommendation is to establish the osd's IOPS capacity using other
benchmark tools (e.g. Fio) and then override
osd_mclock_max_capacity_iops_[hdd|ssd].
12/8/25 08:53 AM [INF] Deploying daemon osd.33 on ceph-osd15
12/8/25 08:53 AM [INF] Found osd claims for drivegroup
dashboard-admin-1633624229976 -> {'ceph-osd15': ['33']}
12/8/25 08:53 AM [INF] Found osd claims -> {'ceph-osd15': ['33']}
12/8/25 08:53 AM [INF] Detected new or changed devices on ceph-osd15
12/8/25 08:52 AM [INF] Successfully zapped devices for osd.33 on ceph-osd15
12/8/25 08:52 AM [INF] Zapping devices for osd.33 on ceph-osd15
12/8/25 08:52 AM [INF] Successfully destroyed old osd.33 on ceph-osd15;
ready for replacement
12/8/25 08:52 AM [INF] Successfully removed osd.33 on ceph-osd15
12/8/25 08:52 AM [INF] Removing key for osd.33
12/8/25 08:52 AM [INF] Removing daemon osd.33 from ceph-osd15 -- ports []
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
