Hi,
I think I found the right place [0]:
---snip---
if any_replace_params:
# mark destroyed in osdmap
if not osd.destroy():
raise orchestrator.OrchestratorError(
f"Could not destroy {osd}")
logger.info(
f"Successfully destroyed old {osd} on
{osd.hostname}; ready for replacement")
if any_replace_params:
osd.zap = True
...
if osd.zap:
# throws an exception if the zap fails
logger.info(f"Zapping devices for {osd} on {osd.hostname}")
osd.do_zap()
---snip---
So if the replace flag is set, Ceph will zap the device(s). I compared
the versions, the change was between 19.2.0 and 19.2.1.
On the one hand, I agree with OP, if Ceph immediately zaps the
drive(s) it will redeploy the destroyed OSD with the faulty disk.
On the other hand, if you don't let cephadm zap the drives you'll need
manual intervention during the actual disk replacement. So the OSD
will be purged, leaving the DB/WAL LVs on the disk.
It would be interesting to learn what led to the decision to implement
it like this, but I also don't see an "optimal" way doing this. I
wonder if it could make sense to zap only DB/WAL devices, not the data
device in case of replacement. Then when the faulty disk gets
replaced, the orchestrator could redeploy an OSD since the data drive
should be clean and there should be space for DB/WAL.
Regards,
Eugen
[0]
https://github.com/ceph/ceph/blob/v19.2.3/src/pybind/mgr/cephadm/services/osd.py#L909
Zitat von Dmitrijs Demidovs <dmitrijs.demid...@carminered.eu>:
Hi List!
We have Squid 19.2.2. It is cephadm/docker based deployment
(recently upgraded from Pacific 16.2.15).
We are using 8 SAS drives for Block and 2 SSD drives for DB on every
OSD Host.
Problem:
One of the SAS Block drives failed on OSD Host and we need to replace it.
When our Ceph cluster was running on Pacific, we usually performed
drive replacement using this steps:
1) Edit -> MARK osd.xx as OUT [re-balancing starts, wait until it is
completed]
2) Edit -> MARK osd.xx as DOWN
3) Edit -> DELETE osd.xx [put check on "Preserve OSD ID" and "Yes, I
am sure"]
4) Edit -> DESTROY osd.xx
5) Edit -> PURGE osd.xx [re-balancing starts, wait until it is completed]
6) Set "noout" and "norebalance" flags. Put OSD Host in maintenance
mode. Shutdown OSD Host. Replace failed drive. Start OSD Host.
7) Wipe old DB Logical Volume (LV) [dd if=/dev/zero
of=/dev/ceph-xxx/osd-db-xxx bs=1M count=10 conv=fsync].
8) Wipe new Block disk. Destroy old DB LV. Wait for automatic
discovery and creation of new osd.xx instance.
In Pacific 16.2.15, after execution of PURGE command, Ceph just
removed old osd.xx instance from cluster without deletion/zapping of
DB and Block LVs.
Now in Squid 19.2.2 we see what Ceph behaves differently.
Execution of step 3 (Edit -> DELETE osd.xx) automatically executes
DESTROY and PURGE, and after that Ceph automatically performs
zapping and deletion of DB LV and Block LV!
And after that it's automatic discovery finds "clean" SAS disk +
free space on SSD drive and happily forms new osd.xx instance form
failed drive what we need to replace :)
Questions:
1) What is correct procedure how-to replace failed Block drive in
Ceph Squid 19.2.2?
2) Is it possible to disable Zapping?
3) Is it possible to temporary disable automatic discovery of new
drives for OSD service?
P.S.
Here is our Pacement Specification for OSD service:
[ceph: root@ceph-mon12 /]# ceph orch ls osd
osd.dashboard-admin-1633624229976 --export
service_type: osd
service_id: dashboard-admin-1633624229976
service_name: osd.dashboard-admin-1633624229976
placement:
host_pattern: '*'
spec:
data_devices:
rotational: true
db_devices:
rotational: false
db_slots: 4
filter_logic: AND
objectstore: bluestore
Logs from Ceph:
12/8/25 08:54 AM [INF] Cluster is now healthy
12/8/25 08:54 AM [INF] Health check cleared: PG_DEGRADED (was:
Degraded data redundancy: 11/121226847 objects degraded (0.000%), 1
pg degraded)
12/8/25 08:54 AM [WRN] Health check update: Degraded data
redundancy: 11/121226847 objects degraded (0.000%), 1 pg degraded
(PG_DEGRADED)
12/8/25 08:54 AM [INF] Health check cleared: PG_AVAILABILITY (was:
Reduced data availability: 1 pg peering)
12/8/25 08:54 AM [WRN] Health check failed: Degraded data
redundancy: 12/121226847 objects degraded (0.000%), 2 pgs degraded
(PG_DEGRADED)
12/8/25 08:54 AM [WRN] Health check failed: Reduced data
availability: 1 pg peering (PG_AVAILABILITY)
12/8/25 08:54 AM [INF] osd.33
[v2:10.10.10.105:6824/297218036,v1:10.10.10.105:6825/297218036] boot
12/8/25 08:54 AM [WRN] OSD bench result of 1909.305644 IOPS is not
within the threshold limit range of 50.000000 IOPS and 500.000000
IOPS for osd.33. IOPS capacity is unchanged at 315.000000 IOPS. The
recommendation is to establish the osd's IOPS capacity using other
benchmark tools (e.g. Fio) and then override
osd_mclock_max_capacity_iops_[hdd|ssd].
12/8/25 08:53 AM [INF] Deploying daemon osd.33 on ceph-osd15
12/8/25 08:53 AM [INF] Found osd claims for drivegroup
dashboard-admin-1633624229976 -> {'ceph-osd15': ['33']}
12/8/25 08:53 AM [INF] Found osd claims -> {'ceph-osd15': ['33']}
12/8/25 08:53 AM [INF] Detected new or changed devices on ceph-osd15
12/8/25 08:52 AM [INF] Successfully zapped devices for osd.33 on ceph-osd15
12/8/25 08:52 AM [INF] Zapping devices for osd.33 on ceph-osd15
12/8/25 08:52 AM [INF] Successfully destroyed old osd.33 on
ceph-osd15; ready for replacement
12/8/25 08:52 AM [INF] Successfully removed osd.33 on ceph-osd15
12/8/25 08:52 AM [INF] Removing key for osd.33
12/8/25 08:52 AM [INF] Removing daemon osd.33 from ceph-osd15 -- ports []
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
