On 27/04/26 16:46, Anthony D'Atri wrote:
Hi to all!
I'm searching the wisdom of the community on how to drain/fill an OSD server
(like for removal, bulk disk replacing, filestore to bluestore conversion...)
without impacting on the performance of the cluster.
You'll impact performance no matter what you do, but to differing degrees.
The least impact is making the least change:
set noout
destroy one OSD
redeploy that OSD
unset noout
wait for recovery
At scale that doesn't, well, scale.
We are using the 'ceph-gentle-reweight' script, which is running fine, but we
are interested to know if there is a newer approach for this job.
What Ceph release are you running? That approach works, but will take a long
time and you need to have enough free capacity on other nodes to hold that
node's share of data.
We are running Quincy (on our upgrade path to Squid/Tentacle) and we
have enough free space to drain at least one node at the time. We used
successfully the gentle-reweight a lot of times, but we are interested
to know if there are other, more efficient, ways to drain/fill a node
(or group of nodes).
Searching the web I found a pdf of the talk of Dan van der Ster about "Mastering
Ceph Operations" (NYC Ceph Day 2024) in which he talks about using the 'balancer' in
'upmap' mode
If you're running, say, Mimic or later, you typically would already have the
balancer on; upmap mode is the default.
Run
ceph balancer status
When using upmaps, you want to avoid "ceph osd reweight-by-utilization" and ensure that the
REWEIGHT column of "ceph osd df" or "ceph osd tree" is all 1.0000
Sure, balancer is already active (in upmap mode) and is doing the job of
balancing :-)
(and a "magic" script) to move the PGs from/to the OSDs in the background.
So the balancer upmap approach can be used to drain a node following this
procedure:
Be sure to disable backfill and rebalancing, you want no backfilling in
progress when you run upmap-remapped.py
I realized (after I sent the email) that you need to have
'norebalancing' active before doinng the upmap-remapped.py, but I didn't
thought about 'nobackfilling', so thank you!
- set balancer tunables (like 'target_max_misplaced_ratio')
- activate the balancer in upmap mode (if not already active)
- 'ceph osd crush reweight-subtree' to 0.1 for the host we wish to drain
- run the magic script 'upmap-remapped.py'
- 'ceph osd crush reweight-subtree' to 0 for the host we wish to drain
- do stuff on the node (remove, reinstall etc etc).
If you are solving for safety over time, sure. Especially if you have
replicated size=2 pools or EC N+1.
Another approach is to set the flags, destroy the OSDs on a single host,
redeploy them, set your backfill/recovery throttles down to 1, and unset the
flags. This will be somewhat more risky as you're reducing redundancy until
the recovery completes, and more will happen in parallel. Whether you can
tolerate that depends in part on your hardware, CRUSH topology, how many hosts,
replication vs EC, etc.
Our first approach was prioritize time over safety, but because we
overran our maintenance window we switched to safety over time (all the
pools in this cluster are replica 3).
But what is the implication for the whole cluster, how this background job
affects its performance?
We need more info.
I assume that 'target_max_misplaced_ratio' is a key parameter to avoid large
data movement in a given time, there is more to look into?
Yes. Setting that lower than default can help avoid certain intermediate
states that can result in a fairly-full cluster's OSDs going
nearfull/backfillfull.
The documentation say about 'target_max_misplaced_ratio' "...the
balancer will incrementally move a small fraction of unbalanced PGs in
order to improve distribution. This fraction will not exceed a certain
threshold that defaults to 5%...". does it mean that off all the PGs
that has to be moved out/in of the node(s), maximum 5% will be moved at
the same time?
Thanks for your help!
Iztok
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
