On Fri, Apr 3, 2026 at 4:06 PM Anthony D'Atri via ceph-users < [email protected]> wrote:
> >> * Are you looking to change the profile to get more usable capacity? > >> * Are you looking to change the profile to get more durability / > sustain more failures? > > Not necessarily, we would rather have a responsive rgw for example > during failures, so when there is backfill/recovery going on that rgw does > not stop responding. > > The wider profile may actually make that worse. > > Since you mention RGW and only mention HDDs, does that mean that your > index pool is on HDDs? There's a good chance that was a factor. HDDs have > very limited IOPS and all of the RGW pools that aren't bucket data really > should be on SSDs. I speculate that client ops, backfill, and RGW ops were > DoSing each other. > Forgot to add, index pools are on NVMEs. Some old OSD nodes have 2 NVMEe inside, newer ones still do not have, so we will have to consider adding NVMEs also. > An expedient way to add those would be something like > > > https://encrypted-tbn0.gstatic.com/shopping?q=tbn:ANd9GcTRl12SIsjlOJTId1V6ZkkEnaG1Rn11hBhnJoEOmLBdwYuslC1jF9davL_ekuJJ4kPkYtzlsEao-eIQVrM95BOAAeCNJaIvllIQXaDVFPMc > > > A PCI-e AIC that accepts 2x NVMe M.2 drives, assuming that your > motherboard can do PCIe bifurcation. You could populate each with like 2x > 1.9T Micron 7450 and place the RGW pools on those. For this purpose having > more OSDs is more beneficial than having larger OSDs. You could easily see > the size of your existing pools to help gauge what you need. > Thx. > > > It happened a few times. e.g. slow/faulty disk from on which reweight > was just set > > You aren't using "ceph osd reweight" or "ceph osd reweight-by-utilization" > are you? Those are legacies and should be avoided. You want the REWEIGHT > value on each OSD to be 1.0000 and to use the manager balancer. > > ceph config set mgr mgr/balancer/upmap_max_deviation 1 > Actually I am doing `ceph osd crush reweight X ...` when using it. But not for faulty OSDs. On faulty OSDs, I set the OSD to out, replace it, activate it. I am using following procedure: 1. systemctl stop ceph-osd@{id} 2. ceph osd down osd.{id} 3. ceph osd out osd.{id} 4. pocakas rebalance 5. ceph osd destroy {id} --yes-i-really-mean-it --- 6. replace disk... 7. ceph-volume lvm prepare --osd-id {id} --data /dev/sdX 8. ceph-volume lvm list 9. ceph-volume lvm activate {id} {osd fsid} Does that mean you're only using RGW? No RBD, no CephFS? This informs the > process by which you would migrate data between pools. If you don't have > capacity concerns, maybe migration isn't worth the trouble. Only RGW for now. We did test CephFS a little bit, but RGW suffices our needs currently. Rok _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
