>>
>> > we expanded the cluster to 12 nodes with 28 osds
>>
>> 2 or 3 OSDs per node? Are these like Raspberry Pis, or maybe converged
>> (Proxmox?)
> 28 osds per node.
Oh, gotcha. The way you wrote sounded MUCH smaller.
So you have 336 OSDs.
>>
>> > but we are still running on EC 3+2, since we started cluster with 6 nodes.
>>
>> There are ways to fake failure domains (hosts, racks, etc) when you *know*
>> that a cluster will be expanding, albeit with increased risk until it does.
> We use host failure domain for now.
>
>> > We would like to change it to EC 6+3 or even EC 8+3? Is this reasonable or
>> > will 8+3 take too much
>> > CPU/network capacities?
>>
>> That depends on several things.
>>
>> * Are you looking to change the profile to get more usable capacity?
>> * Are you looking to change the profile to get more durability / sustain
>> more failures?
> Not necessarily, we would rather have a responsive rgw for example during
> failures, so when there is backfill/recovery going on that rgw does not stop
> responding.
The wider profile may actually make that worse.
Since you mention RGW and only mention HDDs, does that mean that your index
pool is on HDDs? There's a good chance that was a factor. HDDs have very
limited IOPS and all of the RGW pools that aren't bucket data really should be
on SSDs. I speculate that client ops, backfill, and RGW ops were DoSing each
other.
An expedient way to add those would be something like
https://encrypted-tbn0.gstatic.com/shopping?q=tbn:ANd9GcTRl12SIsjlOJTId1V6ZkkEnaG1Rn11hBhnJoEOmLBdwYuslC1jF9davL_ekuJJ4kPkYtzlsEao-eIQVrM95BOAAeCNJaIvllIQXaDVFPMc
A PCI-e AIC that accepts 2x NVMe M.2 drives, assuming that your motherboard can
do PCIe bifurcation. You could populate each with like 2x 1.9T Micron 7450 and
place the RGW pools on those. For this purpose having more OSDs is more
beneficial than having larger OSDs. You could easily see the size of your
existing pools to help gauge what you need.
> It happened a few times. e.g. slow/faulty disk from on which reweight was
> just set
You aren't using "ceph osd reweight" or "ceph osd reweight-by-utilization" are
you? Those are legacies and should be avoided. You want the REWEIGHT value on
each OSD to be 1.0000 and to use the manager balancer.
ceph config set mgr mgr/balancer/upmap_max_deviation 1
> , was the reason that rgw stopped responding, since some queue filled up ...
> because ceph was unable to read from faulty pgs or smth...
Probably saturation, one of the false economies of HDDs.
>> * What is your hardware like? CPU, drive size/model, network links
>
> There is a mix of:
> - Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz with 20Gb network bonds
Check your bonds' xmit_hash_policy. This often defaults to layer2, but often
should be 2+3 or 3+4. Run iptraf-ng, downarrow to General interface
statistics, and see if TX traffic is roughly balanced.
> - AMD EPYC 9254 24-Core Processor with 50Gb network bonds
>
> with mostly old 6T Seagate ST6000NM0095 drives, but we are just in the
> process rotating them.
>
>> * What kind of data are you storing? RBD volumes? CephFS? Big files? Tiny
>> files?
>
> RGW, S3 users, backup data, nextcloud storage or just storage for placing
> files. Mostly tiny files, however backups with >100GB or even TB files can be
> found also.
Does that mean you're only using RGW? No RBD, no CephFS? This informs the
process by which you would migrate data between pools. If you don't have
capacity concerns, maybe migration isn't worth the trouble.
>
> usage: 775 TiB used, 1.0 PiB / 1.8 PiB avail
>
>> * Which Ceph release are you running?
>>
> Reef 18.2.8
>
> Rok
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
