> we expanded the cluster to 12 nodes with 28 osds
2 or 3 OSDs per node? Are these like Raspberry Pis, or maybe converged (Proxmox?) > but we are still running on EC 3+2, since we started cluster with 6 nodes. There are ways to fake failure domains (hosts, racks, etc) when you *know* that a cluster will be expanding, albeit with increased risk until it does. > We would like to change it to EC 6+3 or even EC 8+3? Is this reasonable or > will 8+3 take too much > CPU/network capacities? That depends on several things. * Are you looking to change the profile to get more usable capacity? * Are you looking to change the profile to get more durability / sustain more failures? * What is your hardware like? CPU, drive size/model, network links * What kind of data are you storing? RBD volumes? CephFS? Big files? Tiny files? * Which Ceph release are you running? > Should I use isa/isa-l instead of jerasure? For new pools I would suggest that. The two are very similar, but ISA is maintained while jerasure is abandonware. > Can anyone share experience with 8+3? It goes to 11 ;) That said, props for having k+m < the number of failure domains. This gives you better RAS, and lets the size of those failure domains vary (somewhat) without stranding capacity. Having k+m=failure domains is a very common ... well I won't go so far as to call it a mistake, but let's say suboptimal strategy. I helped someone with 10 dense HDD nodes running 8+2. Adding one more (and later more) node made a big difference. If you are storing mostly very small files/ RGW objects, 8+3 will "waste" a high percentage of underlying storage. https://markhpc.github.io/2019/11/21/BlueFS-Space-Amplification.html If there are a few small files mixed in with much bigger ones, that becomes less of a concern. This space amplification drove the reduction of the default min_alloc_size from 64KB to 4KB in the O/P era. If you are running Tentacle and the pool has the ec optimization flag turned on, this space amp will be much reduced since RADOS will no longer add null padding. A wide EC profile will make scrubs and recovery slower due to the gridlock effect -- scrubs and recovery rely on reservations, and there will be more collision and thus queueing. A wide EC profile will result in slower writes. Your media and workload are factors in whether or not that's a problem for you. Whether your data is on CephFS, RGW, and/or RBD informs the actual process for moving. Please share more detail. There has been talk of being able to do this transparently with Umbrella, but it's hearsay until it's merged. > > Kind regards, > Rok > _______________________________________________ > ceph-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
