Hi all, We're planning a migration from conventional CRUSH rules (chooseleaf firstn/indep) to MSR rules (choosemsr) on large scale production clusters (thousands of OSDs). We've validated the procedure on a test cluster using pg_upmap pinning to avoid data movement during the rule change, then releasing pins in batches to control backfill.
I'd appreciate hearing from anyone who has done this at scale. A few specific areas we're trying to get more data on: 1. Peering storm on rule change — changing a pool's crush_rule triggers re-peering for pretty much every PG in that pool. How long did peering take to settle on your clusters, and was client I/O meaningfully impacted? 2. OSDMap bloat from massive pg_upmap — did you hit any issues with OSD map distribution lag, OSDs falling behind on epochs, or MON store growth? Did you batch the upmap application, and if so, what batch size worked well? 3. pg_upmap apply rate — how fast can ceph osd pg-upmap commands be applied in practice at scale? Did you see any MON bottlenecks when applying tens of thousands of upmaps in rapid succession? 4. Pin release strategy — When releasing pg_upmap pins to allow data movement to the new MSR placement, what batch size and pacing worked for you? 5. Anything else we should watch for? For context, in our testing on a Squid cluster, we confirmed that pg_upmap_items does NOT survive a rule change (they are pruned from the OSDMap), so we're using pg_upmap (full acting set override). Thanks in advance for any data points or stories! -Jane _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
