Thanks, Sage. I will take a look at osdmaptool.
Thanks. Zhi Zhang (David) ---------------------------------------- > Date: Tue, 22 Sep 2015 11:41:24 -0700 > From: [email protected] > To: [email protected] > CC: [email protected] > Subject: Re: use crushtool to simulate pg distribution for a specific pool > > Hi Zhang, > > On Tue, 22 Sep 2015, Z Zhang wrote: >> Hi ceph-devel, >> >> We enhanced the crushtool to simulate real pg distribution for a >> specific pool. Recently we had encountered pg uneven issue again after >> expanding our cluster, although we had re-weighted the cluster to make >> pg evenly-distribution at the time we built up the cluster. >> >> It could be painful to re-weighted osds after expanding cluster because >> in order to achieve pg evenly-distribution, we may need to re-weight >> couple of times and each time will trigger data movement. >> >> Now we could use this crushtool to simulate pg distribution against >> crush map, re-weight osds and test the crush map again and again until >> we satisfy with pg distribution. Then we could set back the final crush >> map and trigger data movement for one time. >> >> Please check this PR: https://github.com/ceph/ceph/pull/6004 if your >> guys are interested. > > This looks like it will work, but I'm not sure it's the right place to > put it. The osdmaptool has a --test-map-pgs option that maps all PGs and > gives you the osd distribution and pg min/max per OSD. The procedure is > then slightly different: > > ceph osd getmap -o om > osdmaptool om --export-crush cm > repeat: > adjust crush map cm... > osdmaptool om --import-crush cm --test-map-pgs > ceph osd setcrushmap -i cm > > ...but it will keep the details of ceph pools out of crushtool (where they > probably don't belong). > > What do you think? > sage
