It is not a crush map thing. What is the PG/OSD ratio? CEPH recommends 100-200 PG (after multiplying the replica number or EC stripe number) per OSD. But even though we also observed about 20-40% differences for PG/OSD distribution. You may try higher PG/OSD ratio but be warned that the messenger system may consume too much system resource.
A workaround is to reweight-by-utilization after the cluster has been filled to certain ratio. It means a lot of data movement and performance penalty to online traffic. From: <Cao>, Buddy <[email protected]<mailto:[email protected]>> Date: Tuesday, July 1, 2014 at 11:52 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: [ceph-users] ceph data replication not even on every osds Hi, I set the same weight for all the hosts, same weight for all the osds under the hosts in crushmap, and set pool replica size to 3. However, after upload 1M/4M/400M/900M files to the pool, I found the data replication is not even on every osds and the utilization for the osds are not the same, they are 25% to 70% respectively. Could you advice, it’s the nature of ceph, or there are some tricky setting in crushmap? Rule r1 { ruleset 0 type replicated min_size 0 max_size 10 step take root step chooseleaf firstn 0 type host step emit } Wei Cao (Buddy)
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
