I seems that with kernel 3.14 and latest source code from github, we still run into troubles when testing multi-mds and directory sharding.
Are there any limits either in the max number of active metadata servers that we could possibly run or in the number of directory entries that we could set for Ceph to trigger a directory split? Is it okay to run 128 or more active metadata servers, for example? Is it okay to let Ceph split directories once a directory has accumulated 200 entries? Cheers, -- Qing Zheng -----Original Message----- From: Yan, Zheng [mailto:[email protected]] Sent: Sunday, April 13, 2014 6:43 PM To: Qing Zheng Cc: [email protected] Subject: Re: [ceph-users] multi-mds and directory sharding On Mon, Apr 14, 2014 at 2:54 AM, Qing Zheng <[email protected]> wrote: > Hi - > > We are currently evaluating CephFS's metadata scalability and > performance. One important feature of CephFS is its support for > running multiple "active" mds instances and partitioning huge > directories into small shards. > > We use mdtest to simulate workloads where multiple parallel client > processes will keep inserting empty files into several large directories. > We found that CephFS is only able to run for the first 5-10 mins, and > then stop making progress -- the clients' "creat" call no longer return. > > We were using Ceph 0.72 and Ubuntu 12.10 with kernel 3.6.6. > Our setup consisted of 8 osds, 3 mds, and 1 mon. All mds were active, > instead of standby, and they were all configured to split directories > once the directory size is greater than 2k. We kernel (not fuse) > mounted CephFS on all 8 osd nodes. 3.6 kernel is too old for cephfs. please use kernel compiled from testing branch https://github.com/ceph/ceph-client and the newest development version of Ceph. There are large number of fixes for directory fragment and multimds. Regards Yan, Zheng > > To test CephFS, we launched 64 client processes on 8 osd nodes (8 > procs per osd). Each client would create 1 directory and then insert > 5k empty files into that directory. In total 64 directories and 320k > files would be created. CephFS gave an avg throughput of 300~1k for > the first 5 minutes, and then stopped making any progress. > > What might go wrong? > > If each client insert 200 files, instead of 5k, then CephFS could > finish the workload with 1.5K ops/s. If each client insert 1k files, > then ~500 ops/s If 2k files (the split threshold), then ~400 ops/s > > Are these numbers reasonable? > > -- Qing > > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
