-----Original Message----- From: Yan, Zheng [mailto:[email protected]] Sent: Wednesday, April 16, 2014 7:44 PM To: Qing Zheng Cc: [email protected] Subject: Re: [ceph-users] multi-mds and directory sharding
On Thu, Apr 17, 2014 at 6:21 AM, Qing Zheng <[email protected]> wrote: > I seems that with kernel 3.14 and latest source code from github, we > still run into troubles when testing multi-mds and directory sharding. > what's problem you encountered? Hi Zheng, We are using mdtest to simulate workloads where multiple parallel client procs will keep inserting empty files into a single newly created directory. We are expecting CephFS to balance its metadata servers, and eventually all metadata servers will get a share of the directory. We currently found that CephFS was only able to run for the first 5-10 mins under such workload, and then stopped making progress -- the clients' "NewFile" calls would no longer return. >From the client point of view, it was like the server was no longer processing any requests. Our test deployment had 32 osds, 8 mds, and 1 mon. CephFS was kernel mounted. Clients were collated with metadata servers. Cheers, -- Qing Zheng > Are there any limits either in the max number of active metadata > servers that we could possibly run or in the number of directory > entries that we could set for Ceph to trigger a directory split? > > Is it okay to run 128 or more active metadata servers, for example? > Is it okay to let Ceph split directories once a directory has > accumulated > 200 entries? > > Cheers, > > -- Qing Zheng > > -----Original Message----- > From: Yan, Zheng [mailto:[email protected]] > Sent: Sunday, April 13, 2014 6:43 PM > To: Qing Zheng > Cc: [email protected] > Subject: Re: [ceph-users] multi-mds and directory sharding > > On Mon, Apr 14, 2014 at 2:54 AM, Qing Zheng <[email protected]> wrote: >> Hi - >> >> We are currently evaluating CephFS's metadata scalability and >> performance. One important feature of CephFS is its support for >> running multiple "active" mds instances and partitioning huge >> directories into small shards. >> >> We use mdtest to simulate workloads where multiple parallel client >> processes will keep inserting empty files into several large directories. >> We found that CephFS is only able to run for the first 5-10 mins, and >> then stop making progress -- the clients' "creat" call no longer return. >> >> We were using Ceph 0.72 and Ubuntu 12.10 with kernel 3.6.6. >> Our setup consisted of 8 osds, 3 mds, and 1 mon. All mds were active, >> instead of standby, and they were all configured to split directories >> once the directory size is greater than 2k. We kernel (not fuse) >> mounted CephFS on all 8 osd nodes. > > 3.6 kernel is too old for cephfs. please use kernel compiled from > testing branch https://github.com/ceph/ceph-client and the newest > development version of Ceph. There are large number of fixes for > directory fragment and multimds. > > Regards > Yan, Zheng > >> >> To test CephFS, we launched 64 client processes on 8 osd nodes (8 >> procs per osd). Each client would create 1 directory and then insert >> 5k empty files into that directory. In total 64 directories and 320k >> files would be created. CephFS gave an avg throughput of 300~1k for >> the first 5 minutes, and then stopped making any progress. >> >> What might go wrong? >> >> If each client insert 200 files, instead of 5k, then CephFS could >> finish the workload with 1.5K ops/s. If each client insert 1k files, >> then ~500 ops/s If 2k files (the split threshold), then ~400 ops/s >> >> Are these numbers reasonable? >> >> -- Qing >> >> _______________________________________________ >> ceph-users mailing list >> [email protected] >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
