Re: [ceph-users] Improving metadata throughput

[Gregory Farnum]

On Wed, Jun 29, 2016 at 2:02 PM, Daniel Davidson
 wrote:
> I am starting to work with and benchmark our ceph cluster.  While throughput
> is so far looking good, metadata performance so far looks to be suffering.
> Is there anything that can be done to speed up the response time of looking
> through a lot of small files and folders?  Right now, I am running four
> metadata servers and the filesystem is mounted via fuse.
>
> We use module to manage environmental variables for the applications on our
> cluster.  When I type "module avail" it takes about 30 minutes to get a
> response the first time with a pair of my monitors running at 100% during
> this time.  Later ones are near instantaneous.

Can you describe your setup and your testing/experience in more detail?
There isn't much reason for your monitors (did you mean MDSes? or the
CPUs, which are colocated with something else?) to be using 100% CPU
as a result of metadata throughput.
I hope the 4 MDSes are 1 active and 3 standbys.
In general, metadata can take a bit of time to be read into the MDS
cache (a RADOS read, ie a disk seek, per directory) but should then be
quick to access, unless it is contending with another client changing
the directories in question (due to cache coherence, that can take
more time). I'm not familiar with Module, but from the man page it
seems to recurse arbitrarily deep once you specify a directory, so if
you've got it searching some large/deep paths, it could take a while.
30 minutes is a *really* long time though, making it sound like you're
colliding with something else to make it slow.
-Greg
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Improving metadata throughput

[Daniel Davidson]

I am starting to work with and benchmark our ceph cluster.  While 
throughput is so far looking good, metadata performance so far looks to 
be suffering.  Is there anything that can be done to speed up the 
response time of looking through a lot of small files and folders?  
Right now, I am running four metadata servers and the filesystem is 
mounted via fuse.


We use module to manage environmental variables for the applications on 
our cluster.  When I type "module avail" it takes about 30 minutes to 
get a response the first time with a pair of my monitors running at 100% 
during this time.  Later ones are near instantaneous.


Dan

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com