On Thu, Sep 15, 2016 at 2:20 PM, Burkhard Linke
<burkhard.li...@computational.bio.uni-giessen.de> wrote:
> Hi,
>
> does CephFS impose an upper limit on the number of files in a directory?
>
>
> We currently have one directory with a large number of subdirectories:
>
> $ ls | wc -l
> 158141
>
> Creating a new subdirectory fails:
>
> $ touch foo
> touch: cannot touch 'foo': No space left on device

This limit was added recently: it's a limit on the size of a directory fragment.

Previously folks were hitting nasty OSD issues with very large
directory fragments, so we added this limit to give a clean failure
instead.

Directory fragmentation (mds_bal_frag setting) is turned off by
default in Jewel: I was planning to get this activated by default in
Kraken, but haven't quite got there yet.  Once fragmentation is
enabled you should find that the threshold for splitting dirfrags is
hit well before you hit the safety limit that gives you ENOSPC.

Note that if you set mds_bal_frag then you also need to use the "ceph
fs set <myfs> allow_dirfrags true" (that command from memory so check
the help if it's wrong), or the MDSs will ignore the setting.

John

>
> Creating files in a different directory does not show any problems. The last
> message in the MDS log relates to the large directory:
>
> 2016-09-15 07:51:54.539216 7f24ef2a6700  0 mds.0.bal replicating dir [dir
> 100008bf9a6 /volumes/biodb/ncbi_genomes/all/ [2,head] auth v=57751905 cv=0/0
> ap=0+2+2 state=1073741826|complete f(v0 m2016-08-22 08:51:34.714570
> 158141=3+158138) n(v182285 rc2016-08-22 11:59:34.976373 b3360569421156
> 2989670=2478235+511435) hs=158141+842,ss=0+0 | child=1 waiter=0 authpin=0
> 0x7f252ca05f00] pop 12842 .. rdp 7353.96 adj 0
>
> Any hints what might go wrong in this case? MDS is taken from the current
> jewel git branch due to some pending backports:
>
> # ceph-mds --version
> ceph version 10.2.2-508-g9bfc0cf (9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b)
>
> CephFS is mounted via kernel implementation:
>
> # uname -a
> Linux waas 4.6.6-040606-generic #201608100733 SMP Wed Aug 10 11:35:29 UTC
> 2016 x86_64 x86_64 x86_64 GNU/Linux
>
> ceph-fuse from jewel is also affected.
>
>
> Regards,
> Burkhard
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Reply via email to