All,
My colleagues in Chile now believe that both of their 2.5.3 file systems are experiencing this same problem with the MDTs filling up with files. We have also come across a report from another user from early 2015 denoting the same issue, also with a 2.5.3 system.


See: https://www.mail-archive.com/search?l=lustre-discuss@lists.lustre.org&q=subject:%22Re%5C%3A+%5C%5Blustre%5C-discuss%5C%5D+MDT+partition+getting+full%22&o=newest

We are confident that these files are not related to the changelog feature.

Does anyone have any other suggestions as to what the cause of this problem could be?

I'm intrigued that the Lustre version involved in all 3 reports is 2.5.3. Could this be a bug?

Thanks,
Jessica


On Thu, Sep 29, 2016 at 8:58 AM, Jessica Otey <jo...@nrao.edu <mailto:jo...@nrao.edu>> wrote:

    Hello all,
    I write on behalf of my colleagues in Chile, who are experiencing
    a bizarre problem with their MDT, namely, it is filling up with 4
    MB files. There is no issue with the number of inodes, of which
    there are hundreds of millions unused. Â

    [root@jaopost-mds ~]# tune2fs -l /dev/sdb2 | grep -i inode
    device /dev/sdb2 mounted by lustre
    Filesystem features: Â  Â  Â has_journal ext_attr resize_inode
    dir_index filetype needs_recovery flex_bg dirdata sparse_super
    large_file huge_file uninit_bg dir_nlink quota
    Inode count: Â  Â  Â  Â  Â  Â  Â 239730688
    Free inodes: Â  Â  Â  Â  Â  Â  Â 223553405
    Inodes per group: Â  Â  Â  Â  32768
    Inode blocks per group: Â  4096
    First inode: Â  Â  Â  Â  Â  Â  Â 11
    Inode size:Â  Â  Â  Â  Â 512
    Journal inode: Â  Â  Â  Â  Â  Â 8
    Journal backup: Â  Â  Â  Â  Â  inode blocks
    User quota inode: Â  Â  Â  Â  3
    Group quota inode: Â  Â  Â  Â 4

    Has anyone ever encountered such a problem? The only thing unusual
    about this cluster is that it is using 2.5.3 MDS/OSSes while still
    using 1.8.9 clients—something I didn't actually believe was
    possible, as I thought the last version to work effectively with
    1.8.9 clients was 2.4.3. However, for all I know, the version gap
    may have nothing to do with this phenomena.

    Any and all advice is appreciated. Any general information on the
    structure of the MDT also welcome, as such info is in short supply
    on the internet.

    Thanks,
    Jessica

    Below is a look inside the O folder at the root of the MDT, where
    there are about 48,000 4MB files:

    [root@jaopost-mds O]# pwd
    /lustrebackup/O
    [root@jaopost-mds O]# tree -L 1
    .
    ├── 1
    ├── 10
    └── 200000003

    3 directories, 0 files

    [root@jaopost-mds O]# ls -l 1
    total 2240
    drwx------ 2 root root 69632 sep 16 16:25 d0
    drwx------ 2 root root 69632 sep 16 16:25 d1
    drwx------ 2 root root 61440 sep 16 17:46 d10
    drwx------ 2 root root 69632 sep 16 17:46 d11
    drwx------ 2 root root 69632 sep 16 18:04 d12
    drwx------ 2 root root 65536 sep 16 18:04 d13
    drwx------ 2 root root 65536 sep 16 18:04 d14
    drwx------ 2 root root 69632 sep 16 18:04 d15
    drwx------ 2 root root 61440 sep 16 18:04 d16
    drwx------ 2 root root 61440 sep 16 18:04 d17
    drwx------ 2 root root 69632 sep 16 18:04 d18
    drwx------ 2 root root 69632 sep 16 18:04 d19
    drwx------ 2 root root 65536 sep 16 16:25 d2
    drwx------ 2 root root 69632 sep 16 18:04 d20
    drwx------ 2 root root 69632 sep 16 18:04 d21
    drwx------ 2 root root 61440 sep 16 18:04 d22
    drwx------ 2 root root 69632 sep 16 18:04 d23
    drwx------ 2 root root 61440 sep 16 16:11 d24
    drwx------ 2 root root 69632 sep 16 16:11 d25
    drwx------ 2 root root 69632 sep 16 16:11 d26
    drwx------ 2 root root 69632 sep 16 16:11 d27
    drwx------ 2 root root 69632 sep 16 16:25 d28
    drwx------ 2 root root 69632 sep 16 16:25 d29
    drwx------ 2 root root 69632 sep 16 16:25 d3
    drwx------ 2 root root 65536 sep 16 16:25 d30
    drwx------ 2 root root 65536 sep 16 16:25 d31
    drwx------ 2 root root 69632 sep 16 16:25 d4
    drwx------ 2 root root 61440 sep 16 16:25 d5
    drwx------ 2 root root 69632 sep 16 16:25 d6
    drwx------ 2 root root 73728 sep 16 16:25 d7
    drwx------ 2 root root 65536 sep 16 17:46 d8
    drwx------ 2 root root 69632 sep 16 17:46 d9
    -rw-r--r-- 1 root root     8 ene  4  2016 LAST_ID

    [root@jaopost-mds d0]# ls -ltr | more
    total 5865240
    -rw-r--r-- 1 root root  252544 ene  4  2016 32
    -rw-r--r-- 1 root root 2396224 ene  9  2016 2720
    -rw-r--r-- 1 root root 4153280 ene  9  2016 2752
    -rw-r--r-- 1 root root 4153280 ene 10  2016 2784
    -rw-r--r-- 1 root root 4153280 ene 10  2016 2816
    -rw-r--r-- 1 root root 4153280 ene 10  2016 2848
    -rw-r--r-- 1 root root 4153280 ene 10  2016 2880
    -rw-r--r-- 1 root root 4153280 ene 10  2016 2944
    -rw-r--r-- 1 root root 4153280 ene 10  2016 2976
    -rw-r--r-- 1 root root 4153280 ene 10  2016 3008
    -rw-r--r-- 1 root root 4153280 ene 10  2016 3040
    -rw-r--r-- 1 root root 4153280 ene 10  2016 3072
    -rw-r--r-- 1 root root 4153280 ene 10  2016 3104
    -rw-r--r-- 1 root root 4153280 ene 10  2016 3136
    -rw-r--r-- 1 root root 4153280 ene 10  2016 3168
    -rw-r--r-- 1 root root 4153280 ene 10  2016 3200
    -rw-r--r-- 1 root root 4153280 ene 10  2016 3232
    -rw-r--r-- 1 root root 4153280 ene 10  2016 3264
    -rw-r--r-- 1 root root 4153280 ene 10  2016 3296
    -rw-r--r-- 1 root root 4153280 ene 10  2016 3328



    _______________________________________________
    lustre-discuss mailing list
    lustre-discuss@lists.lustre.org
    <mailto:lustre-discuss@lists.lustre.org>
    http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
    <http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org>





_______________________________________________
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

_______________________________________________
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Reply via email to