Hi, There was a bug (sorry don't recall which) that would leave the llog files present and full of logs which should have been cleared, remount was the solution. I don't recall the details here but I'm sure you'd be able to find the LU ticket after some searching.
-cf On Sat, Oct 15, 2016 at 3:49 PM, Pawel Dziekonski <dzi...@wcss.pl> wrote: > Hi, > > we had the same problem on 2.5.3. Robinhood was supposed to > consume changelog but it wasn't. Don't know why. Simply > disabling changelog was not enough - we had to remount MDT. > We did it by simply doing failover to other MDS node (HA > pair). > > The other issue we had with MDT was the size of inodes - > they are (were at that time) created with 512 bytes by > default and when you use the stripe count then it will not > accommodate the lfsck and xattr data on that single inode > and it starts utilizing the disk space. So you have to > create the inodes with proper size, then whole data will be > saved in that same inode and will not occupy additional disk > space. AFAIK, this is a known issue since 2.x. > Unfortunately the only solution was to reformat MDT offline. > > P > > > > > (via failover for > On pią, 14 paź 2016 at 06:46:59 -0400, Jessica Otey wrote: > > All, > > My colleagues in Chile now believe that both of their 2.5.3 file > > systems are experiencing this same problem with the MDTs filling up > > with files. We have also come across a report from another user from > > early 2015 denoting the same issue, also with a 2.5.3 system. > > > > See: https://urldefense.proofpoint.com/v2/url?u=https-3A__www. > mail-2Darchive.com_search-3Fl-3Dlustre-2Ddiscuss-40lists. > lustre.org-26q-3Dsubject-3A-2522Re-255C-253A-2B-255C- > 255Blustre-255C-2Ddiscuss-255C-255D-2BMDT-2Bpartition- > 2Bgetting-2Bfull-2522-26o-3Dnewest&d=DQIGaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r= > x9pM59OqndbWw-lPPdr8w1Vud29EZigcxcNkz0uw5oQ&m=hrnhInO0YKCrI7g- > bxkr6YelhXQXywcc8cF4G8x3rR4&s=ulbEYTQ5HJrTLe_tNQ3LdH_Ylc_ > qwxZOWD3MlN_a5Bc&e= > > > > We are confident that these files are not related to the changelog > feature. > > > > Does anyone have any other suggestions as to what the cause of this > > problem could be? > > > > I'm intrigued that the Lustre version involved in all 3 reports is > > 2.5.3. Could this be a bug? > > > > Thanks, > > Jessica > > > > > > >On Thu, Sep 29, 2016 at 8:58 AM, Jessica Otey <jo...@nrao.edu > > ><mailto:jo...@nrao.edu>> wrote: > > > > > > Hello all, > > > I write on behalf of my colleagues in Chile, who are experiencing > > > a bizarre problem with their MDT, namely, it is filling up with 4 > > > MB files. There is no issue with the number of inodes, of which > > > there are hundreds of millions unused.  > > > > > > [root@jaopost-mds ~]# tune2fs -l /dev/sdb2 | grep -i inode > > > device /dev/sdb2 mounted by lustre > > > Filesystem features:    has_journal ext_attr resize_inode > > > dir_index filetype needs_recovery flex_bg dirdata sparse_super > > > large_file huge_file uninit_bg dir_nlink quota > > > Inode count:        239730688 > > > Free inodes:        223553405 > > > Inodes per group:     32768 > > > Inode blocks per group:  4096 > > > First inode:        11 > > > Inode size:     512 > > > Journal inode:       8 > > > Journal backup:      inode blocks > > > User quota inode:     3 > > > Group quota inode:     4 > > > > > > Has anyone ever encountered such a problem? The only thing unusual > > > about this cluster is that it is using 2.5.3 MDS/OSSes while still > > > using 1.8.9 clients—something I didn't actually believe was > > > possible, as I thought the last version to work effectively with > > > 1.8.9 clients was 2.4.3. However, for all I know, the version gap > > > may have nothing to do with this phenomena. > > > > > > Any and all advice is appreciated. Any general information on the > > > structure of the MDT also welcome, as such info is in short supply > > > on the internet. > > > > > > Thanks, > > > Jessica > > > > > -- > Pawel Dziekonski <pawel.dziekon...@wcss.pl> > Wroclaw Centre for Networking & Supercomputing, HPC Department > phone: +48 71 320 37 39, fax: +48 71 322 57 97, > https://urldefense.proofpoint.com/v2/url?u=http-3A__www. > wcss.pl&d=DQIGaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=x9pM59OqndbWw- > lPPdr8w1Vud29EZigcxcNkz0uw5oQ&m=hrnhInO0YKCrI7g- > bxkr6YelhXQXywcc8cF4G8x3rR4&s=Ob17Q3c2GcDNu-2FIvqyDwKxVQ- > u7xeBzVTMo9we3XI&e= > _______________________________________________ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists. > lustre.org_listinfo.cgi_lustre-2Ddiscuss-2Dlustre.org& > d=DQIGaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=x9pM59OqndbWw- > lPPdr8w1Vud29EZigcxcNkz0uw5oQ&m=hrnhInO0YKCrI7g- > bxkr6YelhXQXywcc8cF4G8x3rR4&s=eHOc6uHZuJVxo_TiHizVTzK4FmtxLwtp0jQjbKWFGK8& > e= >
_______________________________________________ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org