Thanks all for the suggestions. Having our metadata NSDs fill up was what prompted this exercise, but space was previously feed up on those by switching them from metadata+data to metadataOnly and using a policy to migrate files out of that pool. So these now have about 30% free space (more if you include fragmented space). The restripe attempt is just to make a final move of any remaining data off those devices. All the NSDs now have free space on them.
df -i shows inode usage at about 84%, so plenty of free inodes for the filesystem as a whole. We did have old .quota files laying around but removing them didn't have any impact. mmlsfileset fs -L -i is taking a while to complete, I'll let it simmer while getting to work. mmrepquota does show about a half-dozen filesets that have hit their quota for space (we don't set quotas on inodes). Once I'm settled in this morning I'll try giving them a little extra space and see what happens. jbh On Thu, Nov 2, 2017 at 4:19 AM, Oesterlin, Robert < [email protected]> wrote: > One thing that I’ve run into before is that on older file systems you had > the “*.quota” files in the file system root. If you upgraded the file > system to a newer version (so these files aren’t used) - There was a bug at > one time where these didn’t get properly migrated during a restripe. > Solution was to just remove them > > > > > > Bob Oesterlin > > Sr Principal Storage Engineer, Nuance > > > > *From: *<[email protected]> on behalf of John > Hanks <[email protected]> > *Reply-To: *gpfsug main discussion list <[email protected]> > *Date: *Wednesday, November 1, 2017 at 5:55 PM > *To: *gpfsug <[email protected]> > *Subject: *[EXTERNAL] [gpfsug-discuss] mmrestripefs "No space left on > device" > > > > Hi all, > > > > I'm trying to do a restripe after setting some nsds to metadataOnly and I > keep running into this error: > > > > Scanning user file metadata ... > > 0.01 % complete on Wed Nov 1 15:36:01 2017 ( 40960 inodes with > total 531689 MB data processed) > > Error processing user file metadata. > > Check file '/var/mmfs/tmp/gsfs0.pit.interestingInodes.12888779708' on > scg-gs0 for inodes with broken disk addresses or failures. > > mmrestripefs: Command failed. Examine previous error messages to determine > cause. > > > > The file it points to says: > > > > This inode list was generated in the Parallel Inode Traverse on Wed Nov 1 > 15:36:06 2017 > > INODE_NUMBER DUMMY_INFO SNAPSHOT_ID ISGLOBAL_SNAPSHOT INDEPENDENT_FSETID > MEMO(INODE_FLAGS FILE_TYPE [ERROR]) > > 53504 0:0 0 1 0 > illreplicated REGULAR_FILE RESERVED Error: 28 No space left on device > > > > > > /var on the node I am running this on has > 128 GB free, all the NSDs have > plenty of free space, the filesystem being restriped has plenty of free > space and if I watch the node while running this no filesystem on it even > starts to get full. Could someone tell me where mmrestripefs is attempting > to write and/or how to point it at a different location? > > > > Thanks, > > > > jbh >
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
