Jason, If there are files located on the full OST that you can delete then delete them without trying to deactivate the OST. There is a process where you can find files solely located on the full OST using lfs then deactivate the full OST, copy the files to new files with some suffix (.new perhaps) so they end up on other OSTs, delete the original files, rename the new files to the old file names (thereby removing the suffix used previously) and then reactivating the deactivated OST. You might be lucky enough to find a few very large single stripe files located on that OST where moving the file gets you easy gains.
You *should* be able to delete a file that resides on an inactive OST but I’m not over a keyboard where I can do it before making the assertion it is possible. —Jeff On Sat, Jan 5, 2019 at 18:49 Jason Williams <jas...@jhu.edu> wrote: > Hello, > > > We have a lustre system (version 2.10.4) that has unfortunately fallen > victim to a 100% full OST... Every time we clear some space on it, the > system fills it right back up again. > > > I have looked around the internet and found you can disable an OST, but > when I have tried that, any writes (including deletes) to the OST hang the > clients indefinitely. Does anyone know a way to make an OST basically > "read-only" with the exception of deletes so we can work to clear out the > OST? Or better yet, a way to "drain" or move files off an OST with a > script (keeping in mind it might not be known if the files are in use at > the time). Or even a way to tell lustre "Hey don't write any new data > here, but reading and removing data is OK." > > > > > -- > Jason Williams > Assistant Director > Systems and Data Center Operations. > Maryland Advanced Research Computing Center (MARCC) > Johns Hopkins University > jas...@jhu.edu > > _______________________________________________ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > -- ------------------------------ Jeff Johnson Co-Founder Aeon Computing jeff.john...@aeoncomputing.com www.aeoncomputing.com t: 858-412-3810 x1001 f: 858-412-3845 m: 619-204-9061 4170 Morena Boulevard, Suite C - San Diego, CA 92117 High-Performance Computing / Lustre Filesystems / Scale-out Storage
_______________________________________________ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org