FWIW, last September in the context of reducing the memory usage on the
mgs, Andreas Dilger had this to say
"ls" is itself fairly inefficient at directory traversal, because all of
the GNU file utilities are bloated and do much more work than necessary
(e.g. "rm" will stat() every file before unlinking it). Using something
like:
find $dir -type f -print0 | xargs -0 munlink
find $dir -type d -print0 | xargs rmdir
will probably be most efficient. "munlink" is a Lustre-specific tool that
just unlinks the filenames passed without stat() on each one. If that is
not available, you can also use "xargs -0 -n 1 unlink" but it will fork a
new unlink process for every file which also adds some overhead.
bob
On 7/10/2015 7:24 PM, Sreedhar wrote:
Hi,
Just like Malcolm suggested, before I implemented Robinhood, I built
the list and removed files (that were not accessed in more than 31
days). I found it much faster than all other methods.
/usr/bin/lfs find /scratch/${folder} -A +31 -p -t f | xargs -0 -n 10
-P 8 /bin/rm
10 and 8 for flags n and p worked well for me on our system.
Sreedhar.
New York University.
On Friday, July 10, 2015, Cowe, Malcolm J <malcolm.j.c...@intel.com
<javascript:_e(%7B%7D,'cvml','malcolm.j.c...@intel.com');>> wrote:
There’s something in the rm command that makes recursive deletes
really expensive, although I don’t know why. I’ve found in the
past that even running a regular find ... –exec rm {} \; has been
quicker. Running lfs find to build the file list would presumably
be quicker still.
Malcolm.
*From:*lustre-discuss
[mailto:lustre-discuss-boun...@lists.lustre.org] *On Behalf Of
*Andrus, Brian Contractor
*Sent:* Saturday, July 11, 2015 8:05 AM
*To:* lustre-discuss@lists.lustre.org
*Subject:* [lustre-discuss] Removing large directory tree
All,
I understand that doing recursive file operations can be taxing on
lustre.
So, I wonder if there is a preferred performance-minded way to
remove an entire directory tree that is several TB in size.
The standard rm -rf ./dir seems to spike the cpu usage on my OSSes
where it sits and sometimes causes clients to be evicted.
Brian Andrus
ITACS/Research Computing
Naval Postgraduate School
Monterey, California
voice: 831-656-6238
_______________________________________________
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
--
Sreedhar Manchu
HPC Systems Administrator
eSystems & Research Services
New York University, New York 10012
*http://unixoperator.blogspot.com*
*https://wikis.nyu.edu/display/~sm4082
<https://wikis.nyu.edu/display/%7Esm4082>*
_______________________________________________
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
_______________________________________________
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
