We've implemented file count quotas in addition to our existing byte quotas to try to avoid this situation. You can improve some things (metadata on SSDs, maybe get an accelerator node if Isilon still offers those) but the fact is that metadata is expensive in terms of CPU (both client and server) and disk.
We chose 1 million objects/TB of allocated disk space. We sort of compete with a storage system offered by our central IT organization, and picked a limit higher than what they would provide. To be honest, though, we're retiring our Isilon systems because the performance/scalability/cost ratios just aren't as great as they used to be. Our new storage is GPFS and mmbackup works much better with huge number of files, though it's still not great. In particular, the filelist generation is based around UNIX sort which is definitely a memory pig, though it can be split across multiple systems so can scale out pretty well. On Thu, Jul 05, 2018 at 02:52:27PM -0400, Zoltan Forray wrote: > As I have mentioned in the past, we have gone through large migrations to > DFS based storage on EMC ISILON hardware. As you may recall, we backup > these DFS mounts (about 90 at last count) using multiple Windows servers > that run multiple ISP nodes (about 30-each) and they access each DFS > mount/filesystem via -object=\\rams.adp.vcu.edu\departmentname. > > This has lead to lots of performance issue with backups and some > departments are now complain that their backups are running into > multiple-days in some cases. > > One such case in a department with 2-nodes with over 30-million objects for > each node. In the past, their backups were able to finish quicker since > they were accessed via dedicated servers and were able to use Journaling to > reduce the scan times. Unless things have changed, I believe Journling is > not an option due to how the files are accessed. > > FWIW, average backups are usually <50k files and <200GB once it finished > scanning..... > > Also, the idea of HSM/SPACEMANAGEMENT has reared its ugly head since many > of these objects haven't been accessed in many years old. But as I > understand it, that won't work either given our current configuration. > > Given the current DFS configuration (previously CIFS), what can we do to > improve backup performance? > > So, any-and-all ideas are up for discussion. There is even discussion on > replacing ISP/TSM due to these issues/limitations. > > -- > *Zoltan Forray* > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator > Xymon Monitor Administrator > VMware Administrator > Virginia Commonwealth University > UCC/Office of Technology Services > www.ucc.vcu.edu > zfor...@vcu.edu - 804-828-4807 > Don't be a phishing victim - VCU and other reputable organizations will > never use email to request that you reply with your password, social > security number or confidential personal information. For more details > visit http://phishing.vcu.edu/ -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine
