Hi Wanda, This is a tough one, probably no "one fell swoop" approach to this.
Some things to consider: - What is the nature of these files that are created every day? Are they of a "temporary" nature so they can be deleted after a period of time? If not, once created, how often are they referenced again? Maybe they are a candidate for archive (and delete after successful archive)? * Is a standard TSM file backup sufficient to meet this applications needs should the files need to be restored? * For that matter, what are the restore requirements? Even if backup was not an issue, can they all be restored in the event of a disk failure or other event that requires a full restore? * One approach could go something like this: - Use two nodes: one node to back up the rest of the system, the second node to back up this application directory. - For the rest of the system, use standard incremental backup.Use EXCLUDE.DIR to skip the application directory. - For the application directory, exclude all directories and files except for the application directory. Then use incremental by date (-incrbydate option) to back up the application directory. I'm sure you are well aware of the caveats associated with incremental by date, but if these files are created once, then never updated again, it might be a reasonable approach. Caveat: if the files are constantly created even during the backup operation, then -incrbydate might not be viable since files created while the backup is running might not get picked up by that backup, and they won't get picked up by the next backup either. * Another approach is a variation on the above: - Use two nodes as above, and back up the rest of the system as above. - Create a script that scans the application directory for files created during the last 24-hour period. For each file that was created in the last 24 hours, write it out to a file that can be fed to a command line client file list backup. For example, let's say the application directory backup begins at 23:00. So the script will search for files created between yesterday at 23:00 and today at 23:00, generate a file list, then kick off the dsmc selective -filelist=<listfile>. * Image backup? Best regards, Andy Raibeck IBM Software Group Tivoli Storage Manager Client Product Development Level 3 Team Lead Internal Notes e-mail: Andrew Raibeck/Hartford/IBM@IBMUS Internet e-mail: [email protected] IBM Tivoli Storage Manager support web page: http://www.ibm.com/support/entry/portal/Overview/Software/Tivoli/Tivoli_Storage_Manager "ADSM: Dist Stor Manager" <[email protected]> wrote on 2011-02-25 14:03:24: > From: "Prather, Wanda" <[email protected]> > To: [email protected] > Date: 2011-02-25 14:04 > Subject: Windows servers with a kazillion files and Win2K8... > Sent by: "ADSM: Dist Stor Manager" <[email protected]> > > I have a site with an application that generates kazillions of tiny > files that are stored forever. > I've already yelled about it, but it's a purchased, customer-facing > black-box app that they really can't change. > (Naturally, when it was bought umpty years ago, nobody thought about > the problem reaching this size or what the ramifications would be.) > Every day the app creates more files. > > They have multiple Win2K3 servers that already have multiple luns > containing over 35M files each, one is over 75M files. > > We are using journaling to back them up successfully (most days). > But it's a struggle just to expand the file tree with Windows > explorer, and there are exposures on the days when the journal gets > overrun (takes 72 hours for TSM to scan the filesystem and > revalidate the journal). > > Looking for anything that might help save our bacon. > > Has anybody had experience with this issue and Win2K8? > Does Win2K8 do any better than Win2K3 at handling huge numbers of > files in 1 NTFS directory? > Upgrading the OS is something application-independent we might be able to do. > > Thanks for any insight! > W > > > Wanda Prather | Senior Technical Specialist | [email protected]< > mailto:[email protected]> | www.jasi.com<www.jasi.com%20> > ICF Jacob & Sundstrom | 401 E. Pratt St, Suite 2214, Baltimore, MD > 21202 | 410.539.1135
