We do our NetApp backups (old GF940 metro cluster) with a combination of snapshots (which are available at both sites) and NDMP here. Our NDMP obviously holds all the long retention data.
The data size isn't huge compared to some (6.5TB, ~40M files), but we've had to resort to multiple policies and multi-streaming to back it up in a reasonable amount of time (less than 60 hours for a full on the wknd). The way it works is: PolicyA (vol1): explicitly lists 37 different paths for the backup selection list, we've empirically determined these are the "smaller" folders PolicyB (vol1-long): explicitly lists 16 paths for the backup selection list, we've determined these are the "large" folders PolicyC (vol1-Catch-missed-directory): we've mounted the root of vol1 on a Linux host, and we back it up via NFS, excluding the 37+16 paths defined above. If this policy's full backup every gets too large (over 10GB or so), we review the contents and add new paths to PolicyA or PolicyB as necessary. This is necessary because you can't specify wildcards on NDMP backups (discussed in this forum several times I believe). We repeat the above process for vol2. The above backup data sits on a deduped VTL for 2 months, and then the data that is held longer than that is duplicated to tape for long-term storage and expired off the VTL. As we run 5 streams concurrently, the throughput is decent, but we peg out the CPU on the NetApp frequently during the backup window, which is a concern. HTH... Jon -----Original Message----- From: veritas-bu-boun...@mailman.eng.auburn.edu [mailto:veritas-bu-boun...@mailman.eng.auburn.edu] On Behalf Of Jeff Cleverley Sent: June 14, 2010 5:57 PM To: rusty.ma...@sungard.com Cc: veritas-bu@mailman.eng.auburn.edu Subject: Re: [Veritas-bu] NDMP and Millions of Files Rusty, If you have a way to use Snapvault to backup to another location, I would use it. We have a number of file systems like what you have. I tried NDMP over TCP and NFS backups using dedicated snapshots mounted on a client. Both used a dedicated 10G network. We basically overran our 6030 filer. We could have jumped through a lot of hoops and split backups over multiple weekends, etc, but we decided it wasn't worth it. I haven't tried Flash Backup for a while but it didn't really buy us much on what we tried to do with it. It may work better now. We backup everything (~200 TB) to NearStores in another building. We use SnapVault instead of SnapMirror. We can still revert our destination volumes to primary r/w file systems if we need to. We don't have the requirement to send tapes off site. If you do, you could still make the tapes from your secondary filer. Using NFS to tape will give you check points and you can run multiple streams to each tape drive. If you are not hitting the filer throughput limits this may work for you. Jeff On Mon, Jun 14, 2010 at 3:24 PM, <rusty.ma...@sungard.com> wrote: > > We have a NetApp filer that has a few TB of data made up largely of millions > of small files (about 30 million or so) and we are using several NDMP > policies to back up this data. The two main problems are length of time it > takes to backup (we usually have 2-3 backups running all day every day) and > when there is a maintenance or other event in the NBU domain, we have to > kill the job, resulting in having to start all over (no checkpoints). > > For those of you who have faced a similar situation, how are you backing up > this data? > > Current thoughts are moving away from NDMP and going with just snapshots and > then getting the snap offsite either by backing it up or replicating it. > We've also thought about backing it up via NFS, but that will probably be > slower, though we would get checkpoints. > > I appreciate any other suggestions anyone has. > > Rusty Major, MCSE, BCFP, VCS ▪ Sr. Storage Engineer ▪ SunGard Availability > Services ▪ 757 N. Eldridge Suite 200, Houston TX 77079 ▪ 281-584-4693 > Keeping People and Information Connected® ▪ http://availability.sungard.com/ > P Think before you print > CONFIDENTIALITY: This e-mail (including any attachments) may contain > confidential, proprietary and privileged information, and unauthorized > disclosure or use is prohibited. If you received this e-mail in error, > please notify the sender and delete this e-mail from your system. > _______________________________________________ > Veritas-bu maillist - veritas...@mailman.eng.auburn.edu > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu > > -- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611 _______________________________________________ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu ==================================================================================== La version française suit le texte anglais. ------------------------------------------------------------------------------------ This email may contain privileged and/or confidential information, and the Bank of Canada does not waive any related rights. Any distribution, use, or copying of this email or the information it contains by other than the intended recipient is unauthorized. If you received this email in error please delete it immediately from your system and notify the sender promptly by email that you have done so. ------------------------------------------------------------------------------------ Le présent courriel peut contenir de l'information privilégiée ou confidentielle. La Banque du Canada ne renonce pas aux droits qui s'y rapportent. Toute diffusion, utilisation ou copie de ce courriel ou des renseignements qu'il contient par une personne autre que le ou les destinataires désignés est interdite. Si vous recevez ce courriel par erreur, veuillez le supprimer immédiatement et envoyer sans délai à l'expéditeur un message électronique pour l'aviser que vous avez éliminé de votre ordinateur toute copie du courriel reçu. _______________________________________________ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu