Try NetApp NDMP streamer, which allows you to leverage the deduplication in NDMP backups.
You need at least PureDisk 6.6.x + NBU 6.5.4 with EEB. On Thu, Jun 17, 2010 at 5:58 PM, <ccosta....@gmail.com> wrote: > Good news is that with 7.01 coming out in a month or so will allow NBU to > multi-stream NDMP data to a single tape drive. > > This will/may alleviate some of the performance issues many of you > experience each day. However I am not sure of any limitations of this > feature patch may have. > > Chris > > Sent from my Verizon Wireless BlackBerry > > -----Original Message----- > From: "Ambrose, Monte" <mambr...@qualcomm.com> > Date: Thu, 17 Jun 2010 14:38:41 > To: Jonathan Dyck<jd...@bank-banque-canada.ca>; Jeff Cleverley< > jeff.clever...@avagotech.com>; rusty.ma...@sungard.com< > rusty.ma...@sungard.com> > Cc: veritas-bu@mailman.eng.auburn.edu<veritas-bu@mailman.eng.auburn.edu> > Subject: Re: [Veritas-bu] NDMP and Millions of Files > > You could also use NetBackups Snap Mirror to tape. You would need a NetApp > snap mirror license. > > The Pros > It uses snapmirror and sends the data off to tape. It is a RAW volume > backup and is extremely fast - 10X in many cases. > It can be fully configured in NetBackup > > The Cons > It backs up the entire volume - so if you have a 1TB volume and only 400GB > are used it will backup 1tb > You cant restore a single file or dir. You have to restore the entire > thing. > You cant mix with incremental backups. > > Monte > > -----Original Message----- > From: veritas-bu-boun...@mailman.eng.auburn.edu [mailto: > veritas-bu-boun...@mailman.eng.auburn.edu] On Behalf Of Jonathan Dyck > Sent: Tuesday, June 15, 2010 7:21 AM > To: Jeff Cleverley; rusty.ma...@sungard.com > Cc: veritas-bu@mailman.eng.auburn.edu > Subject: Re: [Veritas-bu] NDMP and Millions of Files > > We do our NetApp backups (old GF940 metro cluster) with a combination of > snapshots (which are available at both sites) and NDMP here. Our NDMP > obviously holds all the long retention data. > > The data size isn't huge compared to some (6.5TB, ~40M files), but we've > had to resort to multiple policies and multi-streaming to back it up in a > reasonable amount of time (less than 60 hours for a full on the wknd). The > way it works is: > > PolicyA (vol1): explicitly lists 37 different paths for the backup > selection list, we've empirically determined these are the "smaller" > folders > PolicyB (vol1-long): explicitly lists 16 paths for the backup selection > list, we've determined these are the "large" folders > PolicyC (vol1-Catch-missed-directory): we've mounted the root of vol1 on a > Linux host, and we back it up via NFS, excluding the 37+16 paths defined > above. If this policy's full backup every gets too large (over 10GB or so), > we review the contents and add new paths to PolicyA or PolicyB as > necessary. This is necessary because you can't specify wildcards on NDMP > backups (discussed in this forum several times I believe). > > We repeat the above process for vol2. > > The above backup data sits on a deduped VTL for 2 months, and then the data > that is held longer than that is duplicated to tape for long-term storage > and expired off the VTL. As we run 5 streams concurrently, the throughput > is decent, but we peg out the CPU on the NetApp frequently during the backup > window, which is a concern. > > HTH... > Jon > > > > -----Original Message----- > From: veritas-bu-boun...@mailman.eng.auburn.edu [mailto: > veritas-bu-boun...@mailman.eng.auburn.edu] On Behalf Of Jeff Cleverley > Sent: June 14, 2010 5:57 PM > To: rusty.ma...@sungard.com > Cc: veritas-bu@mailman.eng.auburn.edu > Subject: Re: [Veritas-bu] NDMP and Millions of Files > > Rusty, > > If you have a way to use Snapvault to backup to another location, I > would use it. We have a number of file systems like what you have. I > tried NDMP over TCP and NFS backups using dedicated snapshots mounted > on a client. Both used a dedicated 10G network. We basically overran > our 6030 filer. We could have jumped through a lot of hoops and split > backups over multiple weekends, etc, but we decided it wasn't worth > it. I haven't tried Flash Backup for a while but it didn't really buy > us much on what we tried to do with it. It may work better now. > > We backup everything (~200 TB) to NearStores in another building. We > use SnapVault instead of SnapMirror. We can still revert our > destination volumes to primary r/w file systems if we need to. We > don't have the requirement to send tapes off site. If you do, you > could still make the tapes from your secondary filer. > > Using NFS to tape will give you check points and you can run multiple > streams to each tape drive. If you are not hitting the filer > throughput limits this may work for you. > > Jeff > > On Mon, Jun 14, 2010 at 3:24 PM, <rusty.ma...@sungard.com> wrote: > > > > We have a NetApp filer that has a few TB of data made up largely of > millions > > of small files (about 30 million or so) and we are using several NDMP > > policies to back up this data. The two main problems are length of time > it > > takes to backup (we usually have 2-3 backups running all day every day) > and > > when there is a maintenance or other event in the NBU domain, we have to > > kill the job, resulting in having to start all over (no checkpoints). > > > > For those of you who have faced a similar situation, how are you backing > up > > this data? > > > > Current thoughts are moving away from NDMP and going with just snapshots > and > > then getting the snap offsite either by backing it up or replicating it. > > We've also thought about backing it up via NFS, but that will probably be > > slower, though we would get checkpoints. > > > > I appreciate any other suggestions anyone has. > > > > Rusty Major, MCSE, BCFP, VCS ▪ Sr. Storage Engineer ▪ SunGard > Availability > > Services ▪ 757 N. Eldridge Suite 200, Houston TX 77079 ▪ 281-584-4693 > > Keeping People and Information Connected® ▪ > http://availability.sungard.com/ > > P Think before you print > > CONFIDENTIALITY: This e-mail (including any attachments) may contain > > confidential, proprietary and privileged information, and unauthorized > > disclosure or use is prohibited. If you received this e-mail in error, > > please notify the sender and delete this e-mail from your system. > > _______________________________________________ > > Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu > > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu > > > > > > > > -- > Jeff Cleverley > Unix Systems Administrator > 4380 Ziegler Road > Fort Collins, Colorado 80525 > 970-288-4611 > _______________________________________________ > Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu > > ==================================================================================== > > La version française suit le texte anglais. > > > ------------------------------------------------------------------------------------ > > This email may contain privileged and/or confidential information, and the > Bank of > Canada does not waive any related rights. Any distribution, use, or copying > of this > email or the information it contains by other than the intended recipient > is > unauthorized. If you received this email in error please delete it > immediately from > your system and notify the sender promptly by email that you have done so. > > > ------------------------------------------------------------------------------------ > > Le présent courriel peut contenir de l'information privilégiée ou > confidentielle. > La Banque du Canada ne renonce pas aux droits qui s'y rapportent. Toute > diffusion, > utilisation ou copie de ce courriel ou des renseignements qu'il contient > par une > personne autre que le ou les destinataires désignés est interdite. Si vous > recevez > ce courriel par erreur, veuillez le supprimer immédiatement et envoyer sans > délai à > l'expéditeur un message électronique pour l'aviser que vous avez éliminé de > votre > ordinateur toute copie du courriel reçu. > _______________________________________________ > Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu > _______________________________________________ > Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu > _______________________________________________ > Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu >
_______________________________________________ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
