Re: [Veritas-bu] NDMP and Millions of Files
Try NetApp NDMP streamer, which allows you to leverage the deduplication in NDMP backups. You need at least PureDisk 6.6.x + NBU 6.5.4 with EEB. On Thu, Jun 17, 2010 at 5:58 PM, ccosta@gmail.com wrote: Good news is that with 7.01 coming out in a month or so will allow NBU to multi-stream NDMP data to a single tape drive. This will/may alleviate some of the performance issues many of you experience each day. However I am not sure of any limitations of this feature patch may have. Chris Sent from my Verizon Wireless BlackBerry -Original Message- From: Ambrose, Monte mambr...@qualcomm.com Date: Thu, 17 Jun 2010 14:38:41 To: Jonathan Dyckjd...@bank-banque-canada.ca; Jeff Cleverley jeff.clever...@avagotech.com; rusty.ma...@sungard.com rusty.ma...@sungard.com Cc: veritas-bu@mailman.eng.auburn.eduveritas-bu@mailman.eng.auburn.edu Subject: Re: [Veritas-bu] NDMP and Millions of Files You could also use NetBackups Snap Mirror to tape. You would need a NetApp snap mirror license. The Pros It uses snapmirror and sends the data off to tape. It is a RAW volume backup and is extremely fast - 10X in many cases. It can be fully configured in NetBackup The Cons It backs up the entire volume - so if you have a 1TB volume and only 400GB are used it will backup 1tb You cant restore a single file or dir. You have to restore the entire thing. You cant mix with incremental backups. Monte -Original Message- From: veritas-bu-boun...@mailman.eng.auburn.edu [mailto: veritas-bu-boun...@mailman.eng.auburn.edu] On Behalf Of Jonathan Dyck Sent: Tuesday, June 15, 2010 7:21 AM To: Jeff Cleverley; rusty.ma...@sungard.com Cc: veritas-bu@mailman.eng.auburn.edu Subject: Re: [Veritas-bu] NDMP and Millions of Files We do our NetApp backups (old GF940 metro cluster) with a combination of snapshots (which are available at both sites) and NDMP here. Our NDMP obviously holds all the long retention data. The data size isn't huge compared to some (6.5TB, ~40M files), but we've had to resort to multiple policies and multi-streaming to back it up in a reasonable amount of time (less than 60 hours for a full on the wknd). The way it works is: PolicyA (vol1): explicitly lists 37 different paths for the backup selection list, we've empirically determined these are the smaller folders PolicyB (vol1-long): explicitly lists 16 paths for the backup selection list, we've determined these are the large folders PolicyC (vol1-Catch-missed-directory): we've mounted the root of vol1 on a Linux host, and we back it up via NFS, excluding the 37+16 paths defined above. If this policy's full backup every gets too large (over 10GB or so), we review the contents and add new paths to PolicyA or PolicyB as necessary. This is necessary because you can't specify wildcards on NDMP backups (discussed in this forum several times I believe). We repeat the above process for vol2. The above backup data sits on a deduped VTL for 2 months, and then the data that is held longer than that is duplicated to tape for long-term storage and expired off the VTL. As we run 5 streams concurrently, the throughput is decent, but we peg out the CPU on the NetApp frequently during the backup window, which is a concern. HTH... Jon -Original Message- From: veritas-bu-boun...@mailman.eng.auburn.edu [mailto: veritas-bu-boun...@mailman.eng.auburn.edu] On Behalf Of Jeff Cleverley Sent: June 14, 2010 5:57 PM To: rusty.ma...@sungard.com Cc: veritas-bu@mailman.eng.auburn.edu Subject: Re: [Veritas-bu] NDMP and Millions of Files Rusty, If you have a way to use Snapvault to backup to another location, I would use it. We have a number of file systems like what you have. I tried NDMP over TCP and NFS backups using dedicated snapshots mounted on a client. Both used a dedicated 10G network. We basically overran our 6030 filer. We could have jumped through a lot of hoops and split backups over multiple weekends, etc, but we decided it wasn't worth it. I haven't tried Flash Backup for a while but it didn't really buy us much on what we tried to do with it. It may work better now. We backup everything (~200 TB) to NearStores in another building. We use SnapVault instead of SnapMirror. We can still revert our destination volumes to primary r/w file systems if we need to. We don't have the requirement to send tapes off site. If you do, you could still make the tapes from your secondary filer. Using NFS to tape will give you check points and you can run multiple streams to each tape drive. If you are not hitting the filer throughput limits this may work for you. Jeff On Mon, Jun 14, 2010 at 3:24 PM, rusty.ma...@sungard.com wrote: We have a NetApp filer that has a few TB of data made up largely of millions of small files (about 30 million or so) and we are using several NDMP policies to back up this data. The two main problems are length of time
Re: [Veritas-bu] NDMP and Millions of Files
On Mon, Jun 21, 2010 at 2:19 AM, Bahadir Kiziltan bahadir.kizil...@gmail.com wrote: Try NetApp NDMP streamer, which allows you to leverage the deduplication in NDMP backups. You need at least PureDisk 6.6.x + NBU 6.5.4 with EEB. The PureDisk requirement is the killer. I've got several 20TB applications with hundreds of millions of files. Purchasing PureDisk licenses for that much data is prohibitively expensive. As long as Symantec insists on a per-TB license for PureDisk, we will continue to make as little use of it as we can. .../Ed Ed Wilts, RHCE, BCFP, BCSD, SCSP, SCSE ewi...@ewilts.org Linkedin http://www.linkedin.com/in/ewilts On Thu, Jun 17, 2010 at 5:58 PM, ccosta@gmail.com wrote: Good news is that with 7.01 coming out in a month or so will allow NBU to multi-stream NDMP data to a single tape drive. This will/may alleviate some of the performance issues many of you experience each day. However I am not sure of any limitations of this feature patch may have. Chris Sent from my Verizon Wireless BlackBerry -Original Message- From: Ambrose, Monte mambr...@qualcomm.com Date: Thu, 17 Jun 2010 14:38:41 To: Jonathan Dyckjd...@bank-banque-canada.ca; Jeff Cleverley jeff.clever...@avagotech.com; rusty.ma...@sungard.com rusty.ma...@sungard.com Cc: veritas-bu@mailman.eng.auburn.eduveritas-bu@mailman.eng.auburn.edu Subject: Re: [Veritas-bu] NDMP and Millions of Files You could also use NetBackups Snap Mirror to tape. You would need a NetApp snap mirror license. The Pros It uses snapmirror and sends the data off to tape. It is a RAW volume backup and is extremely fast - 10X in many cases. It can be fully configured in NetBackup The Cons It backs up the entire volume - so if you have a 1TB volume and only 400GB are used it will backup 1tb You cant restore a single file or dir. You have to restore the entire thing. You cant mix with incremental backups. Monte -Original Message- From: veritas-bu-boun...@mailman.eng.auburn.edu [mailto: veritas-bu-boun...@mailman.eng.auburn.edu] On Behalf Of Jonathan Dyck Sent: Tuesday, June 15, 2010 7:21 AM To: Jeff Cleverley; rusty.ma...@sungard.com Cc: veritas-bu@mailman.eng.auburn.edu Subject: Re: [Veritas-bu] NDMP and Millions of Files We do our NetApp backups (old GF940 metro cluster) with a combination of snapshots (which are available at both sites) and NDMP here. Our NDMP obviously holds all the long retention data. The data size isn't huge compared to some (6.5TB, ~40M files), but we've had to resort to multiple policies and multi-streaming to back it up in a reasonable amount of time (less than 60 hours for a full on the wknd). The way it works is: PolicyA (vol1): explicitly lists 37 different paths for the backup selection list, we've empirically determined these are the smaller folders PolicyB (vol1-long): explicitly lists 16 paths for the backup selection list, we've determined these are the large folders PolicyC (vol1-Catch-missed-directory): we've mounted the root of vol1 on a Linux host, and we back it up via NFS, excluding the 37+16 paths defined above. If this policy's full backup every gets too large (over 10GB or so), we review the contents and add new paths to PolicyA or PolicyB as necessary. This is necessary because you can't specify wildcards on NDMP backups (discussed in this forum several times I believe). We repeat the above process for vol2. The above backup data sits on a deduped VTL for 2 months, and then the data that is held longer than that is duplicated to tape for long-term storage and expired off the VTL. As we run 5 streams concurrently, the throughput is decent, but we peg out the CPU on the NetApp frequently during the backup window, which is a concern. HTH... Jon -Original Message- From: veritas-bu-boun...@mailman.eng.auburn.edu [mailto: veritas-bu-boun...@mailman.eng.auburn.edu] On Behalf Of Jeff Cleverley Sent: June 14, 2010 5:57 PM To: rusty.ma...@sungard.com Cc: veritas-bu@mailman.eng.auburn.edu Subject: Re: [Veritas-bu] NDMP and Millions of Files Rusty, If you have a way to use Snapvault to backup to another location, I would use it. We have a number of file systems like what you have. I tried NDMP over TCP and NFS backups using dedicated snapshots mounted on a client. Both used a dedicated 10G network. We basically overran our 6030 filer. We could have jumped through a lot of hoops and split backups over multiple weekends, etc, but we decided it wasn't worth it. I haven't tried Flash Backup for a while but it didn't really buy us much on what we tried to do with it. It may work better now. We backup everything (~200 TB) to NearStores in another building. We use SnapVault instead of SnapMirror. We can still revert our destination volumes to primary r/w file systems if we need to. We don't have the requirement to send tapes off site. If you do, you could still make
Re: [Veritas-bu] NDMP and Millions of Files
You could also use NetBackups Snap Mirror to tape. You would need a NetApp snap mirror license. The Pros It uses snapmirror and sends the data off to tape. It is a RAW volume backup and is extremely fast - 10X in many cases. It can be fully configured in NetBackup The Cons It backs up the entire volume - so if you have a 1TB volume and only 400GB are used it will backup 1tb You cant restore a single file or dir. You have to restore the entire thing. You cant mix with incremental backups. Monte -Original Message- From: veritas-bu-boun...@mailman.eng.auburn.edu [mailto:veritas-bu-boun...@mailman.eng.auburn.edu] On Behalf Of Jonathan Dyck Sent: Tuesday, June 15, 2010 7:21 AM To: Jeff Cleverley; rusty.ma...@sungard.com Cc: veritas-bu@mailman.eng.auburn.edu Subject: Re: [Veritas-bu] NDMP and Millions of Files We do our NetApp backups (old GF940 metro cluster) with a combination of snapshots (which are available at both sites) and NDMP here. Our NDMP obviously holds all the long retention data. The data size isn't huge compared to some (6.5TB, ~40M files), but we've had to resort to multiple policies and multi-streaming to back it up in a reasonable amount of time (less than 60 hours for a full on the wknd). The way it works is: PolicyA (vol1): explicitly lists 37 different paths for the backup selection list, we've empirically determined these are the smaller folders PolicyB (vol1-long): explicitly lists 16 paths for the backup selection list, we've determined these are the large folders PolicyC (vol1-Catch-missed-directory): we've mounted the root of vol1 on a Linux host, and we back it up via NFS, excluding the 37+16 paths defined above. If this policy's full backup every gets too large (over 10GB or so), we review the contents and add new paths to PolicyA or PolicyB as necessary. This is necessary because you can't specify wildcards on NDMP backups (discussed in this forum several times I believe). We repeat the above process for vol2. The above backup data sits on a deduped VTL for 2 months, and then the data that is held longer than that is duplicated to tape for long-term storage and expired off the VTL. As we run 5 streams concurrently, the throughput is decent, but we peg out the CPU on the NetApp frequently during the backup window, which is a concern. HTH... Jon -Original Message- From: veritas-bu-boun...@mailman.eng.auburn.edu [mailto:veritas-bu-boun...@mailman.eng.auburn.edu] On Behalf Of Jeff Cleverley Sent: June 14, 2010 5:57 PM To: rusty.ma...@sungard.com Cc: veritas-bu@mailman.eng.auburn.edu Subject: Re: [Veritas-bu] NDMP and Millions of Files Rusty, If you have a way to use Snapvault to backup to another location, I would use it. We have a number of file systems like what you have. I tried NDMP over TCP and NFS backups using dedicated snapshots mounted on a client. Both used a dedicated 10G network. We basically overran our 6030 filer. We could have jumped through a lot of hoops and split backups over multiple weekends, etc, but we decided it wasn't worth it. I haven't tried Flash Backup for a while but it didn't really buy us much on what we tried to do with it. It may work better now. We backup everything (~200 TB) to NearStores in another building. We use SnapVault instead of SnapMirror. We can still revert our destination volumes to primary r/w file systems if we need to. We don't have the requirement to send tapes off site. If you do, you could still make the tapes from your secondary filer. Using NFS to tape will give you check points and you can run multiple streams to each tape drive. If you are not hitting the filer throughput limits this may work for you. Jeff On Mon, Jun 14, 2010 at 3:24 PM, rusty.ma...@sungard.com wrote: We have a NetApp filer that has a few TB of data made up largely of millions of small files (about 30 million or so) and we are using several NDMP policies to back up this data. The two main problems are length of time it takes to backup (we usually have 2-3 backups running all day every day) and when there is a maintenance or other event in the NBU domain, we have to kill the job, resulting in having to start all over (no checkpoints). For those of you who have faced a similar situation, how are you backing up this data? Current thoughts are moving away from NDMP and going with just snapshots and then getting the snap offsite either by backing it up or replicating it. We've also thought about backing it up via NFS, but that will probably be slower, though we would get checkpoints. I appreciate any other suggestions anyone has. Rusty Major, MCSE, BCFP, VCS ▪ Sr. Storage Engineer ▪ SunGard Availability Services ▪ 757 N. Eldridge Suite 200, Houston TX 77079 ▪ 281-584-4693 Keeping People and Information Connected® ▪ http://availability.sungard.com/ P Think before you print CONFIDENTIALITY: This e-mail (including any attachments) may contain confidential
Re: [Veritas-bu] NDMP and Millions of Files
Good news is that with 7.01 coming out in a month or so will allow NBU to multi-stream NDMP data to a single tape drive. This will/may alleviate some of the performance issues many of you experience each day. However I am not sure of any limitations of this feature patch may have. Chris Sent from my Verizon Wireless BlackBerry -Original Message- From: Ambrose, Monte mambr...@qualcomm.com Date: Thu, 17 Jun 2010 14:38:41 To: Jonathan Dyckjd...@bank-banque-canada.ca; Jeff Cleverleyjeff.clever...@avagotech.com; rusty.ma...@sungard.comrusty.ma...@sungard.com Cc: veritas-bu@mailman.eng.auburn.eduveritas-bu@mailman.eng.auburn.edu Subject: Re: [Veritas-bu] NDMP and Millions of Files You could also use NetBackups Snap Mirror to tape. You would need a NetApp snap mirror license. The Pros It uses snapmirror and sends the data off to tape. It is a RAW volume backup and is extremely fast - 10X in many cases. It can be fully configured in NetBackup The Cons It backs up the entire volume - so if you have a 1TB volume and only 400GB are used it will backup 1tb You cant restore a single file or dir. You have to restore the entire thing. You cant mix with incremental backups. Monte -Original Message- From: veritas-bu-boun...@mailman.eng.auburn.edu [mailto:veritas-bu-boun...@mailman.eng.auburn.edu] On Behalf Of Jonathan Dyck Sent: Tuesday, June 15, 2010 7:21 AM To: Jeff Cleverley; rusty.ma...@sungard.com Cc: veritas-bu@mailman.eng.auburn.edu Subject: Re: [Veritas-bu] NDMP and Millions of Files We do our NetApp backups (old GF940 metro cluster) with a combination of snapshots (which are available at both sites) and NDMP here. Our NDMP obviously holds all the long retention data. The data size isn't huge compared to some (6.5TB, ~40M files), but we've had to resort to multiple policies and multi-streaming to back it up in a reasonable amount of time (less than 60 hours for a full on the wknd). The way it works is: PolicyA (vol1): explicitly lists 37 different paths for the backup selection list, we've empirically determined these are the smaller folders PolicyB (vol1-long): explicitly lists 16 paths for the backup selection list, we've determined these are the large folders PolicyC (vol1-Catch-missed-directory): we've mounted the root of vol1 on a Linux host, and we back it up via NFS, excluding the 37+16 paths defined above. If this policy's full backup every gets too large (over 10GB or so), we review the contents and add new paths to PolicyA or PolicyB as necessary. This is necessary because you can't specify wildcards on NDMP backups (discussed in this forum several times I believe). We repeat the above process for vol2. The above backup data sits on a deduped VTL for 2 months, and then the data that is held longer than that is duplicated to tape for long-term storage and expired off the VTL. As we run 5 streams concurrently, the throughput is decent, but we peg out the CPU on the NetApp frequently during the backup window, which is a concern. HTH... Jon -Original Message- From: veritas-bu-boun...@mailman.eng.auburn.edu [mailto:veritas-bu-boun...@mailman.eng.auburn.edu] On Behalf Of Jeff Cleverley Sent: June 14, 2010 5:57 PM To: rusty.ma...@sungard.com Cc: veritas-bu@mailman.eng.auburn.edu Subject: Re: [Veritas-bu] NDMP and Millions of Files Rusty, If you have a way to use Snapvault to backup to another location, I would use it. We have a number of file systems like what you have. I tried NDMP over TCP and NFS backups using dedicated snapshots mounted on a client. Both used a dedicated 10G network. We basically overran our 6030 filer. We could have jumped through a lot of hoops and split backups over multiple weekends, etc, but we decided it wasn't worth it. I haven't tried Flash Backup for a while but it didn't really buy us much on what we tried to do with it. It may work better now. We backup everything (~200 TB) to NearStores in another building. We use SnapVault instead of SnapMirror. We can still revert our destination volumes to primary r/w file systems if we need to. We don't have the requirement to send tapes off site. If you do, you could still make the tapes from your secondary filer. Using NFS to tape will give you check points and you can run multiple streams to each tape drive. If you are not hitting the filer throughput limits this may work for you. Jeff On Mon, Jun 14, 2010 at 3:24 PM, rusty.ma...@sungard.com wrote: We have a NetApp filer that has a few TB of data made up largely of millions of small files (about 30 million or so) and we are using several NDMP policies to back up this data. The two main problems are length of time it takes to backup (we usually have 2-3 backups running all day every day) and when there is a maintenance or other event in the NBU domain, we have to kill the job, resulting in having to start all over (no checkpoints). For those of you who have faced a similar
Re: [Veritas-bu] NDMP and Millions of Files
We do our NetApp backups (old GF940 metro cluster) with a combination of snapshots (which are available at both sites) and NDMP here. Our NDMP obviously holds all the long retention data. The data size isn't huge compared to some (6.5TB, ~40M files), but we've had to resort to multiple policies and multi-streaming to back it up in a reasonable amount of time (less than 60 hours for a full on the wknd). The way it works is: PolicyA (vol1): explicitly lists 37 different paths for the backup selection list, we've empirically determined these are the smaller folders PolicyB (vol1-long): explicitly lists 16 paths for the backup selection list, we've determined these are the large folders PolicyC (vol1-Catch-missed-directory): we've mounted the root of vol1 on a Linux host, and we back it up via NFS, excluding the 37+16 paths defined above. If this policy's full backup every gets too large (over 10GB or so), we review the contents and add new paths to PolicyA or PolicyB as necessary. This is necessary because you can't specify wildcards on NDMP backups (discussed in this forum several times I believe). We repeat the above process for vol2. The above backup data sits on a deduped VTL for 2 months, and then the data that is held longer than that is duplicated to tape for long-term storage and expired off the VTL. As we run 5 streams concurrently, the throughput is decent, but we peg out the CPU on the NetApp frequently during the backup window, which is a concern. HTH... Jon -Original Message- From: veritas-bu-boun...@mailman.eng.auburn.edu [mailto:veritas-bu-boun...@mailman.eng.auburn.edu] On Behalf Of Jeff Cleverley Sent: June 14, 2010 5:57 PM To: rusty.ma...@sungard.com Cc: veritas-bu@mailman.eng.auburn.edu Subject: Re: [Veritas-bu] NDMP and Millions of Files Rusty, If you have a way to use Snapvault to backup to another location, I would use it. We have a number of file systems like what you have. I tried NDMP over TCP and NFS backups using dedicated snapshots mounted on a client. Both used a dedicated 10G network. We basically overran our 6030 filer. We could have jumped through a lot of hoops and split backups over multiple weekends, etc, but we decided it wasn't worth it. I haven't tried Flash Backup for a while but it didn't really buy us much on what we tried to do with it. It may work better now. We backup everything (~200 TB) to NearStores in another building. We use SnapVault instead of SnapMirror. We can still revert our destination volumes to primary r/w file systems if we need to. We don't have the requirement to send tapes off site. If you do, you could still make the tapes from your secondary filer. Using NFS to tape will give you check points and you can run multiple streams to each tape drive. If you are not hitting the filer throughput limits this may work for you. Jeff On Mon, Jun 14, 2010 at 3:24 PM, rusty.ma...@sungard.com wrote: We have a NetApp filer that has a few TB of data made up largely of millions of small files (about 30 million or so) and we are using several NDMP policies to back up this data. The two main problems are length of time it takes to backup (we usually have 2-3 backups running all day every day) and when there is a maintenance or other event in the NBU domain, we have to kill the job, resulting in having to start all over (no checkpoints). For those of you who have faced a similar situation, how are you backing up this data? Current thoughts are moving away from NDMP and going with just snapshots and then getting the snap offsite either by backing it up or replicating it. We've also thought about backing it up via NFS, but that will probably be slower, though we would get checkpoints. I appreciate any other suggestions anyone has. Rusty Major, MCSE, BCFP, VCS ▪ Sr. Storage Engineer ▪ SunGard Availability Services ▪ 757 N. Eldridge Suite 200, Houston TX 77079 ▪ 281-584-4693 Keeping People and Information Connected® ▪ http://availability.sungard.com/ P Think before you print CONFIDENTIALITY: This e-mail (including any attachments) may contain confidential, proprietary and privileged information, and unauthorized disclosure or use is prohibited. If you received this e-mail in error, please notify the sender and delete this e-mail from your system. ___ Veritas-bu maillist - veritas...@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu -- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611 ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu La version française suit le texte anglais
Re: [Veritas-bu] NDMP and Millions of Files
We’ve been testing backup methods for millions of small files on NAS and DAS storage for YEARS here. Without exception, the best backup method has been NFS mount via a “proxy” client. I don’t manage the NetApp Storage, but we saw an increase from 10MB/sec to 17-19MB/sec when switching from NDMP to NFS on a FAS2040. I’ve heard there is a FAS 31XX coming to replace the FAS2040 and hopefully that gives us better performance still. Using NFS gets you checkpoints, but more importantly for us, exclusions. Our filers have something like 70-100 million files, but we only backup 45-50 million of them. Off the top of my head, my best test runs per device are: Apple xRaid DAS (unknown model) – 30MB/sec via 2 streams NFS Dell MD3000 DAS (RedHat 5) – 30MB/sec via single stream Sun Unified Storage System 7210 – 25-29MB/sec via 16 streams via NFS NetApp FAS2040 – 19MB/sec Sun StorageTek 5310 – 8MB/sec via single stream NFS We’re currently using the Dell MD3000 / Redhat 5 rig above to test Linux Flashbackup, but so far we’re only running 24MB/sec. The most important lesson I’ve learned converting NDMP to NFS is to test thoroughly. Every filer is different, and it can take weeks to find a “sweet spot” between number or streams and client configuration. Good luck! -Jonathan From: veritas-bu-boun...@mailman.eng.auburn.edu [mailto:veritas-bu-boun...@mailman.eng.auburn.edu] On Behalf Of rusty.ma...@sungard.com Sent: Monday, June 14, 2010 5:25 PM To: veritas-bu@mailman.eng.auburn.edu Subject: [Veritas-bu] NDMP and Millions of Files We have a NetApp filer that has a few TB of data made up largely of millions of small files (about 30 million or so) and we are using several NDMP policies to back up this data. The two main problems are length of time it takes to backup (we usually have 2-3 backups running all day every day) and when there is a maintenance or other event in the NBU domain, we have to kill the job, resulting in having to start all over (no checkpoints). For those of you who have faced a similar situation, how are you backing up this data? Current thoughts are moving away from NDMP and going with just snapshots and then getting the snap offsite either by backing it up or replicating it. We've also thought about backing it up via NFS, but that will probably be slower, though we would get checkpoints. I appreciate any other suggestions anyone has. Rusty Major, MCSE, BCFP, VCS ▪ Sr. Storage Engineer ▪ SunGard Availability Services ▪ 757 N. Eldridge Suite 200, Houston TX 77079 ▪ 281-584-4693 Keeping People and Information Connected® ▪ http://availability.sungard.com/ http://availability.sungard.com/ P Think before you print CONFIDENTIALITY: This e-mail (including any attachments) may contain confidential, proprietary and privileged information, and unauthorized disclosure or use is prohibited. If you received this e-mail in error, please notify the sender and delete this e-mail from your system. ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
Re: [Veritas-bu] NDMP and Millions of Files
Rusty, If you have a way to use Snapvault to backup to another location, I would use it. We have a number of file systems like what you have. I tried NDMP over TCP and NFS backups using dedicated snapshots mounted on a client. Both used a dedicated 10G network. We basically overran our 6030 filer. We could have jumped through a lot of hoops and split backups over multiple weekends, etc, but we decided it wasn't worth it. I haven't tried Flash Backup for a while but it didn't really buy us much on what we tried to do with it. It may work better now. We backup everything (~200 TB) to NearStores in another building. We use SnapVault instead of SnapMirror. We can still revert our destination volumes to primary r/w file systems if we need to. We don't have the requirement to send tapes off site. If you do, you could still make the tapes from your secondary filer. Using NFS to tape will give you check points and you can run multiple streams to each tape drive. If you are not hitting the filer throughput limits this may work for you. Jeff On Mon, Jun 14, 2010 at 3:24 PM, rusty.ma...@sungard.com wrote: We have a NetApp filer that has a few TB of data made up largely of millions of small files (about 30 million or so) and we are using several NDMP policies to back up this data. The two main problems are length of time it takes to backup (we usually have 2-3 backups running all day every day) and when there is a maintenance or other event in the NBU domain, we have to kill the job, resulting in having to start all over (no checkpoints). For those of you who have faced a similar situation, how are you backing up this data? Current thoughts are moving away from NDMP and going with just snapshots and then getting the snap offsite either by backing it up or replicating it. We've also thought about backing it up via NFS, but that will probably be slower, though we would get checkpoints. I appreciate any other suggestions anyone has. Rusty Major, MCSE, BCFP, VCS ▪ Sr. Storage Engineer ▪ SunGard Availability Services ▪ 757 N. Eldridge Suite 200, Houston TX 77079 ▪ 281-584-4693 Keeping People and Information Connected® ▪ http://availability.sungard.com/ P Think before you print CONFIDENTIALITY: This e-mail (including any attachments) may contain confidential, proprietary and privileged information, and unauthorized disclosure or use is prohibited. If you received this e-mail in error, please notify the sender and delete this e-mail from your system. ___ Veritas-bu maillist - veritas...@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu -- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611 ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
