Spectrum Protect backup (under the hood of mmbackup) and Spectrum Protect for Space Management (HSM) can be combined on the same data. There are some valuable integration topics between the products that can reduce the overall network traffic if using backup and HSM on the same files. With the combination of the products you have the ability to free file system space from cold data and migrate them out to tape and to have several versions of frequently used files in backup in the same file system.
Greetings, Dominic. ______________________________________________________________________________________________________________ Dominic Mueller-Wicke | IBM Spectrum Protect Development | Technical Lead | +49 7034 64 32794 | [email protected] Vorsitzende des Aufsichtsrats: Martina Koederitz; Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen; Registergericht: Amtsgericht Stuttgart, HRB 243294 From: [email protected] To: [email protected] Date: 11.04.2016 17:11 Subject: gpfsug-discuss Digest, Vol 51, Issue 9 Sent by: [email protected] Send gpfsug-discuss mailing list submissions to [email protected] To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to [email protected] You can reach the person managing the list at [email protected] When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. backup and disaster recovery solutions (Damir Krstic) 2. Re: backup and disaster recovery solutions (Jaime Pinto) 3. Re: backup and disaster recovery solutions (Jonathan Buzzard) 4. Re: backup and disaster recovery solutions (Marc A Kaplan) ----- Message from Damir Krstic <[email protected]> on Mon, 11 Apr 2016 13:15:30 +0000 ----- To: gpfsug main discussion list <[email protected]> Subject: [gpfsug-discuss] backup and disaster recovery solutions We have implemented 1.5PB ESS solution recently in our HPC environment. Today we are kicking of backup and disaster recovery discussions so I was wondering what everyone else is using for their backup? In our old storage environment we simply rsync-ed home and software directories and projects were not backed up. With ESS we are looking for more of a GPFS based backup solution - something to tape possibly and also something that will have life cycle feature - so if the file is not touched for number of days, it's moved to a tape (something like LTFS). Thanks in advance. DAmir ----- Message from Jaime Pinto <[email protected]> on Mon, 11 Apr 2016 10:34:54 -0400 ----- To: gpfsug main discussion list <[email protected]>, Damir Krstic <[email protected]> Subject: Re: [gpfsug-discuss] backup and disaster recovery solutions Do you want backups or periodic frozen snapshots of the file system? Backups can entail some level of version control, so that you or end-users can get files back on certain points in time, in case of accidental deletions. Besides 1.5PB is a lot of material, so you may not want to take full snapshots that often. In that case, a combination of daily incremental backups using TSM with GPFS's mmbackup can be a good option. TSM also does a very good job at controlling how material is distributed across multiple tapes, and that is something that requires a lot of micro-management if you want a home grown solution of rsync+LTFS. On the other hand, you could use gpfs built-in tools such a mmapplypolicy to identify candidates for incremental backup, and send them to LTFS. Just more micro management, and you may have to come up with your own tool to let end-users restore their stuff, or you'll have to act on their behalf. Jaime Quoting Damir Krstic <[email protected]>: > We have implemented 1.5PB ESS solution recently in our HPC environment. > Today we are kicking of backup and disaster recovery discussions so I was > wondering what everyone else is using for their backup? > > In our old storage environment we simply rsync-ed home and software > directories and projects were not backed up. > > With ESS we are looking for more of a GPFS based backup solution - > something to tape possibly and also something that will have life cycle > feature - so if the file is not touched for number of days, it's moved to a > tape (something like LTFS). > > Thanks in advance. > > DAmir > ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.org University of Toronto 256 McCaul Street, Room 235 Toronto, ON, M5T1W5 P: 416-978-2755 C: 416-505-1477 ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. ----- Message from Jonathan Buzzard <[email protected]> on Mon, 11 Apr 2016 16:02:45 +0100 ----- To: [email protected] Subject: Re: [gpfsug-discuss] backup and disaster recovery solutions On Mon, 2016-04-11 at 10:34 -0400, Jaime Pinto wrote: > Do you want backups or periodic frozen snapshots of the file system? > > Backups can entail some level of version control, so that you or > end-users can get files back on certain points in time, in case of > accidental deletions. Besides 1.5PB is a lot of material, so you may > not want to take full snapshots that often. In that case, a > combination of daily incremental backups using TSM with GPFS's > mmbackup can be a good option. TSM also does a very good job at > controlling how material is distributed across multiple tapes, and > that is something that requires a lot of micro-management if you want > a home grown solution of rsync+LTFS. Is there any other viable option other than TSM for backing up 1.5PB of data? All other backup software does not handle this at all well. > On the other hand, you could use gpfs built-in tools such a > mmapplypolicy to identify candidates for incremental backup, and send > them to LTFS. Just more micro management, and you may have to come up > with your own tool to let end-users restore their stuff, or you'll > have to act on their behalf. > I was not aware of a way of letting end users restore their stuff from *backup* for any of the major backup software while respecting the file system level security of the original file system. If you let the end user have access to the backup they can restore any file to any location which is generally not a good idea. I do have a concept of creating a read only Fuse mounted file system from a TSM point in time synthetic backup, and then using the shadow copy feature of Samba to enable restores using the "Previous Versions" feature of windows file manager. I got as far as getting a directory tree you could browse through but then had an enforced change of jobs and don't have access to a TSM server any more to continue development. Note if anyone from IBM is listening that would be a super cool feature. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. ----- Message from "Marc A Kaplan" <[email protected]> on Mon, 11 Apr 2016 11:11:24 -0400 ----- To: gpfsug main discussion list <[email protected]> Subject: Re: [gpfsug-discuss] backup and disaster recovery solutions Since you write "so if the file is not touched for number of days, it's moved to a tape" - that is what we call the HSM feature. This is additional function beyond backup. IBM has two implementations. (1) TSM/HSM now called IBM Spectrum Protect. http://www-03.ibm.com/software/products/en/spectrum-protect-for-space-management (2) HPSS http://www.hpss-collaboration.org/ The GPFS (Spectrum Scale File System) policy feature supports both, so that mmapplypolicy and GPFS policy rules can be used to perform accelerated metadata scans to identify which files should be migrated. Also, GPFS supports on-demand recall (on application reads) of data from long term storage (tape) to GPFS storage (disk or SSD). See also DMAPI. Marc A Kaplan From: Damir Krstic <[email protected]> To: gpfsug main discussion list <[email protected]> Date: 04/11/2016 09:16 AM Subject: [gpfsug-discuss] backup and disaster recovery solutions Sent by: [email protected] We have implemented 1.5PB ESS solution recently in our HPC environment. Today we are kicking of backup and disaster recovery discussions so I was wondering what everyone else is using for their backup? In our old storage environment we simply rsync-ed home and software directories and projects were not backed up. With ESS we are looking for more of a GPFS based backup solution - something to tape possibly and also something that will have life cycle feature - so if the file is not touched for number of days, it's moved to a tape (something like LTFS). Thanks in advance. DAmir _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
