Bo, this is easy if you just use the right filesystem to hold the data :-) We have multiple very large scale customers using IBM Spectrum Scale for this.
Environment: Medium IBM Spectrum Scale installations with a single digit number of nodes and file systems. IBM Spectrum Protect backup archive client installed on several cluster nodes • Scalability: Tens of millions of files, Tens of terrabytes of data, up to 1.000.000.000 Objects (empirical value) • Processing: IBM Spectrum Scale mmbackup scans file system and IBM Spectrum Protect data base and generates list of backup candidates. IBM Spectrum Protect backup archive client used from mmbackup to perform file system backup. • Pros: Simple setup and usage, Good performance and scalability • Cons: All data goes to one IBM Spectrum Protect server Backup cycle: • After start mmbackup evaluates the cluster environment and verifies product versions and settings • Optional the Spectrum Protect server is queried for existing backup information. In other cases existing shadow DB is used for processing • The GPFS (Spectrum Scale) policy engine is used to generate a list files currently eligible for backup activities • Compare existing shadow DB and scan result to calculate file lists for required backup activities • Expire all files deleted in the file system since last backup run • Incremental backup all files with changed metadata in the file system since last backup run • Selective backup all files with changed data in the file system since last backup run • While backup activities ongoing update shadow DB inline • Analyse backup results from all used cluster nodes and finish backup cycle by selective backup the current shadow DB If this is not enough..see: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Tivoli%20Storage%20Manager/page/Petascale%20Data%20Protection -frank- Frank Kraemer IBM Consulting IT Specialist / Client Technical Architect Am Weiher 24, 65451 Kelsterbach mailto:[email protected] voice: +49-(0)171-3043699 / +4970342741078 IBM Germany
