You will have some complexity with the size of your data and the size of your loader. Unless your data compresses really well. Does it have more than one tape drive? Your total loader capacity is 48 TBytes raw, and you need 2x your full size to do Consolidations or new Fulls or you have gaps in your protection.
If I’m reading this right you want an off site copy. If that’s correct I would go about this two different ways, * Get a much bigger loader with 2 drives or * Expand backups server raid6 to have Full + growth*Growth Window capacity I would then use migrate+Archive jobs to make my off site and copy to tape. In the first case you can avoid the extra migrate, just do an archive to a pool of tapes you eject. Workflow case 1 Bigger Tape loader 2 or more tape drives. * Spool to Disk * AI-Consolidated Pool Tape * AI-Incremental Pool Disk * Offsite Pool Tape Fulls and Consolidations backups go to AI-Consolidated Tape pool, Your daily go to disk until they are consolidated into tape. To create your off sites you can use a copy job of whatever full you want. https://docs.bareos.org/TasksAndConcepts/AlwaysIncrementalBackupScheme.html#copy-jobs I personally for offsite to avoid issues with Always Incremental jobs use an Archive job https://docs.bareos.org/TasksAndConcepts/AlwaysIncrementalBackupScheme.html#virtual-full-jobs This avoids the offsite tapes being upgraded to the primary copy when the Consolidate job prunes the older jobs. To do a VirtualFull archive job like this though you need enough tape for 2x the data otherwise your swapping tapes every 5.5 hours. I would then use a script called at the end of the archive job to eject the volumes and make volumestatus=Used from that Offsite pool. Load new tapes label and put back in Offsite for the next archive. To do AI jobs this way and the VirtualFull archive job you need 2 tape drives to read from one pool/tape and write to the other. It will be fast as you get LTO speeds. No second Full backup. Workflow case 2 Bigger Disk in Backup Server * Spool if you want to avoid failed jobs cluttering disk volumes, maybe skip. * AI-Consolidated Pool Disk * AI-Incremental Pool Disk * Offsite Pool Tape This would work similar, but your Fulls and Consolidations all happen on Disk. This now means your disk needs to be 2x Full + Incremental. As your Full or VirtualFull consolidation will read all the data and make a second copy before purging the old copy from the catalog. So you won’t need the disk space long, but you will need it. As for Offsite I would do the archive job again, Make a VirtualFull mark it as archive to the Offsite pool in the Tape loader. Eject all 7-8 tapes and load new ones. Assuming you don’t want to be doing a new full Offsite every time, you can do copy jobs from your pools but I have found Bareos to behave in an unintuitive way when it comes Consolidations and offline media Copies. You end up having to run the consolidate job twice (once primary copy, Copy job then becomes primary and is picked up in next consolidation, where it wants to read from that media now) When you really just want the Copy jobs to be purged in most cases along with the primary copy on a Consolidate. So I hoped you wanted just a full offsite say 1/month. Another option if you wanted is to setup a second SD with disk off site, and send just your incremental there as part of a Copy job. Sneakernet the tapes with Full less often. In general Always Incremental with copies for off site needs some work unless you are ok having a full snapshot every so often with an Archive job. I really hope Bareos improves on this in the future. Lastly if you do use the archive job. It’s it’s own stand alone job, As your using WORM and cannot reuse your tapes, you probably want to crank up their retention so you can easly restore files form any batch of tapes. I would also for all of these run an extra backup of your catalog and append it to the offsite copies so you can pull catalog data in the event everything is lost. If you are encrypting (you are right?) be sure to keep a second copy of all your keys, I keep mine in OnePassword. I also upload a dump of the catalog to a cloud storage provider (Secure as you will). Doing a bscan of 35 TBytes will take a while, so please please please keep extra copies of your catlog and keys. Also do a lot of testing with the archive jobs. How to turn them back into backup jobs to make a new VirtualFull for the real job, etc. If your Full’s ever screw up, you will want to do this (what I do for road warriors where Fulls are not really possible) to avoid the long time to do a new full from the source. Putting an Archive job back into a backup job, running a manual virtual full of a job will suck the files defined in the archive job in (won’t if it’s an archive, thus the change to update job type=B) and rebuild a full, then do an incremental. I personally do a version of case 2, but I use Migrate jobs to move the AI-Consolidated to a tape pool, but my jobs full’s are much smaller than yours. I also only do an offsite archive job once a month. Otherwise Incremental and AI-Consolidated on Disk, Migrate to tape pool purge AI-Consolidated jobs quickly. So when I consolidate it’s Tape + Disk —> Disk —> Migrate Tape —> Archive Tape # copy job to long term tape Job { Name = "Migrate-To-Offsite-AI-Consolidated" Client = myth-fd Type = Migrate Purge Migration Job = yes Pool = AI-Consolidated Level = Full Next Pool = Offsite Schedule = WeeklyCycleAfterBackup Allow Duplicate Jobs = no Priority = 4 #before catalog dump Messages = Standard Selection Type = PoolTime # 7 days Spool Data = No Selection Pattern = "." RunAfterJob = "/usr/local/bin/prune.sh” } The script forces the volumes to truncate/prune after I move the jobs off. Brock Palen 1 (989) 277-6075 bro...@mlds-networks.com www.mlds-networks.com Websites, Linux, Hosting, Joomla, Consulting > On Aug 5, 2020, at 5:40 AM, a.br...@mail.de <a.bra...@mail.de> wrote: > > > Hello everybody, > > I'm in the process of developing a regular backup-strategy and found that I > need some assistance. Here are the parameters in short: > - 35TB of medical imaging data > - daily increment of 50-60GB > - one site, 10Gb/s Backbone > - Overland NEOs LTO7 Storageloader, 8-bay. > attached to > - dedicated backupserver with 20TB RAID6, will be enhanced as needed > > I have already backed up all data on LTO7 tape (WORM, for legal reasons) as > of Dec'19. > A 2nd generation, also LTO7 WORM, is currently in progress (VERY! slow, > ~12MB/s, different story). Tapes are/will be stored offsite. > After that I'm planning to do a 3rd generation on disk inhouse and amend the > 1st and 2nd gen on tape so that I end up with three generations identical to > a certain cutoff date. > > Then, what to do next? How could a daily routine look like? > The radiologists are very concerned about their data and would like to see a > daily tape change with the ejected tape being taken offsite in the evening. A > 1-bay LTO8 drive could be purchased then, the daily tape change would be done > by an apprentice or so... > > So I thought about an Always-Incremental-B2D2T-strategy starting with the > above cutoff day. But I still have too less experience and so I'm struggeling > hard to develop a clear structure in my head - WHAT is WHEN being copied > WHERE - let alone transform that into a working bareos configuration. > > Do my thoughts appear reasonable up to that point? > BTW: can a daily tape change be realized at all, where you just push the > button to eject the TUE-tape, insert the WED-tape, and so on without having > to stop the storage-daemon in order to unlock the tape? > > Thanks for helping me with the first steps to bring this under way. > > Andreas > > > > -- > You received this message because you are subscribed to the Google Groups > "bareos-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to bareos-users+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/bareos-users/64fe4cfa-6c87-4d28-aed9-9e30de285ee1n%40googlegroups.com. -- You received this message because you are subscribed to the Google Groups "bareos-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to bareos-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/bareos-users/579569DB-324E-4D9E-90E0-9D20A4B4F723%40mlds-networks.com.
