Re: Looking for suggestions to deal with large backups not completing in 24-hours: the GWDG solution briefly explained
Hi Skylar, Skylar Thompson wrote: One thing to be aware of with partial incremental backups is the danger of backing up data multiple times if the mount points are nested. For instance, /mnt/backup/some-dir /mnt/backup/some-dir/another-dir Under normal operation, a node with DOMAIN set to "/mnt/backup/some-dir /mnt/backup/some-dir/another-dir" will backup the contents of /mnt/backup/some-dir/another-dir as a separate filespace, *and also* will backup another-dir as a subdirectory of the /mnt/backup/some-dir filespace. We reported this as a bug, and IBM pointed us at this flag that can be passed as a scheduler option to prevent this: -TESTFLAG=VMPUNDERNFSENABLED good point, even if my script works a little bit differently: by now the starting folder is not red from the "dsm.opt" file but given in the configuration file for my script "dsmci.cfg". so one run can work for one node starting on a subfolder (done si as windows has no VIRTUALMOUNTPOINT option) Within the this config file several starting folders can be declared and my script creates in the first step a global list of all folders to be backed up "partially incremental" => well, i'm not sure if i check for multiple entries in the list => and if the nesting is done on a deeper level than the list is created from, i think i won't be aware of such a set-up i will check this -- thanks for the advice! best Bjørn On Tue, Jul 17, 2018 at 04:12:17PM +0200, Bjrn Nachtwey wrote: Hi Zoltan, OK, i will translate my text as there are some more approaches discussed :-) breaking up the filesystems in several nodes will work as long as the nodes are of suffiecient size. I'm not sure if a PROXY node will solve the problem, because each "member node" will backup the whole mountpoint. You will need to do partial incremental backups. I expect you will do this based on folders, do you? So, some questions: 1) how will you distribute the folders to the nodes? 2) how will you ensure new folders are processed by one of your "member nodes"? On our filers many folders are created and deleted, sometimes a whole bunch every day. So for me, it was no option to maintain the option file manually. The approach from my script / "MAGS" does this somehow "automatically". 3) what happens if the folders grew not evenly and all the big ones are backed up by one of your nodes? (OK you can change the distribution or even add another node) 4) Are you going to map each backupnode to different nodes of the isilon cluster to distribute the traffic / workload for the isilon nodes? best Bjørn -- -- Bjørn Nachtwey Arbeitsgruppe "IT-Infrastruktur“ Tel.: +49 551 201-2181, E-Mail: bjoern.nacht...@gwdg.de -- Gesellschaft für wissenschaftliche Datenverarbeitung mbH Göttingen (GWDG) Am Faßberg 11, 37077 Göttingen, URL: http://www.gwdg.de Tel.: +49 551 201-1510, Fax: +49 551 201-2150, E-Mail: g...@gwdg.de Service-Hotline: Tel.: +49 551 201-1523, E-Mail: supp...@gwdg.de Geschäftsführer: Prof. Dr. Ramin Yahyapour Aufsichtsratsvorsitzender: Prof. Dr. Norbert Lossau Sitz der Gesellschaft: Göttingen Registergericht: Göttingen, Handelsregister-Nr. B 598 -- Zertifiziert nach ISO 9001 --
Re: Looking for suggestions to deal with large backups not completing in 24-hours: the GWDG solution briefly explained
One thing to be aware of with partial incremental backups is the danger of backing up data multiple times if the mount points are nested. For instance, /mnt/backup/some-dir /mnt/backup/some-dir/another-dir Under normal operation, a node with DOMAIN set to "/mnt/backup/some-dir /mnt/backup/some-dir/another-dir" will backup the contents of /mnt/backup/some-dir/another-dir as a separate filespace, *and also* will backup another-dir as a subdirectory of the /mnt/backup/some-dir filespace. We reported this as a bug, and IBM pointed us at this flag that can be passed as a scheduler option to prevent this: -TESTFLAG=VMPUNDERNFSENABLED On Tue, Jul 17, 2018 at 04:12:17PM +0200, Bjrn Nachtwey wrote: > Hi Zoltan, > > OK, i will translate my text as there are some more approaches discussed :-) > > breaking up the filesystems in several nodes will work as long as the nodes > are of suffiecient size. > > I'm not sure if a PROXY node will solve the problem, because each "member > node" will backup the whole mountpoint. You will need to do partial > incremental backups. I expect you will do this based on folders, do you? > So, some questions: > 1) how will you distribute the folders to the nodes? > 2) how will you ensure new folders are processed by one of your "member > nodes"? On our filers many folders are created and deleted, sometimes a > whole bunch every day. So for me, it was no option to maintain the option > file manually. The approach from my script / "MAGS" does this somehow > "automatically". > 3) what happens if the folders grew not evenly and all the big ones are > backed up by one of your nodes? (OK you can change the distribution or even > add another node) > 4) Are you going to map each backupnode to different nodes of the isilon > cluster to distribute the traffic / workload for the isilon nodes? > > best > Bjørn -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine
Re: Looking for suggestions to deal with large backups not completing in 24-hours: the GWDG solution briefly explained
Hi Zoltan, OK, i will translate my text as there are some more approaches discussed :-) breaking up the filesystems in several nodes will work as long as the nodes are of suffiecient size. I'm not sure if a PROXY node will solve the problem, because each "member node" will backup the whole mountpoint. You will need to do partial incremental backups. I expect you will do this based on folders, do you? So, some questions: 1) how will you distribute the folders to the nodes? 2) how will you ensure new folders are processed by one of your "member nodes"? On our filers many folders are created and deleted, sometimes a whole bunch every day. So for me, it was no option to maintain the option file manually. The approach from my script / "MAGS" does this somehow "automatically". 3) what happens if the folders grew not evenly and all the big ones are backed up by one of your nodes? (OK you can change the distribution or even add another node) 4) Are you going to map each backupnode to different nodes of the isilon cluster to distribute the traffic / workload for the isilon nodes? best Bjørn
Re: Looking for suggestions to deal with large backups not completing in 24-hours: the GWDG solution briefly explained
Bjørn, Thank you for the details. As the common consensus is, we need to break-up the number of directories/files each node processes/scans. Also seem to need the use of the PROXY NODE process to consolidate access into one node/client since 5+ nodes will be required to process what is now being attempted through 1-node. On Tue, Jul 17, 2018 at 8:05 AM Nachtwey, Bjoern wrote: > Hi Zoltan, > > i will come back to the approach Jonas mentioned (as I'm the author of > that text: thanks to Jonas for doing this ;-) ) > > the text is in german of course, but the script has some comments in > English and will be understandable -- I hope so :-) > > the text describes first the problem everybody on this list will know: the > treewalk takes more times than we have. > TSM/ISP has some opportunities to speed up, such as "-incrbydate", but > they do not work properly. > > So for me the only solution is to parallelize the tree walk and do partial > incremental backups. > First tried to write it with BASH commands, but multithreading was not > easy to implement and second it won't run on windows -- but our largest > filers ( 500 TB - 1.2 PB) need to be accessed via CIFS to store the ACL > information. > My first steps with PowerShell for the Windows cost lots of time and were > disappointing. > Using PERL made everything really easy as it runs on windows with the > strawberry perl software and within the script there are only a few > if-conditions needed to determine between Linux and Windows. > > I did some tests according to the depth or the level of the filetree to > dive in: > As the subfolders are of unequal size, diving just below the mount point > and parallelize on the folders of this "first level" mostly does not work > well, there's (nearly) always one folder taking all the time. > On the other hand diving into all levels will take a certain amount of > additional time. > > The best performance I do see using 3 to 4 levels and 4 to 6 parallel > threads for each node. Due to separating users and for accounting I have > several nodes on such large file systems. So in total there are about 20 to > 40 streams in parallel. > > Rudi Wüst mentioned in my text figured out a p520 server running AIX6 will > support up to 2,000 parallel streams, but as mentioned by Grant using an > isilon system the filer will be the bottle neck. > > As mentioned by Del, you may also test a commercial software "MAGS" by > general storage, it can addresses multiple isilon nodes in parallel > > If there're any questions -- just ask or have a look on the script: > https://gitlab.gwdg.de/bnachtw/dsmci > > // even if the last submit is about 4 month old, the project is still in > development ;-) > > > > ==> maybe I should update and translate the text from the "GWDG news" to > English? Any interest? > > > Best > Bjørn > > > p.s. > A Result from the wild (weekly backup of a node from a 343 TB Quantum > StorNext File System) : > >> > Process ID:12988 > Path processed: > - > Start time: 2018-07-14 12:00 > End time : 2018-07-15 06:07 > total processing time : 3d 15h 59m 23s > total wallclock time : 18h 7m 30s > effective speedup :4.855 using 6 parallel threads > datatransfertime ratio:3.575 % > - > Objects inspected : 92061596 > Objects backed up : 9774876 > Objects updated :0 > Objects deleted :0 > Objects expired : 7696 > Objects failed:0 > Bytes inspected :52818.242 (GB) > Bytes transferred : 5063.620 (GB) > - > Number of Errors :0 > Number of Warnings: 43 > # of severe Errors:0 > # Out-of-Space Errors :0 > << > > -- > > Bjørn Nachtwey > > Arbeitsgruppe "IT-Infrastruktur“ > Tel.: +49 551 201-2181, E-Mail: bjoern.nacht...@gwdg.de > -- > > Gesellschaft für wissenschaftliche Datenverarbeitung mbH Göttingen (GWDG) > Am Faßberg 11, 37077 Göttingen, URL: http://www.gwdg.de > Tel.: +49 551 201-1510, Fax: +49 551 201-2150, E-Mail: g...@gwdg.de > Service-Hotline: Tel.: +49 551 201-1523, E-Mail: supp...@gwdg.de > Geschäftsführer: Prof. Dr. Ramin Yahyapour > Aufsichtsratsvorsitzender: Prof. Dr. Norbert Lossau > Sitz der Gesellschaft: Göttingen > Registergericht: Göttingen, Handelsregister-Nr. B 598 > -- > > Zertifiziert nach ISO 9001 > >
