Re: Looking for suggestions to deal with large backups not completing in 24-hours: the GWDG solution briefly explained

2018-07-18 Thread Bjørn Nachtwey
Hi Skylar, Skylar Thompson wrote: One thing to be aware of with partial incremental backups is the danger of backing up data multiple times if the mount points are nested. For instance, /mnt/backup/some-dir /mnt/backup/some-dir/another-dir Under normal operation, a node with DOMAIN set to

Re: Looking for suggestions to deal with large backups not completing in 24-hours: the GWDG solution briefly explained

2018-07-17 Thread Skylar Thompson
One thing to be aware of with partial incremental backups is the danger of backing up data multiple times if the mount points are nested. For instance, /mnt/backup/some-dir /mnt/backup/some-dir/another-dir Under normal operation, a node with DOMAIN set to "/mnt/backup/some-dir

Re: Looking for suggestions to deal with large backups not completing in 24-hours: the GWDG solution briefly explained

2018-07-17 Thread Bjørn Nachtwey
Hi Zoltan, OK, i will translate my text as there are some more approaches discussed :-) breaking up the filesystems in several nodes will work as long as the nodes are of suffiecient size. I'm not sure if a PROXY node will solve the problem, because each "member node" will backup the whole

Re: Looking for suggestions to deal with large backups not completing in 24-hours: the GWDG solution briefly explained

2018-07-17 Thread Zoltan Forray
Bjørn, Thank you for the details. As the common consensus is, we need to break-up the number of directories/files each node processes/scans. Also seem to need the use of the PROXY NODE process to consolidate access into one node/client since 5+ nodes will be required to process what is now