Re: [Bacula-users] Backup of large volumes never completes - keeps restarting

[Radosław Korzeniewski]

Hello

2018-02-23 9:32 GMT+01:00 Fourie Joubert :

> Hi Folks
>
> I know similar topics have been addressed before in many posts, but none
> of them provide me with a workable solution…
>
> - We are backing up fairly large volumes: 250TB up to 1.5 PB over 40Gbps
> Infiniband
> - Bacula backups up to a target PB-scale ZFS pool
> - The volumes created are each 100GB in size
>
> We are having trouble getting the first full backups to finish
> successfully in one job (due to various IT issues that we do not have
> control over).
>
> The result is that although the backups are configured as incrementals,
> there is never a successful full backup in a single job, and the next job
> starts backing up everything all over again. This happens over and over so
> we never get a backup of all the content and we fill up our ZFS backup
> target pool with all the uncompleted attempts.
>
> Is there a way to prevent this, so that despite a backup job being flagged
> as unsuccessfully terminated, the next session will be forced to only be
> incremental?
>

As others already pointed out in this thread you are requesting an
impossible as incremental backup in any real backup software require a
successful full. But you can resolve from this situation in the following
ways:
- resolve all the issues which prevents your full to complete successfully
- restart the failed full job which should backup all remaining data only -
it is possible when a full job finishes as incomplete and not failed one.
You can repeat this as long as all data will be archived without problem
- use a client side deduplication (available in BEE only) which should
backup the data unavailable on backup server until all data will be
archived successfully
- split your backup jobs into a number of smaller ones which has a higher
level of probability to complete with success, you can run all this jobs
concurrently to achieve a required throughput

best regards
-- 
Radosław Korzeniewski
rados...@korzeniewski.net
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Backup of large volumes never completes - keeps restarting

[Bill Arlofski]

On 02/23/2018 01:32 AM, Fourie Joubert wrote:
> Hi Folks
> 
> I know similar topics have been addressed before in many posts, but none of
> them provide me with a workable solution…
> 
> - We are backing up fairly large volumes: 250TB up to 1.5 PB over 40Gbps
> Infiniband
> - Bacula backups up to a target PB-scale ZFS pool
> - The volumes created are each 100GB in size
> 
> We are having trouble getting the first full backups to finish successfully in
> one job (due to various IT issues that we do not have control over).
> 
> The result is that although the backups are configured as incrementals, there
> is never a successful full backup in a single job, and the next job starts
> backing up everything all over again. This happens over and over so we never
> get a backup of all the content and we fill up our ZFS backup target pool with
> all the uncompleted attempts.
> 
> Is there a way to prevent this, so that despite a backup job being flagged as
> unsuccessfully terminated, the next session will be forced to only be 
> incremental?
> 
> Any advice would be sincerely appreciated!

Hi Fourie,

Your request is impossible. Unless perhaps some form of quantum mechanics is
applied to the situation. :)   (I hope I am not the only one laughing a that
joke :)

Let me explain what I mean:

You cannot have an Incremental backup that is based on no prior good Full
backup. This is simply an impossible request.

If your Fulls never complete, you can never have an Incremental (or a
Differential for that matter).

When an Incremental backup job starts, Bacula queries the catalog database to
see when the last Full, Incremental, or Differential job was successfully
completed. If there is no such job that has terminated "Backup OK", then this
Incremental will automatically be upgraded to a Full.

The situation you are describing "due to various IT issues that we do not have
control over" needs to be resolved so that you can get a Full backup job.

Perhaps you can solve this problem by splitting this very large single job
into several smaller jobs that run concurrently?

Best regards,

Bill


-- 
Bill Arlofski
http://www.revpol.com/bacula
-- Not responsible for anything below this line --

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Backup of large volumes never completes - keeps restarting

[Wanderlei Huttel]

Hello Fouri

Please, inform your Bacula version?

Atenciosamente

*Wanderlei Hüttel*
http://www.huttel.com.br

2018-02-23 5:32 GMT-03:00 Fourie Joubert :

> Hi Folks
>
> I know similar topics have been addressed before in many posts, but none
> of them provide me with a workable solution…
>
> - We are backing up fairly large volumes: 250TB up to 1.5 PB over 40Gbps
> Infiniband
> - Bacula backups up to a target PB-scale ZFS pool
> - The volumes created are each 100GB in size
>
> We are having trouble getting the first full backups to finish
> successfully in one job (due to various IT issues that we do not have
> control over).
>
> The result is that although the backups are configured as incrementals,
> there is never a successful full backup in a single job, and the next job
> starts backing up everything all over again. This happens over and over so
> we never get a backup of all the content and we fill up our ZFS backup
> target pool with all the uncompleted attempts.
>
> Is there a way to prevent this, so that despite a backup job being flagged
> as unsuccessfully terminated, the next session will be forced to only be
> incremental?
>
> Any advice would be sincerely appreciated!
>
> Best regards,
>
> Fourie
>
>
>
> This message and attachments are subject to a disclaimer.
> Please refer to http://upnet.up.ac.za/services/it/documentation/docs/
> 004167.pdf for full details.
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Bacula-users mailing list
> Bacula-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] Backup of large volumes never completes - keeps restarting

[Fourie Joubert]

Hi Folks

I know similar topics have been addressed before in many posts, but none of
them provide me with a workable solution…

- We are backing up fairly large volumes: 250TB up to 1.5 PB over 40Gbps
Infiniband
- Bacula backups up to a target PB-scale ZFS pool
- The volumes created are each 100GB in size

We are having trouble getting the first full backups to finish successfully
in one job (due to various IT issues that we do not have control over).

The result is that although the backups are configured as incrementals,
there is never a successful full backup in a single job, and the next job
starts backing up everything all over again. This happens over and over so
we never get a backup of all the content and we fill up our ZFS backup
target pool with all the uncompleted attempts.

Is there a way to prevent this, so that despite a backup job being flagged
as unsuccessfully terminated, the next session will be forced to only be
incremental?

Any advice would be sincerely appreciated!

Best regards,

Fourie

-- 
This message and attachments are subject to a disclaimer.
Please refer to 
http://upnet.up.ac.za/services/it/documentation/docs/004167.pdf for full 
details.
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users