Re: [Bacula-users] Progressive Virtual Fulls...

[Marco Gaiarin]

> full pool). But still i think that there's no way to have ONE full copy...
> clearly without 'scripting' something around (create a scratch pool/volume,
> consolidate, migrate back to original pool/volume, delete scratch).

Exactly.

 27-Sep 17:29 lnfbacula-dir JobId 11634: Start Virtual Backup JobId 11634, 
Job=FVG-SV-Obito.2023-09-27_17.29.35_58
 27-Sep 17:29 lnfbacula-dir JobId 11634: Warning: This Job is not an Accurate 
backup so is not equivalent to a Full backup.
 27-Sep 17:29 lnfbacula-dir JobId 11634: Consolidating JobIds=11594,11633
 27-Sep 17:29 lnfbacula-dir JobId 11634: Found 43953 files to consolidate into 
Virtual Full.
 27-Sep 17:29 lnfbacula-dir JobId 11634: Using Device "FileStorage" to read.
 27-Sep 17:29 lnfbacula-dir JobId 11634: Using Device "VirtualFileStorage" to 
write.
 27-Sep 17:29 svpve3-sd JobId 11634: Ready to read from volume 
"Obito_Full_0001" on File device "FileStorage" (/rpool-backup/bacula).
 27-Sep 17:29 svpve3-sd JobId 11634: Job FVG-SV-Obito.2023-09-27_17.29.35_58 is 
waiting. Cannot find any appendable volumes.
 Please use the "label" command to create a new Volume for:
 Storage:  "VirtualFileStorage" (/rpool-backup/bacula)
 Pool: FVG-SV-ObitoFilePoolFull
 Media type:   File

job need to read from 'Obito_Full_0001', and clearly cannot write on them.
So effectivaly there's no way to have a single full backup (apart, again,
doing some scripting around).


This make sense, but still i really don't understand how to manage jobs and
vlumes...


The only route seems Progessive Virtual Full, indeed:

https://www.bacula.org/whitepapers/PVF.pdf

because old job get deleted automaticaly after consolidation, probably get
deleted also the full one, right?

So, using PVF i can achive the 'one full' goal, clearly at the cost of a
temporary full for the consolidation (obviously).


Right?! Thanks.

-- 
  Io, Yolande Mukagasana, dichiaro di fronte all'umanita` che chiunque
  non voglia prendere conoscenza del calvario del popolo rwandese e`
  complice dei carnefici.   (Yolande Mukagasana)




___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Difference between purge volume and purge jobs volume

[Felix Brack]

Hi Rob,

On 27.09.23 16:20, Rob Gerber wrote:
> I suppose that would be a very thorough way to ensure no records remain.
> Remember that I am not 100% certain about what those commands do, and
> that others here may have more wisdom.
> 
For now the result of issuing the three purge commands seems to do
exactly what I want. It's probably kind of overkill.

> I have read in the docs somewhere about a way to get Bacula to reuse a
> tape that had been previously used before Bacula would usually do such a
> thing. By default I believe Bacula's behavior is to overwrite previously
> written data on a volume as an absolutely last resort (when that data
> has been previously expunged). Somewhere in the docs someone reported
> learning that they could rewind the tape, then write a record to the
> beginning of the tape that would signal to bacula that the tape was
> unused. Maybe writing an EOF marker? Check the docs. You would do this
> thing I describe to get bacula to use the whole tape sooner than it
> would usually do so.
>
Maybe this could be done but things should work without bypassing
bacula's director and storage daemon.

> Now, I don't know for sure that you're using tapes. Could be file volumes.
> 
I'm using an LTO drive for the backups.

> Robert Gerber
> 402-237-8692
> r...@craeon.net 
> 
> On Wed, Sep 27, 2023, 3:54 AM Felix Brack  > wrote:
> 
> Hello Robert,
> 
> Many thanks for the fast response!
> 
> On 26.09.23 17:36, Rob Gerber wrote:
> > My guess is that purge volume=xyz will purge any records regarding
> that
> > volume. However, purge jobs volume=xyz will purge any jobs related to
> > that volume, but not the volume itself.
> >
> > I have learned from this list that there are retention periods for
> > records pertaining to files, jobs, and volumes. This implies that
> bacula
> > considers database entries regarding files, jobs, and volumes to be
> > separate from each other. As a result, I am not sure that purging a
> > volume record would remove all associated file and job records
> from the
> > database as well. 
> >
> > I would say with some confidence that purge jobs volume=xyz will purge
> > job records related to volume xyz, but might not purge file records
> > related to that volume, and almost certainly won't purge volume
> records
> > related to that volume.
> >
> So, to make manually sure nothing prevents volume xyz to remain in
> 'used' state or turn back to 'used' state after one single job, it would
> be best to enter _all_ of the following commands:
> 
> purge volume=xyz
> purge jobs volume=xyz
> purge files volume=xyz
> 
> > Keep in mind that purging one record might result in Bacula removing
> > other records automatically. Like maybe if a volume has no jobs or
> files
> > associated, the volume might be pruned and recycled automatically. I
> > don't know if Bacula does this, but it's plausible.
> >
> > Keep in mind that purge is dangerous and ignores configured retention
> > periods. Prune is safer.
> >
> Thanks for the tip. I use purge intentionally to bypass the configured
> retention policy.
> 
> regards Felix
> 
> > Those who are more knowledgeable should be able to contribute more.
> >
> > Robert Gerber
> > 402-237-8692
> > r...@craeon.net   >
> >
> > On Tue, Sep 26, 2023, 6:58 AM Felix Brack  
> > >> wrote:
> >
> >     Hi,
> >
> >     What is the difference between:
> >
> >     purge volume="xyz"
> >
> >     and
> >
> >     purge jobs volume="xyz"
> >
> >     Or are they equivalent with respect to the result?
> >
> >     regards Felix
> >
> >
> >     ___
> >     Bacula-users mailing list
> >     Bacula-users@lists.sourceforge.net
> 
> >      >
> >     https://lists.sourceforge.net/lists/listinfo/bacula-users
> 
> >      >
> >
> 

-- 
Felix Brack
LTEC AG
Hauptstrasse 66
CH-4436 Oberdorf
SWITZERLAND
Internet: www.ltec.ch

Tel. +41 61 9631414
Fax. +41 61 9631413
E-Mail: f...@ltec.ch


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Progressive Virtual Fulls...

[Josh Fisher via Bacula-users]

On 9/26/23 12:48, Marco Gaiarin wrote:

Mandi! Rados??aw Korzeniewski
   In chel di` si favelave...


Because of this. To make Virtual Full Bacula needs to read all backup jobs
starting from Full, Diff and all Incrementals. Your Full (as volume
suggests) is stored on Obito_Full_0001 which has an improper media type.
Correct your configuration and backups and start again.

I'm really getting mad. This make sense for the behaviour (the first
VirtualFull worked because read full and incremental for the same pool) but
still the docs confuse me.

 https://www.bacula.org/9.4.x-manuals/en/main/Migration_Copy.html



No. Not because they were in the same pool, but rather because the 
volumes were all loadable and readable by the same device.




Say in 'Migration and Copy':

  For migration to work properly, you should use Pools containing only Volumes 
of the same Media Type for all migration jobs.

but in 'Important Migration Considerations':

  * Each Pool into which you migrate Jobs or Volumes must contain Volumes of 
only one Media Type.



While true, I find that misleading. It is true only because a Device 
resource can specify only a single Media Type. What is really going on 
is that Bacula makes a one-time decision when the job starts as to which 
device it will read from and which device it will write to. Once that 
read device is established, all volumes that need to be read must be 
readable by that particular device. Hence, all volumes must have the 
same Media Type, because they must all be read by the same read device.




  [...]
  * Bacula currently does only minimal Storage conflict resolution, so you must 
take care to ensure that you don't try to read
and write to the same device or Bacula may block waiting to reserve a drive 
that it will never find.
In general, ensure that all your migration pools contain only one Media 
Type, and that you always migrate to pools with different Media Types.

and in 'Virtual Backup Consolidation':

  In some respects the Virtual Backup feature works similar to a Migration job, 
in that Bacula normally reads the data from the pool
  specified in the Job resource, and writes it to the Next Pool specified in 
the Job resource. Note, this means that usually the output
  from the Virtual Backup is written into a different pool from where your 
prior backups are saved. Doing it this way guarantees that you
  will not get a deadlock situation attempting to read and write to the same 
volume in the Storage daemon. If you then want to do
  subsequent backups, you may need to move the Virtual Full Volume back to your 
normal backup pool. Alternatively, you can set your
  Next Pool to point to the current pool. This will cause Bacula to read and 
write to Volumes in the current pool. In general, this will
  work, because Bacula will not allow reading and writing on the same Volume. 
In any case, once a VirtualFull has been created, and a
  restore is done involving the most current Full, it will read the Volume or 
Volumes by the VirtualFull regardless of in which Pool the
  Volume is found.



In a nutshell, you must have multiple devices and you must ensure that 
one device reads all of the existing volumes and another different 
device writes the new virtual full volumes. This is why it is not 
possible to do virtual fulls or migration with only a single tape drive.




So, after an aspirin, seems to me that:

1) for a virtual backup, the reading part need to READ jobs from volumes
  with the same 'Media Type'; so i can use different pools/storage, but have
to use the same 'Media Type'.



Yes. Also, if you have multiple devices with the same Media Type, you 
must make certain that any volume with that Media Type can be loaded 
into any of those devices.




2) i can use the same pool, but clearly i cannot guarantee that i'll have
  only TWO full backup; error in rotation, ... could sooner or later lead to
a virtualfull job will write to an empty volume, making another copy of
data; surely i need TWO full copy (one for reading, one for writing).



A new full backup is made each time the virtual full job runs. You can 
purge volumes in a RunAfter script or you limit the number of volumes in 
the pool, but I guess the only way to guarantee there are only two 
copies is to manually purge old volumes.




3) i can use different pool, but with the same media type; in this way i can
  somewhat limit the number of full backup to two (using two media in the
full pool). But still i think that there's no way to have ONE full copy...
clearly without 'scripting' something around (create a scratch pool/volume,
consolidate, migrate back to original pool/volume, delete scratch).



I don't understand the goal, I think, but MaximumVolumeJobs in the Pool 
resource might work.






I'm missing something? Thanks.




___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net

Re: [Bacula-users] Difference between purge volume and purge jobs volume

[Rob Gerber]

I suppose that would be a very thorough way to ensure no records remain.
Remember that I am not 100% certain about what those commands do, and that
others here may have more wisdom.

I have read in the docs somewhere about a way to get Bacula to reuse a tape
that had been previously used before Bacula would usually do such a thing.
By default I believe Bacula's behavior is to overwrite previously written
data on a volume as an absolutely last resort (when that data has been
previously expunged). Somewhere in the docs someone reported learning that
they could rewind the tape, then write a record to the beginning of the
tape that would signal to bacula that the tape was unused. Maybe writing an
EOF marker? Check the docs. You would do this thing I describe to get
bacula to use the whole tape sooner than it would usually do so.

Now, I don't know for sure that you're using tapes. Could be file volumes.

Robert Gerber
402-237-8692
r...@craeon.net

On Wed, Sep 27, 2023, 3:54 AM Felix Brack  wrote:

> Hello Robert,
>
> Many thanks for the fast response!
>
> On 26.09.23 17:36, Rob Gerber wrote:
> > My guess is that purge volume=xyz will purge any records regarding that
> > volume. However, purge jobs volume=xyz will purge any jobs related to
> > that volume, but not the volume itself.
> >
> > I have learned from this list that there are retention periods for
> > records pertaining to files, jobs, and volumes. This implies that bacula
> > considers database entries regarding files, jobs, and volumes to be
> > separate from each other. As a result, I am not sure that purging a
> > volume record would remove all associated file and job records from the
> > database as well.
> >
> > I would say with some confidence that purge jobs volume=xyz will purge
> > job records related to volume xyz, but might not purge file records
> > related to that volume, and almost certainly won't purge volume records
> > related to that volume.
> >
> So, to make manually sure nothing prevents volume xyz to remain in
> 'used' state or turn back to 'used' state after one single job, it would
> be best to enter _all_ of the following commands:
>
> purge volume=xyz
> purge jobs volume=xyz
> purge files volume=xyz
>
> > Keep in mind that purging one record might result in Bacula removing
> > other records automatically. Like maybe if a volume has no jobs or files
> > associated, the volume might be pruned and recycled automatically. I
> > don't know if Bacula does this, but it's plausible.
> >
> > Keep in mind that purge is dangerous and ignores configured retention
> > periods. Prune is safer.
> >
> Thanks for the tip. I use purge intentionally to bypass the configured
> retention policy.
>
> regards Felix
>
> > Those who are more knowledgeable should be able to contribute more.
> >
> > Robert Gerber
> > 402-237-8692
> > r...@craeon.net 
> >
> > On Tue, Sep 26, 2023, 6:58 AM Felix Brack  > > wrote:
> >
> > Hi,
> >
> > What is the difference between:
> >
> > purge volume="xyz"
> >
> > and
> >
> > purge jobs volume="xyz"
> >
> > Or are they equivalent with respect to the result?
> >
> > regards Felix
> >
> >
> > ___
> > Bacula-users mailing list
> > Bacula-users@lists.sourceforge.net
> > 
> > https://lists.sourceforge.net/lists/listinfo/bacula-users
> > 
> >
>
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Difference between purge volume and purge jobs volume

[Felix Brack]

Hello Robert,

Many thanks for the fast response!

On 26.09.23 17:36, Rob Gerber wrote:
> My guess is that purge volume=xyz will purge any records regarding that
> volume. However, purge jobs volume=xyz will purge any jobs related to
> that volume, but not the volume itself.
> 
> I have learned from this list that there are retention periods for
> records pertaining to files, jobs, and volumes. This implies that bacula
> considers database entries regarding files, jobs, and volumes to be
> separate from each other. As a result, I am not sure that purging a
> volume record would remove all associated file and job records from the
> database as well. 
> 
> I would say with some confidence that purge jobs volume=xyz will purge
> job records related to volume xyz, but might not purge file records
> related to that volume, and almost certainly won't purge volume records
> related to that volume.
> 
So, to make manually sure nothing prevents volume xyz to remain in
'used' state or turn back to 'used' state after one single job, it would
be best to enter _all_ of the following commands:

purge volume=xyz
purge jobs volume=xyz
purge files volume=xyz

> Keep in mind that purging one record might result in Bacula removing
> other records automatically. Like maybe if a volume has no jobs or files
> associated, the volume might be pruned and recycled automatically. I
> don't know if Bacula does this, but it's plausible.
> 
> Keep in mind that purge is dangerous and ignores configured retention
> periods. Prune is safer.
> 
Thanks for the tip. I use purge intentionally to bypass the configured
retention policy.

regards Felix

> Those who are more knowledgeable should be able to contribute more.
> 
> Robert Gerber
> 402-237-8692
> r...@craeon.net 
> 
> On Tue, Sep 26, 2023, 6:58 AM Felix Brack  > wrote:
> 
> Hi,
> 
> What is the difference between:
> 
> purge volume="xyz"
> 
> and
> 
> purge jobs volume="xyz"
> 
> Or are they equivalent with respect to the result?
> 
> regards Felix
> 
> 
> ___
> Bacula-users mailing list
> Bacula-users@lists.sourceforge.net
> 
> https://lists.sourceforge.net/lists/listinfo/bacula-users
> 
> 


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users