Re: [Bacula-users] Storage Management

[Ana Emília M . Arruda]

Great news! You're welcome Nick!
Nice to hear everything is working fine now :-)

Best,
Ana

On Thu, Dec 1, 2022 at 4:49 PM Nick Bright  wrote:

> Previously I had tried truncating and the output was essentially 'nothing
> to truncate', but today it's truncating volumes - it may have just needed
> some time to pass before running successfully.
>
> Thank you very much for your help on this!
>
>  - Nick Bright
> On 12/1/22 05:53, Ana Emília M. Arruda wrote:
>
> Hello Nick,
>
> It seems to me that everything is working fine now.
>
> The job log you have sent here is from November 23th, and I see the
> Vol-0001 has been used on November 25th:
>
> |   1 | Vol-0001   | Full  |   1 | 53,687,078,657 |   12 |
>  604,800 |   1 |0 | 0 | File1 |   1 |0
> | 2022-11-25 08:55:06 |   258,265 |
>
> Have you tried to truncate the volumes?
>
> * truncate volume pool=File storage=File1
>
> Volumes seem to be getting recycled. Can you please confirm?
>
> Thanks!
> Best,
> Ana
>
>
>
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Storage Management

[Nick Bright]

Previously I had tried truncating and the output was essentially 
'nothing to truncate', but today it's truncating volumes - it may have 
just needed some time to pass before running successfully.


Thank you very much for your help on this!

 - Nick Bright

On 12/1/22 05:53, Ana Emília M. Arruda wrote:

Hello Nick,

It seems to me that everything is working fine now.

The job log you have sent here is from November 23th, and I see the 
Vol-0001 has been used on November 25th:


|       1 | Vol-0001 | Full      |       1 | 53,687,078,657 |       12 
|  604,800 |       1 |    0 |         0 | File1     |       1 |       
 0 | 2022-11-25 08:55:06 |   258,265 |


Have you tried to truncate the volumes?

* truncate volume pool=File storage=File1

Volumes seem to be getting recycled. Can you please confirm?

Thanks!
Best,
Ana
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Storage Management

[Ana Emília M . Arruda]

Hello Nick,

It seems to me that everything is working fine now.

The job log you have sent here is from November 23th, and I see the
Vol-0001 has been used on November 25th:

|   1 | Vol-0001   | Full  |   1 | 53,687,078,657 |   12 |
 604,800 |   1 |0 | 0 | File1 |   1 |0
| 2022-11-25 08:55:06 |   258,265 |

Have you tried to truncate the volumes?

* truncate volume pool=File storage=File1

Volumes seem to be getting recycled. Can you please confirm?

Thanks!
Best,
Ana

On Tue, Nov 29, 2022 at 4:17 PM Nick Bright  wrote:

> On 11/29/22 07:40, Ana Emília M. Arruda wrote:
>
> If you have volumes in Purged status, the truncate command should be able
> to truncate volumes. Unless there is a misconfiguration related to media
> types and storages.
>
> Can you please share with us the following?
>
> -  list media output
> - "File1"configuration in both the bacula-dir.conf and bacula-sd.conf files
>
> Thank you!
> Best,
> Ana
>
> I've attached the list media output as it was too substantial to paste
> here.
>
> Here are the configuration references to "File1" from bacula-dir.conf and
> bacula-ds.conf:
>
> Autochanger {
>   Name = File1
>   Address = MYHOSTNAME
>   SDPort = 9103
>   Password = "MYPASSWORD"
>   Device = FileChgr1
>   Media Type = File1
>   Maximum Concurrent Jobs = 10
>   Autochanger = File1
> }
> Device {
>   Name = FileChgr1-Dev1
>   Media Type = File1
>   Archive Device = /var/lib/bacula
>   LabelMedia = yes;
>   Random Access = Yes;
>   AutomaticMount = yes;
>   RemovableMedia = no;
>   AlwaysOpen = no;
>   Maximum Concurrent Jobs = 5
> }
> Device {
>   Name = FileChgr1-Dev2
>   Media Type = File1
>   Archive Device = /var/lib/bacula
>   LabelMedia = yes;
>   Random Access = Yes;
>   AutomaticMount = yes;
>   RemovableMedia = no;
>   AlwaysOpen = no;
>   Maximum Concurrent Jobs = 5
> }
>
> --
> -
> -  Nick Bright  -
> -  KwiKom Communications-
> -  Office 800-379-7292  -
> -  Direct 620-228-5653  -
> -  Web https://www.kwikom.com/  -
> -
>
>
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Storage Management

[Ana Emília M . Arruda]

Hello Nick,

If you have volumes in Purged status, the truncate command should be able
to truncate volumes. Unless there is a misconfiguration related to media
types and storages.

Can you please share with us the following?

-  list media output
- "File1"configuration in both the bacula-dir.conf and bacula-sd.conf files

Thank you!
Best,
Ana

On Fri, Nov 25, 2022 at 5:08 PM Nick Bright  wrote:

> On 11/25/22 09:20, Ana Emília M. Arruda wrote:
>
> Can you share your pool resource configuration here? Bacula has many
> directives to different ways to recycle volumes, it would be nice to see
> how you have the pool configured.
>
> Sure, here is my pool configuration:
>
> Pool {
>   Name = File
>   Pool Type = Backup
>   Recycle = yes
>   Recycle Oldest Volume = yes
>   AutoPrune = yes
>   Volume Retention = 7 days
>   Maximum Volume Bytes = 50G
>   Maximum Volumes = 310
>   Label Format = "Vol-"
>   Action On Purge = Truncate
>   Volume Use Duration = 14h
> }
>
> It's notable that this configuration leaves about 450GB of free space on
> the pool partition, so there is plenty of working room. I had initially
> grown the Maximum Volumes setting several times before stopping and
> investigating why volumes weren't being reused; presuming that completely
> filling the partition would likely be A Bad Idea(tm).
>
> After that, you will be able to "truncate volumes in Purged status" using
>> the following command:
>>
>> * truncate volume allpools storage=File1
>>
>> This command reports "No volumes found to perform the command"
>>
> This command will only apply to volumes in "Purged" status. If they have
> been reused, there will be no volume to truncate. Please check if it works
> having at least one volume in Purged status.
>
> After the steps taken so far, I'm showing that I have a significant number
> of volumes in "Purged" status - so, if I understand correctly, those will
> be available for re-use as Bacula requires more space for more backups.
>
> So; with your help I see how to manually prune files, but this seems
>> inelegant. The system should automatically be pruning or at least allowing
>> me to overwrite the old data - it ought to be cyclical - use the available
>> disk space (17T) to back up systems, letting old backups fall off
>> automatically so that new backups can take their place.
>>
>> Is needing to manually remove stale backups the intended behavior of the
>> system, or have I done something incorrectly?
>>
> Bacula will automatically reuse volumes, but it will not destroy data if
> the retention hasn't expired yet.
> It would be nice if you could share the pool resource configuration and a
> job log reporting no volume available to use.
> The best approach is to reuse volumes as soon as they have retention
> expired,  but to not keep the pool too tight as it may happen Bacula can't
> create a new volume or reuse any.
> Limiting the pool size (in bytes or in number of volumes) and keeping a
> retention that can fit into this pool size is ideal.
>
> The pool configuration is listed above, here's an example of a job log
> reporting no volumes available (this is before files were purged allowing
> the currently pending backups to run - they all finished)
>
> 2022-11-23 13:01:31 baculaserver-dir JobId 904: Start Backup JobId 904,
> Job=client.2022-11-23_13.01.29_31
> 2022-11-23 13:01:31 baculaserver-dir JobId 904: Connected to Storage
> "File1" at baculaserver:9103 with TLS
> 2022-11-23 13:01:31 baculaserver-dir JobId 904: Pruning oldest volume
> "Vol-0001"
> 2022-11-23 13:01:31 baculaserver-dir JobId 904: Found no Job associated
> with the Volume "Vol-0001" to prune
> 2022-11-23 13:01:31 baculaserver-dir JobId 904: Using Device
> "FileChgr1-Dev1" to write.
> 2022-11-23 13:01:31 baculaserver-dir JobId 904: Connected to Client
> "client" at IPADDR:9102 with TLS
> 2022-11-23 13:01:31 client JobId 904: Connected to Storage at
> baculaserver:9103 with TLS
> 2022-11-23 13:01:32 baculaserver-dir JobId 904: Pruning oldest volume
> "Vol-0001"
> 2022-11-23 13:01:32 baculaserver-dir JobId 904: Found no Job associated
> with the Volume "Vol-0001" to prune
> 2022-11-23 13:01:32 baculaserver-dir JobId 904: Pruning oldest volume
> "Vol-0001"
> 2022-11-23 13:01:32 baculaserver-dir JobId 904: Found no Job associated
> with the Volume "Vol-0001" to prune
> 2022-11-23 13:01:32 baculaserver-sd JobId 904: Job
> client.2022-11-23_13.01.29_31 is waiting. Cannot find any appendable
> volumes.
> Please use the "label" command to create a new Volume for:
> Storage:  "FileChgr1-Dev1" (/var/lib/bacula)
> Pool: File
> Media type:   File1
> 2022-11-23 13:03:47 baculaserver-sd JobId 904: Error: bsock.c:282 Bsock
> send while terminated=1 on call to client:IPADDR:9103
> 2022-11-23 13:03:47 baculaserver-sd JobId 904: askdir.c:859 Job was
> stopped by the user.
>
> Thanks,
>
> --
> -
> -  Nick Bright  -
> -  KwiKom Communications-
> -  Office 

Re: [Bacula-users] Storage Management

[Nick Bright]

On 11/25/22 09:20, Ana Emília M. Arruda wrote:
Can you share your pool resource configuration here? Bacula has many 
directives to different ways to recycle volumes, it would be nice to 
see how you have the pool configured.


Sure, here is my pool configuration:

Pool {
  Name = File
  Pool Type = Backup
  Recycle = yes
  Recycle Oldest Volume = yes
  AutoPrune = yes
  Volume Retention = 7 days
  Maximum Volume Bytes = 50G
  Maximum Volumes = 310
  Label Format = "Vol-"
  Action On Purge = Truncate
  Volume Use Duration = 14h
}

It's notable that this configuration leaves about 450GB of free space on 
the pool partition, so there is plenty of working room. I had initially 
grown the Maximum Volumes setting several times before stopping and 
investigating why volumes weren't being reused; presuming that 
completely filling the partition would likely be A Bad Idea(tm).



After that, you will be able to "truncate volumes in Purged
status" using the following command:

* truncate volume allpools storage=File1


This command reports "No volumes found to perform the command"

This command will only apply to volumes in "Purged" status. If they 
have been reused, there will be no volume to truncate. Please check if 
it works having at least one volume in Purged status.


After the steps taken so far, I'm showing that I have a significant 
number of volumes in "Purged" status - so, if I understand correctly, 
those will be available for re-use as Bacula requires more space for 
more backups.



So; with your help I see how to manually prune files, but this
seems inelegant. The system should automatically be pruning or at
least allowing me to overwrite the old data - it ought to be
cyclical - use the available disk space (17T) to back up systems,
letting old backups fall off automatically so that new backups can
take their place.

Is needing to manually remove stale backups the intended behavior
of the system, or have I done something incorrectly?

Bacula will automatically reuse volumes, but it will not destroy data 
if the retention hasn't expired yet.
It would be nice if you could share the pool resource configuration 
and a job log reporting no volume available to use.
The best approach is to reuse volumes as soon as they have retention 
expired,  but to not keep the pool too tight as it may happen Bacula 
can't create a new volume or reuse any.
Limiting the pool size (in bytes or in number of volumes) and keeping 
a retention that can fit into this pool size is ideal.


The pool configuration is listed above, here's an example of a job log 
reporting no volumes available (this is before files were purged 
allowing the currently pending backups to run - they all finished)


2022-11-23 13:01:31 baculaserver-dir JobId 904: Start Backup JobId 904, 
Job=client.2022-11-23_13.01.29_31
2022-11-23 13:01:31 baculaserver-dir JobId 904: Connected to Storage 
"File1" at baculaserver:9103 with TLS
2022-11-23 13:01:31 baculaserver-dir JobId 904: Pruning oldest volume 
"Vol-0001"
2022-11-23 13:01:31 baculaserver-dir JobId 904: Found no Job associated 
with the Volume "Vol-0001" to prune
2022-11-23 13:01:31 baculaserver-dir JobId 904: Using Device 
"FileChgr1-Dev1" to write.
2022-11-23 13:01:31 baculaserver-dir JobId 904: Connected to Client 
"client" at IPADDR:9102 with TLS
2022-11-23 13:01:31 client JobId 904: Connected to Storage at 
baculaserver:9103 with TLS
2022-11-23 13:01:32 baculaserver-dir JobId 904: Pruning oldest volume 
"Vol-0001"
2022-11-23 13:01:32 baculaserver-dir JobId 904: Found no Job associated 
with the Volume "Vol-0001" to prune
2022-11-23 13:01:32 baculaserver-dir JobId 904: Pruning oldest volume 
"Vol-0001"
2022-11-23 13:01:32 baculaserver-dir JobId 904: Found no Job associated 
with the Volume "Vol-0001" to prune
2022-11-23 13:01:32 baculaserver-sd JobId 904: Job 
client.2022-11-23_13.01.29_31 is waiting. Cannot find any appendable 
volumes.

Please use the "label" command to create a new Volume for:
    Storage:  "FileChgr1-Dev1" (/var/lib/bacula)
    Pool: File
    Media type:   File1
2022-11-23 13:03:47 baculaserver-sd JobId 904: Error: bsock.c:282 Bsock 
send while terminated=1 on call to client:IPADDR:9103
2022-11-23 13:03:47 baculaserver-sd JobId 904: askdir.c:859 Job was 
stopped by the user.


Thanks,

--
-
-  Nick Bright  -
-  KwiKom Communications-
-  Office 800-379-7292  -
-  Direct 620-228-5653  -
-  Webhttps://www.kwikom.com/   -
-
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Storage Management

[Josip Deanovic]

On 2022-11-25 16:00, Nick Bright wrote:
[...]

inelegant. The system should automatically be pruning or at least
allowing me to overwrite the old data - it ought to be cyclical - use
the available disk space (17T) to back up systems, letting old backups
fall off automatically so that new backups can take their place.


This is exactly what I am doing using these directives in the pool 
resource

configuration:

Maximum Volume Bytes = 10G
Maximum Volumes =   # to be calculated from disk 
space
   # and Maximum Volume Bytes (make 
sure
   # to leave extra space for at 
least

   # 2 x Maximum Volume Bytes)
AutoPrune = yes
Volume Retention = 
Recycle = yes
Purge Oldest Volume = yes

The configuration above would recycle the oldest volume when needed,
regardless of the retention period.
This is potentially dangerous. You should have enough volumes in your
pool to minimize the chance of losing useful (relatively recent) backup
data.


Regards

--
Josip Deanovic


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Storage Management

[Ana Emília M . Arruda]

Hello Nick,

On Fri, Nov 25, 2022 at 4:00 PM Nick Bright  wrote:

> On 11/25/22 04:45, Ana Emília M. Arruda wrote:
>
> The easiest way is to allow Bacula to automatically prune Jobs and Files
> from the Catalog. It means to have "AutoPrune = Yes" in both the Client
> resource and in the Pool resource.
>
> I already had AutoPrune = yes in both the client and pool; I had found
> this in my searches before posting here.
>
> All backups over the weekend failed with "need more volumes".
>
Can you share your pool resource configuration here? Bacula has many
directives to different ways to recycle volumes, it would be nice to see
how you have the pool configured.

> Now, to force the prune of volumes that have the VolumeRetention (value
> defined in the Pool resource) expired, the command is:
>
> * prune volume expired yes
>
> This command is dangerous, it will delete all jobs and files associated
> with any volume by comparing only the "VolumeRetention" of the volume. This
> value is defined in the Pool resource, but it is specific for a volume and
> Bacula will check the value in the "volretention" field in the Media table
> in the Bacula database for each volume.
>
> This command still reports nothing to prune for all 310 volumes as of this
> morning, and I had run "update volume fromallpools" last week after
> reducing retention times; though it does now report the volumes as "Expired"
>
> Then running the command again, it gives me some prompts about pruning
> files/jobs/volume/stats/snapshots/events then a client to choose.
>
Right. This is expected.

> I choose a client, it reports "Pruned Files from 6 Jobs for client" and
> pending backups started executing.
>
Ok, so you got volumes pruned and reused.

>
> Then, to truncate volumes you must have the following configuration in the
> Pool resource:
>
> ActionOnPurge = Truncate
>
> Already had this present as well.
>
Good.

>
> After that, you will be able to "truncate volumes in Purged status" using
> the following command:
>
> * truncate volume allpools storage=File1
>
> This command reports "No volumes found to perform the command"
>
This command will only apply to volumes in "Purged" status. If they have
been reused, there will be no volume to truncate. Please check if it works
having at least one volume in Purged status.

> So; with your help I see how to manually prune files, but this seems
> inelegant. The system should automatically be pruning or at least allowing
> me to overwrite the old data - it ought to be cyclical - use the available
> disk space (17T) to back up systems, letting old backups fall off
> automatically so that new backups can take their place.
>
> Is needing to manually remove stale backups the intended behavior of the
> system, or have I done something incorrectly?
>
Bacula will automatically reuse volumes, but it will not destroy data if
the retention hasn't expired yet.
It would be nice if you could share the pool resource configuration and a
job log reporting no volume available to use.
The best approach is to reuse volumes as soon as they have retention
expired,  but to not keep the pool too tight as it may happen Bacula can't
create a new volume or reuse any.
Limiting the pool size (in bytes or in number of volumes) and keeping a
retention that can fit into this pool size is ideal.

Best,
Ana

-- 
> -
> -  Nick Bright  -
> -  KwiKom Communications-
> -  Office 800-379-7292  -
> -  Direct 620-228-5653  -
> -  Web https://www.kwikom.com/  -
> -
>
>
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Storage Management

[Nick Bright]

On 11/25/22 04:45, Ana Emília M. Arruda wrote:
The easiest way is to allow Bacula to automatically prune Jobs and 
Files from the Catalog. It means to have "AutoPrune = Yes" in both the 
Client resource and in the Pool resource.


I already had AutoPrune = yes in both the client and pool; I had found 
this in my searches before posting here.


All backups over the weekend failed with "need more volumes".

Now, to force the prune of volumes that have the VolumeRetention 
(value defined in the Pool resource) expired, the command is:


* prune volume expired yes

This command is dangerous, it will delete all jobs and files 
associated with any volume by comparing only the "VolumeRetention" of 
the volume. This value is defined in the Pool resource, but it is 
specific for a volume and Bacula will check the value in the 
"volretention" field in the Media table in the Bacula database for 
each volume.


This command still reports nothing to prune for all 310 volumes as of 
this morning, and I had run "update volume fromallpools" last week after 
reducing retention times; though it does now report the volumes as "Expired"


Then running the command again, it gives me some prompts about pruning 
files/jobs/volume/stats/snapshots/events then a client to choose.


I choose a client, it reports "Pruned Files from 6 Jobs for client" and 
pending backups started executing.




Then, to truncate volumes you must have the following configuration in 
the Pool resource:


ActionOnPurge = Truncate


Already had this present as well.



After that, you will be able to "truncate volumes in Purged status" 
using the following command:


* truncate volume allpools storage=File1


This command reports "No volumes found to perform the command"

So; with your help I see how to manually prune files, but this seems 
inelegant. The system should automatically be pruning or at least 
allowing me to overwrite the old data - it ought to be cyclical - use 
the available disk space (17T) to back up systems, letting old backups 
fall off automatically so that new backups can take their place.


Is needing to manually remove stale backups the intended behavior of the 
system, or have I done something incorrectly?


--
-
-  Nick Bright  -
-  KwiKom Communications-
-  Office 800-379-7292  -
-  Direct 620-228-5653  -
-  Webhttps://www.kwikom.com/   -
-
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Storage Management

[Ana Emília M . Arruda]

Hello Nick,

Bill, I'm here :-) Sorry for not jumping in earlier.

Nick, I will try to summarize a bit about retention values and
pruning/truncation in Bacula. Pruning is about "the deletion of jobs and
files from Catalog". Automatic or manual prune will not touch the data in
the volumes, but only the jobs and files records in the Catalog are deleted.

The easiest way is to allow Bacula to automatically prune Jobs and Files
from the Catalog. It means to have "AutoPrune = Yes" in both the Client
resource and in the Pool resource.

1) having "AutoPrune = Yes" in the Client resource -> at the end of each
job run associated to the Client/Pool of the job, Bacula will check if the
retention have passed for all the jobids associated to the Client/Pool and
the jobs and files are deleted from the Catalog if the retention has
passed. You will see this kind of message:

| bacula-dir JobId 5104: Begin pruning Jobs older than 6 months . |
| bacula-dir JobId 5104: No Jobs found to prune.  |
| bacula-dir JobId 5104: Begin pruning Files. |
| bacula-dir JobId 5104: No Files found to prune. |
| bacula-dir JobId 5104: End auto prune.  |

If any job/file has reached the retention based on the values defined in
the Client and/or Pool resources. The values set in a Pool resource will
take precedence over the values set in a Client resource.

2) having "AutoPrune = Yes" in the Pool resource -> when Bacula needs to
reuse a volume for a job (if Bacula can't create a new volume because the
max number of volumes in the pool is reached and there is no volume in
Append, Purged or Recycle status), the recycling algorithm will try to
prune jobs and files to reuse a volume. It will start searching for the
oldest volume in the pool. Then, Bacula will check if this volume has no
more jobs associated with it in the Catalog (it means the jobs and files
have been pruned so this volume can be reused). If so, you will see
messages like:

04-Nov 15:15 bacula-dir JobId 3560: There are no more Jobs associated with
Volume "Vol-0038". Marking it purged.
04-Nov 15:15 bacula-dir JobId 3560: All records pruned from Volume
"Vol-0038"; marking it "Purged"
04-Nov 15:15 bacula-dir JobId 3560: Recycled volume "Vol-0038"

This is a job log where Bacula couldn't create a new volume for jobid=3560
and it reused Vol-0038 because the jobs associated with it had been pruned
(by automatic prune after the job run) previously.

Now, to force the prune of volumes that have the VolumeRetention (value
defined in the Pool resource) expired, the command is:

* prune volume expired yes

This command is dangerous, it will delete all jobs and files associated
with any volume by comparing only the "VolumeRetention" of the volume. This
value is defined in the Pool resource, but it is specific for a volume and
Bacula will check the value in the "volretention" field in the Media table
in the Bacula database for each volume.

This is why when you change the "VolumeRetention" value in the Pool
resource the value of the already existing volumes is not modified and you
have to run:

* update volume fromallpools
All Volume defaults updated from "Default" Pool record.
All Volume defaults updated from "File" Pool record.
All Volume defaults updated from "Scratch" Pool record.

After that, the volretention values in the Catalog for the existing volumes
is updated to the value defined in the Pool configuration and the "prune
expired volume yes" will use the new value to compare with.

Then, to truncate volumes you must have the following configuration in the
Pool resource:

ActionOnPurge = Truncate

Please remember to reload the configuration in bconsole using the reload
command after adding the above line.

After that, you will be able to "truncate volumes in Purged status" using
the following command:

* truncate volume allpools storage=File1

Please let us know if it helps.

Best,
Ana


On Wed, Nov 23, 2022 at 11:04 PM Nick Bright  wrote:

> I changed the Volume retention to 7 days, repeated all of the commands,
> and ran a BackupCatalog job; still reports "The job needs media"
>
> On 11/23/22 13:39, Bill Arlofski via Bacula-users wrote:
> > On 11/23/22 12:27, Nick Bright wrote:
> >> In the Pool definition there is a Volume Retention of 365 days; however
> >> each client has it's own File Retention and Job Retention (7 days each)
> >>
> >> Shouldn't this result in the data contained within the volumes being
> >> expired, and thus the volume could be recycled/purged/truncated once
> >> its' containing datas' retention period has expired?
> >
> >
> >
> > A couple things to consider here:
> >
> >
> > For File and Job retentions, manual says:
> >
> > - When this time period expires, and if AutoPrune is set to yes,
> > Bacula will prune (remove) File|Job records that are older
> > than the specified File Retention period
> >
> > - The shortest retention period of the three 

Re: [Bacula-users] Storage Management

[Nick Bright]

I changed the Volume retention to 7 days, repeated all of the commands, 
and ran a BackupCatalog job; still reports "The job needs media"


On 11/23/22 13:39, Bill Arlofski via Bacula-users wrote:

On 11/23/22 12:27, Nick Bright wrote:

In the Pool definition there is a Volume Retention of 365 days; however
each client has it's own File Retention and Job Retention (7 days each)

Shouldn't this result in the data contained within the volumes being
expired, and thus the volume could be recycled/purged/truncated once
its' containing datas' retention period has expired?




A couple things to consider here:


For File and Job retentions, manual says:

- When this time period expires, and if AutoPrune is set to yes, 
Bacula will prune (remove) File|Job records that are older

than the specified File Retention period

- The shortest retention period of the three takes precedence.

So, I seem to recall that manual prunes do not trigger this action. I 
could be wrong here. I have seen some complicated
things with retention periods and pruning in the years I have worked 
with Bacula. :)


Of all the people I know, my colleague Ana is the expert I refer to for
retention and pruning questions. :)  ANA! Where are
you?  :D


Also, something else to consider since I do not know your full 
environment:


If you have jobs on a volume from several different clients, and the 
files/jobs of one client meet their retention period and
are auto-pruned, consider that other jobs from clients with longer 
retention periods will prevent this volume from being

pruned/purged/recycled/re-used.

I will ping Ana off list to see if she might chime in here.


Best regards,
Bill


--
Bill Arlofski
w...@protonmail.com



___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users


--
-
-  Nick Bright  -
-  KwiKom Communications-
-  Office 800-379-7292  -
-  Direct 620-228-5653  -
-  Web https://www.kwikom.com/  -
-



___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Storage Management

[Chris Miller]

If your volumes were allowed to fill the disk to 100%, that might be part
of your problem. Your (max volumes * volume size) should leave some buffer
space on the disk allocated to Bacula. If your working directory is on the
same disk, you need to be even more generous with that buffer (I often use
a separate disk for this). Also your sql database should be on a different
volume. All these can cumulatively exceed your storage limit.

I ran into this problem before, but I don't recall exactly how I fixed
it, so I'll just talk about this out loud here. The retention time may need
to be adjusted on a per client basis in addition to a global value,
otherwise this may be holding the volumes as in-use. Beyond changing your
retention time, you probably need to restart bacula if you haven't already.

You need to get the oldest backups in the database to reflect (fall outside
of) the updated retention period through the console (this should be a
function of the retention time updates to the client definitions). Bacula
has a notion of "recycling" volumes, but it won't delete them. This is one
of those "you never want to get here" kind of things.

Once you can "expire" (not a Bacula term) the oldest backups, you should
end up with one or more volumes which can be recycled, they will show a
"purged" status in the console I believe. If any backup is still valid, it
will hold that volume as in-use.  Once the target volume is reported as
"Purged" you can use the "delete volume" command, and hopefully you are
good to go.

On a related note, with respect to pruning, this does not recover space or
delete volumes which it seems you may need to do. From the manual

:

"By setting AutoPrune to yes you will permit Bacula to automatically prune
all Volumes in the Pool when a Job needs another Volume. Volume pruning
means removing records from the catalog. It does not shrink the size of the
Volume or affect the Volume data until the Volume gets overwritten. When a
Job requests another volume and there are no Volumes with Volume Status
Append available, Bacula will begin volume pruning. This means that all
Jobs that are older than the VolumeRetention period will be pruned from
every Volume that has Volume Status Full or Used and has Recycle set to
yes. Pruning consists of deleting the corresponding Job, File, and JobMedia
records from the catalog database. No change to the physical data on the
Volume occurs during the pruning process. When all files are pruned from a
Volume (i.e. no records in the catalog), the Volume will be marked as
Purged implying that no Jobs remain on the volume."

I hope this helps.

Chris

On Wed, Nov 23, 2022 at 9:47 AM Nick Bright  wrote:

> Greetings,
>
> I'm new to Bacula, and haven't used tape based backups before (though
> I've used BackupPC for many years.) I'm running Bacula 11.0.6 on Debian 11.
>
> Currently I'm backing up about 30 servers to Bacula, and this seems to
> be working well; however I've filled up my 17TB of disk, and backups are
> now failing citing that there are no available volumes to append to.
>
> I've spent the morning searching for various storage space management
> strategies within Bacula, but not finding anything useful. I suspect my
> failure to find useful information may be that I don't know enough about
> what needs to happen to properly phrase the questions I need to be
> asking - results have centered around truncating volumes, but all
> command just return essentially 'nothing to truncate'.
>
> Speculating this has to do with retention, I turned my retention for
> both Jobs and Files down to 7 days, but still 'nothing to truncate' is
> the output.
>
> Specifically running the bconsole command "prune expired volume yes"
> results in all Volumes reporting "Found no Job associated with the
> Volume to prune", and the bconsole command "purge volume action=truncate
> allpools storage=File1" reports "No Volumes found to perform the command"
>
> Adding storage isn't a tenable solution (all disk bays are full), so I
> would appreciate any specific recommendations for storage management
> strategies.
>
> Thanks,
>
> --
> -
> -  Nick Bright  -
> -  KwiKom Communications-
> -  Office 800-379-7292  -
> -  Direct 620-228-5653  -
> -  Web https://www.kwikom.com/  -
> -
>
>
>
> ___
> Bacula-users mailing list
> Bacula-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>


-- 

Regards,
Chris

*We do the things you want to do and don't know how*
*We do the things you know how to do and don't have the time*

Chris MillerCEORocket Scientist831.480.7199 (office)831.428.2556 (cell)


  

Re: [Bacula-users] Storage Management

[Josip Deanovic]

On 2022-11-23 20:02, Nick Bright wrote:

Thank you for the replies.

The physical disk has about 2% free space (446GB) so it shouldn't be
locked preventing any maintenance from occurring due to insufficient
storage.


Hi Nick

If your file system is configured to reserve some percentage of blocks
for root user, you will not be able to use all the space in case your
bacula-sd is running as a non-root user.

For example, with ext3/ext4 file system you might check that using the
command: dumpe2fs -h /your/file/system | grep "Reserved block count"


Regards!

--
Josip Deanovic


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Storage Management

[Bill Arlofski via Bacula-users]

On 11/23/22 12:27, Nick Bright wrote:

In the Pool definition there is a Volume Retention of 365 days; however
each client has it's own File Retention and Job Retention (7 days each)

Shouldn't this result in the data contained within the volumes being
expired, and thus the volume could be recycled/purged/truncated once
its' containing datas' retention period has expired?




A couple things to consider here:


For File and Job retentions, manual says:

- When this time period expires, and if AutoPrune is set to yes, Bacula will 
prune (remove) File|Job records that are older
than the specified File Retention period

- The shortest retention period of the three takes precedence.

So, I seem to recall that manual prunes do not trigger this action. I could be 
wrong here. I have seen some complicated
things with retention periods and pruning in the years I have worked with 
Bacula. :)

Of all the people I know, my colleague Ana is the expert I refer to for
retention and pruning questions. :)  ANA! Where are
you?  :D


Also, something else to consider since I do not know your full environment:

If you have jobs on a volume from several different clients, and the files/jobs 
of one client meet their retention period and
are auto-pruned, consider that other jobs from clients with longer retention 
periods will prevent this volume from being
pruned/purged/recycled/re-used.

I will ping Ana off list to see if she might chime in here.


Best regards,
Bill


--
Bill Arlofski
w...@protonmail.com



signature.asc
Description: OpenPGP digital signature
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Storage Management

[Nick Bright]

In the Pool definition there is a Volume Retention of 365 days; however 
each client has it's own File Retention and Job Retention (7 days each)


Shouldn't this result in the data contained within the volumes being 
expired, and thus the volume could be recycled/purged/truncated once 
its' containing datas' retention period has expired?


On 11/23/22 13:15, Bill Arlofski via Bacula-users wrote:

Hello Nick,

For the pool you showed us, the VolRetention is `31,536,000` seconds.

This is not 7 days, but 1 year.

You may need to restart your Director, then perform the steps in my 
previous email again to see the results you are looking for.


For VolRetention, you should see `604800` which is the 7 days you 
mentioned.



Hope this helps,
Bill

--
Bill Arlofski
w...@protonmail.com



___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users


--
-
-  Nick Bright  -
-  KwiKom Communications-
-  Office 800-379-7292  -
-  Direct 620-228-5653  -
-  Web https://www.kwikom.com/  -
-



___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Storage Management

[Bill Arlofski via Bacula-users]

Hello Nick,

For the pool you showed us, the VolRetention is `31,536,000` seconds.

This is not 7 days, but 1 year.

You may need to restart your Director, then perform the steps in my previous 
email again to see the results you are looking for.

For VolRetention, you should see `604800` which is the 7 days you mentioned.


Hope this helps,
Bill

--
Bill Arlofski
w...@protonmail.com



signature.asc
Description: OpenPGP digital signature
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Storage Management

[Nick Bright]

Thank you for the replies.

The physical disk has about 2% free space (446GB) so it shouldn't be 
locked preventing any maintenance from occurring due to insufficient 
storage.


I issued the "update pool" command, which output

++--+-+-+-++-+--++++-++--+---+-+---+--+---+-+-+---+---+++---+---+
| PoolId | Name | NumVols | MaxVols | UseOnce | UseCatalog | 
AcceptAnyVolume | VolRetention | CacheRetention | VolUseDuration | 
MaxVolJobs | MaxVolFiles | MaxVolBytes    | MaxPoolBytes | AutoPrune | 
Recycle | ActionOnPurge | PoolType | LabelType | LabelFormat | Enabled | 
ScratchPoolId | RecyclePoolId | NextPoolId | MigrationHighBytes | 
MigrationLowBytes | MigrationTime |

++--+-+-+-++-+--++++-++--+---+-+---+--+---+-+-+---+---+++---+---+
|  2 | File | 310 | 310 |   0 |  1 
|   0 |   31,536,000 |  0 | 50,400 
|  0 |   0 | 53,687,091,200 |    0 | 1 
|   1 | 1 | Backup   | 0 | Vol-    |   1 
| 0 | 0 |  0 |  0 
| 0 | 0 |

++--+-+-+-++-+--++++-++--+---+-+---+--+---+-+-+---+---+++---+---+

then

* update volume fromallpools
All Volume defaults updated from "Default" Pool record.
All Volume defaults updated from "File" Pool record.
All Volume defaults updated from "Scratch" Pool record.

then

* prune expired volume yes

all Volumes reported "Found no Job associated with the Volume to prune"

then

* purge volume action=truncate allpools storage=File1
No Volumes found to perform the command.

I then attempted to run an incremental backup, and again received "The 
job needs media. Please create a new volume for storage"


Apparently I've done something wrongly, but I'm not sure what. Your 
instructions were quite easy to understand.


On 11/23/22 12:50, Bill Arlofski via Bacula-users wrote:

On 11/23/22 10:19, Nick Bright wrote:
>

Speculating this has to do with retention, I turned my retention for
both Jobs and Files down to 7 days, but still 'nothing to truncate' is
the output.


Yes...

After reloading the Director's config and doing "update pool=" for 
the pool(s) you have updated (per Pedro's comment),
you will need to update all the volumes that exist in these pools with 
the new retention periods.



If you do:

* ll volume=

For some existing volumes you will see that the retention is still the 
old, longer one.



Then, do:

* update volume fromallpools

And then re-check those volumes and you will see they have the new 
retention periods set.


Then you can try another prune and should see some new, better results.


Hope this helps,
Bill

--
Bill Arlofski
w...@protonmail.com



___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users


--
-
-  Nick Bright  -
-  KwiKom Communications-
-  Office 800-379-7292  -
-  Direct 620-228-5653  -
-  Web https://www.kwikom.com/  -
-



___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Storage Management

[Bill Arlofski via Bacula-users]

On 11/23/22 10:19, Nick Bright wrote:
>

Speculating this has to do with retention, I turned my retention for
both Jobs and Files down to 7 days, but still 'nothing to truncate' is
the output.


Yes...

After reloading the Director's config and doing "update pool=" for the 
pool(s) you have updated (per Pedro's comment),
you will need to update all the volumes that exist in these pools with the new 
retention periods.


If you do:

* ll volume=

For some existing volumes you will see that the retention is still the old, 
longer one.


Then, do:

* update volume fromallpools

And then re-check those volumes and you will see they have the new retention 
periods set.

Then you can try another prune and should see some new, better results.


Hope this helps,
Bill

--
Bill Arlofski
w...@protonmail.com



signature.asc
Description: OpenPGP digital signature
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Storage Management

[Pedro Oliveira]

Hi Nick

Please execute update pool from resource command in bconsole so Bacula can
reflect the retention perio change in the existing volumes. Only after you
can execute the truncate command.

Best

Pedro

Nick Bright  escreveu em qua., 23/11/2022 às 17:49 :

> Greetings,
>
> I'm new to Bacula, and haven't used tape based backups before (though
> I've used BackupPC for many years.) I'm running Bacula 11.0.6 on Debian 11.
>
> Currently I'm backing up about 30 servers to Bacula, and this seems to
> be working well; however I've filled up my 17TB of disk, and backups are
> now failing citing that there are no available volumes to append to.
>
> I've spent the morning searching for various storage space management
> strategies within Bacula, but not finding anything useful. I suspect my
> failure to find useful information may be that I don't know enough about
> what needs to happen to properly phrase the questions I need to be
> asking - results have centered around truncating volumes, but all
> command just return essentially 'nothing to truncate'.
>
> Speculating this has to do with retention, I turned my retention for
> both Jobs and Files down to 7 days, but still 'nothing to truncate' is
> the output.
>
> Specifically running the bconsole command "prune expired volume yes"
> results in all Volumes reporting "Found no Job associated with the
> Volume to prune", and the bconsole command "purge volume action=truncate
> allpools storage=File1" reports "No Volumes found to perform the command"
>
> Adding storage isn't a tenable solution (all disk bays are full), so I
> would appreciate any specific recommendations for storage management
> strategies.
>
> Thanks,
>
> --
> -
> -  Nick Bright  -
> -  KwiKom Communications-
> -  Office 800-379-7292  -
> -  Direct 620-228-5653  -
> -  Web https://www.kwikom.com/  -
> -
>
>
>
> ___
> Bacula-users mailing list
> Bacula-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>
-- 

Create your own email signature


‌
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users