Re: [Bacula-users] Storage Management
On 11/25/22 09:20, Ana Emília M. Arruda wrote:
Can you share your pool resource configuration here? Bacula has many
directives to different ways to recycle volumes, it would be nice to
see how you have the pool configured.
Sure, here is my pool configuration:
Pool {
Name = File
Pool Type = Backup
Recycle = yes
Recycle Oldest Volume = yes
AutoPrune = yes
Volume Retention = 7 days
Maximum Volume Bytes = 50G
Maximum Volumes = 310
Label Format = "Vol-"
Action On Purge = Truncate
Volume Use Duration = 14h
}
It's notable that this configuration leaves about 450GB of free space on
the pool partition, so there is plenty of working room. I had initially
grown the Maximum Volumes setting several times before stopping and
investigating why volumes weren't being reused; presuming that
completely filling the partition would likely be A Bad Idea(tm).
After that, you will be able to "truncate volumes in Purged
status" using the following command:
* truncate volume allpools storage=File1
This command reports "No volumes found to perform the command"
This command will only apply to volumes in "Purged" status. If they
have been reused, there will be no volume to truncate. Please check if
it works having at least one volume in Purged status.
After the steps taken so far, I'm showing that I have a significant
number of volumes in "Purged" status - so, if I understand correctly,
those will be available for re-use as Bacula requires more space for
more backups.
So; with your help I see how to manually prune files, but this
seems inelegant. The system should automatically be pruning or at
least allowing me to overwrite the old data - it ought to be
cyclical - use the available disk space (17T) to back up systems,
letting old backups fall off automatically so that new backups can
take their place.
Is needing to manually remove stale backups the intended behavior
of the system, or have I done something incorrectly?
Bacula will automatically reuse volumes, but it will not destroy data
if the retention hasn't expired yet.
It would be nice if you could share the pool resource configuration
and a job log reporting no volume available to use.
The best approach is to reuse volumes as soon as they have retention
expired, but to not keep the pool too tight as it may happen Bacula
can't create a new volume or reuse any.
Limiting the pool size (in bytes or in number of volumes) and keeping
a retention that can fit into this pool size is ideal.
The pool configuration is listed above, here's an example of a job log
reporting no volumes available (this is before files were purged
allowing the currently pending backups to run - they all finished)
2022-11-23 13:01:31 baculaserver-dir JobId 904: Start Backup JobId 904,
Job=client.2022-11-23_13.01.29_31
2022-11-23 13:01:31 baculaserver-dir JobId 904: Connected to Storage
"File1" at baculaserver:9103 with TLS
2022-11-23 13:01:31 baculaserver-dir JobId 904: Pruning oldest volume
"Vol-0001"
2022-11-23 13:01:31 baculaserver-dir JobId 904: Found no Job associated
with the Volume "Vol-0001" to prune
2022-11-23 13:01:31 baculaserver-dir JobId 904: Using Device
"FileChgr1-Dev1" to write.
2022-11-23 13:01:31 baculaserver-dir JobId 904: Connected to Client
"client" at IPADDR:9102 with TLS
2022-11-23 13:01:31 client JobId 904: Connected to Storage at
baculaserver:9103 with TLS
2022-11-23 13:01:32 baculaserver-dir JobId 904: Pruning oldest volume
"Vol-0001"
2022-11-23 13:01:32 baculaserver-dir JobId 904: Found no Job associated
with the Volume "Vol-0001" to prune
2022-11-23 13:01:32 baculaserver-dir JobId 904: Pruning oldest volume
"Vol-0001"
2022-11-23 13:01:32 baculaserver-dir JobId 904: Found no Job associated
with the Volume "Vol-0001" to prune
2022-11-23 13:01:32 baculaserver-sd JobId 904: Job
client.2022-11-23_13.01.29_31 is waiting. Cannot find any appendable
volumes.
Please use the "label" command to create a new Volume for:
Storage: "FileChgr1-Dev1" (/var/lib/bacula)
Pool: File
Media type: File1
2022-11-23 13:03:47 baculaserver-sd JobId 904: Error: bsock.c:282 Bsock
send while terminated=1 on call to client:IPADDR:9103
2022-11-23 13:03:47 baculaserver-sd JobId 904: askdir.c:859 Job was
stopped by the user.
Thanks,
--
-
- Nick Bright -
- KwiKom Communications-
- Office 800-379-7292 -
- Direct 620-228-5653 -
- Webhttps://www.kwikom.com/ -
-
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Storage Management
On 2022-11-25 16:00, Nick Bright wrote: [...] inelegant. The system should automatically be pruning or at least allowing me to overwrite the old data - it ought to be cyclical - use the available disk space (17T) to back up systems, letting old backups fall off automatically so that new backups can take their place. This is exactly what I am doing using these directives in the pool resource configuration: Maximum Volume Bytes = 10G Maximum Volumes = # to be calculated from disk space # and Maximum Volume Bytes (make sure # to leave extra space for at least # 2 x Maximum Volume Bytes) AutoPrune = yes Volume Retention = Recycle = yes Purge Oldest Volume = yes The configuration above would recycle the oldest volume when needed, regardless of the retention period. This is potentially dangerous. You should have enough volumes in your pool to minimize the chance of losing useful (relatively recent) backup data. Regards -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Storage Management
Hello Nick, On Fri, Nov 25, 2022 at 4:00 PM Nick Bright wrote: > On 11/25/22 04:45, Ana Emília M. Arruda wrote: > > The easiest way is to allow Bacula to automatically prune Jobs and Files > from the Catalog. It means to have "AutoPrune = Yes" in both the Client > resource and in the Pool resource. > > I already had AutoPrune = yes in both the client and pool; I had found > this in my searches before posting here. > > All backups over the weekend failed with "need more volumes". > Can you share your pool resource configuration here? Bacula has many directives to different ways to recycle volumes, it would be nice to see how you have the pool configured. > Now, to force the prune of volumes that have the VolumeRetention (value > defined in the Pool resource) expired, the command is: > > * prune volume expired yes > > This command is dangerous, it will delete all jobs and files associated > with any volume by comparing only the "VolumeRetention" of the volume. This > value is defined in the Pool resource, but it is specific for a volume and > Bacula will check the value in the "volretention" field in the Media table > in the Bacula database for each volume. > > This command still reports nothing to prune for all 310 volumes as of this > morning, and I had run "update volume fromallpools" last week after > reducing retention times; though it does now report the volumes as "Expired" > > Then running the command again, it gives me some prompts about pruning > files/jobs/volume/stats/snapshots/events then a client to choose. > Right. This is expected. > I choose a client, it reports "Pruned Files from 6 Jobs for client" and > pending backups started executing. > Ok, so you got volumes pruned and reused. > > Then, to truncate volumes you must have the following configuration in the > Pool resource: > > ActionOnPurge = Truncate > > Already had this present as well. > Good. > > After that, you will be able to "truncate volumes in Purged status" using > the following command: > > * truncate volume allpools storage=File1 > > This command reports "No volumes found to perform the command" > This command will only apply to volumes in "Purged" status. If they have been reused, there will be no volume to truncate. Please check if it works having at least one volume in Purged status. > So; with your help I see how to manually prune files, but this seems > inelegant. The system should automatically be pruning or at least allowing > me to overwrite the old data - it ought to be cyclical - use the available > disk space (17T) to back up systems, letting old backups fall off > automatically so that new backups can take their place. > > Is needing to manually remove stale backups the intended behavior of the > system, or have I done something incorrectly? > Bacula will automatically reuse volumes, but it will not destroy data if the retention hasn't expired yet. It would be nice if you could share the pool resource configuration and a job log reporting no volume available to use. The best approach is to reuse volumes as soon as they have retention expired, but to not keep the pool too tight as it may happen Bacula can't create a new volume or reuse any. Limiting the pool size (in bytes or in number of volumes) and keeping a retention that can fit into this pool size is ideal. Best, Ana -- > - > - Nick Bright - > - KwiKom Communications- > - Office 800-379-7292 - > - Direct 620-228-5653 - > - Web https://www.kwikom.com/ - > - > > ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Storage Management
On 11/25/22 04:45, Ana Emília M. Arruda wrote: The easiest way is to allow Bacula to automatically prune Jobs and Files from the Catalog. It means to have "AutoPrune = Yes" in both the Client resource and in the Pool resource. I already had AutoPrune = yes in both the client and pool; I had found this in my searches before posting here. All backups over the weekend failed with "need more volumes". Now, to force the prune of volumes that have the VolumeRetention (value defined in the Pool resource) expired, the command is: * prune volume expired yes This command is dangerous, it will delete all jobs and files associated with any volume by comparing only the "VolumeRetention" of the volume. This value is defined in the Pool resource, but it is specific for a volume and Bacula will check the value in the "volretention" field in the Media table in the Bacula database for each volume. This command still reports nothing to prune for all 310 volumes as of this morning, and I had run "update volume fromallpools" last week after reducing retention times; though it does now report the volumes as "Expired" Then running the command again, it gives me some prompts about pruning files/jobs/volume/stats/snapshots/events then a client to choose. I choose a client, it reports "Pruned Files from 6 Jobs for client" and pending backups started executing. Then, to truncate volumes you must have the following configuration in the Pool resource: ActionOnPurge = Truncate Already had this present as well. After that, you will be able to "truncate volumes in Purged status" using the following command: * truncate volume allpools storage=File1 This command reports "No volumes found to perform the command" So; with your help I see how to manually prune files, but this seems inelegant. The system should automatically be pruning or at least allowing me to overwrite the old data - it ought to be cyclical - use the available disk space (17T) to back up systems, letting old backups fall off automatically so that new backups can take their place. Is needing to manually remove stale backups the intended behavior of the system, or have I done something incorrectly? -- - - Nick Bright - - KwiKom Communications- - Office 800-379-7292 - - Direct 620-228-5653 - - Webhttps://www.kwikom.com/ - - ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Storage Management
Hello Nick, Bill, I'm here :-) Sorry for not jumping in earlier. Nick, I will try to summarize a bit about retention values and pruning/truncation in Bacula. Pruning is about "the deletion of jobs and files from Catalog". Automatic or manual prune will not touch the data in the volumes, but only the jobs and files records in the Catalog are deleted. The easiest way is to allow Bacula to automatically prune Jobs and Files from the Catalog. It means to have "AutoPrune = Yes" in both the Client resource and in the Pool resource. 1) having "AutoPrune = Yes" in the Client resource -> at the end of each job run associated to the Client/Pool of the job, Bacula will check if the retention have passed for all the jobids associated to the Client/Pool and the jobs and files are deleted from the Catalog if the retention has passed. You will see this kind of message: | bacula-dir JobId 5104: Begin pruning Jobs older than 6 months . | | bacula-dir JobId 5104: No Jobs found to prune. | | bacula-dir JobId 5104: Begin pruning Files. | | bacula-dir JobId 5104: No Files found to prune. | | bacula-dir JobId 5104: End auto prune. | If any job/file has reached the retention based on the values defined in the Client and/or Pool resources. The values set in a Pool resource will take precedence over the values set in a Client resource. 2) having "AutoPrune = Yes" in the Pool resource -> when Bacula needs to reuse a volume for a job (if Bacula can't create a new volume because the max number of volumes in the pool is reached and there is no volume in Append, Purged or Recycle status), the recycling algorithm will try to prune jobs and files to reuse a volume. It will start searching for the oldest volume in the pool. Then, Bacula will check if this volume has no more jobs associated with it in the Catalog (it means the jobs and files have been pruned so this volume can be reused). If so, you will see messages like: 04-Nov 15:15 bacula-dir JobId 3560: There are no more Jobs associated with Volume "Vol-0038". Marking it purged. 04-Nov 15:15 bacula-dir JobId 3560: All records pruned from Volume "Vol-0038"; marking it "Purged" 04-Nov 15:15 bacula-dir JobId 3560: Recycled volume "Vol-0038" This is a job log where Bacula couldn't create a new volume for jobid=3560 and it reused Vol-0038 because the jobs associated with it had been pruned (by automatic prune after the job run) previously. Now, to force the prune of volumes that have the VolumeRetention (value defined in the Pool resource) expired, the command is: * prune volume expired yes This command is dangerous, it will delete all jobs and files associated with any volume by comparing only the "VolumeRetention" of the volume. This value is defined in the Pool resource, but it is specific for a volume and Bacula will check the value in the "volretention" field in the Media table in the Bacula database for each volume. This is why when you change the "VolumeRetention" value in the Pool resource the value of the already existing volumes is not modified and you have to run: * update volume fromallpools All Volume defaults updated from "Default" Pool record. All Volume defaults updated from "File" Pool record. All Volume defaults updated from "Scratch" Pool record. After that, the volretention values in the Catalog for the existing volumes is updated to the value defined in the Pool configuration and the "prune expired volume yes" will use the new value to compare with. Then, to truncate volumes you must have the following configuration in the Pool resource: ActionOnPurge = Truncate Please remember to reload the configuration in bconsole using the reload command after adding the above line. After that, you will be able to "truncate volumes in Purged status" using the following command: * truncate volume allpools storage=File1 Please let us know if it helps. Best, Ana On Wed, Nov 23, 2022 at 11:04 PM Nick Bright wrote: > I changed the Volume retention to 7 days, repeated all of the commands, > and ran a BackupCatalog job; still reports "The job needs media" > > On 11/23/22 13:39, Bill Arlofski via Bacula-users wrote: > > On 11/23/22 12:27, Nick Bright wrote: > >> In the Pool definition there is a Volume Retention of 365 days; however > >> each client has it's own File Retention and Job Retention (7 days each) > >> > >> Shouldn't this result in the data contained within the volumes being > >> expired, and thus the volume could be recycled/purged/truncated once > >> its' containing datas' retention period has expired? > > > > > > > > A couple things to consider here: > > > > > > For File and Job retentions, manual says: > > > > - When this time period expires, and if AutoPrune is set to yes, > > Bacula will prune (remove) File|Job records that are older > > than the specified File Retention period > > > > - The shortest retention period of the three
