Part 2/2 follows:
IMO, the best way to manage bacula file volume space usage is through
bacula itself, without any direct interaction with the filesystem. This
keeps bacula fully in the loop so it knows what catalog entries it can
remove, etc. This includes the actual actions that free disc space.
However, the functions and options that do this aren't always intuitive.
You know this already. You use bacula.
With that in mind, what you probably want to do is:
0. Maybe: update the volumes to add a parameter to set 'action on purge =
truncate' (probably only needed if you haven't previously defined 'action
on purge = truncate' BEFORE the file volume was created).
1. Tell bacula a volume isn't needed any longer (delete all jobs on the
volume OR purge the volume OR delete the volume from the catalog, then
delete the file volume on disk)
(if you choose delete, your process stops here)
2. Prune the volume
3. Truncate the volume (this changes the volume contents from 'volume
header + job 1 data + job 2 data + ...' to 'volume header').
Lets talk about retention periods. The subject and how it's handled is
relevant to this.
Bacula has 3 types of catalog entries that are subject to some sort of
retention period. These are:
File records
Job records
Volume records.
You can see the sort of logic there. Volume records reference jobs that
wrote to that volume. Job records reference file records that list files
the job wrote to the volume.
Bacula will prune all file entries related to a job if the job entry is
pruned.
Bacula will prune all job and file entries related to a volume if the
volume entry is pruned.
Pruning file entries does not trigger the pruning of any job or volume
entries. It is possible for a job to exist without any file entries. You'd
just have to restore the whole job to see what files are in it, and
restoring a single file is not possible. The primary rationale for doing
this is to save catalog space, if that is a concern. Most people should set
job and file retention to the same amount of time.
Pruning deletes catalog entries, but the term itself doesn't describe what
is done with the volume catalog entry (ie, is it placed in 'purged' status,
or is the entry flat out deleted?).
A volume is subject to pruning if the volume has expired, or was previously
used and all jobs that previously referenced that volume have been pruned.
A simple way to make a volume subject to pruning, therefore, is to delete
all the jobs that reference the volume. Bacularis makes it easy to find the
jobs that used a given volume.
So to actually talk through the steps to truncate a set of volumes that
you've determined can be safely eliminated:
1. Set volumes to allow truncation once the volume is purged. Choose your
own adventure:
(you only need to do A or B, not both)
A: Update specific volumes only. in bconsole: 'update volume=volname
actiononpurge=truncate'
B: Configure your pool to include the option 'ActionOnPurge = Truncate'.
in bconsole: 'reload' (restart director to apply config changes)
in bconsole: 'update volumes'
(if the volume has not previously been configured to allow truncation with
this option, in the pool setting prior to volume creation, or by using
'update volumes')
2. Choose your own adventure (bconsole edition - this is easier to do on a
small scale in bacularis or baculum):
(you only need to do A or B, not both)
A: Purge the volumes in question WITH NO REGARD TO WHETHER THE DATA INSIDE
HAS ACTUALLY PASSED A RETENTION PERIOD:
in bconsole: 'purge volume=volname' (immediately eliminates all catalog
entries for the given volume, and marks the volume as status purged).
repeat above for each volume you want to eliminate.
B: Delete all the jobs associated with a volume, then prune the volumes.
in bconsole: 'delete jobid=yourjobid', repeated for every job followed by:
in bconsole 'prune volume allpools yes' (just once, it should remove any
expired entries from the catalog, whether they're related to the job
deletions you justdid above or not. This should automatically purge
eligible volumes.
3. Truncate all previously pruned volumes that are marked as purged, and
have 'action on purge = truncate'.
in bconsole: 'truncate allpools storage=yourstorage'. This will truncate
all available file volumes, freeing disk space.
The operation should be done. Check your disk space usage. Plenty of useful
output should be present in the console.
Fun fact: it is possible to use echo to pipe commands directly to bconsole.
This enables easy automation of laborious command entries, or dumping the
output to a file for later parsing.
like this:
echo "your bconsole command" | bconsole
echo "your bconsole command" | bconsole > filename.ext
You can also pass newline characters along to simulate interactive option
selection like this:
echo -e "your command\nAnswers it demanded\nquit\n" | bconsole
You should know that Bill A (experienced member on this list) asserts that
some versions of bacula have had issues if their bconsole sessions weren't
'quit'. Just to be safe, I make sure to configure any bconsole commands ran
from a script in this fashion. I mostly just use the 'echo "some command" |
bconsole" format if I'm casually doing something in the shell. Especially
useful is 'echo reload | bconsole'.
If you're going to run a script containing bconsole commands from a bacula
admin job, you'll want to to specify the paths to the bconsole binary and
the bconsole config file.
Script would look like this:
bcbin="/opt/bacula/bin/bconsole" #set your path to the
bconsole binary
bccfg="/opt/bacula/etc/bconsole.conf" #set your path to the
bconsole.conf file
echo -e "your command\nquit\n" | ${bcbin} -c ${bccfg}
*I don't recommend using bconsole this way when first exploring a command,
but if you have a lot of jobs or volumes to act upon, this can be very
useful. *
You can automate the issuing of commands by using scripts and bacula admin
jobs. An admin job is a job that only runs a script or takes some
administrative function.
Sample admin job (place in your bacula-dir.conf file)
# admin job definition begins
Job {
Name = "sample-admin-job"
Type = admin
Level = Full # Level doesn't mean anything in this context, but must be
defined for the config parser.
Schedule = "aScheduleThatExists" # If you don't want this job to be
scheduled, just remove or comment out this line. Any schedule specified
must exist
# I am using Bill A's dummy resources, so I actually have 'dummy'
storage, fileset, and pools defined with name="None"
# Such resources are convenient for quickly seeing at a glance whether
the job is really using a specific resource type.
Storage = "None" # This job doesn't use this, but the config parser
requires that it be defined, and must exist.
Fileset = "None" # This job doesn't use this, but the config parser
requires that it be defined, and must exist.
Pool = "None" # This job doesn't use this, but the config parser
requires that it be defined, and must exist.
Runscript {
RunsWhen = before
RunsOnClient = no # Default yes, there is no client in an Admin job &
Admin Job RunScripts *only* run on the Director :)
Command = "/opt/bacula/etc/your-script.sh" # Make sure the script's
permissions allow the bacula user (default: bacula) to read and execute the
script.
}
Priority = 30
}
# admin job definition ends
It is possible to add 'console' line to an admin script and directly input
commands into the admin job definition (no script file required). It sounds
easier and cleaner, but it is actually worse because commands ran via
'console' are listed in the log as coming from 'jobid 0', making them
really hard to troubleshoot. So don't use 'console' in an admin job, even
though it looks simpler.
All script output is added to the bacula log, so you'll be able to go back
and see what the script did or didn't manage to do, all from within the
bacula log or joblog.
Also, if a script exits with a non-zero status, the bacula admin job will
terminate with a fatal error, bringing that to your attention. I have one
use case where I want to know if something doesn't go the way I want. I
have a bconsole script tee all its output to a logfile, then after bconsole
is done the script parses the logfile for conditions I want to know about.
If it finds anything troublesome, it echos a useful error message (which is
printed in the bacula log), then exits exits 1. In this fashion, the script
does everything it can, then essentially flags the job for my review.
In closing, I have totally used the 'delete target volumes from bacularis,
then delete the file volumes from disk in a shell' method before. It worked
great, but I was really careful, and in my case I was very certain that I
didn't need those files.
I have also implemented the 'prune, then truncate' method as an automated,
long-term solution to this problem. For this to work, I had to make sure
appropriate retention periods were defined, then run admin jobs to prune,
then truncate volumes (in that order, using priority 1 and 2,
respectively). The early runtime and order of operations ensures that any
relevant volumes are dealt with before any jobs run, so I don't run into
disk issues. I'm also using copy jobs to copy backups to the cloud at one
site, so I want to truncate any expired volumes ASAP so we can stop paying
for unneeded cloud storage.
Anyhow, I hope this helps. Let us know if you have any more questions.
Regards,
Robert Gerber
402-237-8692
[email protected]
_______________________________________________
Bacula-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/bacula-users
