message continues:
###########################
bacula-sd.conf excerpts:
I am using object locking, to prevent the deletion of cloud-based volumes
before their retention period has expired.
This is for ransomware mitigation. Implementation varies between cloud
providers, but generally it is much more difficult to delete an object
locked object from the cloud resource.
An object which is older than the object lock period can be deleted as
normal.
Bacula actually has no idea that I am using object locking. The easiest way
I found to set object locks was on a per-bucket basis.
I have to make sure that my Full, Diff, and Inc rentention periods align
with my bucket object lock durations.
Full, Diff, and Inc buckets each have their own object lock durations, to
match their pool retention periods.
For each bucket, bacula-sd.conf must have a cloud resource, autochanger,
and device resources dedicated to that bucket.
Additionally, Backblaze has an option to keep all versions of a file. I do
not want this, so I have configured it to only keep the latest version of
the file.
If an object lock applies to a file, Backblaze will still keep multiple
versions of the file.
If an object locked volume part is deleted by bacula, Backblaze will tell
bacula that the volume was deleted. The Backblaze bucket file browser will
still show you that a 0B volume part file has overlaid on the original part
file. Both files are accessible from the bucket file browser.
I don't know how other cloud storage providers handle these concepts, but I
assume they are likely present, and should be understood.
I only have a single local autochanger definition, and a number of device
resources dedicated to the local autochanger.
In other words, the local storage resources don't have devices and
autochangers dedicated to different job levels.
###########################
Cloud {
Name = "B2-TD-Full"
Driver = "Amazon" # this is the new cloud driver, available as of bacula
CE 15.x. It is much more reliable than the old S3 driver.
HostName = "s3.us-west-004.backblazeb2.com"
BlobEndpoint = "https://s3.us-west-004.backblazeb2.com/"; # this is
essential for non-AWS cloud providers
BucketName = "TD-Bacula-Full"
AccessKey = "REDACTED"
SecretKey = "REDACTED"
Protocol = "HTTPS"
UriStyle = "VirtualHost"
TruncateCache = "AfterUpload" # All local copies of the cloud volume
parts (except the header, *.part1) will be deleted after upload.
Upload = "No" # Copy volumes will not be uploaded after a job completes.
Upload must be launched by a separate admin script, which runs a bconsole
command.
}
Cloud {
Name = "B2-TD-Diff"
...
BucketName = "TD-Bacula-Diff"
}
Cloud {
Name = "B2-TD-Inc"
...
BucketName = "TD-Bacula-Inc"
}
Autochanger {
Name = "B2-TD-Full"
Device = "B2-TD-Full-001"
Device = "B2-TD-Full-002"
Device = "B2-TD-Full-003"
Device = "B2-TD-Full-004"
Device = "B2-TD-Full-005"
Device = "B2-TD-Full-006"
ChangerDevice = "/dev/null"
ChangerCommand = "/dev/null"
}
Autochanger {
Name = "B2-TD-Diff"
...
}
Autochanger {
Name = "B2-TD-Inc"
...
}
Device {
Name = "B2-TD-Full-001"
Description = "B2-TD-Full"
MediaType = "B2-TD-Full"
DeviceType = "Cloud"
ArchiveDevice = "/mnt/synology/bacula/B2-TD-Full"
RemovableMedia = no
RandomAccess = yes
AutomaticMount = yes
LabelMedia = yes
Autochanger = yes
ReadOnly = no
MaximumFileSize = 5368709120
MaximumConcurrentJobs = 1
DriveIndex = 0
MaximumPartSize = 5368709120
Cloud = "B2-TD-Full"
VolumeEncryption = yes # Remove this line if you haven't set up
volume encryption. Only available in bacula CE 15.x and beyond.
}
Device {
Name = "B2-TD-Full-002"
...
}
Device {
Name = "B2-TD-Full-003"
...
}
Device {
Name = "B2-TD-Full-004"
...
}
Device {
Name = "B2-TD-Full-005"
...
}
Device {
Name = "B2-TD-Full-006"
...
AutoSelect = no # this drive will not be used by backups so it will
remain available for restores.
}
# (devices for the diff and inc storage resources not shown, but they must
exist and are the same as the Full devices, except for the level names)
#####################
bacula-sd.conf snippet ends
#####################
Please see next message.
Regards,
Robert Gerber
402-237-8692
[email protected]
On Sat, Jul 4, 2026 at 6:40 PM Rob Gerber <[email protected]> wrote:
> Andrea,
> Sorry for the delay in reply. I'm going to have to post this in multiple
> pieces due to message length limits.
>
> One of my bacula installations runs on a synology appliance and uploads to
> the Backblaze B2, an S3 compatible cloud service. You should be able to use
> any S3 compatible cloud service.
> I separate out different job levels into their own pool for the local
> storage. This structure is mirrored in the cloud. I have a bucket for each
> cloud pool. Each cloud pool must have its own autochanger and device
> resources in bacula-sd.conf, and its own storage resource in
> bacula-dir.conf. Each local pool has an equivalent cloud pool. The local
> pools reference the cloud pools using 'NextPool='.I use selection method
> PoolUncopiedJobs to choose which jobs are uploaded. I am using bacula 15.x,
> and the amazon cloud driver and volume encryption introduced in that
> version of bacula. You do not need to use volume encryption if you are not
> concerned about this, but if you do you should know that each volume is
> encrypted with a unique key. If these keys are not backed up separately,
> recovery from your cloud volumes alone will be impossible.
>
> I delete all local copies of the copy jobs after upload. I do not upload
> at the time when the job is ran. I upload afterward with a copy volume part
> sweeper script. I previously uploaded at time of copy job execution, but I
> found that bacula does not retry failed part uploads during that process,
> and bacula also does not throw an error when a part fails to upload.
> Without a part sweeper script to instruct bconsole to upload any cloud
> volume parts not previously uploaded, any such part would simply be left
> unuploaded forever. My part sweeper script also outputs to a log file, then
> reads its own log to check for errors. The part sweeper script exits 1 once
> it's done all its work if it finds any failure to upload a cloud volume
> part.
>
> My basic psuedocode config structure (real examples further down):
> bacula-sd.conf:
> Cloud resource for bucket Cloud-Full (TruncateCache = "AfterUpload";
> Upload = "no")
> Cloud resource for bucket Cloud-Diff (TruncateCache = "AfterUpload";
> Upload = "no")
> Cloud resource for bucket Cloud-Inc (TruncateCache = "AfterUpload";
> Upload = "no")
> Autochanger resource for Cloud resource Cloud-Full
> Autochanger resource for Cloud resource Cloud-Diff
> Autochanger resource for Cloud resource Cloud-Inc
> Device resources for Cloud resource Cloud-Full
> Device resources for Cloud resource Cloud-Diff
> Device resources for Cloud resource Cloud-Inc
>
> Autochanger resource for local file based storage
> Device resources for local file based storage
>
> bacula-dir.conf:
> Storage for Cloud-Full
> Storage for Cloud-Diff
> Storage for Cloud-Inc
> Storage for local file based storage
> Pool Local-Full (MaximumVolumeJobs = 1; NextPool references the equivalent
> cloud pool)
> Pool Local-Diff (MaximumVolumeJobs = 1; NextPool references the equivalent
> cloud pool)
> Pool Local-Inc (MaximumVolumeJobs = 1; NextPool references the equivalent
> cloud pool)
> Pool Cloud-Full (MaximumVolumeJobs = 1)
> Pool Cloud-Diff (MaximumVolumeJobs = 1)
> Pool Cloud-Inc (MaximumVolumeJobs = 1)
> Schedule: Daily at 23:05. Used with backup jobs. Jobdefs selects the
> correct level.
> Schedule: Copy-End-Of-Day at 23:50. Copy jobs. Doesn't actually need to be
> separate at this point. Priority ensures that this will run at the correct
> point in the process.
> Schedule: NightlyAfterCopy at 23:54. Used to launch the part sweeper after
> copy jobs are finished. Not actually needed - priority ensures this will
> run at the correct point in the process.
> Jobdefs: Specifies local full, diff, inc pools; maximum interval between
> levels full and diff; no duplicate jobs allowed; ReRunFailedLevels = yes
> admin-prune-allpools-job: Priority 1. Script with bconsole command to
> prune all eligible jobs.
> admin-truncate-volumes-job: Priority 2. Script with bconsole commands to
> truncate all pools on each storage. Minimizes cloud storage usage.
> Job 1
> Job 2
> admin-copy-control-launcher-job (lower priority than main backup jobs;
> launches the copy jobs after main jobs have finished. Otherwise the copy
> jobs will only queue up yesterdays jobs for copying, because todays backup
> jobs have not finished at the time when the copy jobs were launched)
> Copy-Control-Full-job (one copy control job per pool. Manual schedule.
> Targets local pool for each level. Copy control job for each level prevents
> cancellation of copy control jobs as duplicate)
> Copy-Control-Diff-job (one copy control job per pool. Manual schedule.
> Targets local pool for each level. Copy control job for each level prevents
> cancellation of copy control jobs as duplicate)
> Copy-Control-Inc-job (one copy control job per pool. Manual schedule.
> Targets local pool for each level. Copy control job for each level prevents
> cancellation of copy control jobs as duplicate)
> admin-cloud-volume-part-sweeper-job (Script with bconsole commands to
> verify all cloud volume parts have been uploaded. Also responsible to
> perform initial upload of copy job volumes (SD cloud resource specifies
> "Upload = no"))
>
> Message continues.
>
> Regards,
> Robert Gerber
> 402-237-8692
> [email protected]
>
>
>
_______________________________________________
Bacula-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/bacula-users
