Hi all,
I have a client with ~2TB of data. A full backup job takes about 15 hours.
Once the backup finishes, a Copy Job is triggered (Selection Type =
PoolUncopiedJobs) to replicate the volume to a remote Storage Daemon —
which takes another 15 hours per volume in the best-case scenario. Finally,
for each backup, I run a Restore job on a replica server (same client
setup), which adds yet another 15 hours.
The trivial approach to parallelize these jobs (Backup, Copy, Restore) has
been to define a separate Storage + Device pair for each Job. The only
difference between them is the Name. All Devices share the same Archive
Device path — which I'm unsure is a good practice.
Is there a better way to parallelize Backup, Copy, and Restore jobs on the
same Pool/Volumes without defining a separate Storage/Device for each Job?
What would be considered a best-practice setup here?
TL;DR
I need to run Backup, Copy, and Restore jobs in parallel for the same
client (Restore happens on a replica server). Ideally without creating
separate Storage/Device/Pool for each job.
Bacula Version: 15.0.2 (21 March 2024)
This is the config that currently allows me to parallelize only Backup and
Restore. Following this logic, I’d need to create additional Storage +
Device just for the Copy jobs.
Client {
Name = "mongodb01-fd"
Address = xx.xx.xx.xx
FDPort = 9102
Catalog = MyCatalog
Password = "PASSWORD"
File Retention = 11 days
Job Retention = 2 months
AutoPrune = yes
Heartbeat Interval = 1m
TLS Require = yes
TLS PSK Enable = yes
}
Job {
Name = "mongodb01-Backup"
JobDefs = "MongoDef"
Pool = "mongodb01-baculaserver01-Pool"
Level = Full
Client = "mongodb01-fd"
Storage = "mongodb01_baculaserver01_Storage"
Schedule = "DisabledSchedule" # This is how I disable this client so
it doesn't affect others using the same JobDef with valid Schedules. I
launch the backup manually using a bconsole command.
@"/opt/bacula/etc/bacula-dir.conf.d/jobdefs/RunCopyToServer.jd" # This
is how I trigger Copy Jobs for clients with small backups. It works well
within a 24-hour window. For this client, I should remove this line.
}
Pool {
Name = "mongodb01-baculaserver01-Pool"
Use Volume Once = yes
Pool Type = Backup
LabelFormat = "mongodb01-"
AutoPrune = yes
Recycle = yes
VolumeRetention = 30 days
Maximum Volumes = 365
Maximum Volume Jobs = 1
Recycle Oldest Volume = yes
Next Pool = "mongodb01-baculaserver02-Pool" # All copies go from
server01 to server02
}
Pool {
Name = "mongodb01-baculaserver02-Pool"
Use Volume Once = yes
Pool Type = Backup
LabelFormat = "mongodb01-"
AutoPrune = yes
Recycle = yes
VolumeRetention = 30 days
Maximum Volumes = 365
Maximum Volume Jobs = 1
Recycle Oldest Volume = yes
Storage = "mongodb01_baculaserver02_Storage"
Next Pool = "mongodb01-baculaserver01-Pool" # If backup runs on
server02, the copy goes to server01. Normally, all main backups are done on
server01.
}
Storage {
Name = "mongodb01_baculaserver01_Storage"
Address = "baculaserver01.domain"
SDPort = 9103
Password = "xxxxxxxxxxxxxxxxxxxxxxx"
Device = "mongodb01_baculaserver01_Device"
Media Type = File1
Maximum Concurrent Jobs = "10"
Heartbeat Interval = 10
}
Storage {
Name = "mongodb01_baculaserver01_Storage_2"
Address = "baculaserver01.domain"
SDPort = 9103
Password = "xxxxxxxxxxxxxxxxxxxxxxxx"
Device = "mongodb01_baculaserver01_Device_2"
Media Type = File1
Maximum Concurrent Jobs = "10"
Heartbeat Interval = 10
}
Storage {
Name = "mongodb01_baculaserver02_Storage"
Address = "baculaserver02.domain"
SDPort = 9103
Password = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
Device = "mongodb01_baculaserver02_Device"
Media Type = File1
Maximum Concurrent Jobs = "10"
Heartbeat Interval = 10
}
Device {
Name = "mongodb01_baculaserver01_Device"
Media Type = File1
Archive Device = "/backup/bacula-storage/mongodb01/"
Maximum Concurrent Jobs = 10
LabelMedia = yes
Random Access = Yes
AutomaticMount = yes
RemovableMedia = no
AlwaysOpen = yes
}
Device {
Name = "mongodb01_baculaserver01_Device_2"
Media Type = File1
Archive Device = "/backup/bacula-storage/mongodb01/"
Maximum Concurrent Jobs = 10
LabelMedia = yes
Random Access = Yes
AutomaticMount = yes
RemovableMedia = no
AlwaysOpen = yes
}
#This Device is stored in baculaserver02 where only a storage daemon is
running (not bacula-dir)
Device {
Name = "mongodb01_baculaserver02_Device"
Media Type = File1
Archive Device = "/backup/bacula-storage/mongodb01/"
LabelMedia = yes;
Random Access = Yes;
AutomaticMount = yes;
RemovableMedia = no;
AlwaysOpen = no;
}
Backup is launched via cron every 3 days using:
echo 'run job=mongodb01-Backup storage="mongodb01_baculaserver01_Storage_2"
yes' | docker exec -i bacula-dir /opt/bacula/bin/bconsole
Restore is triggered by another cron using the default Storage defined in
the Job.
Any advice or shared experience would be greatly appreciated.
Thanks,
_______________________________________________
Bacula-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/bacula-users
