Hi Robert! Thanks for your time and input! What you suggested was exactly what came to my mind yesterday. I tested it and saw something very strange: I was able to start one Backup Job and 5 Copy Jobs in parallel, and everything looked fine at first. However, on the remote SD, the file sizes stopped growing independently, and a new file appeared that seems to be aggregating all 5 Copy Jobs into a single volume:
-rw-r----- 1 root root 4.6T Jul 16 12:35 mongodb01-2510 ← this should be 1.5T max, seems to be merging copies -rw-r----- 1 root root 190K Jul 15 20:05 mongodb01-2499 -rw-r----- 1 root root 127K Jul 15 20:04 mongodb01-2495 -rw-r----- 1 root root 64K Jul 15 20:04 mongodb01-2452 -rw-r----- 1 root root 318 Jul 15 20:04 mongodb01-2441 -rw-r----- 1 root root 44G Jul 15 20:03 mongodb01-2439 -rw-r----- 1 root root 1.5T Jun 29 18:43 mongodb01-2273 ← expected result -rw-r----- 1 root root 1.5T Jun 29 00:55 mongodb01-2268 ← expected result So it seems the volumes from the parallel Copy Jobs are being merged unintentionally. I added an Autochanger with 5 Devices, declared a Storage using the Autochanger, and pointed my Copy Jobs to this new Storage resource. You can find my config here: https://gist.github.com/lean2206/54b37b8d599243f4b95e047fa4a8b47a Any idea what might be causing this behavior? Meanwhile, until I find a better approach, I'll run the Restore Job using a RunAfterJob from the Backup, and trigger the Copy Jobs via cron — both using separate Devices, in parallel. So far, this setup would require 2 Devices. Thanks again! On Tue, Jul 15, 2025 at 2:03 PM Rob Gerber <r...@craeon.net> wrote: > Leandro, > > The following assumes you are using disk based backup, where the "drives" > and "autochangers" are virtual. > > I would suggest creating multiple "drive" devices in your bacula-sd.conf > for each SD destination, then tie them together with a single autochanger > resource in those respective bacula-sd.conf files. In your bacula-dir.conf, > in your storage resources, reference the respective autochanger resources. > If your jobs are ran with the same priority, and all the various daemons > have enough "allow simultaneous jobs" specified (not the actual option, I > can't recall the exact wording used by Bacula), then you should see better > parallelism. I believe you will then be limited by your disk / netwoek > bandwidth, and single core performance on various Bacula tasks like hashing > files. > > At least, unless I misunderstand something. > > I would provide config examples, but I am afk right now. > > Robert Gerber > 402-237-8692 > r...@craeon.net > > On Tue, Jul 15, 2025, 9:51 AM Leandro Saldivar via Bacula-users < > bacula-users@lists.sourceforge.net> wrote: > >> Hi all, >> >> I have a client with ~2TB of data. A full backup job takes about 15 >> hours. Once the backup finishes, a Copy Job is triggered (Selection Type = >> PoolUncopiedJobs) to replicate the volume to a remote Storage Daemon — >> which takes another 15 hours per volume in the best-case scenario. Finally, >> for each backup, I run a Restore job on a replica server (same client >> setup), which adds yet another 15 hours. >> >> The trivial approach to parallelize these jobs (Backup, Copy, Restore) >> has been to define a separate Storage + Device pair for each Job. The only >> difference between them is the Name. All Devices share the same Archive >> Device path — which I'm unsure is a good practice. >> >> Is there a better way to parallelize Backup, Copy, and Restore jobs on >> the same Pool/Volumes without defining a separate Storage/Device for each >> Job? What would be considered a best-practice setup here? >> >> TL;DR >> I need to run Backup, Copy, and Restore jobs in parallel for the same >> client (Restore happens on a replica server). Ideally without creating >> separate Storage/Device/Pool for each job. >> >> Bacula Version: 15.0.2 (21 March 2024) >> >> This is the config that currently allows me to parallelize only Backup >> and Restore. Following this logic, I’d need to create additional Storage + >> Device just for the Copy jobs. >> >> Client { >> Name = "mongodb01-fd" >> Address = xx.xx.xx.xx >> FDPort = 9102 >> Catalog = MyCatalog >> Password = "PASSWORD" >> File Retention = 11 days >> Job Retention = 2 months >> AutoPrune = yes >> Heartbeat Interval = 1m >> TLS Require = yes >> TLS PSK Enable = yes >> } >> >> Job { >> Name = "mongodb01-Backup" >> JobDefs = "MongoDef" >> Pool = "mongodb01-baculaserver01-Pool" >> Level = Full >> Client = "mongodb01-fd" >> Storage = "mongodb01_baculaserver01_Storage" >> Schedule = "DisabledSchedule" # This is how I disable this client so >> it doesn't affect others using the same JobDef with valid Schedules. I >> launch the backup manually using a bconsole command. >> @"/opt/bacula/etc/bacula-dir.conf.d/jobdefs/RunCopyToServer.jd" # >> This is how I trigger Copy Jobs for clients with small backups. It works >> well within a 24-hour window. For this client, I should remove this line. >> } >> >> Pool { >> Name = "mongodb01-baculaserver01-Pool" >> Use Volume Once = yes >> Pool Type = Backup >> LabelFormat = "mongodb01-" >> AutoPrune = yes >> Recycle = yes >> VolumeRetention = 30 days >> Maximum Volumes = 365 >> Maximum Volume Jobs = 1 >> Recycle Oldest Volume = yes >> Next Pool = "mongodb01-baculaserver02-Pool" # All copies go from >> server01 to server02 >> } >> >> Pool { >> Name = "mongodb01-baculaserver02-Pool" >> Use Volume Once = yes >> Pool Type = Backup >> LabelFormat = "mongodb01-" >> AutoPrune = yes >> Recycle = yes >> VolumeRetention = 30 days >> Maximum Volumes = 365 >> Maximum Volume Jobs = 1 >> Recycle Oldest Volume = yes >> Storage = "mongodb01_baculaserver02_Storage" >> Next Pool = "mongodb01-baculaserver01-Pool" # If backup runs on >> server02, the copy goes to server01. Normally, all main backups are done on >> server01. >> } >> >> Storage { >> Name = "mongodb01_baculaserver01_Storage" >> Address = "baculaserver01.domain" >> SDPort = 9103 >> Password = "xxxxxxxxxxxxxxxxxxxxxxx" >> Device = "mongodb01_baculaserver01_Device" >> Media Type = File1 >> Maximum Concurrent Jobs = "10" >> Heartbeat Interval = 10 >> } >> >> Storage { >> Name = "mongodb01_baculaserver01_Storage_2" >> Address = "baculaserver01.domain" >> SDPort = 9103 >> Password = "xxxxxxxxxxxxxxxxxxxxxxxx" >> Device = "mongodb01_baculaserver01_Device_2" >> Media Type = File1 >> Maximum Concurrent Jobs = "10" >> Heartbeat Interval = 10 >> } >> >> Storage { >> Name = "mongodb01_baculaserver02_Storage" >> Address = "baculaserver02.domain" >> SDPort = 9103 >> Password = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" >> Device = "mongodb01_baculaserver02_Device" >> Media Type = File1 >> Maximum Concurrent Jobs = "10" >> Heartbeat Interval = 10 >> } >> >> Device { >> Name = "mongodb01_baculaserver01_Device" >> Media Type = File1 >> Archive Device = "/backup/bacula-storage/mongodb01/" >> Maximum Concurrent Jobs = 10 >> LabelMedia = yes >> Random Access = Yes >> AutomaticMount = yes >> RemovableMedia = no >> AlwaysOpen = yes >> } >> >> Device { >> Name = "mongodb01_baculaserver01_Device_2" >> Media Type = File1 >> Archive Device = "/backup/bacula-storage/mongodb01/" >> Maximum Concurrent Jobs = 10 >> LabelMedia = yes >> Random Access = Yes >> AutomaticMount = yes >> RemovableMedia = no >> AlwaysOpen = yes >> } >> >> #This Device is stored in baculaserver02 where only a storage daemon is >> running (not bacula-dir) >> Device { >> Name = "mongodb01_baculaserver02_Device" >> Media Type = File1 >> Archive Device = "/backup/bacula-storage/mongodb01/" >> LabelMedia = yes; >> Random Access = Yes; >> AutomaticMount = yes; >> RemovableMedia = no; >> AlwaysOpen = no; >> } >> >> >> Backup is launched via cron every 3 days using: >> echo 'run job=mongodb01-Backup >> storage="mongodb01_baculaserver01_Storage_2" yes' | docker exec -i >> bacula-dir /opt/bacula/bin/bconsole >> Restore is triggered by another cron using the default Storage defined in >> the Job. >> >> Any advice or shared experience would be greatly appreciated. >> >> Thanks, >> _______________________________________________ >> Bacula-users mailing list >> Bacula-users@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/bacula-users >> >
_______________________________________________ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
