Hi Andreas, Yep, each FileSet has a separate job. Thanks for the tips. I discovered more clues, but backups are still not running flawlessly. I think the "scale problem" might actually be simply that bareos is somehow goofed up and doesn't successfully finish backups (anymore).
I've pared back to just a few clients and have noticed that most of the "running" backups don't finish successfully. They get stuck in some state where Some parts of bareos think a job is finished. Other parts say it is still running and the job never finishes. This leads to the queued jobs waiting forever. Here's one example.: *list jobs jobstatus=running [...] | 10,045 | backup-trog.physics.wisc.edu | trog.physics.wisc.edu | 2019-11-17 15:37:02 | B | I | 0 | 0 | R | [...] *status client=trog.physics.wisc.edu Connecting to Client trog.physics.wisc.edu at trog.physics.wisc.edu:9102 [...] trog-fd Version: 18.2.5 (30 January 2019) Linux-4.4.92-6.18-default [...] Terminated Jobs: [...] 10045 Incr 50,882 3.717 G OK 17-Nov-19 15:46 backup-trog.physics.wisc.edu (notice the client says that the backup finished successfully.) *status storage=PhysHDFS [...] Terminated Jobs: [...] 10045 Incr 50,882 3.740 G OK 17-Nov-19 15:47 backup-trog.physics.wisc.edu [...] *list joblog jobid=10045 2019-11-17 15:37:00 bareos-dir JobId 10045: Start Backup JobId 10045, Job=backup-trog.physics.wisc.edu.2019-11-17_15.37.00_09 2019-11-17 15:37:00 bareos-dir JobId 10045: Connected Storage daemon at thedoctor.physics.wisc.edu:9103, encryption: TLS_CHACHA20_POLY1305_SHA256 2019-11-17 15:37:00 bareos-dir JobId 10045: Using Device "PhysGdrive1" to write. 2019-11-17 15:37:00 bareos-dir JobId 10045: Probing client protocol... (result will be saved until config reload) 2019-11-17 15:37:00 bareos-dir JobId 10045: Connected Client: trog.physics.wisc.edu at trog.physics.wisc.edu:9102, encryption: TLS_CHACHA20_POLY1305_SHA256 2019-11-17 15:37:00 bareos-dir JobId 10045: Handshake: Immediate TLS 2019-11-17 15:37:00 bareos-dir JobId 10045: Encryption: TLS_CHACHA20_POLY1305_SHA256 2019-11-17 15:37:03 bareos-dir JobId 10045: Sending Accurate information. 2019-11-17 15:37:02 trog-fd JobId 10045: Connected Storage daemon at thedoctor.physics.wisc.edu:9103, encryption: TLS_CHACHA20_POLY1305_SHA256 2019-11-17 15:38:42 trog-fd JobId 10045: Extended attribute support is enabled 2019-11-17 15:38:42 trog-fd JobId 10045: ACL support is enabled 2019-11-17 15:38:42 bareos-sd JobId 10045: Spooling data ... 2019-11-17 15:46:22 bareos-sd JobId 10045: Committing spooled data to Volume "Merged-1547". Despooling 3,747,231,638 bytes ... 2019-11-17 15:47:15 bareos-sd JobId 10045: Despooling elapsed time = 00:00:53, Transfer rate = 70.70 M Bytes/second 2019-11-17 15:47:15 bareos-sd JobId 10045: Releasing device "PhysGdrive1" (/scratch/gdrive-merged). 2019-11-17 15:47:15 bareos-sd JobId 10045: Elapsed time=00:08:33, Transfer rate=7.291 M Bytes/second 2019-11-17 15:47:15 bareos-sd JobId 10045: Sending spooled attrs to the Director. Despooling 15,057,696 bytes ... 2019-11-17 15:47:22 bareos-dir JobId 10045: Insert of attributes batch table with 2 entries start Thanks for having a look! Chad. -- You received this message because you are subscribed to the Google Groups "bareos-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/bareos-users/e96967b0-f5d1-ef96-73c0-3ded28b40e2a%40physics.wisc.edu.
