Occasionally I see cloud backup errors of the general form below (copy/pasted from Baculum job log so a few lines are out of order) where the part.x upload errors. This will happen maybe 1 in 10 times. The backup will terminate with error or sometimes with OK and warnings. A restore runs OK but because I do not expire the local cache, the restore is I think coming from local cache and not the cloud storage. I don't see much network in traffic during the restore so it seems that's the case. I can use the bconsole cloud upload command to manually upload the failed volume and this invariably works.
My question is, is the cloud backup data good after this upload or is there some way to verify it? This won't be an issue until the cache disk needs replacing but when that happens, I'd like to know that the cloud data is good. 2022-11-18 01:17:54 raspberrypi-dir JobId 1031: End auto prune. 2022-11-18 01:17:54 raspberrypi-dir JobId 1031: Pruned Files from 1 Jobs for client pi-fd from catalog. 2022-11-18 01:17:54 raspberrypi-dir JobId 1031: Begin pruning Files. 2022-11-18 01:17:54 raspberrypi-dir JobId 1031: No Jobs found to prune. 2022-11-18 01:17:54 raspberrypi-dir JobId 1031: Begin pruning Jobs older than 1 day . 2022-11-18 01:17:54 raspberrypi-dir JobId 1031: Bacula raspberrypi-dir 11.0.6 (10Mar22): Build OS: aarch64-unknown-linux-gnu debian 11.3 JobId: 1031 Job: pi <http://kinsham.ddnsfree.com:7130/web/job/pi>.2022-11-18_01.15.00_53 Backup Level: Incremental, since=2022-11-17 01:15:03 Client: "pi-fd <http://kinsham.ddnsfree.com:7130/web/client/pi-fd>" 11.0.6 (10Mar22) aarch64-unknown-linux-gnu,debian,11.3 FileSet: "pi <http://kinsham.ddnsfree.com:7130/web/fileset/pi>" 2022-10-30 23:29:03 Pool: "pi-incr <http://kinsham.ddnsfree.com:7130/web/pool/pi-incr>" (From Job IncPool override) Catalog: "MyCatalog" (From Pool resource) Storage: "cloud-sd <http://kinsham.ddnsfree.com:7130/web/storage/cloud-sd>" (From Pool resource) Scheduled time: 18-Nov-2022 01:15:00 Start time: 18-Nov-2022 01:15:02 End time: 18-Nov-2022 01:17:54 Elapsed time: 2 mins 52 secs Priority: 10 FD Files Written: 1,225 SD Files Written: 1,225 FD Bytes Written: 637,589,362 (637.5 MB) SD Bytes Written: 637,765,966 (637.7 MB) Rate: 3706.9 KB/s Software Compression: 60.2% 2.5:1 Comm Line Compression: 5.1% 1.1:1 Snapshot/VSS: no Encryption: no Accurate: no Volume name(s): pi-incr0668 <http://kinsham.ddnsfree.com:7130/web/volume/pi-incr0668>|pi-incr0669 <http://kinsham.ddnsfree.com:7130/web/volume/pi-incr0669>|pi-incr0670 <http://kinsham.ddnsfree.com:7130/web/volume/pi-incr0670>|pi-incr0671 <http://kinsham.ddnsfree.com:7130/web/volume/pi-incr0671>|pi-incr0672 <http://kinsham.ddnsfree.com:7130/web/volume/pi-incr0672>|pi-incr0634 <http://kinsham.ddnsfree.com:7130/web/volume/pi-incr0634>|pi-incr0635 <http://kinsham.ddnsfree.com:7130/web/volume/pi-incr0635>| Volume Session Id: 78 Volume Session Time: 1667943458 Last Volume Bytes: 9,457,928 (9.457 MB) Non-fatal FD errors: 0 SD Errors: 1 FD termination status: OK SD termination status: OK Termination: Backup OK -- Upload to Cloud failed 2022-11-18 01:17:54 raspberrypi-sd JobId 1031: Sending spooled attrs to the Director. Despooling 299,382 bytes ... 2022-11-18 01:17:54 raspberrypi-sd JobId 1031: pi-incr0635/part.2 state=done size=9.457 MB duration=5s 2022-11-18 01:17:54 raspberrypi-sd JobId 1031: pi-incr0635/part.1 state=done size=240 B duration=2s 2022-11-18 01:17:54 raspberrypi-sd JobId 1031: pi-incr0634/part.2 state=done size=104.8 MB duration=38s 2022-11-18 01:17:49 raspberrypi-sd JobId 1031: pi-incr0634/part.1 state=done size=240 B duration=5s 2022-11-18 01:17:49 raspberrypi-sd JobId 1031: Error: pi-incr0672/part.2 state=error size=104.8 MB duration=52s msg=S3_put_object ERR=InternalError CURL Effective URL: https://s3.eu-central-003.backblazeb2.com/bacula01/pi-incr0672/part.2 CURL Effective URL: https://s3.eu-central-003.backblazeb2.com/bacula01/pi-incr0672/part.2 2022-11-18 01:17:38 raspberrypi-sd JobId 1031: pi-incr0671/part.2 state=done size=104.8 MB duration=49s 2022-11-18 01:17:10 raspberrypi-sd JobId 1031: pi-incr0671/part.1 state=done size=240 B duration=1s 2022-11-18 01:17:10 raspberrypi-sd JobId 1031: pi-incr0670/part.2 state=done size=104.8 MB duration=56s 2022-11-18 01:16:55 raspberrypi-sd JobId 1031: pi-incr0670/part.1 state=done size=240 B duration=2s 2022-11-18 01:16:55 raspberrypi-sd JobId 1031: pi-incr0669/part.2 state=done size=104.8 MB duration=48s 2022-11-18 01:16:48 raspberrypi-sd JobId 1031: pi-incr0669/part.1 state=done size=240 B duration=6s 2022-11-18 01:16:48 raspberrypi-sd JobId 1031: pi-incr0668/part.2 state=done size=104.8 MB duration=42s 2022-11-18 01:16:06 raspberrypi-sd JobId 1031: pi-incr0668/part.1 state=done size=240 B duration=0s 2022-11-18 01:16:06 raspberrypi-sd JobId 1031: Cloud Upload transfers: 2022-11-18 01:16:06 raspberrypi-sd JobId 1031: pi-incr0635/part.1 state=done size=240 B duration=1s 2022-11-18 01:16:06 raspberrypi-sd JobId 1031: pi-incr0634/part.1 state=done size=240 B duration=0s 2022-11-18 01:16:06 raspberrypi-sd JobId 1031: pi-incr0672/part.1 state=done size=240 B duration=0s 2022-11-18 01:16:06 raspberrypi-sd JobId 1031: pi-incr0671/part.1 state=done size=240 B duration=0s 2022-11-18 01:16:06 raspberrypi-sd JobId 1031: pi-incr0670/part.1 state=done size=240 B duration=0s 2022-11-18 01:16:06 raspberrypi-sd JobId 1031: pi-incr0669/part.1 state=done size=240 B duration=0s 2022-11-18 01:16:06 raspberrypi-sd JobId 1031: pi-incr0668/part.1 state=done size=240 B duration=0s 2022-11-18 01:16:06 raspberrypi-sd JobId 1031: Cloud Download transfers: 2022-11-18 01:16:05 raspberrypi-sd JobId 1031: Elapsed time=00:00:48, Transfer rate=13.28 M Bytes/second 2022-11-18 01:16:05 raspberrypi-sd JobId 1031: New volume "pi-incr0635" mounted on device "B2Bucket" (/mnt/usb16tb/cache) at 18-Nov-2022 01:16. 2022-11-18 01:16:05 raspberrypi-sd JobId 1031: Recycled volume "pi-incr0635" on Cloud device "B2Bucket" (/mnt/usb16tb/cache), all previous data lost. 2022-11-18 01:16:02 raspberrypi-dir JobId 1031: Recycled volume "pi-incr0635" 2022-11-18 01:16:02 raspberrypi-sd JobId 1031: End of medium on Volume "pi-incr0634" Bytes=104,832,236 Blocks=1,625 at 18-Nov-2022 01:16. 2022-11-18 01:15:58 raspberrypi-sd JobId 1031: New volume "pi-incr0634" mounted on device "B2Bucket" (/mnt/usb16tb/cache) at 18-Nov-2022 01:15. 2022-11-18 01:15:57 raspberrypi-sd JobId 1031: Recycled volume "pi-incr0634" on Cloud device "B2Bucket" (/mnt/usb16tb/cache), all previous data lost. 2022-11-18 01:15:57 raspberrypi-dir JobId 1031: Recycled volume "pi-incr0634" 2022-11-18 01:15:57 raspberrypi-sd JobId 1031: End of medium on Volume "pi-incr0672" Bytes=104,832,236 Blocks=1,625 at 18-Nov-2022 01:15. 2022-11-18 01:15:43 raspberrypi-fd JobId 1031: /boot is a different filesystem. Will not descend from / into it. 2022-11-18 01:15:43 raspberrypi-fd JobId 1031: /dev is a different filesystem. Will not descend from / into it. 2022-11-18 01:15:43 raspberrypi-fd JobId 1031: /mnt/usb16tb is a different filesystem. Will not descend from / into it. 2022-11-18 01:15:43 raspberrypi-fd JobId 1031: /run is a different filesystem. Will not descend from / into it. 2022-11-18 01:15:43 raspberrypi-fd JobId 1031: /sys is a different filesystem. Will not descend from / into it. 2022-11-18 01:15:43 raspberrypi-fd JobId 1031: /proc is a different filesystem. Will not descend from / into it. 2022-11-18 01:15:42 raspberrypi-sd JobId 1031: New volume "pi-incr0672" mounted on device "B2Bucket" (/mnt/usb16tb/cache) at 18-Nov-2022 01:15. 2022-11-18 01:15:42 raspberrypi-sd JobId 1031: Recycled volume "pi-incr0672" on Cloud device "B2Bucket" (/mnt/usb16tb/cache), all previous data lost. 2022-11-18 01:15:41 raspberrypi-fd JobId 1031: /var/lib/docker/overlay2/0d62a59f5b606d027733103031f03e74e021af20ba1517421cdbfb92afde214f/merged is a different filesystem. Will not descend from / into it. 2022-11-18 01:15:41 raspberrypi-fd JobId 1031: /var/lib/docker/overlay2/a8922ffbda7eba7303a64d45297cbbef992dbfb90a6a7b938523e7f3b0e50fd0/merged is a different filesystem. Will not descend from / into it. 2022-11-18 01:15:41 raspberrypi-fd JobId 1031: /var/lib/docker/overlay2/1f65ecc75564cf09e3c4069eb6296731a22cfc4ddca6a8082360c04059eb0305/merged is a different filesystem. Will not descend from / into it. 2022-11-18 01:15:41 raspberrypi-fd JobId 1031: /var/lib/docker/overlay2/bc4cdfc1942f062cb54133610b7066e8e17033612c7ab858b9aa5c8a0bcf90c4/merged is a different filesystem. Will not descend from / into it. 2022-11-18 01:15:40 raspberrypi-dir JobId 1031: Recycled volume "pi-incr0672" 2022-11-18 01:15:40 raspberrypi-sd JobId 1031: End of medium on Volume "pi-incr0671" Bytes=104,832,225 Blocks=1,625 at 18-Nov-2022 01:15. 2022-11-18 01:15:33 raspberrypi-sd JobId 1031: New volume "pi-incr0671" mounted on device "B2Bucket" (/mnt/usb16tb/cache) at 18-Nov-2022 01:15. 2022-11-18 01:15:33 raspberrypi-sd JobId 1031: Recycled volume "pi-incr0671" on Cloud device "B2Bucket" (/mnt/usb16tb/cache), all previous data lost. 2022-11-18 01:15:32 raspberrypi-dir JobId 1031: Recycled volume "pi-incr0671" 2022-11-18 01:15:32 raspberrypi-sd JobId 1031: End of medium on Volume "pi-incr0670" Bytes=104,832,226 Blocks=1,625 at 18-Nov-2022 01:15. 2022-11-18 01:15:27 raspberrypi-sd JobId 1031: New volume "pi-incr0670" mounted on device "B2Bucket" (/mnt/usb16tb/cache) at 18-Nov-2022 01:15. 2022-11-18 01:15:27 raspberrypi-sd JobId 1031: Recycled volume "pi-incr0670" on Cloud device "B2Bucket" (/mnt/usb16tb/cache), all previous data lost. 2022-11-18 01:15:24 raspberrypi-dir JobId 1031: Recycled volume "pi-incr0670" 2022-11-18 01:15:24 raspberrypi-sd JobId 1031: End of medium on Volume "pi-incr0669" Bytes=104,832,225 Blocks=1,625 at 18-Nov-2022 01:15. 2022-11-18 01:15:10 raspberrypi-sd JobId 1031: New volume "pi-incr0669" mounted on device "B2Bucket" (/mnt/usb16tb/cache) at 18-Nov-2022 01:15. 2022-11-18 01:15:10 raspberrypi-sd JobId 1031: Recycled volume "pi-incr0669" on Cloud device "B2Bucket" (/mnt/usb16tb/cache), all previous data lost. 2022-11-18 01:15:07 raspberrypi-dir JobId 1031: Recycled volume "pi-incr0669" 2022-11-18 01:15:07 raspberrypi-sd JobId 1031: End of medium on Volume "pi-incr0668" Bytes=104,832,236 Blocks=1,625 at 18-Nov-2022 01:15. 2022-11-18 01:15:04 raspberrypi-sd JobId 1031: Using S3 cloud driver Host=s3.eu-central-003.backblazeb2.com Bucket=bacula01 2022-11-18 01:15:04 raspberrypi-sd JobId 1031: Recycled volume "pi-incr0668" on Cloud device "B2Bucket" (/mnt/usb16tb/cache), all previous data lost. 2022-11-18 01:15:02 raspberrypi-dir JobId 1031: Using Device "B2Bucket" to write. 2022-11-18 01:15:02 raspberrypi-dir JobId 1031: Recycled volume "pi-incr0668" 2022-11-18 01:15:02 raspberrypi-dir JobId 1031: Max configured use duration=50,400 sec. exceeded. Marking Volume "pi-incr0667" as Used. 2022-11-18 01:15:02 raspberrypi-dir JobId 1031: Start Backup JobId 1031, Job=pi.2022-11-18_01.15.00_53 2022-11-18 01:17:38 raspberrypi-sd JobId 1031: pi-incr0672/part.1 state=done size=240 B duration=0s The SD conf is (Backblaze B2): pi@raspberrypi:~ $ cat /etc/bacula/bacula-sd.conf Director { Name = "raspberrypi-dir" Password = xx } Storage { Name = "raspberrypi-sd" SdAddress = zz WorkingDirectory = "/var/lib/bacula" PidDirectory = "/var/run/bacula" PluginDirectory = "/usr/lib" MaximumConcurrentJobs = 2 HeartbeatInterval = 300 } Device { Name = "B2Bucket" MediaType = "Cloud" DeviceType = "Cloud" ArchiveDevice = "/mnt/usb16tb/cache" RemovableMedia = no RandomAccess = yes AutomaticMount = yes LabelMedia = yes AlwaysOpen = yes MaximumPartSize = 500000000 Cloud = "B2" } Messages { Name = "Standard" Director = raspberrypi-dir = All } Cloud { Name = "B2" Driver = "S3" HostName = "s3.eu-central-003.backblazeb2.com" BucketName = xx Region = "eu-central-003" AccessKey = xx SecretKey = yy Protocol = "HTTPS" UriStyle = "Path" TruncateCache = “No” #keep the cache Upload = "AtEndOfJob" MaximumUploadBandwidth = 10000000 } Regards Chris Wilkinson
_______________________________________________ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
