Hi,
I have just upgraded my Bacula to 9.4.2 and wanted to switch from my old
s3fs-based cloud backups to the new S3 cloud support in Bacula. Though,
it doesn't work the way I would like it to work.
I have unlimited space on my S3 bucket and limited space for my local
cache. I have configured bacula to use 1GB volumes with 10MB parts and
to upload each part and truncate cache after upload.
The problem is S3 uploads are slower than the volumes are filled and I
run out of the cache space during backup. And Bacula handling of this
condition is far from being graceful – not only it stopped writing new
volumes, it also stopped uploading the existing ones, so the space won't
ever be freed.
05-Mar 09:24 jajo-dir JobId 15038: Max Volume jobs=1 exceeded. Marking
Volume "Full-0021" as Used.
05-Mar 09:24 jajo-sd JobId 15038: New volume "Full-0021" mounted on
device "S3-Cloud" (/var/spool/bacula/s3) at 05-Mar-2019 09:24.
05-Mar 09:24 jajo-sd JobId 15038: Fatal error: [SF0209] Out of freespace
caused End of Volume "Full-0021" at 53:9289728 on device "S3-Cloud"
(/var/spool/bacula/s3). Write of 64512 bytes got 24576.
05-Mar 09:24 jajo-sd JobId 15038: Elapsed time=00:21:09, Transfer
rate=16.12 M Bytes/second
05-Mar 09:24 jajo-sd JobId 15038: Cloud Upload transfers:
05-Mar 09:24 jajo-sd JobId 15038: Full-0001/part.1 state=done
size=233 B duration=1s
05-Mar 09:24 jajo-sd JobId 15038: Full-0001/part.2 state=done
size=9.999 MB duration=17s
05-Mar 09:24 jajo-sd JobId 15038: Full-0001/part.3 state=done
size=9.999 MB duration=14s
[...]
05-Mar 09:24 jajo-sd JobId 15038: Full-0001/part.84 state=done
size=9.999 MB duration=16s
05-Mar 09:24 jajo-sd JobId 15038: Full-0001/part.85 state=done
size=9.999 MB duration=12s
05-Mar 09:24 jajo-sd JobId 15038: Error: Full-0001/part.86
state=error size=9.999 MB duration=11s msg=S3_put_object
ERR=AbortedByCallback
05-Mar 09:24 jajo-sd JobId 15038: Error: Full-0001/part.87
state=error size=9.999 MB duration=0s msg=S3_put_object
ERR=AbortedByCallback
05-Mar 09:24 jajo-sd JobId 15038: Error: Full-0001/part.88
state=error size=9.999 MB duration=0s msg=S3_put_object
ERR=AbortedByCallback
[...]
05-Mar 09:27 jajo-sd JobId 15038: Error: Full-0007/part.84
state=error size=9.999 MB duration=0s msg=S3_put_object
ERR=AbortedByCallback
05-Mar 09:27 jajo-sd JobId 15038: Error: Full-0007/part.85
state=error size=9.999 MB duration=1s msg=S3_put_object
ERR=AbortedByCallback
[...]
My storage configuration.
Device {
Name = S3-Cloud
Device Type = Cloud
Cloud = S3-Cloud
Archive Device = /var/spool/bacula/s3
Maximum File Size = 10 MB
Maximum Part Size = 10 MB
Maximum Volume Size = 1 GB
Media Type = CloudVolume
LabelMedia = yes
Random Access = yes
AutomaticMount = yes
RemovableMedia = no
}
Cloud {
Name = S3-Cloud
Driver = "S3"
HostName = "s3.amazonaws.com"
BucketName = ...
AccessKey = ...
SecretKey = ...
Protocol = HTTPS
UriStyle = VirtualHost
Truncate Cache = AfterUpload
Upload = EachPart
Region = ...
MaximumUploadBandwidth = 8MB/s
}
Is there any way to make Bacula wait until enough volumes are uploaded
and removed before writing new ones to the cache? Maybe some virtual
autochanger magic?
Greets,
Jacek
_______________________________________________
Bacula-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/bacula-users
