After some tests and research, I found a workaround for me. Maybe it can help somebody else...
I am using Scaleway Object Storage (S3 compatible). I set volsize to 1GB but always get a "Broken pipe" error when uploading large sigtar files (>8GB) Tried to use boto backend and "--s3-use-multiprocessing" whith no luck (got a bytes/string TypeError with _boto_multi.py) So I switch to boto3 backend to get multiprocessing working "Boto3 always attempts to multiprocessing when it is believed it will be more efficient" Source : http://duplicity.nongnu.org/vers8/duplicity.1.html But still get the same error as the begin : "Broken pipe" After some research I found a limitation with Scaleway : "Object Storage supports multipart upload. We recommend uploading by chunks, in a limit of 1000 chunks per upload and 5TB per object." Source : https://www.scaleway.com/en/faq/how-can-i-upload-large-objects/ The default boto3 multipart_chunksize is 8MB and my sigtar file is 8.8GB So I tweak multipart_threshold & multipart_chunksize boto3 variables (from 8MB to 50MB) and everything works fine ! :) -- You received this bug notification because you are a member of duplicity-team, which is subscribed to a duplicate bug report (502609). https://bugs.launchpad.net/bugs/385495 Title: Large backup signature and manifest files should be split with --volsize too Status in Déjà Dup: Triaged Status in Duplicity: In Progress Bug description: With new 0.6.0 release, the signature & manifest archive should be split to respect the volume size's command line option. Without this, it's not possible to backup to limited filesize backend, like imap, or ftp. To manage notifications about this bug go to: https://bugs.launchpad.net/deja-dup/+bug/385495/+subscriptions _______________________________________________ Mailing list: https://launchpad.net/~duplicity-team Post to : duplicity-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~duplicity-team More help : https://help.launchpad.net/ListHelp
