On Tuesday, April 21, 2015 at 2:58:29 AM UTC+2, Nikolaus Rath wrote: > > On Apr 20 2015, Hans Reinders <[email protected] <javascript:>> wrote: > > Hello, > > > > When using s3ql for backup purpose of files (movies) with size of 1 - 2 > GB, > > is it possible / advisable to increase the max-obj-size to a very high > > value (say 1GB)? > > Possible yes. Advisable I'm not sure. >
Because I have the cache on a 2GB tmpfs (mounted in memory), the tests failed, and so I have discarded this. > > > Because it will be only for backing up, the download > > granularity argument isn't the case. It also seems to me deduplication > will > > not be an argument to use the default max-obj-size of 10MB. > > Why not? > Output of s3ql after 4TB with the default max-obj-size of 10MB: Directory entries: 18952 Inodes: 18954 Data blocks: 449412 Total data size: 4.23 TB After de-duplication: 4.23 TB (100.00% of total) After compression: 4.23 TB (100.00% of total, 100.00% of de-duplicated) Database size: 62.8 MiB (uncompressed) Cache usage: 1016 MiB (dirty: 0 bytes) So deduplication is indeed not happening. > > > Main reason for this question is the fact that a TB or more of this kind > of > > files results in hundreds of thousands of objects with the default > > max-obj-size of 10MB. > > Why is this an issue? > Yes, you're right, with the 4TB I am not noticing remarkable slower speeds of the filesystem. > > > The backend used will be OpenStack Swift (Hubic). > > Are you planning to use the experimental patches from > https://bitbucket.org/nikratio/s3ql/issue/132/, or do you have a method > to make it work out of the box? > I have it working out of the box, For the OAuth2 authentication I use a PHP script HubiC2SwiftGate which I found on the Hubic forums, and has its home at https://github.com/oderwat/hubic2swiftgate. With this script it is also possible to use tools such as Cyberduck for creating the container. > > > Best, > -Nikolaus > > -- > GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F > Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F > > »Time flies like an arrow, fruit flies like a Banana.« > -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
