On Dec 09 2015, Daniel Jagszent <[email protected]> wrote: > AFAIK S3QL does not do directory/bucket/container listings on the Swift > backend. So it does not matter if there are thousands or millions of > data blocks on your Swift storage.
Well, fsck.s3ql does such listings. But they are paginated, so there should be no issues. > I opted to not use that filesystem anymore because S3QL and the backup > application that filled up that filesystem (Burp, http://burp.grke.org/) > are not a good fit. At the end the filesystem had 16 million directory > entries and 1.5 million inodes (Burp uses hard links excessively) and > the sqlite database that S3QL uses to store the filesystem structure was > 1.2 GB uncompressed. This is not unreasonable though. Note that ext4 would require at least 5 GB of metadata as well - just to store the inodes (assuming 4096 bytes inode size). That's not yet counting directory entry *names*. > Also S3QL scales not very good with parallel file accesses but Burp > does a ton of those. (The sqlite database is not thread safe and thus > every read/write access to the database gets serialized by S3QL). Both is true, but one is not the cause of another. Most reads/writes don't require access to the database and could run in parallel. However, S3QL itself is mostly single threaded at the moment so the requests are indeed serialized. However, I have plans in the drawer to fix this at some point. The idea is to handle reads/writes for blocks that are already cached entirely at the C level. This will allow concurrenty *and* at the same time boost single-threaded performance as well. Just need to find the time... Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
