Hi Isaac, Hi Riku, are you sure that this limit affects you? Did you tried creating more than 50.000 data blocks?
AFAIK S3QL does not do directory/bucket/container listings on the Swift backend. So it does not matter if there are thousands or millions of data blocks on your Swift storage. > [...] i think you guys should add the directory feature in the > filesystem backend files. atless add a parameter in the > -backend-options specific only for openstack that allow create > automatic directories only to fix the 50K files limit. [...] There is no thing as a folder/directory in Swift terminology. Swift is (the same as S3 and all other object storages) a flat filesystem. You can have many buckets (or in Swift they are called container) but in each bucket you can only store files (or objects in object storage terminology). So a recursive directory structure is not possible. Swift clients can (any most of them do) opt to show you a virtual folder structure underneath a container. If an object is named “folder/structure/file.name” the Swift client can show you a virtual folder structure with one top level folder “folder” that has a sub-folder “structure” and a file “file.name” in that sub-folder. I suspect the 50.000 files per directory limit is a limit that not the hubic backend has but one or more of the hubic clients (the web app or the sync client probably). If you use that hubic account for nothing else as S3QL filesystems you should be ok. Otherwise you might want to experiment with the “prefix” option of S3QL <http://www.rath.org/s3ql-docs/mkfs.html> to put all S3QL files in a virtual folder of your hubic store and do not touch this “folder” in any other application that accesses your hubic account. I have had a S3QL file system on OVH Object Storage (the “pro” version of hubic – https://www.ovh.co.uk/cloud/storage/object-storage.xml ) with approx. 3 million data blocks / Swift objects in the container (but only approx. 400GB storage, many small and very good compressible files). The Swift backend did not mind that many objects in a single container. I’m currently deleting the filesystem – that involves doing a listing of all objects and issuing a HTTP DELETE requests for each of them. This takes many hours and is not finished yet but otherwise works OK. I opted to not use that filesystem anymore because S3QL and the backup application that filled up that filesystem (Burp, http://burp.grke.org/) are not a good fit. At the end the filesystem had 16 million directory entries and 1.5 million inodes (Burp uses hard links excessively) and the sqlite database that S3QL uses to store the filesystem structure was 1.2 GB uncompressed. A |s3qlstat| or |df| on that filesystem took several seconds because of the huge database size. Also S3QL scales not very good with parallel file accesses but Burp does a ton of those. (The sqlite database is not thread safe and thus every read/write access to the database gets serialized by S3QL). Now I use Bareos and 4 (in the future possibly more) different S3QL mounts (on 4 different container but you could do that on one container with different prefixes for every filesystem). Bareos distributes the read/write access on the 4 S3QL mounts and backing up the same data as before the combined uncompressed database sizes of all 4 S3QL filesystems is only 10 MB. -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
