Hello, I love the design of S3QL so much that I want to use it in my homelab to replace my whole storage pool. But I have encountered a problem that when storing over 10 million files in a mount point, S3QL creates so many objects in a single bucket, without any prefix to divide these files into sub-directories. This behavior is causing my object storage backend (self-hosted MINIO) creating over 10 million files under a single directory, which is very slow on any operation that need list folder (any filesystem could not handle so many files in a single directory). I have checked the document carefully, but I could not find any instruction on how could I change this behavior. What I want is let S3QL stores objects with prefix, like when it want to store s3ql_data_1197116, the directory structure would be s3ql_data_1/s3ql_data_1197/s3ql_data_1197116, so that any single directory would have no more than 1k files/sub-directories. Could you please help to take a look and share some idea? I know this could be a backend specific problem, that AWS or other cloud provider do not have this issue. But I indeed appreciate the design of S3QL and want to use it in every place I could. Thanks a lot!
Best regards, Moon -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/s3ql/C080486A-797E-45B0-928E-5FC08C1A6B6E%40gmail.com.
