Any time using shared storage like S3 or cephfs/nfs/gluster/etc the absolute rule that I refuse to break is to never rely on a directory listing to know where objects/files are. You should be maintaining a database of some sort or a deterministic naming scheme. The only time a full listing of a directory should be required is if you feel like your tooling is orphaning files and you want to clean them up. If I had someone with a bucket with 2B objects, I would force them to use an index-less bucket.
That's me, though. I'm sure there are ways to manage a bucket in other ways, but it sounds awful. On Tue, May 1, 2018 at 10:10 AM Robert Stanford <[email protected]> wrote: > > Listing will always take forever when using a high shard number, AFAIK. > That's the tradeoff for sharding. Are those 2B objects in one bucket? > How's your read and write performance compared to a bucket with a lower > number (thousands) of objects, with that shard number? > > On Tue, May 1, 2018 at 7:59 AM, Katie Holly <[email protected]> wrote: > >> One of our radosgw buckets has grown a lot in size, `rgw bucket stats >> --bucket $bucketname` reports a total of 2,110,269,538 objects with the >> bucket index sharded across 32768 shards, listing the root context of the >> bucket with `s3 ls s3://$bucketname` takes more than an hour which is the >> hard limit to first-byte on our nginx reverse proxy and the aws-cli times >> out long before that timeout limit is hit. >> >> The software we use supports sharding the data across multiple s3 buckets >> but before I go ahead and enable this, has anyone ever had that many >> objects in a single RGW bucket and can let me know how you solved the >> problem of RGW taking a long time to read the full index? >> >> -- >> Best regards >> >> Katie Holly >> _______________________________________________ >> ceph-users mailing list >> [email protected] >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
