Hi Folks, Thanks for sharing your ceph.conf along with the behavior.
There are some odd things there. 1. rgw_num_rados_handles is deprecated--it should be 1 (the default), but changing it may require you to check and retune the values for objecter_inflight_ops and objecter_inflight_op_bytes to be larger 2. you have very different rgw_thread_pool_size values on these to gateways; a value between 512 and 1024 is usually best (future rgws will not rely on large thread pools) 3. the actual behavior with 128 shards might be assisted by listing in unordered mode--HOWEVER, there was a bug in this feature which caused a perf regression and masked the benefit--make sure you have applied the fix for https://tracker.ceph.com/issues/39393 before evaluating regards, Matt On Fri, May 3, 2019 at 4:57 AM EDH - Manuel Rios Fernandez <mrios...@easydatahost.com> wrote: > > Hi, > > > > We got a ceph deployment 13.2.5 version, but several bucket with millions of > files. > > > > services: > > mon: 3 daemons, quorum CEPH001,CEPH002,CEPH003 > > mgr: CEPH001(active) > > osd: 106 osds: 106 up, 106 in > > rgw: 2 daemons active > > > > data: > > pools: 17 pools, 7120 pgs > > objects: 106.8 M objects, 271 TiB > > usage: 516 TiB used, 102 TiB / 619 TiB avail > > pgs: 7120 active+clean > > > > We done a test in a spare RGW server for this case. > > > > > > Customer report us that is unable to list their buckets, we tested in a > monitor with the command: > > > > s3cmd ls s3://[bucket] --no-ssl --limit 20 > > > > Takes 1m and 2 secs. > > > > RGW log in debug mode = 2 > > > > 2019-05-03 10:40:25.449 7f65f63e1700 1 ====== starting new request > req=0x55eba26e8970 ===== > > 2019-05-03 10:40:25.449 7f65f63e1700 2 req 113:0s::GET > /[bucketname]/::initializing for trans_id = > tx000000000000000000071-005ccbfe79-e6283e-default > > 2019-05-03 10:40:25.449 7f65f63e1700 2 req 113:0s:s3:GET > /[bucketname]/::getting op 0 > > 2019-05-03 10:40:25.449 7f65f63e1700 2 req 113:0s:s3:GET > /[bucketname]/:list_bucket:verifying requester > > 2019-05-03 10:40:25.449 7f65f63e1700 2 req 113:0s:s3:GET > /[bucketname]/:list_bucket:normalizing buckets and tenants > > 2019-05-03 10:40:25.449 7f65f63e1700 2 req 113:0s:s3:GET > /[bucketname]/:list_bucket:init permissions > > 2019-05-03 10:40:25.449 7f65f63e1700 2 req 113:0s:s3:GET > /[bucketname]/:list_bucket:recalculating target > > 2019-05-03 10:40:25.449 7f65f63e1700 2 req 113:0s:s3:GET > /[bucketname]/:list_bucket:reading permissions > > 2019-05-03 10:40:25.449 7f65f63e1700 2 req 113:0s:s3:GET > /[bucketname]/:list_bucket:init op > > 2019-05-03 10:40:25.449 7f65f63e1700 2 req 113:0s:s3:GET > /[bucketname]/:list_bucket:verifying op mask > > 2019-05-03 10:40:25.449 7f65f63e1700 2 req 113:0s:s3:GET > /[bucketname]/:list_bucket:verifying op permissions > > 2019-05-03 10:40:25.449 7f65f63e1700 2 req 113:0s:s3:GET > /[bucketname]/:list_bucket:verifying op params > > 2019-05-03 10:40:25.449 7f65f63e1700 2 req 113:0s:s3:GET > /[bucketname]/:list_bucket:pre-executing > > 2019-05-03 10:40:25.449 7f65f63e1700 2 req 113:0s:s3:GET > /[bucketname]/:list_bucket:executing > > 2019-05-03 10:40:41.026 7f660e411700 2 > RGWDataChangesLog::ChangesRenewThread: start > > 2019-05-03 10:41:03.026 7f660e411700 2 > RGWDataChangesLog::ChangesRenewThread: start > > 2019-05-03 10:41:25.026 7f660e411700 2 > RGWDataChangesLog::ChangesRenewThread: start > > 2019-05-03 10:41:47.026 7f660e411700 2 > RGWDataChangesLog::ChangesRenewThread: start > > 2019-05-03 10:41:49.395 7f65f63e1700 2 req 113:83.9461s:s3:GET > /[bucketname]/:list_bucket:completing > > 2019-05-03 10:41:49.395 7f65f63e1700 2 req 113:83.9461s:s3:GET > /[bucketname]/:list_bucket:op status=0 > > 2019-05-03 10:41:49.395 7f65f63e1700 2 req 113:83.9461s:s3:GET > /[bucketname]/:list_bucket:http status=200 > > 2019-05-03 10:41:49.395 7f65f63e1700 1 ====== req done req=0x55eba26e8970 op > status=0 http_status=200 ====== > > > > > > time s3cmd ls s3://[bucket] --no-ssl --limit 100 > > real 4m26.318s > > > > > > 2019-05-03 10:42:36.439 7f65f33db700 1 ====== starting new request > req=0x55eba26e8970 ===== > > 2019-05-03 10:42:36.439 7f65f33db700 2 req 115:0s::GET > /[bucketname]/::initializing for trans_id = > tx000000000000000000073-005ccbfefc-e6283e-default > > 2019-05-03 10:42:36.439 7f65f33db700 2 req 115:0s:s3:GET > /[bucketname]/::getting op 0 > > 2019-05-03 10:42:36.439 7f65f33db700 2 req 115:0s:s3:GET > /[bucketname]/:list_bucket:verifying requester > > 2019-05-03 10:42:36.439 7f65f33db700 2 req 115:0s:s3:GET > /[bucketname]/:list_bucket:normalizing buckets and tenants > > 2019-05-03 10:42:36.439 7f65f33db700 2 req 115:0s:s3:GET > /[bucketname]/:list_bucket:init permissions > > 2019-05-03 10:42:36.439 7f65f33db700 2 req 115:0s:s3:GET > /[bucketname]/:list_bucket:recalculating target > > 2019-05-03 10:42:36.439 7f65f33db700 2 req 115:0s:s3:GET > /[bucketname]/:list_bucket:reading permissions > > 2019-05-03 10:42:36.439 7f65f33db700 2 req 115:0s:s3:GET > /[bucketname]/:list_bucket:init op > > 2019-05-03 10:42:36.439 7f65f33db700 2 req 115:0s:s3:GET > /[bucketname]/:list_bucket:verifying op mask > > 2019-05-03 10:42:36.439 7f65f33db700 2 req 115:0s:s3:GET > /[bucketname]/:list_bucket:verifying op permissions > > 2019-05-03 10:42:36.439 7f65f33db700 2 req 115:0s:s3:GET > /[bucketname]/:list_bucket:verifying op params > > 2019-05-03 10:42:36.439 7f65f33db700 2 req 115:0s:s3:GET > /[bucketname]/:list_bucket:pre-executing > > 2019-05-03 10:42:36.439 7f65f33db700 2 req 115:0s:s3:GET > /[bucketname]/:list_bucket:executing > > 2019-05-03 10:42:53.026 7f660e411700 2 > RGWDataChangesLog::ChangesRenewThread: start > > 2019-05-03 10:43:15.027 7f660e411700 2 > RGWDataChangesLog::ChangesRenewThread: start > > 2019-05-03 10:43:37.028 7f660e411700 2 > RGWDataChangesLog::ChangesRenewThread: start > > 2019-05-03 10:43:59.027 7f660e411700 2 > RGWDataChangesLog::ChangesRenewThread: start > > 2019-05-03 10:44:21.028 7f660e411700 2 > RGWDataChangesLog::ChangesRenewThread: start > > 2019-05-03 10:44:43.027 7f660e411700 2 > RGWDataChangesLog::ChangesRenewThread: start > > 2019-05-03 10:45:05.027 7f660e411700 2 > RGWDataChangesLog::ChangesRenewThread: start > > 2019-05-03 10:45:18.260 7f660cc0e700 2 object expiration: start > > 2019-05-03 10:45:18.779 7f660cc0e700 2 object expiration: stop > > 2019-05-03 10:45:27.027 7f660e411700 2 > RGWDataChangesLog::ChangesRenewThread: start > > 2019-05-03 10:45:49.027 7f660e411700 2 > RGWDataChangesLog::ChangesRenewThread: start > > 2019-05-03 10:46:11.027 7f660e411700 2 > RGWDataChangesLog::ChangesRenewThread: start > > 2019-05-03 10:46:33.027 7f660e411700 2 > RGWDataChangesLog::ChangesRenewThread: start > > 2019-05-03 10:46:55.028 7f660e411700 2 > RGWDataChangesLog::ChangesRenewThread: start > > 2019-05-03 10:47:02.092 7f65f33db700 2 req 115:265.652s:s3:GET > /[bucketname]/:list_bucket:completing > > 2019-05-03 10:47:02.092 7f65f33db700 2 req 115:265.652s:s3:GET > /[bucketname]/:list_bucket:op status=0 > > 2019-05-03 10:47:02.092 7f65f33db700 2 req 115:265.652s:s3:GET > /[bucketname]/:list_bucket:http status=200 > > 2019-05-03 10:47:02.092 7f65f33db700 1 ====== req done req=0x55eba26e8970 op > status=0 http_status=200 ====== > > > > > > radosgw-admin bucket limit check > > } > > "bucket": "[BUCKETNAME]", > > "tenant": "", > > "num_objects": 7126133, > > "num_shards": 128, > > "objects_per_shard": 55672, > > "fill_status": "OK" > > }, > > > > > > We ‘realy don’t know who to solve that , looks like a timeout or slow > performance for that bucket. > > > > Our RGW section in ceph.conf > > > > [client.rgw.ceph-rgw01] > > host = ceph-rgw01 > > rgw enable usage log = true > > rgw dns name = XXXXXX > > rgw frontends = "beast port=7480" > > rgw resolve cname = false > > rgw thread pool size = 128 > > rgw num rados handles = 1 > > rgw op thread timeout = 120 > > > > > > [client.rgw.ceph-rgw03] > > host = ceph-rgw03 > > rgw enable usage log = true > > rgw dns name = XXXXXXXX > > rgw frontends = "beast port=7480" > > rgw resolve cname = false > > rgw thread pool size = 640 > > rgw num rados handles = 16 > > rgw op thread timeout = 120 > > > > > > Best Regards, > > > > Manuel > > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
