I think I would not override the default value for "rgw list buckets max chunk", I have no experience doing that, though I can see why it might be plausible.
Matt On Fri, May 3, 2019 at 9:39 AM EDH - Manuel Rios Fernandez <mrios...@easydatahost.com> wrote: > > From changes right know we got some other errors... > > 2019-05-03 15:37:28.604 7f499a2e8700 1 ====== starting new request > req=0x55f326692970 ===== > 2019-05-03 15:37:28.604 7f499a2e8700 2 req 23651:0s::GET > /CBERRY/MBS-69e38e26-5516-4b83-864c-d2b05b9db5af/CBB_DFTDI/CBB_DiskImage/Disk_011c3bdb-ac85-41f4-b727-c263001ba42f/Volume_Unknown_fbf0ea7a-af96-4dd4-9ad5-dbf6efdeefdc%24/20190430074414/0.cbrevision::initializing > for trans_id = tx000000000000000005c63-005ccc4418-e76558-default > 2019-05-03 15:37:28.604 7f499a2e8700 2 req 23651:0s:s3:GET > /CBERRY/MBS-69e38e26-5516-4b83-864c-d2b05b9db5af/CBB_DFTDI/CBB_DiskImage/Disk_011c3bdb-ac85-41f4-b727-c263001ba42f/Volume_Unknown_fbf0ea7a-af96-4dd4-9ad5-dbf6efdeefdc%24/20190430074414/0.cbrevision::getting > op 0 > 2019-05-03 15:37:28.604 7f499a2e8700 2 req 23651:0s:s3:GET > /CBERRY/MBS-69e38e26-5516-4b83-864c-d2b05b9db5af/CBB_DFTDI/CBB_DiskImage/Disk_011c3bdb-ac85-41f4-b727-c263001ba42f/Volume_Unknown_fbf0ea7a-af96-4dd4-9ad5-dbf6efdeefdc%24/20190430074414/0.cbrevision:get_obj:verifying > requester > 2019-05-03 15:37:28.604 7f499a2e8700 2 req 23651:0s:s3:GET > /CBERRY/MBS-69e38e26-5516-4b83-864c-d2b05b9db5af/CBB_DFTDI/CBB_DiskImage/Disk_011c3bdb-ac85-41f4-b727-c263001ba42f/Volume_Unknown_fbf0ea7a-af96-4dd4-9ad5-dbf6efdeefdc%24/20190430074414/0.cbrevision:get_obj:normalizing > buckets and tenants > 2019-05-03 15:37:28.604 7f499a2e8700 2 req 23651:0s:s3:GET > /CBERRY/MBS-69e38e26-5516-4b83-864c-d2b05b9db5af/CBB_DFTDI/CBB_DiskImage/Disk_011c3bdb-ac85-41f4-b727-c263001ba42f/Volume_Unknown_fbf0ea7a-af96-4dd4-9ad5-dbf6efdeefdc%24/20190430074414/0.cbrevision:get_obj:init > permissions > 2019-05-03 15:37:28.604 7f499a2e8700 2 req 23651:0s:s3:GET > /CBERRY/MBS-69e38e26-5516-4b83-864c-d2b05b9db5af/CBB_DFTDI/CBB_DiskImage/Disk_011c3bdb-ac85-41f4-b727-c263001ba42f/Volume_Unknown_fbf0ea7a-af96-4dd4-9ad5-dbf6efdeefdc%24/20190430074414/0.cbrevision:get_obj:recalculating > target > 2019-05-03 15:37:28.604 7f499a2e8700 2 req 23651:0s:s3:GET > /CBERRY/MBS-69e38e26-5516-4b83-864c-d2b05b9db5af/CBB_DFTDI/CBB_DiskImage/Disk_011c3bdb-ac85-41f4-b727-c263001ba42f/Volume_Unknown_fbf0ea7a-af96-4dd4-9ad5-dbf6efdeefdc%24/20190430074414/0.cbrevision:get_obj:reading > permissions > 2019-05-03 15:37:28.607 7f499a2e8700 2 req 23651:0.003s:s3:GET > /CBERRY/MBS-69e38e26-5516-4b83-864c-d2b05b9db5af/CBB_DFTDI/CBB_DiskImage/Disk_011c3bdb-ac85-41f4-b727-c263001ba42f/Volume_Unknown_fbf0ea7a-af96-4dd4-9ad5-dbf6efdeefdc%24/20190430074414/0.cbrevision:get_obj:init > op > 2019-05-03 15:37:28.607 7f499a2e8700 2 req 23651:0.003s:s3:GET > /CBERRY/MBS-69e38e26-5516-4b83-864c-d2b05b9db5af/CBB_DFTDI/CBB_DiskImage/Disk_011c3bdb-ac85-41f4-b727-c263001ba42f/Volume_Unknown_fbf0ea7a-af96-4dd4-9ad5-dbf6efdeefdc%24/20190430074414/0.cbrevision:get_obj:verifying > op mask > 2019-05-03 15:37:28.607 7f499a2e8700 2 req 23651:0.003s:s3:GET > /CBERRY/MBS-69e38e26-5516-4b83-864c-d2b05b9db5af/CBB_DFTDI/CBB_DiskImage/Disk_011c3bdb-ac85-41f4-b727-c263001ba42f/Volume_Unknown_fbf0ea7a-af96-4dd4-9ad5-dbf6efdeefdc%24/20190430074414/0.cbrevision:get_obj:verifying > op permissions > 2019-05-03 15:37:28.607 7f499a2e8700 2 req 23651:0.003s:s3:GET > /CBERRY/MBS-69e38e26-5516-4b83-864c-d2b05b9db5af/CBB_DFTDI/CBB_DiskImage/Disk_011c3bdb-ac85-41f4-b727-c263001ba42f/Volume_Unknown_fbf0ea7a-af96-4dd4-9ad5-dbf6efdeefdc%24/20190430074414/0.cbrevision:get_obj:verifying > op params > 2019-05-03 15:37:28.607 7f499a2e8700 2 req 23651:0.003s:s3:GET > /CBERRY/MBS-69e38e26-5516-4b83-864c-d2b05b9db5af/CBB_DFTDI/CBB_DiskImage/Disk_011c3bdb-ac85-41f4-b727-c263001ba42f/Volume_Unknown_fbf0ea7a-af96-4dd4-9ad5-dbf6efdeefdc%24/20190430074414/0.cbrevision:get_obj:pre-executing > 2019-05-03 15:37:28.607 7f499a2e8700 2 req 23651:0.003s:s3:GET > /CBERRY/MBS-69e38e26-5516-4b83-864c-d2b05b9db5af/CBB_DFTDI/CBB_DiskImage/Disk_011c3bdb-ac85-41f4-b727-c263001ba42f/Volume_Unknown_fbf0ea7a-af96-4dd4-9ad5-dbf6efdeefdc%24/20190430074414/0.cbrevision:get_obj:executing > 2019-05-03 15:37:28.958 7f4a68484700 -1 write_data failed: Connection reset > by peer > 2019-05-03 15:37:28.959 7f4a68484700 0 ERROR: flush_read_list(): > d->client_cb->handle_data() returned -104 > 2019-05-03 15:37:28.959 7f4a68484700 2 req 23574:41.87s:s3:GET > /CBERRY/MBS-69e38e26-5516-4b83-864c-d2b05b9db5af/CBB_DFTDI/CBB_DiskImage/Disk_011c3bdb-ac85-41f4-b727-c263001ba42f/Volume_Unknown_fbf0ea7a-af96-4dd4-9ad5-dbf6efdeefdc%24/20190430074414/0.cbrevision:get_obj:completing > 2019-05-03 15:37:28.959 7f4a68484700 2 req 23574:41.87s:s3:GET > /CBERRY/MBS-69e38e26-5516-4b83-864c-d2b05b9db5af/CBB_DFTDI/CBB_DiskImage/Disk_011c3bdb-ac85-41f4-b727-c263001ba42f/Volume_Unknown_fbf0ea7a-af96-4dd4-9ad5-dbf6efdeefdc%24/20190430074414/0.cbrevision:get_obj:op > status=-104 > 2019-05-03 15:37:28.959 7f4a68484700 2 req 23574:41.87s:s3:GET > /CBERRY/MBS-69e38e26-5516-4b83-864c-d2b05b9db5af/CBB_DFTDI/CBB_DiskImage/Disk_011c3bdb-ac85-41f4-b727-c263001ba42f/Volume_Unknown_fbf0ea7a-af96-4dd4-9ad5-dbf6efdeefdc%24/20190430074414/0.cbrevision:get_obj:http > status=206 > 2019-05-03 15:37:28.959 7f4a68484700 1 ====== req done req=0x55f2fde20970 op > status=-104 http_status=206 ====== > > > -----Mensaje original----- > De: EDH - Manuel Rios Fernandez <mrios...@easydatahost.com> > Enviado el: viernes, 3 de mayo de 2019 15:12 > Para: 'Matt Benjamin' <mbenj...@redhat.com> > CC: 'ceph-users' <ceph-users@lists.ceph.com> > Asunto: RE: [ceph-users] RGW Bucket unable to list buckets 100TB bucket > > Hi Matt, > > Thanks for your help, > > We have done the changes plus a reboot of MONs and RGW they look like strange > stucked , now we're able to list 250 directories. > > time s3cmd ls s3://datos101 --no-ssl --limit 150 > real 2m50.854s > user 0m0.147s > sys 0m0.042s > > > Is there any recommendation of max_shard ? > > Our main goal is cold storage, normally our usage are backups or customers > tons of files. This cause that customers in single bucket store millions > objetcs. > > Its strange because this issue started on Friday without any warning error at > OSD / RGW logs. > > When you should warning customer that will not be able to list their > directory if they reach X Millions objetcs? > > Our current ceph.conf > > #Normal-Memory 1/5 > debug rgw = 2 > #Disable > debug osd = 0 > debug journal = 0 > debug ms = 0 > > fsid = e1ee8086-7cce-43fd-a252-3d677af22428 > mon_initial_members = CEPH001, CEPH002, CEPH003 mon_host = > 172.16.2.10,172.16.2.11,172.16.2.12 > auth_cluster_required = cephx > auth_service_required = cephx > auth_client_required = cephx > osd pool default pg num = 128 > osd pool default pgp num = 128 > > public network = 172.16.2.0/24 > cluster network = 172.16.1.0/24 > > osd pool default size = 2 > osd pool default min size = 1 > > rgw_dynamic_resharding = true > #Increment to 128 > rgw_override_bucket_index_max_shards = 128 > > #Default: 1000 > rgw list buckets max chunk = 5000 > > > > [osd] > osd mkfs type = xfs > osd op threads = 12 > osd disk threads = 12 > > osd recovery threads = 4 > osd recovery op priority = 1 > osd recovery max active = 2 > osd recovery max single start = 1 > > osd max backfills = 4 > osd backfill scan max = 16 > osd backfill scan min = 4 > osd client op priority = 63 > > > osd_memory_target = 2147483648 > > osd_scrub_begin_hour = 23 > osd_scrub_end_hour = 6 > osd_scrub_load_threshold = 0.25 #low load scrubbing osd_scrub_during_recovery > = false #scrub during recovery > > [mon] > mon allow pool delete = true > mon osd min down reporters = 3 > > [mon.a] > host = CEPH001 > public bind addr = 172.16.2.10 > mon addr = 172.16.2.10:6789 > mon allow pool delete = true > > [mon.b] > host = CEPH002 > public bind addr = 172.16.2.11 > mon addr = 172.16.2.11:6789 > mon allow pool delete = true > > [mon.c] > host = CEPH003 > public bind addr = 172.16.2.12 > mon addr = 172.16.2.12:6789 > mon allow pool delete = true > > [client.rgw] > rgw enable usage log = true > > > [client.rgw.ceph-rgw01] > host = ceph-rgw01 > rgw enable usage log = true > rgw dns name = > rgw frontends = "beast port=7480" > rgw resolve cname = false > rgw thread pool size = 512 > rgw num rados handles = 1 > rgw op thread timeout = 600 > > > [client.rgw.ceph-rgw03] > host = ceph-rgw03 > rgw enable usage log = true > rgw dns name = > rgw frontends = "beast port=7480" > rgw resolve cname = false > rgw thread pool size = 512 > rgw num rados handles = 1 > rgw op thread timeout = 600 > > > On Thursday customer tell us that listing were instant, and now their > programs delay until timeout. > > Best Regards > > Manuel > > -----Mensaje original----- > De: Matt Benjamin <mbenj...@redhat.com> > Enviado el: viernes, 3 de mayo de 2019 14:00 > Para: EDH - Manuel Rios Fernandez <mrios...@easydatahost.com> > CC: ceph-users <ceph-users@lists.ceph.com> > Asunto: Re: [ceph-users] RGW Bucket unable to list buckets 100TB bucket > > Hi Folks, > > Thanks for sharing your ceph.conf along with the behavior. > > There are some odd things there. > > 1. rgw_num_rados_handles is deprecated--it should be 1 (the default), but > changing it may require you to check and retune the values for > objecter_inflight_ops and objecter_inflight_op_bytes to be larger 2. you have > very different rgw_thread_pool_size values on these to gateways; a value > between 512 and 1024 is usually best (future rgws will not rely on large > thread pools) 3. the actual behavior with 128 shards might be assisted by > listing in unordered mode--HOWEVER, there was a bug in this feature which > caused a perf regression and masked the benefit--make sure you have applied > the fix for https://tracker.ceph.com/issues/39393 before evaluating > > regards, > > Matt > > On Fri, May 3, 2019 at 4:57 AM EDH - Manuel Rios Fernandez > <mrios...@easydatahost.com> wrote: > > > > Hi, > > > > > > > > We got a ceph deployment 13.2.5 version, but several bucket with millions > > of files. > > > > > > > > services: > > > > mon: 3 daemons, quorum CEPH001,CEPH002,CEPH003 > > > > mgr: CEPH001(active) > > > > osd: 106 osds: 106 up, 106 in > > > > rgw: 2 daemons active > > > > > > > > data: > > > > pools: 17 pools, 7120 pgs > > > > objects: 106.8 M objects, 271 TiB > > > > usage: 516 TiB used, 102 TiB / 619 TiB avail > > > > pgs: 7120 active+clean > > > > > > > > We done a test in a spare RGW server for this case. > > > > > > > > > > > > Customer report us that is unable to list their buckets, we tested in a > > monitor with the command: > > > > > > > > s3cmd ls s3://[bucket] --no-ssl --limit 20 > > > > > > > > Takes 1m and 2 secs. > > > > > > > > RGW log in debug mode = 2 > > > > > > > > 2019-05-03 10:40:25.449 7f65f63e1700 1 ====== starting new request > > req=0x55eba26e8970 ===== > > > > 2019-05-03 10:40:25.449 7f65f63e1700 2 req 113:0s::GET > > /[bucketname]/::initializing for trans_id = > > tx000000000000000000071-005ccbfe79-e6283e-default > > > > 2019-05-03 10:40:25.449 7f65f63e1700 2 req 113:0s:s3:GET > > /[bucketname]/::getting op 0 > > > > 2019-05-03 10:40:25.449 7f65f63e1700 2 req 113:0s:s3:GET > > /[bucketname]/:list_bucket:verifying requester > > > > 2019-05-03 10:40:25.449 7f65f63e1700 2 req 113:0s:s3:GET > > /[bucketname]/:list_bucket:normalizing buckets and tenants > > > > 2019-05-03 10:40:25.449 7f65f63e1700 2 req 113:0s:s3:GET > > /[bucketname]/:list_bucket:init permissions > > > > 2019-05-03 10:40:25.449 7f65f63e1700 2 req 113:0s:s3:GET > > /[bucketname]/:list_bucket:recalculating target > > > > 2019-05-03 10:40:25.449 7f65f63e1700 2 req 113:0s:s3:GET > > /[bucketname]/:list_bucket:reading permissions > > > > 2019-05-03 10:40:25.449 7f65f63e1700 2 req 113:0s:s3:GET > > /[bucketname]/:list_bucket:init op > > > > 2019-05-03 10:40:25.449 7f65f63e1700 2 req 113:0s:s3:GET > > /[bucketname]/:list_bucket:verifying op mask > > > > 2019-05-03 10:40:25.449 7f65f63e1700 2 req 113:0s:s3:GET > > /[bucketname]/:list_bucket:verifying op permissions > > > > 2019-05-03 10:40:25.449 7f65f63e1700 2 req 113:0s:s3:GET > > /[bucketname]/:list_bucket:verifying op params > > > > 2019-05-03 10:40:25.449 7f65f63e1700 2 req 113:0s:s3:GET > > /[bucketname]/:list_bucket:pre-executing > > > > 2019-05-03 10:40:25.449 7f65f63e1700 2 req 113:0s:s3:GET > > /[bucketname]/:list_bucket:executing > > > > 2019-05-03 10:40:41.026 7f660e411700 2 > > RGWDataChangesLog::ChangesRenewThread: start > > > > 2019-05-03 10:41:03.026 7f660e411700 2 > > RGWDataChangesLog::ChangesRenewThread: start > > > > 2019-05-03 10:41:25.026 7f660e411700 2 > > RGWDataChangesLog::ChangesRenewThread: start > > > > 2019-05-03 10:41:47.026 7f660e411700 2 > > RGWDataChangesLog::ChangesRenewThread: start > > > > 2019-05-03 10:41:49.395 7f65f63e1700 2 req 113:83.9461s:s3:GET > > /[bucketname]/:list_bucket:completing > > > > 2019-05-03 10:41:49.395 7f65f63e1700 2 req 113:83.9461s:s3:GET > > /[bucketname]/:list_bucket:op status=0 > > > > 2019-05-03 10:41:49.395 7f65f63e1700 2 req 113:83.9461s:s3:GET > > /[bucketname]/:list_bucket:http status=200 > > > > 2019-05-03 10:41:49.395 7f65f63e1700 1 ====== req done > > req=0x55eba26e8970 op status=0 http_status=200 ====== > > > > > > > > > > > > time s3cmd ls s3://[bucket] --no-ssl --limit 100 > > > > real 4m26.318s > > > > > > > > > > > > 2019-05-03 10:42:36.439 7f65f33db700 1 ====== starting new request > > req=0x55eba26e8970 ===== > > > > 2019-05-03 10:42:36.439 7f65f33db700 2 req 115:0s::GET > > /[bucketname]/::initializing for trans_id = > > tx000000000000000000073-005ccbfefc-e6283e-default > > > > 2019-05-03 10:42:36.439 7f65f33db700 2 req 115:0s:s3:GET > > /[bucketname]/::getting op 0 > > > > 2019-05-03 10:42:36.439 7f65f33db700 2 req 115:0s:s3:GET > > /[bucketname]/:list_bucket:verifying requester > > > > 2019-05-03 10:42:36.439 7f65f33db700 2 req 115:0s:s3:GET > > /[bucketname]/:list_bucket:normalizing buckets and tenants > > > > 2019-05-03 10:42:36.439 7f65f33db700 2 req 115:0s:s3:GET > > /[bucketname]/:list_bucket:init permissions > > > > 2019-05-03 10:42:36.439 7f65f33db700 2 req 115:0s:s3:GET > > /[bucketname]/:list_bucket:recalculating target > > > > 2019-05-03 10:42:36.439 7f65f33db700 2 req 115:0s:s3:GET > > /[bucketname]/:list_bucket:reading permissions > > > > 2019-05-03 10:42:36.439 7f65f33db700 2 req 115:0s:s3:GET > > /[bucketname]/:list_bucket:init op > > > > 2019-05-03 10:42:36.439 7f65f33db700 2 req 115:0s:s3:GET > > /[bucketname]/:list_bucket:verifying op mask > > > > 2019-05-03 10:42:36.439 7f65f33db700 2 req 115:0s:s3:GET > > /[bucketname]/:list_bucket:verifying op permissions > > > > 2019-05-03 10:42:36.439 7f65f33db700 2 req 115:0s:s3:GET > > /[bucketname]/:list_bucket:verifying op params > > > > 2019-05-03 10:42:36.439 7f65f33db700 2 req 115:0s:s3:GET > > /[bucketname]/:list_bucket:pre-executing > > > > 2019-05-03 10:42:36.439 7f65f33db700 2 req 115:0s:s3:GET > > /[bucketname]/:list_bucket:executing > > > > 2019-05-03 10:42:53.026 7f660e411700 2 > > RGWDataChangesLog::ChangesRenewThread: start > > > > 2019-05-03 10:43:15.027 7f660e411700 2 > > RGWDataChangesLog::ChangesRenewThread: start > > > > 2019-05-03 10:43:37.028 7f660e411700 2 > > RGWDataChangesLog::ChangesRenewThread: start > > > > 2019-05-03 10:43:59.027 7f660e411700 2 > > RGWDataChangesLog::ChangesRenewThread: start > > > > 2019-05-03 10:44:21.028 7f660e411700 2 > > RGWDataChangesLog::ChangesRenewThread: start > > > > 2019-05-03 10:44:43.027 7f660e411700 2 > > RGWDataChangesLog::ChangesRenewThread: start > > > > 2019-05-03 10:45:05.027 7f660e411700 2 > > RGWDataChangesLog::ChangesRenewThread: start > > > > 2019-05-03 10:45:18.260 7f660cc0e700 2 object expiration: start > > > > 2019-05-03 10:45:18.779 7f660cc0e700 2 object expiration: stop > > > > 2019-05-03 10:45:27.027 7f660e411700 2 > > RGWDataChangesLog::ChangesRenewThread: start > > > > 2019-05-03 10:45:49.027 7f660e411700 2 > > RGWDataChangesLog::ChangesRenewThread: start > > > > 2019-05-03 10:46:11.027 7f660e411700 2 > > RGWDataChangesLog::ChangesRenewThread: start > > > > 2019-05-03 10:46:33.027 7f660e411700 2 > > RGWDataChangesLog::ChangesRenewThread: start > > > > 2019-05-03 10:46:55.028 7f660e411700 2 > > RGWDataChangesLog::ChangesRenewThread: start > > > > 2019-05-03 10:47:02.092 7f65f33db700 2 req 115:265.652s:s3:GET > > /[bucketname]/:list_bucket:completing > > > > 2019-05-03 10:47:02.092 7f65f33db700 2 req 115:265.652s:s3:GET > > /[bucketname]/:list_bucket:op status=0 > > > > 2019-05-03 10:47:02.092 7f65f33db700 2 req 115:265.652s:s3:GET > > /[bucketname]/:list_bucket:http status=200 > > > > 2019-05-03 10:47:02.092 7f65f33db700 1 ====== req done > > req=0x55eba26e8970 op status=0 http_status=200 ====== > > > > > > > > > > > > radosgw-admin bucket limit check > > > > } > > > > "bucket": "[BUCKETNAME]", > > > > "tenant": "", > > > > "num_objects": 7126133, > > > > "num_shards": 128, > > > > "objects_per_shard": 55672, > > > > "fill_status": "OK" > > > > }, > > > > > > > > > > > > We ‘realy don’t know who to solve that , looks like a timeout or slow > > performance for that bucket. > > > > > > > > Our RGW section in ceph.conf > > > > > > > > [client.rgw.ceph-rgw01] > > > > host = ceph-rgw01 > > > > rgw enable usage log = true > > > > rgw dns name = XXXXXX > > > > rgw frontends = "beast port=7480" > > > > rgw resolve cname = false > > > > rgw thread pool size = 128 > > > > rgw num rados handles = 1 > > > > rgw op thread timeout = 120 > > > > > > > > > > > > [client.rgw.ceph-rgw03] > > > > host = ceph-rgw03 > > > > rgw enable usage log = true > > > > rgw dns name = XXXXXXXX > > > > rgw frontends = "beast port=7480" > > > > rgw resolve cname = false > > > > rgw thread pool size = 640 > > > > rgw num rados handles = 16 > > > > rgw op thread timeout = 120 > > > > > > > > > > > > Best Regards, > > > > > > > > Manuel > > > > > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > -- > > Matt Benjamin > Red Hat, Inc. > 315 West Huron Street, Suite 140A > Ann Arbor, Michigan 48103 > > http://www.redhat.com/en/technologies/storage > > tel. 734-821-5101 > fax. 734-769-8938 > cel. 734-216-5309 > -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
