***bump*** this is pretty broken and urgent. thanks
----- Original Message ----- > From: "Andrei Mikhailovsky" <[email protected]> > To: "Yoann Moulin" <[email protected]> > Cc: "ceph-users" <[email protected]> > Sent: Wednesday, 9 November, 2016 23:27:17 > Subject: Re: [ceph-users] radosgw - http status 400 while creating a bucket > Hi Yoann, > > I am running 10.2.3 on all nodes. > > Andrei > > ----- Original Message ----- >> From: "Yoann Moulin" <[email protected]> >> To: "ceph-users" <[email protected]> >> Sent: Wednesday, 9 November, 2016 21:20:45 >> Subject: Re: [ceph-users] radosgw - http status 400 while creating a bucket > >> Hello, >> >>> many thanks for your help. I've tried setting the zone to master, followed >>> by >>> the period update --commit command. This is what i've had: >> >> maybe it's related to this issue : >> >> http://tracker.ceph.com/issues/16839 (fixe in Jewel 10.2.3) >> >> or this one : >> >> http://tracker.ceph.com/issues/17239 >> >> the "id" of the zonegroup shouldn't be "default" but an uuid afaik >> >> Best regards >> >> Yoann Moulin >> >>> root@arh-ibstorage1-ib:~# radosgw-admin zonegroup get >>> --rgw-zonegroup=default >>> { >>> "id": "default", >>> "name": "default", >>> "api_name": "", >>> "is_master": "true", >>> "endpoints": [], >>> "hostnames": [], >>> "hostnames_s3website": [], >>> "master_zone": "default", >>> "zones": [ >>> { >>> "id": "default", >>> "name": "default", >>> "endpoints": [], >>> "log_meta": "false", >>> "log_data": "false", >>> "bucket_index_max_shards": 0, >>> "read_only": "false" >>> } >>> ], >>> "placement_targets": [ >>> { >>> "name": "default-placement", >>> "tags": [] >>> } >>> ], >>> "default_placement": "default-placement", >>> "realm_id": "5b41b1b2-0f92-463d-b582-07552f83e66c" >>> } >>> >>> >>> root@arh-ibstorage1-ib:~# radosgw-admin period update --commit >>> cannot commit period: period does not have a master zone of a master >>> zonegroup >>> failed to commit period: (22) Invalid argument >>> >>> >>> root@arh-ibstorage1-ib:~# radosgw-admin zonegroup get >>> --rgw-zonegroup=default >>> { >>> "id": "default", >>> "name": "default", >>> "api_name": "", >>> "is_master": "true", >>> "endpoints": [], >>> "hostnames": [], >>> "hostnames_s3website": [], >>> "master_zone": "", >>> "zones": [ >>> { >>> "id": "default", >>> "name": "default", >>> "endpoints": [], >>> "log_meta": "false", >>> "log_data": "false", >>> "bucket_index_max_shards": 0, >>> "read_only": "false" >>> } >>> ], >>> "placement_targets": [ >>> { >>> "name": "default-placement", >>> "tags": [] >>> } >>> ], >>> "default_placement": "default-placement", >>> "realm_id": "" >>> } >>> >>> >>> >>> >>> >>> The strange thing as you can see, following the "radosgw-admin period update >>> --commit" command, the master_zone and the realm_id values reset to blank. >>> What >>> could be causing this? >>> >>> Here is my ceph infrastructure setup, perhaps it will help with finding the >>> issue?: >>> >>> ceph osd and mon servers: >>> arh-ibstorage1-ib (also radosgw server) >>> arh-ibstorage2-ib (also radosgw server) >>> arh-ibstorage3-ib >>> >>> ceph mon server: >>> arh-cloud13-ib >>> >>> >>> >>> Thus, overall, i have 4 mon servers, 3 osd servers and 2 radosgw servers >>> >>> Thanks >>> >>> >>> >>> ----- Original Message ----- >>>> From: "Yehuda Sadeh-Weinraub" <[email protected]> >>>> To: "Andrei Mikhailovsky" <[email protected]> >>>> Cc: "ceph-users" <[email protected]> >>>> Sent: Wednesday, 9 November, 2016 17:12:30 >>>> Subject: Re: [ceph-users] radosgw - http status 400 while creating a bucket >>> >>>> On Wed, Nov 9, 2016 at 1:30 AM, Andrei Mikhailovsky <[email protected]> >>>> wrote: >>>>> Hi Yehuda, >>>>> >>>>> just tried to run the command to set the master_zone to default followed >>>>> by the >>>>> bucket create without doing the restart and I still have the same error >>>>> on the >>>>> client: >>>>> >>>>> <?xml version="1.0" >>>>> encoding="UTF-8"?><Error><Code>InvalidArgument</Code><BucketName>my-new-bucket-31337</BucketName><RequestId>tx000000000000000000010-005822ebbd-9951ad8-default</RequestId><HostId>9951ad8-default-default</HostId></Error> >>>>> >>>> >>>> After setting the master zone, try running: >>>> >>>> $ radosgw-admin period update --commit >>>> >>>> Yehuda >>>> >>>>> >>>>> Andrei >>>>> >>>>> ----- Original Message ----- >>>>>> From: "Yehuda Sadeh-Weinraub" <[email protected]> >>>>>> To: "Andrei Mikhailovsky" <[email protected]> >>>>>> Cc: "ceph-users" <[email protected]> >>>>>> Sent: Wednesday, 9 November, 2016 01:13:48 >>>>>> Subject: Re: [ceph-users] radosgw - http status 400 while creating a >>>>>> bucket >>>>> >>>>>> On Tue, Nov 8, 2016 at 5:05 PM, Andrei Mikhailovsky <[email protected]> >>>>>> wrote: >>>>>>> Hi Yehuda, >>>>>>> >>>>>>> I don't have a multizone setup. The radosgw service was configured >>>>>>> about two >>>>>>> years ago according to the documentation on ceph.com and haven't >>>>>>> changed with >>>>>>> numerous version updates. All was working okay until i've upgraded to >>>>>>> version >>>>>>> 10.2.x. >>>>>>> >>>>>>> Could you please point me in the right direction what exactly needs to >>>>>>> be done? >>>>>>> >>>>>>> >>>>>>> # radosgw-admin zonegroup get --rgw-zonegroup=default >>>>>>> { >>>>>>> "id": "default", >>>>>>> "name": "default", >>>>>>> "api_name": "", >>>>>>> "is_master": "true", >>>>>>> "endpoints": [], >>>>>>> "hostnames": [], >>>>>>> "hostnames_s3website": [], >>>>>>> "master_zone": "", >>>>>>> "zones": [ >>>>>>> { >>>>>>> "id": "default", >>>>>>> "name": "default", >>>>>>> "endpoints": [], >>>>>>> "log_meta": "false", >>>>>>> "log_data": "false", >>>>>>> "bucket_index_max_shards": 0, >>>>>>> "read_only": "false" >>>>>>> } >>>>>>> ], >>>>>>> "placement_targets": [ >>>>>>> { >>>>>>> "name": "default-placement", >>>>>>> "tags": [] >>>>>>> } >>>>>>> ], >>>>>>> "default_placement": "default-placement", >>>>>>> "realm_id": "" >>>>>>> } >>>>>> >>>>>> Try: >>>>>> >>>>>> $ radosgw-admin zonegroup get --rgw-zonegroup=default > zonegroup.json >>>>>> >>>>>> ... modify the master_zone to be "default" >>>>>> >>>>>> $ radosgw-admin zonegroup set --rgw-zonegroup=default < zonegroup.json >>>>>> >>>>>> (restart radosgw) >>>>>> >>>>>> Yehuda >>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> # radosgw-admin zone get --rgw-zone=default >>>>>>> { >>>>>>> "id": "default", >>>>>>> "name": "default", >>>>>>> "domain_root": ".rgw", >>>>>>> "control_pool": ".rgw.control", >>>>>>> "gc_pool": ".rgw.gc", >>>>>>> "log_pool": ".log", >>>>>>> "intent_log_pool": ".intent-log", >>>>>>> "usage_log_pool": ".usage", >>>>>>> "user_keys_pool": ".users", >>>>>>> "user_email_pool": ".users.email", >>>>>>> "user_swift_pool": ".users.swift", >>>>>>> "user_uid_pool": ".users.uid", >>>>>>> "system_key": { >>>>>>> "access_key": "", >>>>>>> "secret_key": "" >>>>>>> }, >>>>>>> "placement_pools": [ >>>>>>> { >>>>>>> "key": "default-placement", >>>>>>> "val": { >>>>>>> "index_pool": ".rgw.buckets.index", >>>>>>> "data_pool": ".rgw.buckets", >>>>>>> "data_extra_pool": "default.rgw.buckets.non-ec", >>>>>>> "index_type": 0 >>>>>>> } >>>>>>> } >>>>>>> ], >>>>>>> "metadata_heap": ".rgw.meta", >>>>>>> "realm_id": "5b41b1b2-0f92-463d-b582-07552f83e66c" >>>>>>> } >>>>>>> >>>>>>> >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> >>>>>>> >>>>>>> ----- Original Message ----- >>>>>>>> From: "Yehuda Sadeh-Weinraub" <[email protected]> >>>>>>>> To: "Andrei Mikhailovsky" <[email protected]> >>>>>>>> Cc: "ceph-users" <[email protected]> >>>>>>>> Sent: Wednesday, 9 November, 2016 00:48:50 >>>>>>>> Subject: Re: [ceph-users] radosgw - http status 400 while creating a >>>>>>>> bucket >>>>>>> >>>>>>>> On Tue, Nov 8, 2016 at 3:36 PM, Andrei Mikhailovsky >>>>>>>> <[email protected]> wrote: >>>>>>>>> Hello >>>>>>>>> >>>>>>>>> I am having issues with creating buckets in radosgw. It started with >>>>>>>>> an >>>>>>>>> upgrade to version 10.2.x >>>>>>>>> >>>>>>>>> When I am creating a bucket I get the following error on the client >>>>>>>>> side: >>>>>>>>> >>>>>>>>> >>>>>>>>> boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request >>>>>>>>> <?xml version="1.0" >>>>>>>>> encoding="UTF-8"?><Error><Code>InvalidArgument</Code><BucketName>my-new-bucket-31337</BucketName><RequestId>tx000000000000000000002-0058225bae-994d148-default</RequestId><HostId>994d148-default-default</HostId></Error> >>>>>>>>> >>>>>>>>> >>>>>>>>> The radosgw logs are (redacted): >>>>>>>>> >>>>>>>>> ############################### >>>>>>>>> >>>>>>>>> 2016-11-08 23:11:42.876862 7f026d953700 20 enqueued request >>>>>>>>> req=0x7f02ba07b0e0 >>>>>>>>> 2016-11-08 23:11:42.876892 7f026d953700 20 RGWWQ: >>>>>>>>> 2016-11-08 23:11:42.876897 7f026d953700 20 req: 0x7f02ba07b0e0 >>>>>>>>> 2016-11-08 23:11:42.876912 7f026d953700 10 allocated request >>>>>>>>> req=0x7f02ba07b140 >>>>>>>>> 2016-11-08 23:11:42.876975 7f026b94f700 20 dequeued request >>>>>>>>> req=0x7f02ba07b0e0 >>>>>>>>> 2016-11-08 23:11:42.876987 7f026b94f700 20 RGWWQ: empty >>>>>>>>> 2016-11-08 23:11:42.877050 7f026b94f700 20 CONTENT_LENGTH=0 >>>>>>>>> 2016-11-08 23:11:42.877060 7f026b94f700 20 >>>>>>>>> CONTEXT_DOCUMENT_ROOT=/var/www >>>>>>>>> 2016-11-08 23:11:42.877062 7f026b94f700 20 CONTEXT_PREFIX= >>>>>>>>> 2016-11-08 23:11:42.877063 7f026b94f700 20 DOCUMENT_ROOT=/var/www >>>>>>>>> 2016-11-08 23:11:42.877081 7f026b94f700 20 FCGI_ROLE=RESPONDER >>>>>>>>> 2016-11-08 23:11:42.877083 7f026b94f700 20 GATEWAY_INTERFACE=CGI/1.1 >>>>>>>>> 2016-11-08 23:11:42.877084 7f026b94f700 20 >>>>>>>>> HTTP_ACCEPT_ENCODING=identity >>>>>>>>> 2016-11-08 23:11:42.877086 7f026b94f700 20 HTTP_AUTHORIZATION=AWS >>>>>>>>> XXXXXXXEDITEDXXXXXX:EDITEDXXXXXXXeWyiacaN26GcME >>>>>>>>> 2016-11-08 23:11:42.877087 7f026b94f700 20 HTTP_DATE=Tue, 08 Nov 2016 >>>>>>>>> 23:11:37 GMT >>>>>>>>> 2016-11-08 23:11:42.877088 7f026b94f700 20 >>>>>>>>> HTTP_HOST=s3service.editedname.com >>>>>>>>> 2016-11-08 23:11:42.877089 7f026b94f700 20 HTTP_USER_AGENT=Boto/2.38.0 >>>>>>>>> Python/2.7.12 Linux/4.8.4-040804-generic >>>>>>>>> 2016-11-08 23:11:42.877090 7f026b94f700 20 HTTPS=on >>>>>>>>> 2016-11-08 23:11:42.877092 7f026b94f700 20 >>>>>>>>> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin >>>>>>>>> 2016-11-08 23:11:42.877093 7f026b94f700 20 proxy-nokeepalive=1 >>>>>>>>> 2016-11-08 23:11:42.877094 7f026b94f700 20 QUERY_STRING= >>>>>>>>> 2016-11-08 23:11:42.877095 7f026b94f700 20 REMOTE_ADDR=192.168.169.91 >>>>>>>>> 2016-11-08 23:11:42.877096 7f026b94f700 20 REMOTE_PORT=45404 >>>>>>>>> 2016-11-08 23:11:42.877097 7f026b94f700 20 REQUEST_METHOD=PUT >>>>>>>>> 2016-11-08 23:11:42.877098 7f026b94f700 20 REQUEST_SCHEME=https >>>>>>>>> 2016-11-08 23:11:42.877099 7f026b94f700 20 >>>>>>>>> REQUEST_URI=/my-new-bucket-31337/ >>>>>>>>> 2016-11-08 23:11:42.877104 7f026b94f700 20 >>>>>>>>> SCRIPT_FILENAME=proxy:fcgi://localhost:9000/my-new-bucket-31337/ >>>>>>>>> 2016-11-08 23:11:42.877105 7f026b94f700 20 >>>>>>>>> SCRIPT_NAME=/my-new-bucket-31337/ >>>>>>>>> 2016-11-08 23:11:42.877107 7f026b94f700 20 >>>>>>>>> SCRIPT_URI=https://s3service.editedname.com/my-new-bucket-31337/ >>>>>>>>> 2016-11-08 23:11:42.877108 7f026b94f700 20 >>>>>>>>> SCRIPT_URL=/my-new-bucket-31337/ >>>>>>>>> 2016-11-08 23:11:42.877109 7f026b94f700 20 SERVER_ADDR=192.168.169.201 >>>>>>>>> 2016-11-08 23:11:42.877110 7f026b94f700 20 >>>>>>>>> [email protected] >>>>>>>>> 2016-11-08 23:11:42.877111 7f026b94f700 20 >>>>>>>>> SERVER_NAME=s3service.editedname.com >>>>>>>>> 2016-11-08 23:11:42.877112 7f026b94f700 20 SERVER_PORT=443 >>>>>>>>> 2016-11-08 23:11:42.877113 7f026b94f700 20 SERVER_PROTOCOL=HTTP/1.1 >>>>>>>>> 2016-11-08 23:11:42.877114 7f026b94f700 20 SERVER_SIGNATURE= >>>>>>>>> 2016-11-08 23:11:42.877115 7f026b94f700 20 >>>>>>>>> SERVER_SOFTWARE=Apache/2.4.18 >>>>>>>>> (Ubuntu) >>>>>>>>> 2016-11-08 23:11:42.877116 7f026b94f700 20 >>>>>>>>> SSL_TLS_SNI=s3service.editedname.com >>>>>>>>> 2016-11-08 23:11:42.877119 7f026b94f700 1 ====== starting new request >>>>>>>>> req=0x7f02ba07b0e0 ===== >>>>>>>>> 2016-11-08 23:11:42.877155 7f026b94f700 2 req 2:0.000035::PUT >>>>>>>>> /my-new-bucket-31337/::initializing for trans_id = >>>>>>>>> tx000000000000000000002-0058225bae-994d148-default >>>>>>>>> 2016-11-08 23:11:42.877175 7f026b94f700 10 rgw api priority: s3=5 >>>>>>>>> s3website=4 >>>>>>>>> 2016-11-08 23:11:42.877179 7f026b94f700 10 >>>>>>>>> host=s3service.editedname.com >>>>>>>>> 2016-11-08 23:11:42.877199 7f026b94f700 20 subdomain= >>>>>>>>> domain=s3service.editedname.com in_hosted_domain=1 >>>>>>>>> in_hosted_domain_s3website=0 >>>>>>>>> 2016-11-08 23:11:42.877203 7f026b94f700 20 final domain/bucket >>>>>>>>> subdomain= >>>>>>>>> domain=s3service.editedname.com in_hosted_domain=1 >>>>>>>>> in_hosted_domain_s3website=0 s->info.domain=s3service.editedname.com >>>>>>>>> s->info.request_uri=/my-new-bucket-31337/ >>>>>>>>> 2016-11-08 23:11:42.877277 7f026b94f700 20 get_handler >>>>>>>>> handler=25RGWHandler_REST_Bucket_S3 >>>>>>>>> 2016-11-08 23:11:42.877286 7f026b94f700 10 >>>>>>>>> handler=25RGWHandler_REST_Bucket_S3 >>>>>>>>> 2016-11-08 23:11:42.877291 7f026b94f700 2 req 2:0.000172:s3:PUT >>>>>>>>> /my-new-bucket-31337/::getting op 1 >>>>>>>>> 2016-11-08 23:11:42.877326 7f026b94f700 10 >>>>>>>>> op=27RGWCreateBucket_ObjStore_S3 >>>>>>>>> 2016-11-08 23:11:42.877334 7f026b94f700 2 req 2:0.000215:s3:PUT >>>>>>>>> /my-new-bucket-31337/:create_bucket:authorizing >>>>>>>>> 2016-11-08 23:11:42.877386 7f026b94f700 20 get_system_obj_state: >>>>>>>>> rctx=0x7f026b94b7c0 obj=.users:XXXXXXXEDITEDXXXXXX >>>>>>>>> state=0x7f02b70cdfe8 >>>>>>>>> s->prefetch_data=0 >>>>>>>>> 2016-11-08 23:11:42.877403 7f026b94f700 10 cache get: >>>>>>>>> name=.users+XXXXXXXEDITEDXXXXXX : miss >>>>>>>>> 2016-11-08 23:11:42.877516 7f026b94f700 1 -- >>>>>>>>> 192.168.168.201:0/3703211654 >>>>>>>>> --> 192.168.168.201:6829/893 -- osd_op(client.160747848.0:1019 >>>>>>>>> 16.e7d6c03f >>>>>>>>> XXXXXXXEDITEDXXXXXX [getxattrs,stat] snapc 0=[] ack+read+known_if_red >>>>>>>>> irected e97148) v7 -- ?+0 0x7f02b70ab600 con 0x7f02c581a800 >>>>>>>>> 2016-11-08 23:11:42.879265 7f02e9405700 1 -- >>>>>>>>> 192.168.168.201:0/3703211654 >>>>>>>>> <== osd.3 192.168.168.201:6829/893 52 ==== osd_op_reply(1019 >>>>>>>>> XXXXXXXEDITEDXXXXXX [getxattrs,stat] v0'0 uv7 ondisk = 0) v7 ==== >>>>>>>>> 182+0+20 >>>>>>>>> (2521936 >>>>>>>>> 738 0 3070622072) 0x7f02c9897280 con 0x7f02c581a800 >>>>>>>>> 2016-11-08 23:11:42.879391 7f026b94f700 10 cache put: >>>>>>>>> name=.users+XXXXXXXEDITEDXXXXXX info.flags=6 >>>>>>>>> 2016-11-08 23:11:42.879421 7f026b94f700 10 adding >>>>>>>>> .users+XXXXXXXEDITEDXXXXXX >>>>>>>>> to cache LRU end >>>>>>>>> 2016-11-08 23:11:42.879442 7f026b94f700 20 get_system_obj_state: >>>>>>>>> s->obj_tag >>>>>>>>> was set empty >>>>>>>>> 2016-11-08 23:11:42.879452 7f026b94f700 10 cache get: >>>>>>>>> name=.users+XXXXXXXEDITEDXXXXXX : type miss (requested=1, cached=6) >>>>>>>>> 2016-11-08 23:11:42.879461 7f026b94f700 20 get_system_obj_state: >>>>>>>>> rctx=0x7f026b94b7c0 obj=.users:XXXXXXXEDITEDXXXXXX >>>>>>>>> state=0x7f02b70cdfe8 >>>>>>>>> s->prefetch_data=0 >>>>>>>>> 2016-11-08 23:11:42.879480 7f026b94f700 20 rados->read ofs=0 >>>>>>>>> len=524288 >>>>>>>>> 2016-11-08 23:11:42.879533 7f026b94f700 1 -- >>>>>>>>> 192.168.168.201:0/3703211654 >>>>>>>>> --> 192.168.168.201:6829/893 -- osd_op(client.160747848.0:1020 >>>>>>>>> 16.e7d6c03f >>>>>>>>> XXXXXXXEDITEDXXXXXX [read 0~524288] snapc 0=[] ack+read+known_if_redi >>>>>>>>> rected e97148) v7 -- ?+0 0x7f02b70ab980 con 0x7f02c581a800 >>>>>>>>> 2016-11-08 23:11:42.880343 7f02e9405700 1 -- >>>>>>>>> 192.168.168.201:0/3703211654 >>>>>>>>> <== osd.3 192.168.168.201:6829/893 53 ==== osd_op_reply(1020 >>>>>>>>> XXXXXXXEDITEDXXXXXX [read 0~37] v0'0 uv7 ondisk = 0) v7 ==== 140+0+37 >>>>>>>>> (3791206703 0 >>>>>>>>> 1897156770) 0x7f02c9897280 con 0x7f02c581a800 >>>>>>>>> 2016-11-08 23:11:42.880433 7f026b94f700 20 rados->read r=0 >>>>>>>>> bl.length=37 >>>>>>>>> 2016-11-08 23:11:42.880464 7f026b94f700 10 cache put: >>>>>>>>> name=.users+XXXXXXXEDITEDXXXXXX info.flags=1 >>>>>>>>> 2016-11-08 23:11:42.880476 7f026b94f700 10 moving >>>>>>>>> .users+XXXXXXXEDITEDXXXXXX >>>>>>>>> to cache LRU end >>>>>>>>> 2016-11-08 23:11:42.880509 7f026b94f700 20 get_system_obj_state: >>>>>>>>> rctx=0x7f026b94b3f0 obj=.users.uid:EDITED - client name >>>>>>>>> state=0x7f02b70cede8 >>>>>>>>> s->prefetch_data=0 >>>>>>>>> 2016-11-08 23:11:42.880519 7f026b94f700 10 cache get: >>>>>>>>> name=.users.uid+EDITED >>>>>>>>> - client name : miss >>>>>>>>> 2016-11-08 23:11:42.880587 7f026b94f700 1 -- >>>>>>>>> 192.168.168.201:0/3703211654 >>>>>>>>> --> 192.168.168.203:6825/25725 -- osd_op(client.160747848.0:1021 >>>>>>>>> 15.d14b4318 >>>>>>>>> EDITED - client name [call version.read,getxattrs,stat] >>>>>>>>> snapc 0=[] ack+read+known_if_redirected e97148) v7 -- ?+0 >>>>>>>>> 0x7f02b70abd00 >>>>>>>>> con 0x7f02b9057180 >>>>>>>>> 2016-11-08 23:11:42.882156 7f02bbefe700 1 -- >>>>>>>>> 192.168.168.201:0/3703211654 >>>>>>>>> <== osd.27 192.168.168.203:6825/25725 33 ==== osd_op_reply(1021 >>>>>>>>> EDITED - >>>>>>>>> client name [call,getxattrs,stat] v0'0 uv14016 ondisk = 0) >>>>>>>>> v7 ==== 237+0+139 (1797997075 0 3786209825) 0x7f02bc847600 con >>>>>>>>> 0x7f02b9057180 >>>>>>>>> 2016-11-08 23:11:42.882322 7f026b94f700 10 cache put: >>>>>>>>> name=.users.uid+EDITED >>>>>>>>> - client name info.flags=22 >>>>>>>>> 2016-11-08 23:11:42.882353 7f026b94f700 10 adding .users.uid+EDITED - >>>>>>>>> client >>>>>>>>> name to cache LRU end >>>>>>>>> 2016-11-08 23:11:42.882363 7f026b94f700 20 get_system_obj_state: >>>>>>>>> s->obj_tag >>>>>>>>> was set empty >>>>>>>>> 2016-11-08 23:11:42.882374 7f026b94f700 10 cache get: >>>>>>>>> name=.users.uid+EDITED >>>>>>>>> - client name : type miss (requested=17, cached=22) >>>>>>>>> 2016-11-08 23:11:42.882383 7f026b94f700 20 get_system_obj_state: >>>>>>>>> rctx=0x7f026b94b3f0 obj=.users.uid:EDITED - client name >>>>>>>>> state=0x7f02b70cede8 >>>>>>>>> s->prefetch_data=0 >>>>>>>>> 2016-11-08 23:11:42.882427 7f026b94f700 20 rados->read ofs=0 >>>>>>>>> len=524288 >>>>>>>>> 2016-11-08 23:11:42.882492 7f026b94f700 1 -- >>>>>>>>> 192.168.168.201:0/3703211654 >>>>>>>>> --> 192.168.168.203:6825/25725 -- osd_op(client.160747848.0:1022 >>>>>>>>> 15.d14b4318 >>>>>>>>> EDITED - client name [call version.check_conds,call ver >>>>>>>>> sion.read,read 0~524288] snapc 0=[] ack+read+known_if_redirected >>>>>>>>> e97148) v7 >>>>>>>>> -- ?+0 0x7f02b70ac400 con 0x7f02b9057180 >>>>>>>>> 2016-11-08 23:11:42.883550 7f02bbefe700 1 -- >>>>>>>>> 192.168.168.201:0/3703211654 >>>>>>>>> <== osd.27 192.168.168.203:6825/25725 34 ==== osd_op_reply(1022 >>>>>>>>> EDITED - >>>>>>>>> client name [call,call,read 0~401] v0'0 uv14016 ondisk = 0) >>>>>>>>> v7 ==== 237+0+449 (131137144 0 372490823) 0x7f02bc847980 con >>>>>>>>> 0x7f02b9057180 >>>>>>>>> 2016-11-08 23:11:42.883653 7f026b94f700 20 rados->read r=0 >>>>>>>>> bl.length=401 >>>>>>>>> 2016-11-08 23:11:42.883695 7f026b94f700 10 cache put: >>>>>>>>> name=.users.uid+EDITED >>>>>>>>> - client name info.flags=17 >>>>>>>>> 2016-11-08 23:11:42.883698 7f026b94f700 10 moving .users.uid+EDITED - >>>>>>>>> client >>>>>>>>> name to cache LRU end >>>>>>>>> 2016-11-08 23:11:42.883726 7f026b94f700 10 chain_cache_entry: >>>>>>>>> cache_locator=.users.uid+EDITED - client name >>>>>>>>> 2016-11-08 23:11:42.883777 7f026b94f700 10 get_canon_resource(): >>>>>>>>> dest=/my-new-bucket-31337/ >>>>>>>>> 2016-11-08 23:11:42.883782 7f026b94f700 10 auth_hdr: >>>>>>>>> PUT >>>>>>>>> >>>>>>>>> Tue, 08 Nov 2016 23:11:37 GMT >>>>>>>>> /my-new-bucket-31337/ >>>>>>>>> 2016-11-08 23:11:42.883849 7f026b94f700 15 calculated >>>>>>>>> digest=EDITEDXXXXXXXeWyiacaN26GcME >>>>>>>>> 2016-11-08 23:11:42.883852 7f026b94f700 15 >>>>>>>>> auth_sign=EDITEDXXXXXXXeWyiacaN26GcME >>>>>>>>> 2016-11-08 23:11:42.883853 7f026b94f700 15 compare=0 >>>>>>>>> 2016-11-08 23:11:42.883858 7f026b94f700 2 req 2:0.006739:s3:PUT >>>>>>>>> /my-new-bucket-31337/:create_bucket:normalizing buckets and tenants >>>>>>>>> 2016-11-08 23:11:42.883864 7f026b94f700 10 s->object=<NULL> >>>>>>>>> s->bucket=my-new-bucket-31337 >>>>>>>>> 2016-11-08 23:11:42.883869 7f026b94f700 2 req 2:0.006750:s3:PUT >>>>>>>>> /my-new-bucket-31337/:create_bucket:init permissions >>>>>>>>> 2016-11-08 23:11:42.883872 7f026b94f700 2 req 2:0.006753:s3:PUT >>>>>>>>> /my-new-bucket-31337/:create_bucket:recalculating target >>>>>>>>> 2016-11-08 23:11:42.883875 7f026b94f700 2 req 2:0.006756:s3:PUT >>>>>>>>> /my-new-bucket-31337/:create_bucket:reading permissions >>>>>>>>> 2016-11-08 23:11:42.883879 7f026b94f700 2 req 2:0.006760:s3:PUT >>>>>>>>> /my-new-bucket-31337/:create_bucket:init op >>>>>>>>> 2016-11-08 23:11:42.883884 7f026b94f700 2 req 2:0.006765:s3:PUT >>>>>>>>> /my-new-bucket-31337/:create_bucket:verifying op mask >>>>>>>>> 2016-11-08 23:11:42.883886 7f026b94f700 20 required_mask= 2 >>>>>>>>> user.op_mask=7 >>>>>>>>> 2016-11-08 23:11:42.883887 7f026b94f700 2 req 2:0.006769:s3:PUT >>>>>>>>> /my-new-bucket-31337/:create_bucket:verifying op permissions >>>>>>>>> 2016-11-08 23:11:42.883939 7f026b94f700 1 -- >>>>>>>>> 192.168.168.201:0/3703211654 >>>>>>>>> --> 192.168.168.203:6840/36134 -- osd_op(client.160747848.0:1023 >>>>>>>>> 15.efcbc969 >>>>>>>>> EDITED - client name.buckets [call user.list_buckets] snapc 0=[] >>>>>>>>> ack+read+known_if_redirected e97148) v7 -- ?+0 0x7f02b70acb00 con >>>>>>>>> 0x7f02cd43a900 >>>>>>>>> 2016-11-08 23:11:42.885157 7f02bbfff700 1 -- >>>>>>>>> 192.168.168.201:0/3703211654 >>>>>>>>> <== osd.25 192.168.168.203:6840/36134 37 ==== osd_op_reply(1023 >>>>>>>>> EDITED - >>>>>>>>> client name.buckets [call] v0'0 uv0 ack = -2 ((2) No such file or >>>>>>>>> directory)) v7 ==== 161+0+0 (2433927993 0 0) 0x7f02b886d280 con >>>>>>>>> 0x7f02cd43a900 >>>>>>>>> 2016-11-08 23:11:42.885307 7f026b94f700 2 req 2:0.008187:s3:PUT >>>>>>>>> /my-new-bucket-31337/:create_bucket:verifying op params >>>>>>>>> 2016-11-08 23:11:42.885338 7f026b94f700 2 req 2:0.008219:s3:PUT >>>>>>>>> /my-new-bucket-31337/:create_bucket:pre-executing >>>>>>>>> 2016-11-08 23:11:42.885342 7f026b94f700 2 req 2:0.008223:s3:PUT >>>>>>>>> /my-new-bucket-31337/:create_bucket:executing >>>>>>>>> 2016-11-08 23:11:42.885377 7f026b94f700 20 get_system_obj_state: >>>>>>>>> rctx=0x7f026b94c6d0 obj=.rgw:my-new-bucket-31337 state=0x7f02b70cdfe8 >>>>>>>>> s->prefetch_data=0 >>>>>>>>> 2016-11-08 23:11:42.885390 7f026b94f700 10 cache get: >>>>>>>>> name=.rgw+my-new-bucket-31337 : miss >>>>>>>>> 2016-11-08 23:11:42.885444 7f026b94f700 1 -- >>>>>>>>> 192.168.168.201:0/3703211654 >>>>>>>>> --> 192.168.168.202:6821/14483 -- osd_op(client.160747848.0:1024 >>>>>>>>> 13.501be1c3 >>>>>>>>> my-new-bucket-31337 [call version.read,getxattrs,stat] snapc 0=[] >>>>>>>>> ack+read+known_if_redirected e97148) v7 -- ?+0 0x7f02b70ace80 con >>>>>>>>> 0x7f02cd43b980 >>>>>>>>> 2016-11-08 23:11:42.886940 7f02bfefe700 1 -- >>>>>>>>> 192.168.168.201:0/3703211654 >>>>>>>>> <== osd.11 192.168.168.202:6821/14483 11 ==== osd_op_reply(1024 >>>>>>>>> my-new-bucket-31337 [call,getxattrs,stat] v0'0 uv0 ack = -2 ((2) No >>>>>>>>> such >>>>>>>>> file or directory)) v7 ==== 223+0+0 (4048274385 0 0) 0x7f02bf480280 >>>>>>>>> con >>>>>>>>> 0x7f02cd43b980 >>>>>>>>> 2016-11-08 23:11:42.887120 7f026b94f700 10 cache put: >>>>>>>>> name=.rgw+my-new-bucket-31337 info.flags=0 >>>>>>>>> 2016-11-08 23:11:42.887135 7f026b94f700 10 adding >>>>>>>>> .rgw+my-new-bucket-31337 >>>>>>>>> to cache LRU end >>>>>>>>> 2016-11-08 23:11:42.887144 7f026b94f700 0 rest connection is invalid >>>>>>>> >>>>>>>> Did you configure multi-site? Seems to me that it tries to connect to >>>>>>>> a master zone, so if you have a single zonegroup, single zone >>>>>>>> configuration you need to make sure that the zone (and zonegroup) are >>>>>>>> set to be the master. >>>>>>>> >>>>>>>> Yehuda >>>>>>>> >>>>>>>>> 2016-11-08 23:11:42.887149 7f026b94f700 2 req 2:0.010030:s3:PUT >>>>>>>>> /my-new-bucket-31337/:create_bucket:completing >>>>>>>>> 2016-11-08 23:11:42.887217 7f026b94f700 2 req 2:0.010098:s3:PUT >>>>>>>>> /my-new-bucket-31337/:create_bucket:op status=-22 >>>>>>>>> 2016-11-08 23:11:42.887221 7f026b94f700 2 req 2:0.010102:s3:PUT >>>>>>>>> /my-new-bucket-31337/:create_bucket:http status=400 >>>>>>>>> 2016-11-08 23:11:42.887244 7f026b94f700 1 ====== req done >>>>>>>>> req=0x7f02ba07b0e0 op status=-22 http_status=400 ====== >>>>>>>>> 2016-11-08 23:11:42.887254 7f026b94f700 20 process_request() returned >>>>>>>>> -22 >>>>>>>>> >>>>>>>>> ############################### >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Apache proxy logs show the following: >>>>>>>>> >>>>>>>>> 192.168.169.91 - - [08/Nov/2016:23:11:42 +0000] "PUT >>>>>>>>> /my-new-bucket-31337/ >>>>>>>>> HTTP/1.1" 400 4379 "-" "Boto/2.38.0 Python/2.7.12 >>>>>>>>> Linux/4.8.4-040804-generic" >>>>>>>>> >>>>>>>>> >>>>>>>>> The existing buckets work perfectly well, i can list and put objects. >>>>>>>>> It's >>>>>>>>> the creation of new buckets that i am having issues with. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Could someone please help me to figure out what the issue is? >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> Andrei >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> ceph-users mailing list >>>>>>>>> [email protected] >>>>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> _______________________________________________ >>> ceph-users mailing list >>> [email protected] >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >> >> >> -- >> Yoann Moulin >> EPFL IC-IT >> _______________________________________________ >> ceph-users mailing list >> [email protected] >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
