Hi Daniel, how would I check what server is the master and how do I set it?
cheers ----- Original Message ----- > From: "Daniel Gryniewicz" <[email protected]> > To: "ceph-users" <[email protected]> > Sent: Thursday, 10 November, 2016 15:08:55 > Subject: Re: [ceph-users] radosgw - http status 400 while creating a bucket > Your RGW doesn't think it's the master, and cannot connect to the > master, thus the create fails. > > Daniel > > On 11/08/2016 06:36 PM, Andrei Mikhailovsky wrote: >> Hello >> >> I am having issues with creating buckets in radosgw. It started with an >> upgrade to version 10.2.x >> >> When I am creating a bucket I get the following error on the client side: >> >> >> boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request >> <?xml version="1.0" >> encoding="UTF-8"?><Error><Code>InvalidArgument</Code><BucketName>my-new-bucket-31337</BucketName><RequestId>tx000000000000000000002-0058225bae-994d148-default</RequestId><HostId>994d148-default-default</HostId></Error> >> >> >> The radosgw logs are (redacted): >> >> ############################### >> >> 2016-11-08 23:11:42.876862 7f026d953700 20 enqueued request >> req=0x7f02ba07b0e0 >> 2016-11-08 23:11:42.876892 7f026d953700 20 RGWWQ: >> 2016-11-08 23:11:42.876897 7f026d953700 20 req: 0x7f02ba07b0e0 >> 2016-11-08 23:11:42.876912 7f026d953700 10 allocated request >> req=0x7f02ba07b140 >> 2016-11-08 23:11:42.876975 7f026b94f700 20 dequeued request >> req=0x7f02ba07b0e0 >> 2016-11-08 23:11:42.876987 7f026b94f700 20 RGWWQ: empty >> 2016-11-08 23:11:42.877050 7f026b94f700 20 CONTENT_LENGTH=0 >> 2016-11-08 23:11:42.877060 7f026b94f700 20 CONTEXT_DOCUMENT_ROOT=/var/www >> 2016-11-08 23:11:42.877062 7f026b94f700 20 CONTEXT_PREFIX= >> 2016-11-08 23:11:42.877063 7f026b94f700 20 DOCUMENT_ROOT=/var/www >> 2016-11-08 23:11:42.877081 7f026b94f700 20 FCGI_ROLE=RESPONDER >> 2016-11-08 23:11:42.877083 7f026b94f700 20 GATEWAY_INTERFACE=CGI/1.1 >> 2016-11-08 23:11:42.877084 7f026b94f700 20 HTTP_ACCEPT_ENCODING=identity >> 2016-11-08 23:11:42.877086 7f026b94f700 20 HTTP_AUTHORIZATION=AWS >> XXXXXXXEDITEDXXXXXX:EDITEDXXXXXXXeWyiacaN26GcME >> 2016-11-08 23:11:42.877087 7f026b94f700 20 HTTP_DATE=Tue, 08 Nov 2016 >> 23:11:37 GMT >> 2016-11-08 23:11:42.877088 7f026b94f700 20 >> HTTP_HOST=s3service.editedname.com >> 2016-11-08 23:11:42.877089 7f026b94f700 20 HTTP_USER_AGENT=Boto/2.38.0 >> Python/2.7.12 Linux/4.8.4-040804-generic >> 2016-11-08 23:11:42.877090 7f026b94f700 20 HTTPS=on >> 2016-11-08 23:11:42.877092 7f026b94f700 20 >> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin >> 2016-11-08 23:11:42.877093 7f026b94f700 20 proxy-nokeepalive=1 >> 2016-11-08 23:11:42.877094 7f026b94f700 20 QUERY_STRING= >> 2016-11-08 23:11:42.877095 7f026b94f700 20 REMOTE_ADDR=192.168.169.91 >> 2016-11-08 23:11:42.877096 7f026b94f700 20 REMOTE_PORT=45404 >> 2016-11-08 23:11:42.877097 7f026b94f700 20 REQUEST_METHOD=PUT >> 2016-11-08 23:11:42.877098 7f026b94f700 20 REQUEST_SCHEME=https >> 2016-11-08 23:11:42.877099 7f026b94f700 20 REQUEST_URI=/my-new-bucket-31337/ >> 2016-11-08 23:11:42.877104 7f026b94f700 20 >> SCRIPT_FILENAME=proxy:fcgi://localhost:9000/my-new-bucket-31337/ >> 2016-11-08 23:11:42.877105 7f026b94f700 20 SCRIPT_NAME=/my-new-bucket-31337/ >> 2016-11-08 23:11:42.877107 7f026b94f700 20 >> SCRIPT_URI=https://s3service.editedname.com/my-new-bucket-31337/ >> 2016-11-08 23:11:42.877108 7f026b94f700 20 SCRIPT_URL=/my-new-bucket-31337/ >> 2016-11-08 23:11:42.877109 7f026b94f700 20 SERVER_ADDR=192.168.169.201 >> 2016-11-08 23:11:42.877110 7f026b94f700 20 >> [email protected] >> 2016-11-08 23:11:42.877111 7f026b94f700 20 >> SERVER_NAME=s3service.editedname.com >> 2016-11-08 23:11:42.877112 7f026b94f700 20 SERVER_PORT=443 >> 2016-11-08 23:11:42.877113 7f026b94f700 20 SERVER_PROTOCOL=HTTP/1.1 >> 2016-11-08 23:11:42.877114 7f026b94f700 20 SERVER_SIGNATURE= >> 2016-11-08 23:11:42.877115 7f026b94f700 20 SERVER_SOFTWARE=Apache/2.4.18 >> (Ubuntu) >> 2016-11-08 23:11:42.877116 7f026b94f700 20 >> SSL_TLS_SNI=s3service.editedname.com >> 2016-11-08 23:11:42.877119 7f026b94f700 1 ====== starting new request >> req=0x7f02ba07b0e0 ===== >> 2016-11-08 23:11:42.877155 7f026b94f700 2 req 2:0.000035::PUT >> /my-new-bucket-31337/::initializing for trans_id = >> tx000000000000000000002-0058225bae-994d148-default >> 2016-11-08 23:11:42.877175 7f026b94f700 10 rgw api priority: s3=5 >> s3website=4 >> 2016-11-08 23:11:42.877179 7f026b94f700 10 host=s3service.editedname.com >> 2016-11-08 23:11:42.877199 7f026b94f700 20 subdomain= >> domain=s3service.editedname.com in_hosted_domain=1 >> in_hosted_domain_s3website=0 >> 2016-11-08 23:11:42.877203 7f026b94f700 20 final domain/bucket >> subdomain= domain=s3service.editedname.com in_hosted_domain=1 >> in_hosted_domain_s3website=0 s->info.domain=s3service.editedname.com >> s->info.request_uri=/my-new-bucket-31337/ >> 2016-11-08 23:11:42.877277 7f026b94f700 20 get_handler >> handler=25RGWHandler_REST_Bucket_S3 >> 2016-11-08 23:11:42.877286 7f026b94f700 10 >> handler=25RGWHandler_REST_Bucket_S3 >> 2016-11-08 23:11:42.877291 7f026b94f700 2 req 2:0.000172:s3:PUT >> /my-new-bucket-31337/::getting op 1 >> 2016-11-08 23:11:42.877326 7f026b94f700 10 op=27RGWCreateBucket_ObjStore_S3 >> 2016-11-08 23:11:42.877334 7f026b94f700 2 req 2:0.000215:s3:PUT >> /my-new-bucket-31337/:create_bucket:authorizing >> 2016-11-08 23:11:42.877386 7f026b94f700 20 get_system_obj_state: >> rctx=0x7f026b94b7c0 obj=.users:XXXXXXXEDITEDXXXXXX state=0x7f02b70cdfe8 >> s->prefetch_data=0 >> 2016-11-08 23:11:42.877403 7f026b94f700 10 cache get: >> name=.users+XXXXXXXEDITEDXXXXXX : miss >> 2016-11-08 23:11:42.877516 7f026b94f700 1 -- >> 192.168.168.201:0/3703211654 --> 192.168.168.201:6829/893 -- >> osd_op(client.160747848.0:1019 16.e7d6c03f XXXXXXXEDITEDXXXXXX >> [getxattrs,stat] snapc 0=[] ack+read+known_if_red >> irected e97148) v7 -- ?+0 0x7f02b70ab600 con 0x7f02c581a800 >> 2016-11-08 23:11:42.879265 7f02e9405700 1 -- >> 192.168.168.201:0/3703211654 <== osd.3 192.168.168.201:6829/893 52 ==== >> osd_op_reply(1019 XXXXXXXEDITEDXXXXXX [getxattrs,stat] v0'0 uv7 ondisk = >> 0) v7 ==== 182+0+20 (2521936 >> 738 0 3070622072) 0x7f02c9897280 con 0x7f02c581a800 >> 2016-11-08 23:11:42.879391 7f026b94f700 10 cache put: >> name=.users+XXXXXXXEDITEDXXXXXX info.flags=6 >> 2016-11-08 23:11:42.879421 7f026b94f700 10 adding >> .users+XXXXXXXEDITEDXXXXXX to cache LRU end >> 2016-11-08 23:11:42.879442 7f026b94f700 20 get_system_obj_state: >> s->obj_tag was set empty >> 2016-11-08 23:11:42.879452 7f026b94f700 10 cache get: >> name=.users+XXXXXXXEDITEDXXXXXX : type miss (requested=1, cached=6) >> 2016-11-08 23:11:42.879461 7f026b94f700 20 get_system_obj_state: >> rctx=0x7f026b94b7c0 obj=.users:XXXXXXXEDITEDXXXXXX state=0x7f02b70cdfe8 >> s->prefetch_data=0 >> 2016-11-08 23:11:42.879480 7f026b94f700 20 rados->read ofs=0 len=524288 >> 2016-11-08 23:11:42.879533 7f026b94f700 1 -- >> 192.168.168.201:0/3703211654 --> 192.168.168.201:6829/893 -- >> osd_op(client.160747848.0:1020 16.e7d6c03f XXXXXXXEDITEDXXXXXX [read >> 0~524288] snapc 0=[] ack+read+known_if_redi >> rected e97148) v7 -- ?+0 0x7f02b70ab980 con 0x7f02c581a800 >> 2016-11-08 23:11:42.880343 7f02e9405700 1 -- >> 192.168.168.201:0/3703211654 <== osd.3 192.168.168.201:6829/893 53 ==== >> osd_op_reply(1020 XXXXXXXEDITEDXXXXXX [read 0~37] v0'0 uv7 ondisk = 0) >> v7 ==== 140+0+37 (3791206703 0 >> 1897156770) 0x7f02c9897280 con 0x7f02c581a800 >> 2016-11-08 23:11:42.880433 7f026b94f700 20 rados->read r=0 bl.length=37 >> 2016-11-08 23:11:42.880464 7f026b94f700 10 cache put: >> name=.users+XXXXXXXEDITEDXXXXXX info.flags=1 >> 2016-11-08 23:11:42.880476 7f026b94f700 10 moving >> .users+XXXXXXXEDITEDXXXXXX to cache LRU end >> 2016-11-08 23:11:42.880509 7f026b94f700 20 get_system_obj_state: >> rctx=0x7f026b94b3f0 obj=.users.uid:EDITED - client name >> state=0x7f02b70cede8 s->prefetch_data=0 >> 2016-11-08 23:11:42.880519 7f026b94f700 10 cache get: >> name=.users.uid+EDITED - client name : miss >> 2016-11-08 23:11:42.880587 7f026b94f700 1 -- >> 192.168.168.201:0/3703211654 --> 192.168.168.203:6825/25725 -- >> osd_op(client.160747848.0:1021 15.d14b4318 EDITED - client name [call >> version.read,getxattrs,stat] >> snapc 0=[] ack+read+known_if_redirected e97148) v7 -- ?+0 >> 0x7f02b70abd00 con 0x7f02b9057180 >> 2016-11-08 23:11:42.882156 7f02bbefe700 1 -- >> 192.168.168.201:0/3703211654 <== osd.27 192.168.168.203:6825/25725 33 >> ==== osd_op_reply(1021 EDITED - client name [call,getxattrs,stat] v0'0 >> uv14016 ondisk = 0) >> v7 ==== 237+0+139 (1797997075 0 3786209825) 0x7f02bc847600 con >> 0x7f02b9057180 >> 2016-11-08 23:11:42.882322 7f026b94f700 10 cache put: >> name=.users.uid+EDITED - client name info.flags=22 >> 2016-11-08 23:11:42.882353 7f026b94f700 10 adding .users.uid+EDITED - >> client name to cache LRU end >> 2016-11-08 23:11:42.882363 7f026b94f700 20 get_system_obj_state: >> s->obj_tag was set empty >> 2016-11-08 23:11:42.882374 7f026b94f700 10 cache get: >> name=.users.uid+EDITED - client name : type miss (requested=17, cached=22) >> 2016-11-08 23:11:42.882383 7f026b94f700 20 get_system_obj_state: >> rctx=0x7f026b94b3f0 obj=.users.uid:EDITED - client name >> state=0x7f02b70cede8 s->prefetch_data=0 >> 2016-11-08 23:11:42.882427 7f026b94f700 20 rados->read ofs=0 len=524288 >> 2016-11-08 23:11:42.882492 7f026b94f700 1 -- >> 192.168.168.201:0/3703211654 --> 192.168.168.203:6825/25725 -- >> osd_op(client.160747848.0:1022 15.d14b4318 EDITED - client name [call >> version.check_conds,call ver >> sion.read,read 0~524288] snapc 0=[] ack+read+known_if_redirected e97148) >> v7 -- ?+0 0x7f02b70ac400 con 0x7f02b9057180 >> 2016-11-08 23:11:42.883550 7f02bbefe700 1 -- >> 192.168.168.201:0/3703211654 <== osd.27 192.168.168.203:6825/25725 34 >> ==== osd_op_reply(1022 EDITED - client name [call,call,read 0~401] v0'0 >> uv14016 ondisk = 0) >> v7 ==== 237+0+449 (131137144 0 372490823) 0x7f02bc847980 con 0x7f02b9057180 >> 2016-11-08 23:11:42.883653 7f026b94f700 20 rados->read r=0 bl.length=401 >> 2016-11-08 23:11:42.883695 7f026b94f700 10 cache put: >> name=.users.uid+EDITED - client name info.flags=17 >> 2016-11-08 23:11:42.883698 7f026b94f700 10 moving .users.uid+EDITED - >> client name to cache LRU end >> 2016-11-08 23:11:42.883726 7f026b94f700 10 chain_cache_entry: >> cache_locator=.users.uid+EDITED - client name >> 2016-11-08 23:11:42.883777 7f026b94f700 10 get_canon_resource(): >> dest=/my-new-bucket-31337/ >> 2016-11-08 23:11:42.883782 7f026b94f700 10 auth_hdr: >> PUT >> >> Tue, 08 Nov 2016 23:11:37 GMT >> /my-new-bucket-31337/ >> 2016-11-08 23:11:42.883849 7f026b94f700 15 calculated >> digest=EDITEDXXXXXXXeWyiacaN26GcME >> 2016-11-08 23:11:42.883852 7f026b94f700 15 >> auth_sign=EDITEDXXXXXXXeWyiacaN26GcME >> 2016-11-08 23:11:42.883853 7f026b94f700 15 compare=0 >> 2016-11-08 23:11:42.883858 7f026b94f700 2 req 2:0.006739:s3:PUT >> /my-new-bucket-31337/:create_bucket:normalizing buckets and tenants >> 2016-11-08 23:11:42.883864 7f026b94f700 10 s->object=<NULL> >> s->bucket=my-new-bucket-31337 >> 2016-11-08 23:11:42.883869 7f026b94f700 2 req 2:0.006750:s3:PUT >> /my-new-bucket-31337/:create_bucket:init permissions >> 2016-11-08 23:11:42.883872 7f026b94f700 2 req 2:0.006753:s3:PUT >> /my-new-bucket-31337/:create_bucket:recalculating target >> 2016-11-08 23:11:42.883875 7f026b94f700 2 req 2:0.006756:s3:PUT >> /my-new-bucket-31337/:create_bucket:reading permissions >> 2016-11-08 23:11:42.883879 7f026b94f700 2 req 2:0.006760:s3:PUT >> /my-new-bucket-31337/:create_bucket:init op >> 2016-11-08 23:11:42.883884 7f026b94f700 2 req 2:0.006765:s3:PUT >> /my-new-bucket-31337/:create_bucket:verifying op mask >> 2016-11-08 23:11:42.883886 7f026b94f700 20 required_mask= 2 user.op_mask=7 >> 2016-11-08 23:11:42.883887 7f026b94f700 2 req 2:0.006769:s3:PUT >> /my-new-bucket-31337/:create_bucket:verifying op permissions >> 2016-11-08 23:11:42.883939 7f026b94f700 1 -- >> 192.168.168.201:0/3703211654 --> 192.168.168.203:6840/36134 -- >> osd_op(client.160747848.0:1023 15.efcbc969 EDITED - client name.buckets >> [call user.list_buckets] snapc 0=[] ack+read+known_if_redirected e97148) >> v7 -- ?+0 0x7f02b70acb00 con 0x7f02cd43a900 >> 2016-11-08 23:11:42.885157 7f02bbfff700 1 -- >> 192.168.168.201:0/3703211654 <== osd.25 192.168.168.203:6840/36134 37 >> ==== osd_op_reply(1023 EDITED - client name.buckets [call] v0'0 uv0 ack >> = -2 ((2) No such file or directory)) v7 ==== 161+0+0 (2433927993 0 0) >> 0x7f02b886d280 con 0x7f02cd43a900 >> 2016-11-08 23:11:42.885307 7f026b94f700 2 req 2:0.008187:s3:PUT >> /my-new-bucket-31337/:create_bucket:verifying op params >> 2016-11-08 23:11:42.885338 7f026b94f700 2 req 2:0.008219:s3:PUT >> /my-new-bucket-31337/:create_bucket:pre-executing >> 2016-11-08 23:11:42.885342 7f026b94f700 2 req 2:0.008223:s3:PUT >> /my-new-bucket-31337/:create_bucket:executing >> 2016-11-08 23:11:42.885377 7f026b94f700 20 get_system_obj_state: >> rctx=0x7f026b94c6d0 obj=.rgw:my-new-bucket-31337 state=0x7f02b70cdfe8 >> s->prefetch_data=0 >> 2016-11-08 23:11:42.885390 7f026b94f700 10 cache get: >> name=.rgw+my-new-bucket-31337 : miss >> 2016-11-08 23:11:42.885444 7f026b94f700 1 -- >> 192.168.168.201:0/3703211654 --> 192.168.168.202:6821/14483 -- >> osd_op(client.160747848.0:1024 13.501be1c3 my-new-bucket-31337 [call >> version.read,getxattrs,stat] snapc 0=[] ack+read+known_if_redirected >> e97148) v7 -- ?+0 0x7f02b70ace80 con 0x7f02cd43b980 >> 2016-11-08 23:11:42.886940 7f02bfefe700 1 -- >> 192.168.168.201:0/3703211654 <== osd.11 192.168.168.202:6821/14483 11 >> ==== osd_op_reply(1024 my-new-bucket-31337 [call,getxattrs,stat] v0'0 >> uv0 ack = -2 ((2) No such file or directory)) v7 ==== 223+0+0 >> (4048274385 0 0) 0x7f02bf480280 con 0x7f02cd43b980 >> 2016-11-08 23:11:42.887120 7f026b94f700 10 cache put: >> name=.rgw+my-new-bucket-31337 info.flags=0 >> 2016-11-08 23:11:42.887135 7f026b94f700 10 adding >> .rgw+my-new-bucket-31337 to cache LRU end >> 2016-11-08 23:11:42.887144 7f026b94f700 0 rest connection is invalid >> 2016-11-08 23:11:42.887149 7f026b94f700 2 req 2:0.010030:s3:PUT >> /my-new-bucket-31337/:create_bucket:completing >> 2016-11-08 23:11:42.887217 7f026b94f700 2 req 2:0.010098:s3:PUT >> /my-new-bucket-31337/:create_bucket:op status=-22 >> 2016-11-08 23:11:42.887221 7f026b94f700 2 req 2:0.010102:s3:PUT >> /my-new-bucket-31337/:create_bucket:http status=400 >> 2016-11-08 23:11:42.887244 7f026b94f700 1 ====== req done >> req=0x7f02ba07b0e0 op status=-22 http_status=400 ====== >> 2016-11-08 23:11:42.887254 7f026b94f700 20 process_request() returned -22 >> >> ############################### >> >> >> >> >> Apache proxy logs show the following: >> >> 192.168.169.91 - - [08/Nov/2016:23:11:42 +0000] "PUT >> /my-new-bucket-31337/ HTTP/1.1" 400 4379 "-" "Boto/2.38.0 Python/2.7.12 >> Linux/4.8.4-040804-generic" >> >> >> The existing buckets work perfectly well, i can list and put objects. >> It's the creation of new buckets that i am having issues with. >> >> >> >> Could someone please help me to figure out what the issue is? >> >> Thanks >> >> Andrei >> >> >> _______________________________________________ >> ceph-users mailing list >> [email protected] >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
