I'm also not certain how to do the tcpdump for this. Do you have any pointers to how to capture that for you?
On Mon, Feb 26, 2018 at 4:09 PM David Turner <[email protected]> wrote: > That's what I set it to in the config file. I probably should have > mentioned that. > > On Mon, Feb 26, 2018 at 4:07 PM Yehuda Sadeh-Weinraub <[email protected]> > wrote: > >> According to the log here, it says that the location constraint it got >> is "cn", can you take a look at a tcpdump, see if that's actually >> what's passed in? >> >> On Mon, Feb 26, 2018 at 12:02 PM, David Turner <[email protected]> >> wrote: >> > I run with `debug rgw = 10` and was able to find these lines at the end >> of a >> > request to create the bucket. >> > >> > Successfully creating a bucket with `bucket_location = US` looks like >> > [1]this. Failing to create a bucket has "ERROR: S3 error: 400 >> > (InvalidLocationConstraint): The specified location-constraint is not >> valid" >> > on the CLI and [2]this (excerpt from the end of the request) in the rgw >> log >> > (debug level 10). "create bucket location constraint" was not found in >> the >> > log for successfully creating the bucket. >> > >> > >> > [1] >> > 2018-02-26 19:52:36.419251 7f4bc9bc8700 10 cache put: >> > >> name=local-atl.rgw.data.root++.bucket.meta.testerton:bef43c26-daf3-47ef-a3a5-e1167e3f88ac.39099765.1 >> > info.flags=0x17 >> > 2018-02-26 19:52:36.419262 7f4bc9bc8700 10 adding >> > >> local-atl.rgw.data.root++.bucket.meta.testerton:bef43c26-daf3-47ef-a3a5-e1167e3f88ac.39099765.1 >> > to cache LRU end >> > 2018-02-26 19:52:36.419266 7f4bc9bc8700 10 updating xattr: >> name=user.rgw.acl >> > bl.length()=141 >> > 2018-02-26 19:52:36.423863 7f4bc9bc8700 10 RGWWatcher::handle_notify() >> > notify_id 344855809097728 cookie 139963970426880 notifier 39099765 >> > bl.length()=361 >> > 2018-02-26 19:52:36.423875 7f4bc9bc8700 10 cache put: >> > name=local-atl.rgw.data.root++testerton info.flags=0x17 >> > 2018-02-26 19:52:36.423882 7f4bc9bc8700 10 adding >> > local-atl.rgw.data.root++testerton to cache LRU end >> > >> > [2] >> > 2018-02-26 19:43:37.340289 7f466bbca700 2 req 428078:0.004204:s3:PUT >> > /testraint/:create_bucket:executing >> > 2018-02-26 19:43:37.340366 7f466bbca700 5 NOTICE: call to >> > do_aws4_auth_completion >> > 2018-02-26 19:43:37.340472 7f466bbca700 10 v4 auth ok -- >> > do_aws4_auth_completion >> > 2018-02-26 19:43:37.340715 7f466bbca700 10 create bucket location >> > constraint: cn >> > 2018-02-26 19:43:37.340766 7f466bbca700 0 location constraint (cn) >> can't be >> > found. >> > 2018-02-26 19:43:37.340794 7f466bbca700 2 req 428078:0.004701:s3:PUT >> > /testraint/:create_bucket:completing >> > 2018-02-26 19:43:37.341782 7f466bbca700 2 req 428078:0.005689:s3:PUT >> > /testraint/:create_bucket:op status=-2208 >> > 2018-02-26 19:43:37.341792 7f466bbca700 2 req 428078:0.005707:s3:PUT >> > /testraint/:create_bucket:http status=400 >> > >> > On Mon, Feb 26, 2018 at 2:36 PM Yehuda Sadeh-Weinraub < >> [email protected]> >> > wrote: >> >> >> >> I'm not sure if the rgw logs (debug rgw = 20) specify explicitly why a >> >> bucket creation is rejected in these cases, but it might be worth >> >> trying to look at these. If not, then a tcpdump of the specific failed >> >> request might shed some light (would be interesting to look at the >> >> generated LocationConstraint). >> >> >> >> Yehuda >> >> >> >> On Mon, Feb 26, 2018 at 11:29 AM, David Turner <[email protected]> >> >> wrote: >> >> > Our problem only appeared to be present in bucket creation. Listing, >> >> > putting, etc objects in a bucket work just fine regardless of the >> >> > bucket_location setting. I ran this test on a few different realms >> to >> >> > see >> >> > what would happen and only 1 of them had a problem. There isn't an >> >> > obvious >> >> > thing that steps out about it. The 2 local realms do not have >> >> > multi-site, >> >> > the internal realm has multi-site and the operations were performed >> on >> >> > the >> >> > primary zone for the zonegroup. >> >> > >> >> > Worked with non 'US' bucket_location for s3cmd to create bucket: >> >> > realm=internal >> >> > zonegroup=internal-ga >> >> > zone=internal-atl >> >> > >> >> > Failed with non 'US' bucket_location for s3cmd to create bucket: >> >> > realm=local-atl >> >> > zonegroup=local-atl >> >> > zone=local-atl >> >> > >> >> > Worked with non 'US' bucket_location for s3cmd to create bucket: >> >> > realm=local >> >> > zonegroup=local >> >> > zone=local >> >> > >> >> > I was thinking it might have to do with all of the parts being named >> the >> >> > same, but I made sure to do the last test to confirm. Interestingly >> >> > it's >> >> > only bucket creation that has a problem and it's fine as long as I >> put >> >> > 'US' >> >> > as the bucket_location. >> >> > >> >> > On Mon, Feb 19, 2018 at 6:48 PM F21 <[email protected]> wrote: >> >> >> >> >> >> I am using the official ceph/daemon docker image. It starts RGW and >> >> >> creates a zonegroup and zone with their names set to an empty >> string: >> >> >> >> >> >> >> >> >> >> https://github.com/ceph/ceph-container/blob/master/ceph-releases/luminous/ubuntu/16.04/daemon/start_rgw.sh#L36:54 >> >> >> >> >> >> $RGW_ZONEGROUP and $RGW_ZONE are both empty strings by default: >> >> >> >> >> >> >> >> >> >> https://github.com/ceph/ceph-container/blob/master/ceph-releases/luminous/ubuntu/16.04/daemon/variables_entrypoint.sh#L46 >> >> >> >> >> >> Here's what I get when I query RGW: >> >> >> >> >> >> $ radosgw-admin zonegroup list >> >> >> { >> >> >> "default_info": "", >> >> >> "zonegroups": [ >> >> >> "default" >> >> >> ] >> >> >> } >> >> >> >> >> >> $ radosgw-admin zone list >> >> >> { >> >> >> "default_info": "", >> >> >> "zones": [ >> >> >> "default" >> >> >> ] >> >> >> } >> >> >> >> >> >> On 20/02/2018 10:33 AM, Yehuda Sadeh-Weinraub wrote: >> >> >> > What is the name of your zonegroup? >> >> >> > >> >> >> > On Mon, Feb 19, 2018 at 3:29 PM, F21 <[email protected]> >> wrote: >> >> >> >> I've done some debugging and the LocationConstraint is not being >> set >> >> >> >> by >> >> >> >> the >> >> >> >> SDK by default. >> >> >> >> >> >> >> >> I do, however, need to set the region on the client to us-east-1 >> for >> >> >> >> it >> >> >> >> to >> >> >> >> work. Anything else will return an InvalidLocationConstraint >> error. >> >> >> >> >> >> >> >> Francis >> >> >> >> >> >> >> >> >> >> >> >> On 20/02/2018 8:40 AM, Yehuda Sadeh-Weinraub wrote: >> >> >> >>> Sounds like the go sdk adds a location constraint to requests >> that >> >> >> >>> don't go to us-east-1. RGW itself is definitely isn't tied to >> >> >> >>> us-east-1, and does not know anything about it (unless you >> happen >> >> >> >>> to >> >> >> >>> have a zonegroup named us-east-1). Maybe there's a way to >> configure >> >> >> >>> the sdk to avoid doing that? >> >> >> >>> >> >> >> >>> Yehuda >> >> >> >>> >> >> >> >>> On Sun, Feb 18, 2018 at 1:54 PM, F21 <[email protected]> >> wrote: >> >> >> >>>> I am using the AWS Go SDK v2 >> >> >> >>>> (https://github.com/aws/aws-sdk-go-v2) >> >> >> >>>> to >> >> >> >>>> talk >> >> >> >>>> to my RGW instance using the s3 interface. I am running ceph in >> >> >> >>>> docker >> >> >> >>>> using >> >> >> >>>> the ceph/daemon docker images in demo mode. The RGW is started >> >> >> >>>> with a >> >> >> >>>> zonegroup and zone with their names set to an empty string by >> the >> >> >> >>>> scripts >> >> >> >>>> in >> >> >> >>>> the image. >> >> >> >>>> >> >> >> >>>> I have ForcePathStyle for the client set to true, because I >> want >> >> >> >>>> to >> >> >> >>>> access >> >> >> >>>> all my buckets using the path: myrgw.instance:8080/somebucket. >> >> >> >>>> >> >> >> >>>> I noticed that if I set the region for the client to anything >> >> >> >>>> other >> >> >> >>>> than >> >> >> >>>> us-east-1, I get this error when creating a bucket: >> >> >> >>>> InvalidLocationConstraint: The specified location-constraint is >> >> >> >>>> not >> >> >> >>>> valid. >> >> >> >>>> >> >> >> >>>> If I set the region in the client to something made up, such as >> >> >> >>>> "ceph" >> >> >> >>>> and >> >> >> >>>> the LocationConstraint to "ceph", I still get the same error. >> >> >> >>>> >> >> >> >>>> The only way to get my buckets to create successfully is to set >> >> >> >>>> the >> >> >> >>>> client's >> >> >> >>>> region to us-east-1. I have grepped the ceph code base and >> cannot >> >> >> >>>> find >> >> >> >>>> any >> >> >> >>>> references to us-east-1. In addition, I looked at the AWS docs >> for >> >> >> >>>> calculating v4 signatures and us-east-1 is the default region >> but >> >> >> >>>> I >> >> >> >>>> can >> >> >> >>>> see >> >> >> >>>> that the region string is used in the calculation (i.e. the >> region >> >> >> >>>> is >> >> >> >>>> not >> >> >> >>>> ignored when calculating the signature if it is set to >> us-east-1). >> >> >> >>>> >> >> >> >>>> Why do my buckets create successfully if I set the region in >> my s3 >> >> >> >>>> client >> >> >> >>>> to >> >> >> >>>> us-east-1, but not otherwise? If I do not want to use >> us-east-1 as >> >> >> >>>> my >> >> >> >>>> default region, for example, if I want us-west-1 as my default >> >> >> >>>> region, >> >> >> >>>> what >> >> >> >>>> should I be configuring in ceph? >> >> >> >>>> >> >> >> >>>> Thanks, >> >> >> >>>> >> >> >> >>>> Francis >> >> >> >>>> >> >> >> >>>> _______________________________________________ >> >> >> >>>> ceph-users mailing list >> >> >> >>>> [email protected] >> >> >> >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> >> >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> >> ceph-users mailing list >> >> >> [email protected] >> >> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
