Re: [ceph-users] CORS not working
hello! yes, dns name is configured and working perfectly, the bucket (in this example static3) is found actually, but RGW can't read CORS configuration due some reason. 2013-09-04 08:07:46.082740 7ff4bf7ee700 2 req 10:0.000275:s3:OPTIONS /::getting op 2013-09-04 08:07:46.082745 7ff4bf7ee700 2 req 10:0.000280:s3:OPTIONS /:options_cors:authorizing 2013-09-04 08:07:46.082753 7ff4bf7ee700 2 req 10:0.000287:s3:OPTIONS /:options_cors:reading permissions 2013-09-04 08:07:46.082790 7ff4bf7ee700 20 get_obj_state: rctx=0x7ff4f8003400 obj=.rgw:static3 state=0x7ff4f8005968 s->prefetch_data=0 2013-09-04 08:07:46.082810 7ff4bf7ee700 10 moving .rgw+static3 to cache LRU end 2013-09-04 08:07:46.082819 7ff4bf7ee700 10 cache get: name=.rgw+static3 : hit 2013-09-04 08:07:46.082840 7ff4bf7ee700 20 get_obj_state: s->obj_tag was set empty 2013-09-04 08:07:46.082845 7ff4bf7ee700 20 Read xattr: user.rgw.acl 2013-09-04 08:07:46.082847 7ff4bf7ee700 20 Read xattr: user.rgw.cors 2013-09-04 08:07:46.082848 7ff4bf7ee700 20 Read xattr: user.rgw.idtag 2013-09-04 08:07:46.082849 7ff4bf7ee700 20 Read xattr: user.rgw.manifest 2013-09-04 08:07:46.082855 7ff4bf7ee700 10 moving .rgw+static3 to cache LRU end 2013-09-04 08:07:46.082857 7ff4bf7ee700 10 cache get: name=.rgw+static3 : hit 2013-09-04 08:07:46.082898 7ff4bf7ee700 20 rgw_get_bucket_info: old bucket info, bucket=static3(@.rgw.buckets2[99137.2]) owner pejotes 2013-09-04 08:07:46.082921 7ff4bf7ee700 15 Read AccessControlPolicyhttp://s3.amazonaws.com/doc/2006-03-01/";>pejotesofehttp://www.w3.org/2001/XMLSchema-instance"; xsi:type="Group"> http://acs.amazonaws.com/groups/global/AllUsersFULL_CONTROLhttp://www.w3.org/2001/XMLSchema-instance"; xsi:type="CanonicalUser">pejotesofeFULL_CONTROL 2013-09-04 08:07:46.082943 7ff4bf7ee700 15 Read AccessControlPolicyhttp://s3.amazonaws.com/doc/2006-03-01/";>pejotesofehttp://www.w3.org/2001/XMLSchema-instance"; xsi:type="Group"> http://acs.amazonaws.com/groups/global/AllUsersFULL_CONTROLhttp://www.w3.org/2001/XMLSchema-instance"; xsi:type="CanonicalUser">pejotesofeFULL_CONTROL 2013-09-04 08:07:46.082951 7ff4bf7ee700 2 req 10:0.000486:s3:OPTIONS /:options_cors:verifying op mask 2013-09-04 08:07:46.082955 7ff4bf7ee700 20 required_mask= 1 user.op_mask=7 2013-09-04 08:07:46.082957 7ff4bf7ee700 2 req 10:0.000492:s3:OPTIONS /:options_cors:verifying op permissions 2013-09-04 08:07:46.082960 7ff4bf7ee700 2 req 10:0.000495:s3:OPTIONS /:options_cors:verifying op params 2013-09-04 08:07:46.082963 7ff4bf7ee700 2 req 10:0.000498:s3:OPTIONS /:options_cors:executing 2013-09-04 08:07:46.082966 7ff4bf7ee700 2 No CORS configuration set yet for this bucket 2013-09-04 08:07:46.083105 7ff4bf7ee700 2 req 10:0.000640:s3:OPTIONS /:options_cors:http status=403 2013-09-04 08:07:46.083548 7ff4bf7ee700 1 == req done req=0xbcd910 http_status=403 == best regards! -- pawel On Tue, Sep 3, 2013 at 5:17 PM, Yehuda Sadeh wrote: > On Tue, Sep 3, 2013 at 3:40 AM, Pawel Stefanski wrote: > > hello! > > > > I've tried with wip-6078 and git dumpling builds and got the same error > > during OPTIONS request. > > > > curl -v -X OPTIONS -H 'Access-Control-Request-Method: PUT' -H "Origin: > > http://X.pl"; http://static3.X.pl/ > > > >> OPTIONS / HTTP/1.1 > >> User-Agent: curl/7.31.0 > >> Host: static3.X.pl > >> Accept: */* > >> Access-Control-Request-Method: PUT > >> Origin: http://X.pl > >> > > < HTTP/1.1 403 Forbidden > > < Date: Tue, 03 Sep 2013 09:34:39 GMT > > * Server Apache/2.2.22 (Ubuntu) is not blacklisted > > < Server: Apache/2.2.22 (Ubuntu) > > < Accept-Ranges: bytes > > < Content-Length: 78 > > < Content-Type: application/xml > > < > > > encoding="UTF-8"?>AccessDenied > > > > Of course CORS was set, but RGW can't find it, dump from log: > > > > 2013-09-03 11:34:39.876809 7f7c6dff3700 2 req 129:0.000385:s3:OPTIONS > > /:options_cors:verifying op mask > > 2013-09-03 11:34:39.876813 7f7c6dff3700 20 required_mask= 1 > user.op_mask=7 > > 2013-09-03 11:34:39.876815 7f7c6dff3700 2 req 129:0.000392:s3:OPTIONS > > /:options_cors:verifying op permissions > > 2013-09-03 11:34:39.876818 7f7c6dff3700 2 req 129:0.000395:s3:OPTIONS > > /:options_cors:verifying op params > > 2013-09-03 11:34:39.876821 7f7c6dff3700 2 req 129:0.000397:s3:OPTIONS > > /:options_cors:executing > > 2013-09-03 11:34:39.876824 7f7c6dff3700 2 No CORS configuration set yet > for > > this bucket > > 2013-09-03 11:34:39.876907 7f7c6dff3700 2 req 129:0.000483:s3:OPTIONS > > /:options_cors:http status=403 > > 2013-09-03 11:34:39.877551 7f7c6dff3700 1 == req done req=0x128e0e0 > > http_status=403 == > > > > Setting CORSfor static3 bucket: > > > > > CORSConfigurationGETPUT > > d>DELETEPOST > http://static3.X.pl > *3000 > > MaxAgeSeconds>GETPUTDELETEPOST > thod>http://static3.X.pl/ > *3000GET > > > PUTDELETEPOST > http://X.pl/ > > gin>*3000 > > 2013-09-03 12:34:02.973999 7f7c44fa1700 20 get_obj_state: > > rctx=0x7f7c38009330 obj=.rgw:static3 state=0x7f7c38008768 > s-
[ceph-users] How to use s3 API in Ceph Object Storage
Hello Guys:
I am working with ceph nowadys and i want to use the ceph object gateway.
So, i tried to use s3 API to fulfill my needs. I use the api for create a
baucet.
PUT /{bucket} HTTP/1.1
Host: kp
x-amz-acl: public-read-write
Authorization: AWS {access-key}:{hash-of-header-and-secret}
curl -XPUT http://kp/mybucket -H'x-amz-acl:public-read-write'
-H'Authorization: AWS {access-key}:{hash-of-header-and-secret}'
the use info is
{"user_id":"johndoe","rados_uid":0,"display_name":"John
Doe","email":"j...@example.com","suspended":0,"subusers":[],"keys":[{"user":"johndoe","access_key":"QFAMEDSJP5DEKJO0DDXY","secret_key":"iaSFLDVvDdQt6lkNzHyW4fPLZugBAI1g17LO0+87"}],"swift_keys":[]}
the {access-key} is QFAMEDSJP5DEKJO0DDXY
and the {hash-of-header-and-secret} is the result of
echo base64_encode(hash_hmac('sha1', 'x-amz-acl:public-read-write',
"iaSFLDVvDdQt6lkNzHyW4fPLZugBAI1g17LO0+87"))
and I get the return 403
Can anyone explain the method to deal with s3 API, thanks!
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] radosgw s3 ACLs
On Tue, Sep 3, 2013 at 2:30 PM, Derek Yarnell wrote: > Hi, > > So say a usera has full control (and is the owner) of a bucket in s3 and > gives userb 'FULL_CONTROL' on the bucket. Userb writes a file and it > seems that by default the ACL for that key is going to be 'FULL_CONTROL' > for userb. When usera iterates the key even just over the ACL for the > key, they get a 403. > > I would think that 'FULL_CONTROL' at the bucket level gives the user at > least the ability to see the ACL. They can list the key itself and the > size and last modified. > > Is this by design? Is there any chance that the default behavior would > be able to changed to by default (without specifying a canned acl) that > the bucket acl would be applied for a key? > > We are looking to provide some secure s3 collaboration space. Groups in > the ACLs would be like pie in the sky but right now just being able to > get sticky ACLs from the bucket would be huge. > The Swift acls provide that, but there's no way to set it through the S3 api. Basically, if you set the bucket permissions through the Swift api it'll be 'sticky' and apply also on the objects. Yehuda ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] radosgw s3 ACLs
Hi, So say a usera has full control (and is the owner) of a bucket in s3 and gives userb 'FULL_CONTROL' on the bucket. Userb writes a file and it seems that by default the ACL for that key is going to be 'FULL_CONTROL' for userb. When usera iterates the key even just over the ACL for the key, they get a 403. I would think that 'FULL_CONTROL' at the bucket level gives the user at least the ability to see the ACL. They can list the key itself and the size and last modified. Is this by design? Is there any chance that the default behavior would be able to changed to by default (without specifying a canned acl) that the bucket acl would be applied for a key? We are looking to provide some secure s3 collaboration space. Groups in the ACLs would be like pie in the sky but right now just being able to get sticky ACLs from the bucket would be huge. Thanks, derek -- --- Derek T. Yarnell University of Maryland Institute for Advanced Computer Studies ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] How to force lost PGs
> > > > Now I'm trying to clear the stale PGs. I've tried removing the OSD from the > > crush maps, the OSD lists etc, without any luck. > > Note that this means that you destroyed all copies of those 3 PGs, which > means this experiment lost data. > > You can make ceph recreate the PGs (empty!) with > > ceph pg force_create_pg > Just out of curiosity, in the rbd case, what does the OS get when it tries to access data from a missing pg? I assume something like 'unrecoverable read error' on read, and some other sort of error on write. And what about when the pg is recreated? Ideally, each 'sector' of the rbd device would still initially be flagged to report as 'unrecoverable read' on read, but then a write would clear that flag, in the same way as a physical disk would, but that would require a lot of extra metadata. This would allow a filesystem with such capabilities to best recover from the data loss. James ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] radosgw warnings in apache error.log
On Tue, Sep 3, 2013 at 9:13 AM, Fuchs, Andreas (SwissTXT) wrote: > We are testing radosgw with cyberduck, so far we see the following issues > > 1. In apache error log for each file put we see: > > [Tue Sep 03 17:35:24 2013] [warn] FastCGI: 193.218.104.138 PUT > https://193.218.100.131/test/tesfile04.iso auth AWS *** > [Tue Sep 03 17:35:24 2013] [warn] FastCGI: JJJ gotCont=1 > > - This is with the ceph version of apache/fastcgi. > - This happens on each file, 1 time at the beginning of the transfer and at > least at large files, several times at the end. > - The standard apache/fastcgi combo does not show this issue. > About what do we get warned here? > That's a debug output that I thought was long gone. We need to remove it. > > 2. Large file transfer of 4GB iso image with cyberduck > Uploads of a 4GB .iso file fail at the end of each upload, the file is then > shown with 0B in directory listing > > At the end of the upload I get the following in rados log > 2013-09-03 18:00:49.427092 7faf87fff700 2 > RGWDataChangesLog::ChangesRenewThread: start > 2013-09-03 18:01:11.427201 7faf87fff700 2 > RGWDataChangesLog::ChangesRenewThread: start > 2013-09-03 18:01:33.427310 7faf87fff700 2 > RGWDataChangesLog::ChangesRenewThread: start > 2013-09-03 18:01:36.557121 7faf46ff5700 10 x>> x-amz-acl:public-read > 2013-09-03 18:01:36.557567 7faf46ff5700 20 get_obj_state: rctx=0x7faf540251c0 > obj=test-han:tesfile05.iso state=0x7faf5412ecd8 s->prefetch_data=0 > 2013-09-03 18:01:36.558581 7faf46ff5700 0 setting object > write_tag=default.6641.36 > 2013-09-03 18:01:36.574302 7faf46ff5700 20 get_obj_state: rctx=0x7faf540251c0 > obj=test-han:_shadow__Bw6LkKRYSmAwEh5VBmxo7fb5Md-IY7d_1 state=0x7faf54028488 > s->prefetch_data=0 > 2013-09-03 18:01:36.575215 7faf46ff5700 20 get_obj_state: s->obj_tag was set > empty > 2013-09-03 18:01:36.575250 7faf46ff5700 20 prepare_atomic_for_write_impl: > state is not atomic. state=0x7faf54028488 > 2013-09-03 18:01:36.579593 7faf46ff5700 20 get_obj_state: rctx=0x7faf540251c0 > obj=test-han:_shadow__Bw6LkKRYSmAwEh5VBmxo7fb5Md-IY7d_2 state=0x7faf5412ea28 > s->prefetch_data=0 > 2013-09-03 18:01:36.580581 7faf46ff5700 20 get_obj_state: s->obj_tag was set > empty > 2013-09-03 18:01:36.580635 7faf46ff5700 20 prepare_atomic_for_write_impl: > state is not atomic. state=0x7faf5412ea28 > 2013-09-03 18:01:36.585058 7faf46ff5700 20 get_obj_state: rctx=0x7faf540251c0 > obj=test-han:_shadow__Bw6LkKRYSmAwEh5VBmxo7fb5Md-IY7d_3 state=0x7faf5412aaa8 > s->prefetch_data=0 > 2013-09-03 18:01:36.585956 7faf46ff5700 20 get_obj_state: s->obj_tag was set > empty > 2013-09-03 18:01:36.586005 7faf46ff5700 20 prepare_atomic_for_write_impl: > state is not atomic. state=0x7faf5412aaa8 > 2013-09-03 18:01:36.589811 7faf46ff5700 20 get_obj_state: rctx=0x7faf540251c0 > obj=test-han:_shadow__Bw6LkKRYSmAwEh5VBmxo7fb5Md-IY7d_4 state=0x7faf5412a068 > s->prefetch_data=0 > 2013-09-03 18:01:36.590886 7faf46ff5700 20 get_obj_state: s->obj_tag was set > empty > 2013-09-03 18:01:36.590921 7faf46ff5700 20 prepare_atomic_for_write_impl: > state is not atomic. state=0x7faf5412a068 > 2013-09-03 18:01:36.594977 7faf46ff5700 20 get_obj_state: rctx=0x7faf540251c0 > obj=test-han:_shadow__Bw6LkKRYSmAwEh5VBmxo7fb5Md-IY7d_5 state=0x7faf54126468 > s->prefetch_data=0 > 2013-09-03 18:01:36.596140 7faf46ff5700 20 get_obj_state: s->obj_tag was set > empty > . > . > hunderts of similar lines > . > . > . > 2013-09-03 18:01:42.210941 7faf46ff5700 20 get_obj_state: rctx=0x7faf540251c0 > obj=test-han:_shadow__Bw6LkKRYSmAwEh5VBmxo7fb5Md-IY7d_1038 > state=0x7faf54172738 s->prefetch_data=0 > 2013-09-03 18:01:42.211888 7faf46ff5700 20 get_obj_state: s->obj_tag was set > empty > 2013-09-03 18:01:42.211947 7faf46ff5700 20 prepare_atomic_for_write_impl: > state is not atomic. state=0x7faf54172738 > 2013-09-03 18:01:42.216903 7faf46ff5700 20 get_obj_state: rctx=0x7faf540251c0 > obj=test-han:_shadow__Bw6LkKRYSmAwEh5VBmxo7fb5Md-IY7d_1039 > state=0x7faf54172a88 s->prefetch_data=0 > 2013-09-03 18:01:42.217757 7faf46ff5700 20 prepare_atomic_for_write_impl: > state is not atomic. state=0x7faf54172a88 > 2013-09-03 18:01:42.221839 7faf46ff5700 0 WARNING: set_req_state_err > err_no=27 resorting to 500 error 27 means EFBIG. I do see a few newish error paths in the osd that may return this. Can you try setting the following on your osd: osd max attr size = 655360 > 2013-09-03 18:01:42.221868 7faf46ff5700 2 req 36:402.848669:s3:PUT > /test-han/tesfile05.iso:put_obj:http status=500 > 2013-09-03 18:01:42.223232 7faf46ff5700 1 == req done req=0x25e24b0 > http_status=500 == > 2013-09-03 18:01:55.427440 7faf87fff700 2 > RGWDataChangesLog::ChangesRenewThread: start > 2013-09-03 18:02:01.331323 7faf9d011780 20 enqueued request req=0x25e2020 > 2013-09-03 18:02:01.331341 7faf9d011780 20 RGWWQ: > 2013-09-03 18:02:01.331343 7faf9d011780 20 req: 0x25e2020 > 2013-09-03 18:02:01.331352 7faf9d011780 10 allocated request req=0x25e2500
Re: [ceph-users] adding SSD only pool to existing ceph cluster
On Mon, Sep 2, 2013 at 5:09 AM, Jens-Christian Fischer
wrote:
> We have a ceph cluster with 64 OSD (3 TB SATA) disks on 10 servers, and run
> an OpenStack cluster.
>
> We are planning to move the images of the running VM instances from the
> physical machines to CephFS. Our plan is to add 10 SSDs (one in each server)
> and create a pool that is backed only by these SSDs and mount that pool in a
> specific location in CephFS.
>
> References perused:
>
> http://www.sebastien-han.fr/blog/2012/12/07/ceph-2-speed-storage-with-crush/
> http://ceph.com/docs/master/rados/operations/crush-map/#placing-different-pools-on-different-osds
>
> The difference between Sebastiens and the Ceph approach is that Sebastien
> has mixed SAS/SSD servers, while the ceph documentation assumes either or
> servers.
>
> We have tried to replicate both approaches by manually editing the CRUSH map
> like so:
>
> Option 1)
>
> Create new "virtual" SSD only servers (where we have a h0 physical server,
> we'd set a h0-ssd for the ssd) in the CRUSH map, together with a related
> server/rack/datacenter/root hierarchy
>
> --- cut ---
> host s1-ssd {
> id -15 # do not change unnecessarily
> # weight 0.500
> alg straw
> hash 0 # rjenkins1
> item osd.36 weight 0.500
> }
> …
>
> rack cla-r71-ssd {
> id -24 # do not change unnecessarily
> # weight 2.500
> alg straw
> hash 0 # rjenkins1
> item s0-ssd weight 0.000
> item s1-ssd weight 0.500
> […]
> item h5-ssd weight 0.000
> }
> root ssd {
> id -25 # do not change unnecessarily
> # weight 2.500
> alg straw
> hash 0 # rjenkins1
> item cla-r71-ssd weight 2.500
> }
>
> rule ssd {
> ruleset 3
> type replicated
> min_size 1
> max_size 10
> step take ssd
> step chooseleaf firstn 0 type host
> step emit
> }
>
> --- cut ---
>
> Option 2)
> Create two pools (SATA and SSD) and list all SSDs manually in them
>
> --- cut ---
> pool ssd {
> id -14 # do not change unnecessarily
> # weight 2.500
> alg straw
> hash 0 # rjenkins1
> item osd.36 weight 0.500
> item osd.65 weight 0.500
> item osd.66 weight 0.500
> item osd.67 weight 0.500
> item osd.68 weight 0.500
> item osd.69 weight 0.500
> }
>
> --- cut ---
>
>
> We extracted the CRUSH map, decompiled, changed, compiled and injected it.
> Both tries didn't seem to "really work" (™) as we saw the cluster go into
> reshuffling mode immediately (probably due to the changed layout (OSD ->
> Host -> Rack -> Root) in both cases.
>
> We reverted to the original CRUSH map and the cluster has been quiet since
> then.
>
> Now the question: What is the best way to handle our use case?
>
> Add 10 SSD drives, create a separate pool with them, don't upset the current
> pools (We don't want the "regular/existing" data to migrate towards the SSD
> pool, and no disruption of service?
If you saw your existing data migrate that means you changed its
hierarchy somehow. It sounds like maybe you reorganized your existing
nodes slightly, and that would certainly do it (although simply adding
single-node higher levels would not). It's also possible that you
introduced your SSD devices/hosts in a way that your existing data
pool rules believed they should make use of them (if, for instance,
your data pool rule starts out at root and you added your SSDs
underneath there). What you'll want to do is add a whole new root for
your SSD nodes, and then make the SSD pool rule (and only that rule)
start out there.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] PG stuck in inconsistent state
On Tue, Sep 3, 2013 at 7:20 AM, Maciej Gałkiewicz
wrote:
> Hi
>
> I have recently discovered that one of my pg is in inconsistent state. I
> have checked filesystem on osd.3, and re-run deep-scrub few times. Osd uses
> xfs. Any suggestions how to fix it?
You can use "repair" instead of "scrub" or "deep-scrub" — but right
now the "fix" just copies the primary's copy to anybody who disagrees.
You need to evaluate data integrity yourself (if you have three
copies, probably just look for the two that match) and if the primary
is the bad copy then delete it on the node before running repair.
At least, that was how it went last time I saw this come up; Sam might
have some newer recommendations.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
>
> # ceph -v
> ceph version 0.61.8 (a6fdcca3bddbc9f177e4e2bf0d9cdd85006b028b)
> # ceph pg deep-scrub 350.5e
>
> 2013-09-03 16:15:51.961722 7f9189d5c700 0 log [ERR] : 350.5e osd.3: soid
> 21b651de/rbd_data.163bb74b0dc51.0007/head//350 digest 1420341435
> != known digest 3593795552
> 2013-09-03 16:15:53.114372 7f9189d5c700 0 log [ERR] : 350.5e deep-scrub 0
> missing, 1 inconsistent objects
> 2013-09-03 16:15:53.114379 7f9189d5c700 0 log [ERR] : 350.5e deep-scrub 1
> errors
>
>
> { "state": "active+clean+inconsistent",
> "epoch": 6429,
> "up": [
> 3,
> 5],
> "acting": [
> 3,
> 5],
> "info": { "pgid": "350.5e",
> "last_update": "6429'972570",
> "last_complete": "6429'972570",
> "log_tail": "6398'969569",
> "last_backfill": "MAX",
> "purged_snaps": "[]",
> "history": { "epoch_created": 2132,
> "last_epoch_started": 6429,
> "last_epoch_clean": 6429,
> "last_epoch_split": 0,
> "same_up_since": 6428,
> "same_interval_since": 6428,
> "same_primary_since": 6428,
> "last_scrub": "6429'972567",
> "last_scrub_stamp": "2013-09-03 16:15:53.114382",
> "last_deep_scrub": "6429'972567",
> "last_deep_scrub_stamp": "2013-09-03 16:15:53.114382",
> "last_clean_scrub_stamp": "2013-09-03 16:04:57.077937"},
> "stats": { "version": "6429'972570",
> "reported": "6429'1178349",
> "state": "active+clean+inconsistent",
> "last_fresh": "2013-09-03 16:16:23.630653",
> "last_change": "2013-09-03 16:15:53.114402",
> "last_active": "2013-09-03 16:16:23.630653",
> "last_clean": "2013-09-03 16:16:23.630653",
> "last_became_active": "0.00",
> "last_unstale": "2013-09-03 16:16:23.630653",
> "mapping_epoch": 6426,
> "log_start": "6398'969569",
> "ondisk_log_start": "6398'969569",
> "created": 2132,
> "last_epoch_clean": 6429,
> "parent": "0.0",
> "parent_split_bits": 0,
> "last_scrub": "6429'972567",
> "last_scrub_stamp": "2013-09-03 16:15:53.114382",
> "last_deep_scrub": "6429'972567",
> "last_deep_scrub_stamp": "2013-09-03 16:15:53.114382",
> "last_clean_scrub_stamp": "2013-09-03 16:04:57.077937",
> "log_size": 0,
> "ondisk_log_size": 0,
> "stats_invalid": "0",
> "stat_sum": { "num_bytes": 571404305,
> "num_objects": 78,
> "num_object_clones": 0,
> "num_object_copies": 0,
> "num_objects_missing_on_primary": 0,
> "num_objects_degraded": 0,
> "num_objects_unfound": 0,
> "num_read": 1037151,
> "num_read_kb": 1982418,
> "num_write": 972739,
> "num_write_kb": 12430480,
> "num_scrub_errors": 1,
> "num_shallow_scrub_errors": 0,
> "num_deep_scrub_errors": 1,
> "num_objects_recovered": 1320,
> "num_bytes_recovered": 10111942741,
> "num_keys_recovered": 124},
> "stat_cat_sum": {},
> "up": [
> 3,
> 5],
> "acting": [
> 3,
> 5]},
> "empty": 0,
> "dne": 0,
> "incomplete": 0,
> "last_epoch_started": 6429},
> "recovery_state": [
> { "name": "Started\/Primary\/Active",
> "enter_time": "2013-09-03 16:14:59.621528",
> "might_have_unfound": [
> { "osd": 5,
> "status": "already probed"}],
> "recovery_progress": { "backfill_target": -1,
> "waiting_on_backfill": 0,
> "backfill_pos": "0\/\/0\/\/-1",
> "backfill_info": { "begin": "0\/\/0\/\/-1",
> "end": "0\/\/0\/\/-1",
> "objects": []},
> "peer_backfill_info": { "begin": "0\/\/0\/\/-1",
> "end": "0\/\/0\/\/-1",
> "objects": []},
> "backfills_in_flight": [],
>
Re: [ceph-users] How to force a monitor time check?
On 09/03/2013 02:35 PM, Da Chun Ng wrote: How often are the time check messages sent between monitors? The monitors will perform a timecheck every 5 minutes, by default. You can adjust this interval using 'mon timecheck interval = SECS'. From: dachun...@outlook.com To: ceph-users@lists.ceph.com Date: Tue, 3 Sep 2013 13:22:54 + Subject: [ceph-users] How to force a monitor time check? Time skews happen frequently when the systems running monitors are restarted. With ntp server configured, the time skew between systems will be fixed over some time. But the ceph monitors won't find it at once if there are no time check messages at that time, so the ceph status will be still shown as time skew warning. Sometimes it will keep for several hours. How can I trigger a monitor time check manually? There are no current provisions to manually trigger a timecheck. Timechecks will be performed every 5 minutes (by default) as long as there's a formed quorum. What version are you running? I don't recall ceph having any bug so far that could lead to such state, thus I find it hard to believe that the monitors will warn over a period of several hours when no clock skews are present. In that case, my guess would be that there's still some server suffering from a clock skew. Please note that all clock skews are determined taking the 'leader' as the point of reference. This means that if you notice that all monitors are displaying the same clock skews while the leader is not, then it's likely that it is the leader that is suffering from said clock skew. -Joao -- Joao Eduardo Luis Software Engineer | http://inktank.com | http://ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] radosgw warnings in apache error.log
We are testing radosgw with cyberduck, so far we see the following issues 1. In apache error log for each file put we see: [Tue Sep 03 17:35:24 2013] [warn] FastCGI: 193.218.104.138 PUT https://193.218.100.131/test/tesfile04.iso auth AWS *** [Tue Sep 03 17:35:24 2013] [warn] FastCGI: JJJ gotCont=1 - This is with the ceph version of apache/fastcgi. - This happens on each file, 1 time at the beginning of the transfer and at least at large files, several times at the end. - The standard apache/fastcgi combo does not show this issue. About what do we get warned here? 2. Large file transfer of 4GB iso image with cyberduck Uploads of a 4GB .iso file fail at the end of each upload, the file is then shown with 0B in directory listing At the end of the upload I get the following in rados log 2013-09-03 18:00:49.427092 7faf87fff700 2 RGWDataChangesLog::ChangesRenewThread: start 2013-09-03 18:01:11.427201 7faf87fff700 2 RGWDataChangesLog::ChangesRenewThread: start 2013-09-03 18:01:33.427310 7faf87fff700 2 RGWDataChangesLog::ChangesRenewThread: start 2013-09-03 18:01:36.557121 7faf46ff5700 10 x>> x-amz-acl:public-read 2013-09-03 18:01:36.557567 7faf46ff5700 20 get_obj_state: rctx=0x7faf540251c0 obj=test-han:tesfile05.iso state=0x7faf5412ecd8 s->prefetch_data=0 2013-09-03 18:01:36.558581 7faf46ff5700 0 setting object write_tag=default.6641.36 2013-09-03 18:01:36.574302 7faf46ff5700 20 get_obj_state: rctx=0x7faf540251c0 obj=test-han:_shadow__Bw6LkKRYSmAwEh5VBmxo7fb5Md-IY7d_1 state=0x7faf54028488 s->prefetch_data=0 2013-09-03 18:01:36.575215 7faf46ff5700 20 get_obj_state: s->obj_tag was set empty 2013-09-03 18:01:36.575250 7faf46ff5700 20 prepare_atomic_for_write_impl: state is not atomic. state=0x7faf54028488 2013-09-03 18:01:36.579593 7faf46ff5700 20 get_obj_state: rctx=0x7faf540251c0 obj=test-han:_shadow__Bw6LkKRYSmAwEh5VBmxo7fb5Md-IY7d_2 state=0x7faf5412ea28 s->prefetch_data=0 2013-09-03 18:01:36.580581 7faf46ff5700 20 get_obj_state: s->obj_tag was set empty 2013-09-03 18:01:36.580635 7faf46ff5700 20 prepare_atomic_for_write_impl: state is not atomic. state=0x7faf5412ea28 2013-09-03 18:01:36.585058 7faf46ff5700 20 get_obj_state: rctx=0x7faf540251c0 obj=test-han:_shadow__Bw6LkKRYSmAwEh5VBmxo7fb5Md-IY7d_3 state=0x7faf5412aaa8 s->prefetch_data=0 2013-09-03 18:01:36.585956 7faf46ff5700 20 get_obj_state: s->obj_tag was set empty 2013-09-03 18:01:36.586005 7faf46ff5700 20 prepare_atomic_for_write_impl: state is not atomic. state=0x7faf5412aaa8 2013-09-03 18:01:36.589811 7faf46ff5700 20 get_obj_state: rctx=0x7faf540251c0 obj=test-han:_shadow__Bw6LkKRYSmAwEh5VBmxo7fb5Md-IY7d_4 state=0x7faf5412a068 s->prefetch_data=0 2013-09-03 18:01:36.590886 7faf46ff5700 20 get_obj_state: s->obj_tag was set empty 2013-09-03 18:01:36.590921 7faf46ff5700 20 prepare_atomic_for_write_impl: state is not atomic. state=0x7faf5412a068 2013-09-03 18:01:36.594977 7faf46ff5700 20 get_obj_state: rctx=0x7faf540251c0 obj=test-han:_shadow__Bw6LkKRYSmAwEh5VBmxo7fb5Md-IY7d_5 state=0x7faf54126468 s->prefetch_data=0 2013-09-03 18:01:36.596140 7faf46ff5700 20 get_obj_state: s->obj_tag was set empty . . hunderts of similar lines . . . 2013-09-03 18:01:42.210941 7faf46ff5700 20 get_obj_state: rctx=0x7faf540251c0 obj=test-han:_shadow__Bw6LkKRYSmAwEh5VBmxo7fb5Md-IY7d_1038 state=0x7faf54172738 s->prefetch_data=0 2013-09-03 18:01:42.211888 7faf46ff5700 20 get_obj_state: s->obj_tag was set empty 2013-09-03 18:01:42.211947 7faf46ff5700 20 prepare_atomic_for_write_impl: state is not atomic. state=0x7faf54172738 2013-09-03 18:01:42.216903 7faf46ff5700 20 get_obj_state: rctx=0x7faf540251c0 obj=test-han:_shadow__Bw6LkKRYSmAwEh5VBmxo7fb5Md-IY7d_1039 state=0x7faf54172a88 s->prefetch_data=0 2013-09-03 18:01:42.217757 7faf46ff5700 20 prepare_atomic_for_write_impl: state is not atomic. state=0x7faf54172a88 2013-09-03 18:01:42.221839 7faf46ff5700 0 WARNING: set_req_state_err err_no=27 resorting to 500 2013-09-03 18:01:42.221868 7faf46ff5700 2 req 36:402.848669:s3:PUT /test-han/tesfile05.iso:put_obj:http status=500 2013-09-03 18:01:42.223232 7faf46ff5700 1 == req done req=0x25e24b0 http_status=500 == 2013-09-03 18:01:55.427440 7faf87fff700 2 RGWDataChangesLog::ChangesRenewThread: start 2013-09-03 18:02:01.331323 7faf9d011780 20 enqueued request req=0x25e2020 2013-09-03 18:02:01.331341 7faf9d011780 20 RGWWQ: 2013-09-03 18:02:01.331343 7faf9d011780 20 req: 0x25e2020 2013-09-03 18:02:01.331352 7faf9d011780 10 allocated request req=0x25e2500 2013-09-03 18:02:01.331377 7faf307c8700 20 dequeued request req=0x25e2020 2013-09-03 18:02:01.331385 7faf307c8700 20 RGWWQ: empty 3. performance While I get great performance from the radosgw to the stoage nodes, the performance from cyberduck to the radosgw is poor (10 times slower) On a rados filesystem mounted on the radosgw I get: arround 100MB/s write troughput (here I know it's the 1Gb sync network between the nodes who is the limiting factor). With a cyberduck test I
Re: [ceph-users] Best way to reformat OSD drives?
On 03.09.2013, at 16:27, Sage Weil wrote:
>> ceph osd create # this should give you back the same osd number as the one
>> you just removed
>
> OSD=`ceph osd create` # may or may not be the same osd id
good point - so far it has been good to us!
>
>>
>> umount ${PART}1
>> parted $PART rm 1 # remove partion and create a new one
>> parted $PART mkpart primary 0% 100% # remove partion and create a new one
>
> I don't think the partition removal/add step is needed.
it isn't - I'm still learning the ropes :)
>
> Otherwise it looks fine!
ok - I have tried a simplified version (that doesn't take the OSD out) that
just "simulates" a disk failure (i.e.. stops the OSD, reformats the drive,
recreates the OSD structure and starts the process again). This (seems) to
work, but is really slow in rebuilding the disk (we see write speed of 4-20
MB/s - and it takes ages to refill around 100GB of data)
I don't dare to run this on multiple OSDs a the same time for fear of loosing
data, so the "slower/longer" process of first marking all OSDs of a server as
out, waiting for them to empty and then batch formatting all OSDs on the server
and waiting for the cluster to be stable again, might be faster in the end
cheers
jc
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] CORS not working
On Tue, Sep 3, 2013 at 3:40 AM, Pawel Stefanski wrote: > hello! > > I've tried with wip-6078 and git dumpling builds and got the same error > during OPTIONS request. > > curl -v -X OPTIONS -H 'Access-Control-Request-Method: PUT' -H "Origin: > http://X.pl"; http://static3.X.pl/ > >> OPTIONS / HTTP/1.1 >> User-Agent: curl/7.31.0 >> Host: static3.X.pl >> Accept: */* >> Access-Control-Request-Method: PUT >> Origin: http://X.pl >> > < HTTP/1.1 403 Forbidden > < Date: Tue, 03 Sep 2013 09:34:39 GMT > * Server Apache/2.2.22 (Ubuntu) is not blacklisted > < Server: Apache/2.2.22 (Ubuntu) > < Accept-Ranges: bytes > < Content-Length: 78 > < Content-Type: application/xml > < > encoding="UTF-8"?>AccessDenied > > Of course CORS was set, but RGW can't find it, dump from log: > > 2013-09-03 11:34:39.876809 7f7c6dff3700 2 req 129:0.000385:s3:OPTIONS > /:options_cors:verifying op mask > 2013-09-03 11:34:39.876813 7f7c6dff3700 20 required_mask= 1 user.op_mask=7 > 2013-09-03 11:34:39.876815 7f7c6dff3700 2 req 129:0.000392:s3:OPTIONS > /:options_cors:verifying op permissions > 2013-09-03 11:34:39.876818 7f7c6dff3700 2 req 129:0.000395:s3:OPTIONS > /:options_cors:verifying op params > 2013-09-03 11:34:39.876821 7f7c6dff3700 2 req 129:0.000397:s3:OPTIONS > /:options_cors:executing > 2013-09-03 11:34:39.876824 7f7c6dff3700 2 No CORS configuration set yet for > this bucket > 2013-09-03 11:34:39.876907 7f7c6dff3700 2 req 129:0.000483:s3:OPTIONS > /:options_cors:http status=403 > 2013-09-03 11:34:39.877551 7f7c6dff3700 1 == req done req=0x128e0e0 > http_status=403 == > > Setting CORSfor static3 bucket: > > CORSConfigurationGETPUT d>DELETEPOSThttp://static3.X.pl*3000 MaxAgeSeconds>GETPUTDELETEPOST thod>http://static3.X.pl/*3000GET > PUTDELETEPOSThttp://X.pl/ gin>*3000 > 2013-09-03 12:34:02.973999 7f7c44fa1700 20 get_obj_state: > rctx=0x7f7c38009330 obj=.rgw:static3 state=0x7f7c38008768 s->prefetch_data=0 > 2013-09-03 12:34:02.974004 7f7c44fa1700 20 state->obj_tag is empty, not > appending atomic test > 2013-09-03 12:34:02.983719 7f7c44fa1700 10 cache put: name=.rgw+static3 > 2013-09-03 12:34:02.983750 7f7c44fa1700 10 moving .rgw+static3 to cache LRU > end > 2013-09-03 12:34:02.983756 7f7c44fa1700 10 appending xattr: > name=user.rgw.cors bl.length()=194 > 2013-09-03 12:34:02.983801 7f7c44fa1700 10 distributing notification > oid=notify.2 bl.length()=389 > 2013-09-03 12:34:02.986649 7f7c9bfff700 10 RGWWatcher::notify() opcode=1 > ver=1 bl.length()=389 > 2013-09-03 12:34:02.986696 7f7c9bfff700 10 cache put: name=.rgw+static3 > 2013-09-03 12:34:02.986699 7f7c9bfff700 10 moving .rgw+static3 to cache LRU > end > 2013-09-03 12:34:02.986703 7f7c9bfff700 10 appending xattr: > name=user.rgw.cors bl.length()=194 > 2013-09-03 12:34:02.987861 7f7c44fa1700 10 cache put: name=.rgw+static3 > 2013-09-03 12:34:02.987872 7f7c44fa1700 10 moving .rgw+static3 to cache LRU > end > 2013-09-03 12:34:02.987876 7f7c44fa1700 10 appending xattr: > name=user.rgw.cors bl.length()=194 > 2013-09-03 12:34:02.987901 7f7c44fa1700 10 distributing notification > oid=notify.2 bl.length()=389 > 2013-09-03 12:34:02.988593 7f7c9bfff700 10 RGWWatcher::notify() opcode=1 > ver=1 bl.length()=389 > 2013-09-03 12:34:02.988624 7f7c9bfff700 10 cache put: name=.rgw+static3 > 2013-09-03 12:34:02.988627 7f7c9bfff700 10 moving .rgw+static3 to cache LRU > end > 2013-09-03 12:34:02.988635 7f7c9bfff700 10 appending xattr: > name=user.rgw.cors bl.length()=194 > 2013-09-03 12:34:02.989550 7f7c44fa1700 2 req 133:0.016683:s3:PUT > /:put_cors:http status=200 > > Checked with: > 0.67.1-8-gaeb4141 (aeb414135326805324d619b507274b1d0cef7067) > 0.67.2-22-ga708c8a (a708c8ab52e5b1476405a1f817c23b8845fbaab3) > Did you configure the virtual bucket name through host name correctly? For that you would have needed to set 'rgw dns name' in your ceph.conf. It seems as if it's not set and you're using it. Yehuda ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] How to force lost PGs
Awesome Sage! I knew I had lost data. I'm trying to find out what will happen when the worst happens (like the ceph administer is an idiot). So those PGs are hanging around in a OSD/pool somewhere with some kind of reference count and they just need to be recreated? Thanks again for unsticking me. -Gaylord On 09/03/2013 10:44 AM, Sage Weil wrote: On Sun, 1 Sep 2013, Gaylord Holder wrote: I created a pool with no replication and an RBD within that pool. I mapped the RBD to a machine, formatted it with a file system and dumped data on it. Just to see what kind of trouble I can get into, I stopped the OSD the RBD was using, marked the OSD as out, and reformatted the OSD tree. When I brought the OSD back up, I now have three stale PGs. Now I'm trying to clear the stale PGs. I've tried removing the OSD from the crush maps, the OSD lists etc, without any luck. Note that this means that you destroyed all copies of those 3 PGs, which means this experiment lost data. You can make ceph recreate the PGs (empty!) with ceph pg force_create_pg sage Running ceph pg 3.1 query ceph pg 3.1 mark_unfound_lost revert ceph explains it doesn't have a PG 3.1 Running ceph osd repair osd.1 hangs after pg 2.3e Running ceph osd lost 1 --yes-i-really-mean-it nukes the osd. Rebuilding osd.1 goes fine, but I still have 3 stale PGs. Any help clearing these stale pages would be appreciated. Thanks, -Gaylord ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] How to force lost PGs
On Sun, 1 Sep 2013, Gaylord Holder wrote: > > I created a pool with no replication and an RBD within that pool. I mapped > the RBD to a machine, formatted it with a file system and dumped data on it. > > Just to see what kind of trouble I can get into, I stopped the OSD the RBD was > using, marked the OSD as out, and reformatted the OSD tree. > > When I brought the OSD back up, I now have three stale PGs. > > Now I'm trying to clear the stale PGs. I've tried removing the OSD from the > crush maps, the OSD lists etc, without any luck. Note that this means that you destroyed all copies of those 3 PGs, which means this experiment lost data. You can make ceph recreate the PGs (empty!) with ceph pg force_create_pg sage > Running > ceph pg 3.1 query > ceph pg 3.1 mark_unfound_lost revert > ceph explains it doesn't have a PG 3.1 > > Running > ceph osd repair osd.1 > hangs after pg 2.3e > > Running > ceph osd lost 1 --yes-i-really-mean-it > nukes the osd. Rebuilding osd.1 goes fine, but I still have 3 stale PGs. > > Any help clearing these stale pages would be appreciated. > > Thanks, > -Gaylord > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Best way to reformat OSD drives?
On Mon, 2 Sep 2013, Jens-Christian Fischer wrote:
> Hi all
> we have a Ceph Cluster with 64 OSD drives in 10 servers. We originally
> formatted the OSDs with btrfs but have had numerous problems (server kernel
> panics) that we could point back to btrfs. We are therefore in the process
> of reformatting our OSDs to XFS. We have a process that works, but I was
> wondering, if there is a simpler / faster way.
>
> Currently we 'ceph osd out' all drives of a server and wait for the data to
> migrate away, then delete the OSD, recreate it and start the OSD processes
> again. This takes at least 1-2 days per server (mostly waiting for the data
> to migrate back and forth)
>
> Here's the script we are using:
>
> --- cut ---
> #! /bin/bash
>
> OSD=$1
> PART=$2
> HOST=$3
> echo "changing partition ${PART}1 to XFS for OSD: $OSD on host: $HOST"
> read -p "continue or CTRL-C"
>
>
> service ceph -a stop osd.$OSD
> ceph osd crush remove osd.$OSD
> ceph auth del osd.$OSD
> ceph osd rm $OSD
> ceph osd create # this should give you back the same osd number as the one
> you just removed
OSD=`ceph osd create` # may or may not be the same osd id
>
> umount ${PART}1
> parted $PART rm 1 # remove partion and create a new one
> parted $PART mkpart primary 0% 100% # remove partion and create a new one
I don't think the partition removal/add step is needed.
> mkfs.xfs -f -i size=2048 ${PART}1 -L osd.$OSD
> mount -o inode64,noatime ${PART}1 /var/lib/ceph/osd/ceph-$OSD
> ceph-osd -i $OSD --mkfs --mkkey --mkjournal
> ceph auth add osd.$OSD osd 'allow *' mon 'allow rwx' -i
> /var/lib/ceph/osd/ceph-${OSD}/keyring
> ceph osd crush set $OSD 1 root=default host=$HOST
> service ceph -a start osd.$OSD
Otherwise it looks fine!
sage
>
> --- cut ---
>
> cheers
> Jens-Christian
>
> --
> SWITCH
> Jens-Christian Fischer, Peta Solutions
> Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland
> phone +41 44 268 15 15, direct +41 44 268 15 71
> jens-christian.fisc...@switch.ch
> http://www.switch.ch
>
> http://www.switch.ch/socialmedia
>
>
> ___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] PG stuck in inconsistent state
Hi
I have recently discovered that one of my pg is in inconsistent state. I
have checked filesystem on osd.3, and re-run deep-scrub few times. Osd uses
xfs. Any suggestions how to fix it?
# ceph -v
ceph version 0.61.8 (a6fdcca3bddbc9f177e4e2bf0d9cdd85006b028b)
# ceph pg deep-scrub 350.5e
2013-09-03 16:15:51.961722 7f9189d5c700 0 log [ERR] : 350.5e osd.3: soid
21b651de/rbd_data.163bb74b0dc51.0007/head//350 digest
1420341435 != known digest 3593795552
2013-09-03 16:15:53.114372 7f9189d5c700 0 log [ERR] : 350.5e deep-scrub 0
missing, 1 inconsistent objects
2013-09-03 16:15:53.114379 7f9189d5c700 0 log [ERR] : 350.5e deep-scrub 1
errors
{ "state": "active+clean+inconsistent",
"epoch": 6429,
"up": [
3,
5],
"acting": [
3,
5],
"info": { "pgid": "350.5e",
"last_update": "6429'972570",
"last_complete": "6429'972570",
"log_tail": "6398'969569",
"last_backfill": "MAX",
"purged_snaps": "[]",
"history": { "epoch_created": 2132,
"last_epoch_started": 6429,
"last_epoch_clean": 6429,
"last_epoch_split": 0,
"same_up_since": 6428,
"same_interval_since": 6428,
"same_primary_since": 6428,
"last_scrub": "6429'972567",
"last_scrub_stamp": "2013-09-03 16:15:53.114382",
"last_deep_scrub": "6429'972567",
"last_deep_scrub_stamp": "2013-09-03 16:15:53.114382",
"last_clean_scrub_stamp": "2013-09-03 16:04:57.077937"},
"stats": { "version": "6429'972570",
"reported": "6429'1178349",
"state": "active+clean+inconsistent",
"last_fresh": "2013-09-03 16:16:23.630653",
"last_change": "2013-09-03 16:15:53.114402",
"last_active": "2013-09-03 16:16:23.630653",
"last_clean": "2013-09-03 16:16:23.630653",
"last_became_active": "0.00",
"last_unstale": "2013-09-03 16:16:23.630653",
"mapping_epoch": 6426,
"log_start": "6398'969569",
"ondisk_log_start": "6398'969569",
"created": 2132,
"last_epoch_clean": 6429,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "6429'972567",
"last_scrub_stamp": "2013-09-03 16:15:53.114382",
"last_deep_scrub": "6429'972567",
"last_deep_scrub_stamp": "2013-09-03 16:15:53.114382",
"last_clean_scrub_stamp": "2013-09-03 16:04:57.077937",
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": "0",
"stat_sum": { "num_bytes": 571404305,
"num_objects": 78,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_degraded": 0,
"num_objects_unfound": 0,
"num_read": 1037151,
"num_read_kb": 1982418,
"num_write": 972739,
"num_write_kb": 12430480,
"num_scrub_errors": 1,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 1,
"num_objects_recovered": 1320,
"num_bytes_recovered": 10111942741,
"num_keys_recovered": 124},
"stat_cat_sum": {},
"up": [
3,
5],
"acting": [
3,
5]},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 6429},
"recovery_state": [
{ "name": "Started\/Primary\/Active",
"enter_time": "2013-09-03 16:14:59.621528",
"might_have_unfound": [
{ "osd": 5,
"status": "already probed"}],
"recovery_progress": { "backfill_target": -1,
"waiting_on_backfill": 0,
"backfill_pos": "0\/\/0\/\/-1",
"backfill_info": { "begin": "0\/\/0\/\/-1",
"end": "0\/\/0\/\/-1",
"objects": []},
"peer_backfill_info": { "begin": "0\/\/0\/\/-1",
"end": "0\/\/0\/\/-1",
"objects": []},
"backfills_in_flight": [],
"pull_from_peer": [],
"pushing": []},
"scrub": { "scrubber.epoch_start": "6428",
"scrubber.active": 0,
"scrubber.block_writes": 0,
"scrubber.finalizing": 0,
"scrubber.waiting_on": 0,
"scrubber.waiting_on_whom": []}},
{ "name": "Started",
"enter_time": "2013-09-03 16:14:58.192177"}]}
regards
--
Maciej Gałkiewicz
Shelly Cloud Sp. z o. o., Sysadmin
http://shellycloud.com/, mac...@shellycloud.com
KRS: 440358 REGON: 101504426
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] How to force a monitor time check?
How often are the time check messages sent between monitors? From: dachun...@outlook.com To: ceph-users@lists.ceph.com Date: Tue, 3 Sep 2013 13:22:54 + Subject: [ceph-users] How to force a monitor time check? Time skews happen frequently when the systems running monitors are restarted.With ntp server configured, the time skew between systems will be fixed over some time. But the ceph monitors won't find it at once if there are no time check messages at that time, so the ceph status will be still shown as time skew warning. Sometimes it will keep for several hours.How can I trigger a monitor time check manually? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph-mon runs on 6800 not 6789.
Am 03.09.2013 14:56, schrieb Joao Eduardo Luis: On 09/03/2013 02:02 AM, 이주헌 wrote: Hi all. I have 1 MDS and 3 OSDs. I installed them via ceph-deploy. (dumpling 0.67.2 version) At first, It works perfectly. But, after I reboot one of OSD, ceph-mon launched on port 6800 not 6789. This has been a recurrent issue I've been completely unable to reproduce so far. Are you able to reproduce this reliably? Could you share the steps you took leading you to this state? The OS automatically closes the previous socket when the old process ends. But often you can't listen on the same port right away because the socket, though closed, remains in the 2MSL state for some amount of time (generally a few minutes). The OS automatically transitions the old socket out of this state when the timeout expires. The problem might occur one some machines and not on others, depending on the system's socket settings (systcl). So is the port dynamically chosen by ceph? Is the SO_REUSEADDR flag set on the socket to workaround the just described problem? Corin ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] How to force a monitor time check?
Time skews happen frequently when the systems running monitors are restarted.With ntp server configured, the time skew between systems will be fixed over some time. But the ceph monitors won't find it at once if there are no time check messages at that time, so the ceph status will be still shown as time skew warning. Sometimes it will keep for several hours.How can I trigger a monitor time check manually? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph-mon runs on 6800 not 6789.
On 09/03/2013 02:02 AM, 이주헌 wrote:
Hi all.
I have 1 MDS and 3 OSDs. I installed them via ceph-deploy. (dumpling
0.67.2 version)
At first, It works perfectly. But, after I reboot one of OSD, ceph-mon
launched on port 6800 not 6789.
This has been a recurrent issue I've been completely unable to reproduce
so far.
Are you able to reproduce this reliably?
Could you share the steps you took leading you to this state?
-Joao
This is a result of 'ceph -s'
---
cluster c59d13fd-c4c9-4cd0-b2ed-b654428b3171
health HEALTH_WARN 1 mons down, quorum 0,1,2
ceph-mds,ceph-osd0,ceph-osd1
monmap e1: 4 mons at
{ceph-mds=192.168.13.135:6789/0,ceph-osd0=192.168.13.136:6789/0,ceph-osd1=192.168.13.137:6789/0,ceph-osd2=192.168.13.138:6789/0
},
election epoch 206, quorum 0,1,2 ceph-mds,ceph-osd0,ceph-osd1
osdmap e22: 4 osds: 2 up, 2 in
pgmap v67: 192 pgs: 192 active+clean; 145 MB data, 2414 MB used,
18043 MB / 20458 MB avail
mdsmap e4: 1/1/1 up {0=ceph-mds=up:active}
---
1 mons down -> It's running on 6800.
This is a /etc/ceph/ceph.conf that is created automatically by ceph-deploy.
---
[global]
fsid = c59d13fd-c4c9-4cd0-b2ed-b654428b3171
mon_initial_members = ceph-mds, ceph-osd0, ceph-osd1, ceph-osd2
mon_host = 192.168.13.135,192.168.13.136,192.168.13.137,192.168.13.138
auth_supported = cephx
osd_journal_size = 1024
filestore_xattr_use_omap = true
---
According to my understanding, ceph-mon's default port is 6789.
Why does it run on 6800 instead of 6789?
Restarting ceph-mon has a same result.
Sorry for my poor english. I don't write or speak english fluently.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Joao Eduardo Luis
Software Engineer | http://inktank.com | http://ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph s3 allowed characters
Thanks for your answer. Regards Dominik On Aug 30, 2013 4:59 PM, "Yehuda Sadeh" wrote: > On Fri, Aug 30, 2013 at 7:44 AM, Dominik Mostowiec > wrote: > > (echo -n 'GET /dysk/files/test.test% > 40op.pl/DOMIWENT%202013/Damian%20DW/dw/Specyfikacja%20istotnych%20warunk%F3w%20zam%F3wienia.doc > > HTTP/1.0'; printf "\r\n\r\n") | nc localhost 88 > > HTTP/1.1 400 Bad Request > > Date: Fri, 30 Aug 2013 14:10:07 GMT > > Server: Apache/2.2.22 (Ubuntu) > > Accept-Ranges: bytes > > Content-Length: 83 > > Connection: close > > Content-Type: application/xml > > > > > encoding="UTF-8"?>InvalidObjectName > > > > > The object name needs to be utf8 encoded. > > Yehuda > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] CORS not working
hello! I've tried with wip-6078 and git dumpling builds and got the same error during OPTIONS request. curl -v -X OPTIONS -H 'Access-Control-Request-Method: PUT' -H "Origin: http://X.pl"; http://static3.X.pl/ > OPTIONS / HTTP/1.1 > User-Agent: curl/7.31.0 > Host: static3.X.pl > Accept: */* > Access-Control-Request-Method: PUT > Origin: http://X.pl > < HTTP/1.1 403 Forbidden < Date: Tue, 03 Sep 2013 09:34:39 GMT * Server Apache/2.2.22 (Ubuntu) is not blacklisted < Server: Apache/2.2.22 (Ubuntu) < Accept-Ranges: bytes < Content-Length: 78 < Content-Type: application/xml < AccessDenied Of course CORS was set, but RGW can't find it, dump from log: 2013-09-03 11:34:39.876809 7f7c6dff3700 2 req 129:0.000385:s3:OPTIONS /:options_cors:verifying op mask 2013-09-03 11:34:39.876813 7f7c6dff3700 20 required_mask= 1 user.op_mask=7 2013-09-03 11:34:39.876815 7f7c6dff3700 2 req 129:0.000392:s3:OPTIONS /:options_cors:verifying op permissions 2013-09-03 11:34:39.876818 7f7c6dff3700 2 req 129:0.000395:s3:OPTIONS /:options_cors:verifying op params 2013-09-03 11:34:39.876821 7f7c6dff3700 2 req 129:0.000397:s3:OPTIONS /:options_cors:executing 2013-09-03 11:34:39.876824 7f7c6dff3700 2 No CORS configuration set yet for this bucket 2013-09-03 11:34:39.876907 7f7c6dff3700 2 req 129:0.000483:s3:OPTIONS /:options_cors:http status=403 2013-09-03 11:34:39.877551 7f7c6dff3700 1 == req done req=0x128e0e0 http_status=403 == Setting CORSfor static3 bucket: CORSConfigurationGETPUTDELETEPOST http://static3.X.pl *3000GETPUTDELETEPOSThttp://static3.X.pl/ *3000GET PUTDELETEPOST http://X.pl/*3000 2013-09-03 12:34:02.973999 7f7c44fa1700 20 get_obj_state: rctx=0x7f7c38009330 obj=.rgw:static3 state=0x7f7c38008768 s->prefetch_data=0 2013-09-03 12:34:02.974004 7f7c44fa1700 20 state->obj_tag is empty, not appending atomic test 2013-09-03 12:34:02.983719 7f7c44fa1700 10 cache put: name=.rgw+static3 2013-09-03 12:34:02.983750 7f7c44fa1700 10 moving .rgw+static3 to cache LRU end 2013-09-03 12:34:02.983756 7f7c44fa1700 10 appending xattr: name=user.rgw.cors bl.length()=194 2013-09-03 12:34:02.983801 7f7c44fa1700 10 distributing notification oid=notify.2 bl.length()=389 2013-09-03 12:34:02.986649 7f7c9bfff700 10 RGWWatcher::notify() opcode=1 ver=1 bl.length()=389 2013-09-03 12:34:02.986696 7f7c9bfff700 10 cache put: name=.rgw+static3 2013-09-03 12:34:02.986699 7f7c9bfff700 10 moving .rgw+static3 to cache LRU end 2013-09-03 12:34:02.986703 7f7c9bfff700 10 appending xattr: name=user.rgw.cors bl.length()=194 2013-09-03 12:34:02.987861 7f7c44fa1700 10 cache put: name=.rgw+static3 2013-09-03 12:34:02.987872 7f7c44fa1700 10 moving .rgw+static3 to cache LRU end 2013-09-03 12:34:02.987876 7f7c44fa1700 10 appending xattr: name=user.rgw.cors bl.length()=194 2013-09-03 12:34:02.987901 7f7c44fa1700 10 distributing notification oid=notify.2 bl.length()=389 2013-09-03 12:34:02.988593 7f7c9bfff700 10 RGWWatcher::notify() opcode=1 ver=1 bl.length()=389 2013-09-03 12:34:02.988624 7f7c9bfff700 10 cache put: name=.rgw+static3 2013-09-03 12:34:02.988627 7f7c9bfff700 10 moving .rgw+static3 to cache LRU end 2013-09-03 12:34:02.988635 7f7c9bfff700 10 appending xattr: name=user.rgw.cors bl.length()=194 2013-09-03 12:34:02.989550 7f7c44fa1700 2 req 133:0.016683:s3:PUT /:put_cors:http status=200 Checked with: 0.67.1-8-gaeb4141 (aeb414135326805324d619b507274b1d0cef7067) 0.67.2-22-ga708c8a (a708c8ab52e5b1476405a1f817c23b8845fbaab3) best regards! -- pawel On Wed, Aug 21, 2013 at 11:25 PM, Jeppesen, Nelson < nelson.jeppe...@disney.com> wrote: > Fantastic! Thank you for the quick response. I'm available if you need > any testing. > > Nelson Jeppesen >Disney Technology Solutions and Services >Phone 206-588-5001 > > -Original Message- > From: Yehuda Sadeh [mailto:yeh...@inktank.com] > Sent: Wednesday, August 21, 2013 2:22 PM > To: Jeppesen, Nelson > Cc: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] CORS not working > > On Wed, Aug 21, 2013 at 1:19 PM, Jeppesen, Nelson < > nelson.jeppe...@disney.com> wrote: > > Hello, > > > > I'm having issues with setting cors on dumpling. I seems like it's not > doing anything. > > > > I have the following CORS rule on test1 bucket: > > > > > > > > > GETPOST > > http://a.a.a > > * > > > > > > > > > > When test with the following I'm missing the Access-Control-Allow-Origin > on the response: > >Curl -H "Origin: http://a.a.a"; https://s3/test1 -vv > > > >< HTTP/1.1 200 OK > >< Date: Wed, 21 Aug 2013 20:13:31 GMT > >< Server: Apache/2.2.22 (Ubuntu) > >< Transfer-Encoding: chunked > >< Content-Type: application/xml > > > > > > > > When I try a pre-flight check I get a 403. > > > > curl -vv -X OPTIONS -H 'Access-Control-Request-Method: GET' -H > > "Origin: http://a.a.a"; https://s3/test1/a > > > > < HTTP/1.1 403 Forbidden > > < Date: Wed, 21 Aug 2013 20:16:12 GMT
[ceph-users] how to get hash-of-header-and-secret
|
hi,
I want to konw how to get the hash-of-header-and-secret, i read the
http://ceph.com/docs/master/radosgw/s3/authentication/ , but i still not
understand , I hope I can get example . if i want PUT a bucket for the user
{"user_id":"johndoe","rados_uid":0,"display_name":"John
Doe","email":"j...@example.com","suspended":0,"subusers":[],"keys":[{"user":"johndoe","access_key":"QFAMEDSJP5DEKJO0DDXY","secret_key":"iaSFLDVvDdQt6lkNzHyW4fPLZugBAI1g17LO0+87"}],"swift_keys":[]}
I use the curl -XPUT http://kp/myNewBucket -H"x-amz-acl: public-read-write"
-H"Authorization: AWS QFAMEDSJP5DEKJO0DDXY:{hash-of-header-and-secret}".
how to get hash-of-header-and-secret
|
|
| | |
|___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Best way to reformat OSD drives?
> Why wait for the data to migrate away? Normally you have replicas of the > whole osd data, so you can simply stop the osd, reformat the disk and restart > it again. It'll join the cluster and automatically get all data it's missing. > Of course the risk of dataloss is a bit higher during that time, but normally > that should be ok, because it's not different from an ordinary disk failure > which can happen any time. > > I just found a similar question from one year ago: > http://www.spinics.net/lists/ceph-devel/msg05915.html I didn't read the whole > thread, but probably you can find some other ideas there. > > service ceph osd stop $OSD > mkfs -t xfs /dev/XXX > ceph-osd -i $OSD --mkfs --mkkey --mkjournal > service ceph osd start $OSD this is what I did now: ceph osd set noout service ceph stop osd.X umount /dev/sdX1 mkfs.xfs -f -i size=2048 /dev/sdX1 -L osd.X vim /etc/fstab # edit line for /dev/sdX1 mount /dev/sdX1 ceph-osd -i X --mkfs --mkkey --mkjournal ceph auth add osd.X osd 'allow *' mon 'allow rwx' -i /var/lib/ceph/osd/ceph-X/keyring service ceph start osd.X seems to work so far, the OSD is busy retrieving data - and I didn't have to wait for the OSD to become empty. cheers jc___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
