Re: [ceph-users] Ceph incomplete pg

Are there any docs on how I can repair the inconsistent pgs? Or any thoughts on the crash of OSD? Thanks! From: Jeppesen, Nelson Sent: Thursday, December 12, 2013 10:58 PM To: 'ceph-users@lists.ceph.com' Subject: Ceph incomplete pg I have an issue with incomplete pgs, I've tri

[ceph-users] Ceph incomplete pg

I have an issue with incomplete pgs, I've tried repairing it but no such luck. Any ideas what to check? Output from 'ceph health detail' HEALTH_ERR 2 pgs inconsistent; 1 pgs recovering; 1 pgs stuck unclean; recovery 15/863113 degraded (0.002%); 5/287707 unfound (0.002%); 4 scrub errors pg 22.ee

Re: [ceph-users] How to use Admin Ops API in Ceph Object Storage

After further investigation I have noticed I can pull info on ANY user with 'GET /admin/user?user=user1' but cannot enumerate users with 'GET /admin/user' Nelson Jeppesen Disney Technology Solutions and Services Phone 206-588-5001 _

Re: [ceph-users] How to use Admin Ops API in Ceph Object Storage

Looks like the thread was striped, let me add it back in: -Mensaje original- De: Yehuda Sadeh [mailto:yehuda at inktank.com] Enviado el: miércoles, 10 de julio de 2013 16:50 Para: Alvaro Izquierdo Jimeno CC: Bright; ceph-users

Re: [ceph-users] How to use Admin Ops API in Ceph Object Storage

Any update on this issue? I'm running into the same problem, I can get usage information but I get 403s when pulling user data, even with user=* caps. Thanks! Nelson Jeppesen Disney Technology Solutions and Services Phone 206-588-5001

Re: [ceph-users] Index document for radosgw buckets?

hnology Solutions and Services Phone 206-588-5001 -Original Message- From: Gregory Farnum [mailto:g...@inktank.com] Sent: Wednesday, September 18, 2013 10:59 AM To: Jeppesen, Nelson Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Index document for radosgw buckets? What do you mean

[ceph-users] Index document for radosgw buckets?

Is there a way to enable index documents for radosgw buckets? If not, is that on the roadmap? I've looked around but have not seen anything. Thanks! Nelson Jeppesen Disney Technology Solutions and Services Phone 206-588-5001 ___ ceph-users mailin

Re: [ceph-users] CORS not working

pesen, Nelson Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] CORS not working On Wed, Aug 21, 2013 at 1:19 PM, Jeppesen, Nelson wrote: > Hello, > > I'm having issues with setting cors on dumpling. I seems like it's not doing > anything. > > I have the fo

[ceph-users] CORS not working

Hello, I'm having issues with setting cors on dumpling. I seems like it's not doing anything. I have the following CORS rule on test1 bucket: GETPOST http://a.a.a * When test with the following I'm missing t

[ceph-users] paxos is_readable spam on monitors?

Hello Ceph-users, Running dumping (upgraded yesterday) and several hours after the upgrade the following type of message repeated over and over in logs. Started about 8 hours ago. 1 mon.1@0(leader).paxos(paxos active c 6005920..6006535) is_readable now=2013-08-16 14:35:53.351282 lease_expire=

Re: [ceph-users] v0.67 Dumpling released

Sage et al, This is an exciting release but I must say I'm a bit confused about some of the new rgw details. Questions: 1) I'd like to understand how regions work. I assume that's how you get multi-site, multi-datacenter support working but must they be part of the same ceph cluster still?

Re: [ceph-users] Why is my mon store.db is 220GB?

Success! It was pretty quick too, maybe 20-30min. It’s now at 100MB. In a matter of min I was able to add two monitors and now I’m back to three monitors. Thank you again, Joao and Sage! I can sleep at night now knowing that a single node won't take down the cluster anymore ☺ __

Re: [ceph-users] Why is my mon store.db is 220GB?

Joao, ceph-monstore-tool --mon-store-path /var/lib/ceph/mon/ceph-2 --out /var/lib/ceph/mon/ceph-1 --command store-copy is running now. It hit 52MB very quickly then nothing with lots of disk read, which is what I'd expect. Its reading fast and expect it to finish in 35min. Just to make sure, t

Re: [ceph-users] Why is my mon store.db is 220GB?

Never mind, I removed --with-rest-bench and it worked. > I built the wip-monstore-copy branch with './configure --with-rest-bench > --with-debug' and 'make'. It worked and I get all the usual stuff but ceph-> > monstore-tool is missing. I see code in ./src/tools/. Did I miss something? _

Re: [ceph-users] Why is my mon store.db is 220GB?

I built the wip-monstore-copy branch with './configure --with-rest-bench --with-debug' and 'make'. It worked and I get all the usual stuff but ceph-monstore-tool is missing. I see code in ./src/tools/. Did I miss something? ___ ceph-users mailing list c

Re: [ceph-users] Why is my mon store.db is 220GB?

Is there an easy way I can find the age and/or expiration of the service ticket on a particular osd? Is that a file or just kept in ram? -Original Message- From: Sage Weil [mailto:s...@inktank.com] Sent: Tuesday, August 13, 2013 9:01 AM To: Jeppesen, Nelson Cc: ceph-users

Re: [ceph-users] Why is my mon store.db is 220GB?

Thanks Joao, Is there a doc somewhere on the dependencies? I assume I'll need to setup the tool chain to compile? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Why is my mon store.db is 220GB?

Interesting, So if I change ' auth service ticket ttl' to 172,800, in theory I could go without a monitor for 48 hours? -Original Message- From: Sage Weil [mailto:s...@inktank.com] Sent: Monday, August 12, 2013 9:50 PM To: Jeppesen, Nelson Cc: ceph-users@lists.ceph.com S

Re: [ceph-users] Why is my mon store.db is 220GB?

Joao, (log file uploaded to http://pastebin.com/Ufrxn6fZ) I had some good luck and some bad luck. I copied the store.db to a new monitor, injected a modified monmap and started it up (This is all on the same host.) Very quickly it reached quorum (as far as I can tell) but didn't respond. Runn

Re: [ceph-users] Why is my mon store.db is 220GB?

Joao, Have you had a chance to look at my monitor issues? I Ran ''ceph-mon -i FOO -compact' last week but it did not improve disk usage. Let me know if there's anything else I dig up. The monitor still at 0.67-rc2 with the OSDs at .0.61.7. On 08/02/2013 12:15 AM, Je

Re: [ceph-users] Why is my mon store.db is 220GB?

Thank you Joao, I'll get you any information you need. I can tell you that I've restarted the mon few times and it does seem to change disk usage. I just ran 'ceph-mon -i 2 --compact' on my monitor, see how that looks in the morning. On 08/02/2013 12:15 AM, Jeppesen, Ne

Re: [ceph-users] Why is my mon store.db is 220GB?

son [mailto:mike.daw...@cloudapt.com] Sent: Thursday, August 01, 2013 4:10 PM To: Jeppesen, Nelson Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Why is my mon store.db is 220GB? 220GB is way, way too big. I suspect your monitors need to go through a successful leveldb compaction. The early re

[ceph-users] Why is my mon store.db is 220GB?

My Mon store.db has been at 220GB for a few months now. Why is this and how can I fix it? I have one monitor in this cluster and I suspect that I can't add monitors to the cluster because it is too big. Thank you. ___ ceph-users mailing list ceph-user

Re: [ceph-users] Issues going from 1 to 3 mons

m -rf /var/lib/ceph/mon/ceph-3 sudo mkdir /var/lib/ceph/mon/ceph-3 ceph auth get mon. -o /tmp/auth ceph mon getmap -o /tmp/map sudo ceph-mon -i 3 --mkfs --monmap /tmp/map --keyring /tmp/auth ceph mon add 10.198.141.203:6789 ceph-mon -i 3 --public-addr 10.198.141.203:6789 From: Jeppesen, Nelson Se

Re: [ceph-users] Issues going from 1 to 3 mons

rbich...@risc-software.at] Sent: Wednesday, July 10, 2013 3:30 AM To: Jeppesen, Nelson Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Issues going from 1 to 3 mons Sorry, no updates on my side. My wife got our second baby and I'm busy with reality (changing nappies and stuff) -- Se

Re: [ceph-users] Issues going from 1 to 3 mons

Any updates on this? My production cluster has been running on one monitor for a while and I'm a little nervous. Can I expect a fix in 0.61.5? Thank you. > (Re-adding the list for future reference) > > Wolfgang, from your log file: > > 2013-06-25 14:58:39.739392 7fa329698780 -1 common/

Re: [ceph-users] Issues going from 1 to 3 mons

onmap /tmp/map --keyring /tmp/auth 4. ceph mon add 1 [:] 5. ceph-mon -i 1 --public-addr {ip:port} Thank you. From: Gregory Farnum [mailto:g...@inktank.com] Sent: Sunday, June 23, 2013 12:59 PM To: Jeppesen, Nelson Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Issues going from 1 to

[ceph-users] Issues going from 1 to 3 mons

Hello, I have cluster that only has one monitor running but I loose quorum when I try add a second monitor. I'm trying to raise the cluster from 1 to 3 monitors. I think it breaks when I run 'ceph mon add [:]' because it loses quorum before the new monitor is online. Thanks.

Re: [ceph-users] Upgrading to .61.3 broke monitors

Resending smaller lsit so it's not held up by body size too big: 2013-06-10 14:36:38.901182 7f8dccebf700 5 mon.1@0(probing) e1 waitlisting message auth(proto 0 27 bytes epoch 1) v1 2013-06-10 14:36:38.901184 7f8dccebf700 20 mon.1@0(probing) e1

Re: [ceph-users] Upgrading to .61.3 broke monitors

I noticed that the mon dir is 500+ gigs big on all three monitors. Do you think this might cause an issue? Cpu usage is about 12%. # du -sh /var/lib/ceph/mon/ceph-2/ 545G/var/lib/ceph/mon/ceph-2/ Nelson Jeppesen Disney Technology Solutions and Services Phone 206-588-5001 _

Re: [ceph-users] Upgrading to .61.3 broke monitors

It's odd, the cluster is seems to be working somewhat. Cant bring down OSDs online, but the un-restarted nodes still work. Ceph -w hangs ceph --admin-daemon /var/run/ceph/ceph-mon.FOO.asok mon_status hangs and nothing in /var/log/ceph/* Mon03 output 57+0+0 (2052948678 0 0) 0x3fa7240 con 0x

Re: [ceph-users] Upgrading to .61.3 broke monitors

I was able to catch on the crashes pn 951901 opn 0) v3 982012+0+0 (3546980463 0 0) 0x5e2ea00 con 0x3701420 -3> 2013-06-10 15:45:47.944085 7f27a5038700 20 mon.3@2(peon) e1 have connection -2> 2013-06-10 15:45:47.944092 7f27a5038700 20 mon.3@2(peon) e1 ms_dispatch existing session Mon

Re: [ceph-users] Upgrading to .61.3 broke monitors

Downgraded nodes to .61.3: Mon2 logs: 2013-06-10 15:36:25.175309 7f7a2686e700 1 mon.2@1(probing) e1 discarding message auth(proto 0 27 bytes epoch 1) v1 and sending client elsewhere 2013-06-10 15:36:25.175316 7f7a2686e700 1 mon.2@1(probing) e1 discarding message auth(proto 0 27 bytes epoch 1)

Re: [ceph-users] Upgrading to .61.3 broke monitors

Forgot to stop the service Mon3 2013-06-10 15:05:27.245156 7f9898aad700 11 mon.3@2(electing) e1 tick 2013-06-10 15:05:28.289125 7f9898aad700 5 mon.3@2(electing).elector(19067) election timer expired 2013-06-10 15:05:28.289140 7f9898aad700 5 mon.3@2(electing).elector(19067) start -- can i be le

Re: [ceph-users] Upgrading to .61.3 broke monitors

I tired upgrading this node to .63, thinking it was a .61.3 bug. ceph-mon -i 3 -d 2>&1 | tee -a /tmp/ceph-FOO.log 2013-06-10 14:59:46.033781 7ff4a9324780 0 ceph version 0.63 (054e96cf79e960894ef7e33a4d13635d3ad2a1b9), process ceph-mon, pid 25092 2013-06-10 14:59:46.036590 7ff4a9324780 10 needs_c

Re: [ceph-users] Upgrading to .61.3 broke monitors

So mon01 is logging and mon_status returns data. Mon2 and mon3 do not log (even with debug) ceph -admini-deamon mon_status hangs. It's interesting that the disk IO for mon2 and mon3 are very high: Mon1 Device:tpskB_read/skB_wrtn/skB_readkB_wrtn sda

Re: [ceph-users] Upgrading to .61.3 broke monitors

Even with debug, mon2 and mon3 have nothing the log dir. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Upgrading to .61.3 broke monitors

Resending smaller lsit so it's not held up by body size too big: 2013-06-10 14:36:38.901297 7f8dccebf700 5 mon.1@0(probing) e1 waitlisting message auth(proto 0 27 bytes epoch 1) v1 2013-06-10 14:36:38.901299 7f8dccebf700 20 mon.1@0(probing) e1

Re: [ceph-users] Upgrading to .61.3 broke monitors

Sorry sent to wrong thread If I restart the monitors it selects a leader, but only for a few seconds. var/log/ceph# ceph --admin-daemon /var/run/ceph/ceph-mon.1.asok mon_status { "name": "1", "rank": 0, "state": "leader", "election_epoch": 19044, "quorum": [ 0, 2], "outs

Re: [ceph-users] Mon store.db size

This is the output from the first node, the other 2 nodes hang. { "name": "1", "rank": 0, "state": "probing", "election_epoch": 19038, "quorum": [], "outside_quorum": [ "1"], "extra_probe_peers": [], "monmap": { "epoch": 1, "fsid": "c5344dc8-b390-420a-bc1c-8b3ba4d9d5eb"

[ceph-users] Upgrading to .61.3 broke monitors

I upgrade my monitors to 61.3 (from 61.2) and the monitors have stopped working. They start but everything hangs. I can't find any errors. - Nothing in the monitor log files - Ceph -w and ceph status just hangs (no stdout or stderr) - OSDs currently running work but

Re: [ceph-users] Help! 61.1 killed my monitors in prod

Thank you, you saved my bacon. I didn't inject the new map properly, the monitor is going nuts but it's recovering. I wonder if I was hit by the .61 race condition. How can I verify that the monitor has upgraded to the 'new' .61 style that uses a single paxos? Thanks. Nelson Jeppesen _

[ceph-users] Help! 61.1 killed my monitors in prod

After upgrading my cluster everything looked good, then I rebooted the farm and all hell broke loose. I have 3 monitors but none are able to start. On all of them the '/usr/bin/python /usr/sbin/ceph-create-keys' command is hanging because none of the nodes can accept quorum. All ceph tools a

Re: [ceph-users] Rados Gateway Pools

n! Nelson Jeppesen -Original Message- From: yehud...@gmail.com [mailto:yehud...@gmail.com] On Behalf Of Yehuda Sadeh Sent: Tuesday, May 07, 2013 3:14 PM To: Jeppesen, Nelson Cc: Yehuda Sadeh; ceph-users@lists.ceph.com Subject: Re: [ceph-users] Rados Gateway Pools On Tue, May 7, 2013 at 2:54

Re: [ceph-users] Rados Gateway Pools

lf Of Yehuda Sadeh Sent: Tuesday, May 07, 2013 2:46 PM To: Jeppesen, Nelson Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Rados Gateway Pools On Tue, May 7, 2013 at 2:26 PM, Jeppesen, Nelson wrote: > Now the .61 is out I have tried getting a second radosgw farm working > but into

Re: [ceph-users] Rados Gateway Pools

Now the .61 is out I have tried getting a second radosgw farm working but into an issue using a custom root/zone pool. The 'radosgw-admin zone set' and ' radosgw-admin zone info' commands are working fine except it keeps defaulting to using .rgw.root. I've tried the two settings, the one you g

[ceph-users] Why would you update 'rgw usage max user shards' setting?

Why would you update 'rgw usage max user shards' setting? I don't really understand what it's for. Thank you. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph -w question

chnology Solutions and Services Phone 206-588-5001 -Original Message- From: Samuel Just [mailto:sam.j...@inktank.com] Sent: Monday, April 15, 2013 10:11 AM To: Jeppesen, Nelson Cc: Gregory Farnum; ceph-users@lists.ceph.com Subject: Re: [ceph-users] ceph -w question Also the ceph osdmap. (c

Re: [ceph-users] ceph -w question

er 18446744073709551615 pool 21 '.users.swift' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 1138 owner 0 Nelson Jeppesen Disney Technology Solutions and Services Phone 206-588-5001 -Original Message- From: Gregory Farnum [mai

[ceph-users] ceph -w question

When I reboot any node in my prod environment with no activity I see incomplete pgs. Is that a concern? Does that mean some data is unavailable? Thank you. # ceph -v ceph version 0.56.4 (63b0f854d1cef490624de5d6cf9039735c7de5ca) # ceph -w 2013-04-15 08:57:27.712065 mon.0 [INF] pgmap v585220: 486

[ceph-users] 2 days to delete a rgw bucket?

I have a large bucket (about million objects) and it takes a few days to delete it. Watching ceph -w, I only see 8 to 30 op/s. What's going on? Thanks. The command: radosgw-admin bucket rm --bucket=testbucket --purge-objects ___ ceph-users mailing list

Re: [ceph-users] More than one rados gateway per ceph cluster?

Thank you Sedeh, that's great to hear! So some of the zone params would be log pool, user pool etc.? -Original Message- From: Yehuda Sadeh [mailto:yeh...@inktank.com] Sent: Friday, March 22, 2013 11:09 AM To: Jeppesen, Nelson Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users]

[ceph-users] More than one rados gateway per ceph cluster?

Is it possible to have two independent rados gw farms in one Ceph environment? I can't figure out how to get two rados gateway servers use different pools. Basically I want two rados gateway farms (internal, external) that's hosted by one ceph environment. Thanks! ___

[ceph-users] Bad drive caused radosgw to timeout with http 500s

Hello Ceph-Users, I was testing our rados gateway and after a few hours rgw started sending http 500 responses for certain uploads. I did some digging and found that a HDD died. The OSD was marked out, but not after a short rgw outage. Start to finish was 60 to 120 seconds. I have a few questi

[ceph-users] btrfs for prod in with 3.8 kernel?

Ubuntu 13.04 will be using a 3.8 kernel. Do you guys think that btrfs is production ready for Ceph in linux 3.8? or would it be safer to use Ubuntu 12.04 and upgrade the kernel to 3.8? or even stick with XFS. Thanks. Nelson Jeppesen Disney Technology Solutions and Services Phone 206-588-5