Re: [ceph-users] 答复: 答复: 答复: 答复: 答复: how to transfer ceph cluster from the old network-and-hosts to a new one

Hi! To have all PGs in active+clean state you need to have enough nodes and OSDs to hold all PG copies (depends on you pools size). If your pools have size 3 (by default) then you need 3 nodes with enough OSDs\space alive. If you want to migrate from the old hardware to a new then I would rec

Re: [ceph-users] blind buckets

Hi, You can modify zone like this: "placement_pools": [ { "key": "default-placement", "val": { "index_pool": “test.rgw.buckets.index", "data_pool": “test.rgw.buckets.data", "data_extra_pool": “test.buckets.non-ec"

Re: [ceph-users] Unknown error (95->500) when creating buckets or putting files to RGW after upgrade from Infernalis to Jewel

Hi Maciej, slowly I am running out of ideas :-) Could you send the output of 'ceph osd lspools' so that I can compare your pools with ours? Maybe someone else got similiar problems and can help? Cheers Nick On Tuesday, July 26, 2016 03:56:39 PM Naruszewicz, Maciej wrote: > Unfortunately none of

Re: [ceph-users] syslog broke my cluster

On Tue, Jul 26, 2016 at 03:48:33PM +0100, Sergio A. de Carvalho Jr. wrote: > As per my previous messages on the list, I was having a strange problem in > my test cluster (Hammer 0.94.6, CentOS 6.5) where my monitors were > literally crawling to a halt, preventing them to ever reach quorum and > cau

[ceph-users] 答复: 答复: 答复: 答复: 答复: how to transfer ceph cluster from the old network-and-hosts to a new one

@Владимир I'll try that thanks. Now when I remove the old OSD, pg remapped but stuck. http://docs.ceph.com/docs/jewel/rados/operations/add-or-rm-osds/ describes this as a "CRUSH corner case where some PGs remain stuck in the active+remapped state". I have tried: ceph osd crush reweight osd.8

Re: [ceph-users] bluestore overlay write failure

Bluestore has evolved a long way and I don’t think we support this overlay anymore. Please try Bluestore with latest master.. Thanks & Regards Somnath From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of ??? Sent: Tuesday, July 26, 2016 7:09 PM To: ceph-users@lists.ceph.com S

[ceph-users] bluestore overlay write failure

Hi All I'm using ceph-10.1.1, when I open overlay write, some OSDs will down and out when I use fio to test 4K IO write of rbd.The default option is below :OPTION(bluestore_overlay_max, OPT_INT, 0) I change the 0 to 512 to make data write that small than 64K processed by overlay, then some OSD

Re: [ceph-users] Ceph performance pattern

Hi epk, Which ioengine are you using? if it's librbd, you might try playing with librbd readahead as well: # don't disable readahead after a certain number of bytes rbd readahead disable after bytes = 0 # Set the librbd readahead to whatever: rbd readahead max bytes = 4194304 If it's with k

Re: [ceph-users] Ceph performance calculator

how did you deploy ceph jewel on debian7? 2016-07-26 1:08 GMT+08:00 Mark Nelson : > Several years ago Mark Kampe proposed doing something like this. I was > never totally convinced we could make something accurate enough quickly > enough for it to be useful. > > If I were to attempt it, I would

Re: [ceph-users] newly osds dying (jewel 10.2.2)

Hi cephers. I think this is solved. The issue is because of puppet. and the new interface naming of centos7. In our puppet configs we defined an iptable module which restricts access to the private ceph network based on src and on destiny interface. We had eth1 hardwired and in this new serv

Re: [ceph-users] Ceph performance pattern

<< Ceph performance in general (without read_ahead_kb) will be lower specially in all flash as the requests will be serialized within a PG I meant to say Ceph sequential performance..Sorry for the spam.. From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Somnath Roy Sent:

Re: [ceph-users] Ceph performance pattern

Not exactly, but, we are seeing some drop with 256K compare to 64K. This is with random reads though in Ubuntu. We had to bump up read_ahead_kb from default 128KB to 512KB to work around that. But, in RHEL we saw all sorts of issues with read_ahead_kb for small block random reads and I think it

Re: [ceph-users] Monitors not reaching quorum

Agreed. When I first had these problems, random stuff would just not work. SSH would take a while to log in, DNS server would process requests slow, our Batch system would freeze and not run jobs. It's now one of my first things to check when services are running weirdly. My failsafe check is to d

Re: [ceph-users] Ceph performance pattern

Thanks Somnath. I am running with CentOS7.2. Have you seen this pattern before? - epk From: Somnath Roy [mailto:somnath@sandisk.com] Sent: Tuesday, July 26, 2016 4:44 PM To: EP Komarla ; ceph-users@lists.ceph.com Subject: RE: Ceph performance pattern Which OS/kernel you are running with ?

Re: [ceph-users] Ceph performance pattern

Which OS/kernel you are running with ? Try setting bigger read_ahead_kb for sequential runs. Thanks & Regards Somnath From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of EP Komarla Sent: Tuesday, July 26, 2016 4:38 PM To: ceph-users@lists.ceph.com Subject: [ceph-users] Ceph

[ceph-users] Ceph performance pattern

Hi, I am showing below fio results for Sequential Read on my Ceph cluster. I am trying to understand this pattern: - why there is a dip in the performance for block sizes 32k-256k? - is this an expected performance graph? - have you seen this kind of pattern before [cid:image001.png@01D1E75C.2

Re: [ceph-users] mon_osd_nearfull_ratio (unchangeable) ?

On Tue, Jul 26, 2016 at 09:37:37AM +0200, Dan van der Ster wrote: > On Tue, Jul 26, 2016 at 3:52 AM, Brad Hubbard wrote: > >> 1./ if I try to change mon_osd_nearfull_ratio from 0.85 to 0.90, I get > >> > >># ceph tell mon.* injectargs "--mon_osd_nearfull_ratio 0.90" > >>mon.rccephmon1: inj

[ceph-users] newly osds dying (jewel 10.2.2)

Hi cephers... Our production cluster is running Jewel 10.2.2. We were running a production cluster with 8 servers each with 8 osds making a gran total of 64 osds. Each server also hosts 2 ssds for journals. Each sshd supports 4 journals. We had 1/3 of our osds above 80% occupied, and we decid

Re: [ceph-users] How to get Active set of OSD Map in serial order of osd index

Why do you want them in serial increasing order? -Sam On Tue, Jul 26, 2016 at 2:43 PM, Samuel Just wrote: > How would such a code work if there were more than 24 osds? > -Sam > > On Tue, Jul 26, 2016 at 2:37 PM, Syed Hussain wrote: > >> Hi, >> >> I'm working to develop an Erasure Code plugin (v

Re: [ceph-users] How to get Active set of OSD Map in serial order of osd index

How would such a code work if there were more than 24 osds? -Sam On Tue, Jul 26, 2016 at 2:37 PM, Syed Hussain wrote: > Hi, > > I'm working to develop an Erasure Code plugin (variation of ISA) that have > typical requirement that the active set of the Erasure Coded pool in serial > order. > For

[ceph-users] How to get Active set of OSD Map in serial order of osd index

Hi, I'm working to develop an Erasure Code plugin (variation of ISA) that have typical requirement that the active set of the Erasure Coded pool in serial order. For example, >ceph osd erasure-code

Re: [ceph-users] cephfs - mds hardware recommendation for 40 million files and 500 users

On Tue, Jul 26, 2016 at 1:53 PM, Mike Miller wrote: > Hi, > > we have started to migrate user homes to cephfs with the mds server 32GB > RAM. With multiple rsync threads copying this seems to be undersized; the > mds process consumes all memory 32GB fitting about 4 million caps. > > Any hardware r

[ceph-users] cephfs - mds hardware recommendation for 40 million files and 500 users

Hi, we have started to migrate user homes to cephfs with the mds server 32GB RAM. With multiple rsync threads copying this seems to be undersized; the mds process consumes all memory 32GB fitting about 4 million caps. Any hardware recommendation for about 40 million files and about 500 users?

[ceph-users] Vote for OpenStack Talks!

Hey cephers, It seems that direct links to specific OpenStack talks have been disabled for the voting period this summit. So, please take a walk through the search interface and pick out your favorite Ceph talks to promote! https://www.openstack.org/summit/barcelona-2016/vote-for-speakers/ There

Re: [ceph-users] Monitors not reaching quorum

The funny thing is that I just restarted the rsyslog daemon on the Ceph hosts and I can now re-enable syslog for Ceph without any issues. It just looks like the rsyslog service had a hiccup, possibly related to problem on one of the central syslog servers, and this in turn prevent the monitors to o

Re: [ceph-users] Monitors not reaching quorum

On 07/26/2016 06:27 PM, Sergio A. de Carvalho Jr. wrote: (Just realised I originally replied to Sean directly, so reposting here for posterity). Bingo! wow. This didn't even cross my mind. D: Thanks for sharing. I turned off syslog and the monitors quickly reached quorum and everything see

[ceph-users] blind buckets

Hi there, We are looking at using Ceph (Jewel) for a use case that is very write heavy strictly as an object store. We've been working with Rados Gateway because we can easily integrate with existing S3 libraries... but we will never be doing any of the bucket listing operations. I am concerned

Re: [ceph-users] Monitors not reaching quorum

(Just realised I originally replied to Sean directly, so reposting here for posterity). Bingo! I turned off syslog and the monitors quickly reached quorum and everything seems back to normal. Thanks so much, Sean. Luckily this is a test cluster. I wonder how I could catch this in a production cl

Re: [ceph-users] Backfilling pgs not making progress

Hmm, nvm, it's not an lfn object anyway. -Sam On Tue, Jul 26, 2016 at 7:07 AM, Brian Felton wrote: > If I search on osd.580, I find > default.421929.15\uTEPP\s84316222-6ddd-4ac9-8283-6fa1cdcf9b88\sbackups\s20160630091353\sp1\s\sShares\sWarehouse\sLondonWarehouse\sLondon\sRon > picture's\sMISCELLA

Re: [ceph-users] Unknown error (95->500) when creating buckets or putting files to RGW after upgrade from Infernalis to Jewel

Unfortunately none of our pools are erasure-code pools - I just double-checked that. I found another issue with deleting (I only can't create buckets or upload files, get/delete work fine) which looks almost identically http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-July/003100.html

[ceph-users] syslog broke my cluster

As per my previous messages on the list, I was having a strange problem in my test cluster (Hammer 0.94.6, CentOS 6.5) where my monitors were literally crawling to a halt, preventing them to ever reach quorum and causing all sort of problems. As it turned out, to my surprise everything went back to

Re: [ceph-users] Recovery stuck after adjusting to recent tunables

I will catch up with the issues in the ML and hopefully with the code. Yes the 2 nodes are very different from the other 2, we are in the middle of restructuring this cluster thus the irregularity. Thanks a lot Dan On 26 July 2016 at 15:25, Dan van der Ster wrote: > Cool, glad that worked. You'

Re: [ceph-users] ceph + vmware

On Thursday, July 21, 2016, Mike Christie wrote: > On 07/21/2016 11:41 AM, Mike Christie wrote: > > On 07/20/2016 02:20 PM, Jake Young wrote: > >> > >> For starters, STGT doesn't implement VAAI properly and you will need to > >> disable VAAI in ESXi. > >> > >> LIO does seem to implement VAAI prop

Re: [ceph-users] Backfilling pgs not making progress

If I search on osd.580, I find default.421929.15\uTEPP\s84316222-6ddd-4ac9-8283-6fa1cdcf9b88\sbackups\s20160630091353\sp1\s\sShares\sWarehouse\sLondonWarehouse\sLondon\sRon picture's\sMISCELLANEOUS\s2014\sOct., 2014\sOct. 1\sDSC04329.JPG__head_981926C1__21__5, which has a non-zero s

Re: [ceph-users] Backfilling pgs not making progress

Did you also confirm that the backfill target does not have any of those dangling links? I'd be looking for a dangling link for 981926c1/default.421929.15_TEPP/84316222-6ddd-4ac9-8283-6fa1cdcf9b88/backups/20160630091353/p1//Shares/Warehouse/LondonWarehouse/London/Ron picture's/MISCELLANEOUS/2014/O

Re: [ceph-users] Recovery stuck after adjusting to recent tunables

Cool, glad that worked. You'll have to read backwards in the ML to find this discussed -- though it is rarely needed, therefore rarely discussed. For code, it's used in src/crush/mapper.c. Most clusters, irrespective of size, work with 50 tries. Clusters that need more than 50 tries usually have s

Re: [ceph-users] Monitors not reaching quorum

On 07/26/2016 12:13 PM, Sergio A. de Carvalho Jr. wrote: I left the 4 nodes running overnight and they just crawled to their knees... to the point that nothing has been written to the logs in the last 11 hours. So I stopped all monitors this morning and started them one by one again, but they're

Re: [ceph-users] Problem with RGW after update to Jewel

radosgw-admin --cluster=pbs zonegroup-map get { "zonegroups": [], "master_zonegroup": "", "bucket_quota": { "enabled": false, "max_size_kb": -1, "max_objects": -1 }, "user_quota": { "enabled": false, "max_size_kb": -1, "max_objects

Re: [ceph-users] Recovery stuck after adjusting to recent tunables

Hello Dan, I increased choose_local_tries to 75 and the misplaced objects reduced to 286. One more increase to 100 to get 141 misplaced objects and one more to 125 for the cluster to fully recover! I also verified that I can now down + out an OSD and the cluster will also fully recover. My problem

Re: [ceph-users] Problem with RGW after update to Jewel

can you get the print out of radosgw-admin zonegroupmap? and radosgw-admin zonegroup get --rgw-zonegroup=default On Tue, Jul 26, 2016 at 12:36 PM, Frank Enderle wrote: > ok - i did now the following: > > radosgw-admin --cluster=pbs realm create --rgw-realm=pbs --default > 2016-07-26 10:34:15.2164

Re: [ceph-users] Monitors not reaching quorum

I left the 4 nodes running overnight and they just crawled to their knees... to the point that nothing has been written to the logs in the last 11 hours. So I stopped all monitors this morning and started them one by one again, but they're are still being extremely slow. Here are their logs:

Re: [ceph-users] Problem with RGW after update to Jewel

ok - i did now the following: radosgw-admin --cluster=pbs realm create --rgw-realm=pbs --default 2016-07-26 10:34:15.216404 7fdf346bc9c0 0 error read_lastest_epoch .rgw.root:periods.d94c5208-fc1f-4e02-9773-bc709e4d8a34.latest_epoch { "id": "98089a5c-6c61-4cc2-a5d8-fce0cb0a9704", "name":

Re: [ceph-users] Problem with RGW after update to Jewel

it doesn't matter, you can call it gold like in the documentation On Tue, Jul 26, 2016 at 12:15 PM, Frank Enderle wrote: > What should I choose for realm name? I never selected one - does it matter > what I put there? > > -- > > anamica GmbH > Heppacher Str. 39 > 71404 Korb > > Telefon: +49 715

Re: [ceph-users] Problem with RGW after update to Jewel

What should I choose for realm name? I never selected one - does it matter what I put there? -- anamica GmbH Heppacher Str. 39 71404 Korb Telefon: +49 7151 1351565 0 Telefax: +49 7151 1351565 9 E-Mail: frank.ende...@anamica.de Internet: www.anamica.de Handelsregister: AG Stuttgart HRB 73235

Re: [ceph-users] Problem with RGW after update to Jewel

Lets try: radosgw-admin realm create --rgw-realm= --default radosgw-admin zonegroup set --rgw-zonegroup=default < json radosgw-admin period update --commit In the next jewel release the upgrade will be smoother. Orit On Tue, Jul 26, 2016 at 11:34 AM, Frank Enderle wrote: > Yes! that worked

Re: [ceph-users] 答复: 答复: 答复: 答复: how to transfer ceph cluster from the old network-and-hosts to a new one

Hello! As far as I know, 'admin node' is just a node with ceph-deploy and initial config/keys directory (if I'm wrong somebody will corrent me, I hope). So you need just to install ceph-deploy (if you are going to use it futher) and move the ceph user's cluster config/keys directory to a new nod

Re: [ceph-users] Recovery stuck after adjusting to recent tunables

Hi, Starting from the beginning... If a 3-replica PG gets stuck with only 2 replicas after changing tunables, it's probably a case where choose_total_tries is too low for your cluster configuration. Try increasing choose_total_tries from 50 to 75. -- Dan On Fri, Jul 22, 2016 at 4:17 PM, Kosti

Re: [ceph-users] Problem with RGW after update to Jewel

Yes! that worked :-) now I changed the master_zone to default like so: { "id": "default", "name": "default", "api_name": "", "is_master": "true", "endpoints": [], "hostnames": [ "***REDACTED***", "***REDACTED***", "***REDACTED***" ], "hostna

Re: [ceph-users] cephfs failed to rdlock, waiting

On Tue, Jul 26, 2016 at 4:30 AM, Oliver Dzombic wrote: > Hi Greg, > > i switched the cache tier to forward, and began to evit everything. > > I restarted the mds, it was switching to another node. > > Still the same issue... > > So how can it be a pg full issue this way ? Have a look at "ceph dae

Re: [ceph-users] Problem with RGW after update to Jewel

does adding --rgw-zonegroup=default helps? On Tue, Jul 26, 2016 at 11:09 AM, Frank Enderle wrote: > I get this error when I try to execute the command: > > radosgw-admin --cluster=pbs zonegroup get > failed to init zonegroup: (2) No such file or directory > > also with > > radosgw-admin --cluster

Re: [ceph-users] Recovery stuck after adjusting to recent tunables

Hi Brad, the cluster recover to about 0.012% after switching to firefly tunables (got stuck again with 1 PG remapped) and after that I increased the pg_num/pgp_num from 128 to 256 to 512, and the status is getting worse: more PGs are getting stuck at a remapped state, so I don't see a reason to ke

Re: [ceph-users] Problem with RGW after update to Jewel

I get this error when I try to execute the command: radosgw-admin --cluster=pbs zonegroup get failed to init zonegroup: (2) No such file or directory also with radosgw-admin --cluster=pbs zonegroup get --rgw-zone=default failed to init zonegroup: (2) No such file or directory -- anamica GmbH

Re: [ceph-users] mon_osd_nearfull_ratio (unchangeable) ?

Try: ceph pg set_nearfull_ratio 0.9 On 26 Jul 2016 08:16, "Goncalo Borges" wrote: > Hello... > > I do not think that these settings are working properly in jewel. Maybe > someone else can confirm. > > So, to summarize: > > 1./ I've restarted mon and osd services (systemctl restart ceph.target)

[ceph-users] 答复: 答复: 答复: 答复: how to transfer ceph cluster from the old network-and-hosts to a new one

Thanks! the cluster becomes active+clean again. Basically this proves OSD and MON could be transferred using rebalance. What about the admin node? After adding new OSD and MON, shutdown old OSD and MON, should I also set up a new admin and "turn off" the new one?

Re: [ceph-users] Problem with RGW after update to Jewel

you need to set the default zone as master zone. you can try: radosgw-admin zonegroup set < zg.json where the json is the json return from radosgw-admin zonegroup get with master_zone field set to "default" Orit On Mon, Jul 25, 2016 at 11:17 PM, Frank Enderle wrote: > It most certainly looks ve

[ceph-users] Monitoring slow requests

Hey, I am wondering how people are monitoring/graphing slow requests ("oldest blocked for > xxx secs") on their clusters? I didn't find related counters to graph. So it looks like mon logs should be parsed for that info? Maybe someone has other ideas?

Re: [ceph-users] Unknown error (95->500) when creating buckets or putting files to RGW after upgrade from Infernalis to Jewel

Fwiw this thread still has me terrified to upgrade my rgw cluster. Just when I thought it was safe. Anyone have any successful problem free rgw infernalis-jewel upgrade reports? On Jul 25, 2016 11:27 PM, "nick" wrote: > Hey Maciej, > I compared the output of your commands with the output on our

Re: [ceph-users] mon_osd_nearfull_ratio (unchangeable) ?

On Tue, Jul 26, 2016 at 3:52 AM, Brad Hubbard wrote: >> 1./ if I try to change mon_osd_nearfull_ratio from 0.85 to 0.90, I get >> >># ceph tell mon.* injectargs "--mon_osd_nearfull_ratio 0.90" >>mon.rccephmon1: injectargs:mon_osd_nearfull_ratio = '0.9' >>(unchangeable) >>mon.rcceph

Re: [ceph-users] mon_osd_nearfull_ratio (unchangeable) ?

Hello... I do not think that these settings are working properly in jewel. Maybe someone else can confirm. So, to summarize: 1./ I've restarted mon and osd services (systemctl restart ceph.target) after setting # grep nearfull /etc/ceph/ceph.conf mon osd nearfull ratio = 0.90 2./ Th

Re: [ceph-users] Jewel + kernel 4.4 Massive performance regression (-50%)

Hello, do you have journal on disk too ? Yes am having journal on same hard disk. ok and could you do bench with kernel 4.2 ? just to see if you have better throughput. Thanks In ubuntu 14 I was running 4.2 kernel. the throughput was the same around 80-90MB/s per osd. I cant tell the difference

Re: [ceph-users] cephfs failed to rdlock, waiting

> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Oliver Dzombic > Sent: 26 July 2016 04:30 > To: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] cephfs failed to rdlock, waiting > > Hi Greg, > > i switched the cache tier to forward