Re: [ceph-users] Mon won't start, possibly due to corrupt disk?

Thanks Greg. Just for posterity, "ceph-kvstore-tool /var/lib/ceph/mon/store.db set auth last_committed ver 0" did the trick and we're back to HEALTH_OK. Cheers, Lincoln Bryant On Jul 18, 2014, at 4:15 PM, Gregory Farnum wrote: > Hmm, this log is just leaving me with more qu

Re: [ceph-users] PG to pool mapping?

On Feb 4, 2015, at 3:27 PM, Gregory Farnum wrote: > On Wed, Feb 4, 2015 at 1:20 PM, Chad William Seys > wrote: >> Hi all, >> How do I determine which pool a PG belongs to? >> (Also, is it the case that all objects in a PG belong to one pool?) > > PGs are of the form "1.a2b3c4". The part prio

Re: [ceph-users] Ceph User Teething Problems

Hi David, I also see only the RBD pool getting created by default in 0.93. With regards to resizing placement groups, I believe you can use: ceph osd pool set [pool name] pg_num ceph osd pool set [pool name] pgp_num Be forewarned, this will trigger data migration. Cheers, Lincol

[ceph-users] Failing to respond to cache pressure?

Hello all, I'm seeing some warnings regarding trimming and cache pressure. We're running 0.94.1 on our cluster, with erasure coding + cache tiering backing our CephFS. health HEALTH_WARN mds0: Behind on trimming (250/30) mds0: Client 74135 failing to respond to cache

Re: [ceph-users] Kernel Bug in 3.13.0-52

Hi Daniel, There are some kernel recommendations here, although it's unclear if they only apply to RBD or also to CephFS. http://ceph.com/docs/master/start/os-recommendations/ --Lincoln On May 13, 2015, at 3:03 PM, Daniel Takatori Ohara wrote: > Thank Gregory for the answer. > > I wil

Re: [ceph-users] Kernel Bug in 3.13.0-52

- Lab. of Bioinformatics > Molecular Oncology Center > Instituto Sírio-Libanês de Ensino e Pesquisa > Hospital Sírio-Libanês > Phone: +55 11 3155-0200 (extension 1927) > R: Cel. Nicolau dos Santos, 69 > São Paulo-SP. 01308-060 > http://www.bioinfo.mochsl.org.br > > > On Wed

Re: [ceph-users] Ceph MDS continually respawning (hammer)

Hi Adam, You can get the MDS to spit out more debug information like so: # ceph mds tell 0 injectargs '--debug-mds 20 --debug-ms 1' At least then you can see where it's at when it crashes. --Lincoln On May 22, 2015, at 9:33 AM, Adam Tygart wrote: > Hello all, > > The ceph-mds servers

Re: [ceph-users] Ceph MDS continually respawning (hammer)

ned. > > -- > Adam > > On Fri, May 22, 2015 at 9:42 AM, Lincoln Bryant wrote: >> Hi Adam, >> >> You can get the MDS to spit out more debug information like so: >> >># ceph mds tell 0 injectargs '--debug-mds 20 --debug-ms 1' >>

Re: [ceph-users] Ceph MDS continually respawning (hammer)

.0 up:standby seq 1 > 20284976: 10.5.38.13:6926/66700 'hobbit13' mds.0.1696 up:replay seq 1 > > -- > Adam > > On Fri, May 22, 2015 at 11:37 AM, Lincoln Bryant > wrote: >> I've experienced MDS issues in the past, but nothing sticks out to me in >

Re: [ceph-users] ceph mount error

Hi, Are you using cephx? If so, does your client have the appropriate key on it? It looks like you have an mds set up and running from your screenshot. Try mounting it like so: mount -t ceph -o name=admin,secret=[your secret] 192.168.1.105:6789:/ /mnt/mycephfs --Lincoln On Jun 7, 2015, at 1

[ceph-users] Erasure Coding + CephFS, objects not being deleted after rm

ticed is that after deleting the files via 'rm' through my CephFS kernel client, the cache is emptied but the objects that were evicted to the EC pool stick around. I've attached an image that demonstrates what I'm seeing. Is this intended behavior, or have I misconfigured so

Re: [ceph-users] Erasure Coding + CephFS, objects not being deleted after rm

Thanks John, Greg. If I understand this correctly, then, doing this: rados -p hotpool cache-flush-evict-all should start appropriately deleting objects from the cache pool. I just started one up, and that seems to be working. Otherwise, the cache's confgured timeouts/limits should get th

Re: [ceph-users] CephFS client issue

Hi Matteo, Are your clients using the FUSE client or the kernel client? If the latter, what kernel version? --Lincoln On 6/14/2015 10:26 AM, Matteo Dacrema wrote: ?Hi all, I'm using CephFS on Hammer and sometimes I need to reboot one or more clients because , as ceph -s tells me, it's "fa

Re: [ceph-users] EC on 1.1PB?

Hi Sean, We have ~1PB of EC storage using Dell R730xd servers with 6TB OSDs. We've got our erasure coding profile set up to be k=10,m=3 which gives us a very reasonable chunk of the raw storage with nice resiliency. I found that CPU usage was significantly higher in EC, but not so much as to be

Re: [ceph-users] EC on 1.1PB?

t; > Sorry for bombarding you with questions I am just curious as to where the 40% > performance comes from. > > On 06/19/2015 11:05 AM, Lincoln Bryant wrote: >> Hi Sean, >> >> We have ~1PB of EC storage using Dell R730xd servers with 6TB OSDs. We've >> got our eras

[ceph-users] Anyone using Ganesha with CephFS?

Hi Cephers, Is anyone successfully using Ganesha for re-exporting CephFS as NFS? I’ve seen some blog posts about setting it up and the basic functionality seems to be there. Just wondering if anyone in the community is actively using it, and could relate some experiences. —Lincoln

Re: [ceph-users] kernel 3.18 io bottlenecks?

Hi German, Is this with CephFS, or RBD? Thanks, Lincoln > On Jun 24, 2015, at 9:44 AM, German Anders wrote: > > Hi all, > >Is there any IO botleneck reported on kernel 3.18.3-031803-generic? since > I'm having a lot of iowait and the cluster is really getting slow, and > actually there'

Re: [ceph-users] CephFS vs RBD

Hi Hadi, AFAIK, you can’t safely mount RBD as R/W on multiple machines. You could re-export the RBD as NFS, but that’ll introduce a bottleneck and probably tank your performance gains over CephFS. For what it’s worth, some of our RBDs are mapped to multiple machines, mounted read-write on one

[ceph-users] Trying to identify performance bottlenecks

Hi all, I'm trying to identify the performance bottlenecks in my experimental Ceph cluster. A little background on my setup: 10 storage servers, each configured with: -(2) dual-core opterons -8 GB of RAM -(6) 750GB disks (1 OSD per disk, 720

Re: [ceph-users] xfsprogs not found in RHEL

Hi, xfsprogs should be included in the EL6 base. Perhaps run "yum clean all" and try again? Cheers, Lincoln On Aug 27, 2013, at 9:16 PM, sriram wrote: > I am trying to install CEPH and I get the following error - > > ---> Package ceph.x86_64 0:0.67.2-0.el6 will be installed > --> Processing

Re: [ceph-users] ceph creating stuck inactive and unclean

Hi Juan, Are the two OSDs that you started with on the same host? I've seen the same problem, which fixed itself after I added more OSDs on a separate host. Cheers, Lincoln On 11/3/2013 12:09 PM, Juan Vega wrote: Ceph Users, I'm trying to create a cluster with 9 OSDs manually (withouth

Re: [ceph-users] Ceph User Committee

Seems interesting to me. I've added my name to the pot :) --Lincoln On Nov 6, 2013, at 11:56 AM, Loic Dachary wrote: > > > On 07/11/2013 01:53, ja...@peacon.co.uk wrote: >> It's a great idea... are there any requirements, to be considered? > > Being a Ceph user seems to be the only requiremen

[ceph-users] pgs stuck in 'incomplete' state, blocked ops, query command hangs

Hi cephers, We have two pgs that are stuck in 'incomplete' state across two different pools: pg 2.525 is stuck inactive since forever, current state incomplete, last acting [55,89] pg 0.527 is stuck inactive since forever, current state incomplete, last acting [55,89] pg 0.527 is stuck unclean

Re: [ceph-users] pgs stuck in 'incomplete' state, blocked ops, query command hangs

24,\ 24,\ 24]\},\ \{ "first": 61,\ "last": 62,\ "maybe_went_rw": 1,\ "up": [\ 89],\

Re: [ceph-users] pgs stuck in 'incomplete' state, blocked ops, query command hangs

"maybe_went_rw": 1, "up": [ 89], "acting": [ 89, 89, 89]}, { "first": 222184, "l

Re: [ceph-users] Troubleshooting Incomplete PGs

Hi Greg, Loic, I think we have seen this as well (sent a mail to the list a week or so ago about incomplete pgs). I ended up giving up on the data and doing a force_create_pgs after doing a find on my OSDs and deleting the relevant pg dirs. If there are any logs etc you'd like to see for debugg

[ceph-users] mds continuously crashing on Firefly

-mds 20 --debug-ms 1'" below [1]. Can anyone help me interpret this error? Thanks for your time, Lincoln Bryant [1] -7> 2014-11-13 10:52:15.064784 7fc49d8ab700 7 mds.0.locker rdlock_start on (ifile sync->mix) on [inode 1000258c3c8 [2,head] /stash/sys/etc/grid-mapfile au

Re: [ceph-users] mds continuously crashing on Firefly

Cheers, Lincoln On Nov 13, 2014, at 11:01 AM, Lincoln Bryant wrote: > Hi Cephers, > > Over night, our MDS crashed, failing over to the standby which also crashed! > Upon trying to restart them this morning, I find that they no longer start > and always seem to crash on the same

Re: [ceph-users] Slow/Hung IOs

Hi BIll, From your log excerpt, it looks like your slow requests are happening on OSDs 14 and 18. Is it always these two OSDs? If you don't have a long recovery time (e.g., the cluster is just full of test data), maybe you could try setting OSDs 14 and 18 out and re-benching? Alternatively I s

Re: [ceph-users] Building Ceph

Hi Pankaj, You can search for the lib using the 'yum provides' command, which accepts wildcards. [root@sl7 ~]# yum provides */lib64/libkeyutils* Loaded plugins: langpacks keyutils-libs-1.5.8-3.el7.x86_64 : Key utilities library Repo: sl Matched from: Filename: /lib64/libkeyutils.so.1

Re: [ceph-users] Ceph status

Hi Ajitha, For one, it looks like you don't have enough OSDs for the number of replicas you have specified in the config file. What is the value of your 'osd pool default size' in ceph.conf? If it's "3", for example, then you need to have at least 3 hosts with 1 OSD each (with the default CRU

Re: [ceph-users] Ceph Testing

Hi Raj, Sébastien Han has done some excellent Ceph benchmarking on his blog here: http://www.sebastien-han.fr/blog/2012/08/26/ceph-benchmarks/ Maybe that's a good place to start for your own testing? Cheers, Lincoln On Jan 28, 2015, at 12:59 PM, Jeripotula, Shashiraj wrote: > Resending, Guys,

Re: [ceph-users] Ceph / Dell hardware recommendation

For our ~400 TB Ceph deployment, we bought: (2) R720s w/ dual X5660s and 96 GB of RAM (1) 10Gb NIC (2 interfaces per card) (4) MD1200s per machine ...and a boat load of 4TB disks! In retrospect, I would almost certainly would have gotten more servers. During heavy

Re: [ceph-users] issues with ceph

Hi Erik, What happens if you try to stat one of the "missing" files (assuming you know the name of the file before you remount raw)? I had a problem where files would disappear and reappear in CephFS, which I believe was fixed in kernel 3.12. Cheers, Lincoln On May 9, 2014, at 9:30 AM, Arones

Re: [ceph-users] issues with ceph

FWIW, I believe the particular/similar bug I was thinking of was fixed by: commit 590fb51f1c (vfs: call d_op->d_prune() before unhashing dentry) --Lincoln On May 9, 2014, at 12:37 PM, Gregory Farnum wrote: > I'm less current on the kernel client, so maybe there are some > since-fixed bu

Re: [ceph-users] Ceph cluster NO read / write performance :: Ops are blocked

Hi Vickey, I had this exact same problem last week, resolved by rebooting all of my OSD nodes. I have yet to figure out why it happened, though. I _suspect_ in my case it's due to a failing controller on a particular box I've had trouble with in the past. I tried setting 'noout', stopping my

Re: [ceph-users] Ceph cluster NO read / write performance :: Ops are blocked

For whatever it’s worth, my problem has returned and is very similar to yours. Still trying to figure out what’s going on over here. Performance is nice for a few seconds, then goes to 0. This is a similar setup to yours (12 OSDs per box, Scientific Linux 6, Ceph 0.94.3, etc) 384 16

Re: [ceph-users] Ceph cluster NO read / write performance :: Ops are blocked

)? Not out of any type of memory (slab, min_free_kbytes)? Not network > link errors, no bad checksums (those are hard to spot, though)? > > Unless you find something I suggest you try disabling offloads on the NICs > and see if the problem goes away. > > Jan > >> On 08 S

Re: [ceph-users] Straw2 kernel version?

Hi Robert, I believe kernel versions 4.1 and beyond support straw2. —Lincoln > On Sep 10, 2015, at 1:43 PM, Robert LeBlanc wrote: > > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Has straw2 landed in the kernel and if so which version? > > Thanks, > - > Robert LeBla

Re: [ceph-users] Hammer reduce recovery impact

On 9/10/2015 5:39 PM, Lionel Bouton wrote: For example deep-scrubs were a problem on our installation when at times there were several going on. We implemented a scheduler that enforces limits on simultaneous deep-scrubs and these problems are gone. Hi Lionel, Out of curiosity, how many was "

Re: [ceph-users] Ceph cluster NO read / write performance :: Ops are blocked

t that didn’t seem to help. Thanks for any insight! Cheers, Lincoln Bryant > On Sep 9, 2015, at 1:09 PM, Lincoln Bryant wrote: > > Hi Jan, > > I’ll take a look at all of those things and report back (hopefully :)) > > I did try setting all of my OSDs to writet

Re: [ceph-users] Ceph cluster NO read / write performance :: Ops are blocked

t; From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of >> Lincoln Bryant >> Sent: 17 September 2015 17:42 >> To: ceph-users@lists.ceph.com >> Subject: Re: [ceph-users] Ceph cluster NO read / write performance :: Ops >> are blocked >> >&g

Re: [ceph-users] Ceph cluster NO read / write performance :: Ops are blocked

nd do some > benchmarks on that to rule out any issue with the hardware and OSD's. > >> -Original Message- >> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of >> Lincoln Bryant >> Sent: 17 September 2015 17:54 >> To: Nick Fis

Re: [ceph-users] Ceph cluster NO read / write performance :: Ops are blocked

I set up a cron to run cache-try-flush-evict-all every night? :)) Another curious thing is that a rolling restart of all OSDs also seems to fix the problem — for a time. I’m not sure how that would fit in if this is the problem. —Lincoln > On Sep 17, 2015, at 12:07 PM, Lincoln Bryant wr

Re: [ceph-users] Ceph cluster NO read / write performance :: Ops are blocked

t > on the test 3 way pool, to rule out any hardware/OS issues. > > >> -Original Message- >> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of >> Lincoln Bryant >> Sent: 17 September 2015 18:36 >> To: Nick Fisk >> Cc: cep

Re: [ceph-users] CephFS "corruption" -- Nulled bytes

Hi Sage, Will this patch be in 0.94.4? We've got the same problem here. -Lincoln > On Oct 8, 2015, at 12:11 AM, Sage Weil wrote: > > On Wed, 7 Oct 2015, Adam Tygart wrote: >> Does this patch fix files that have been corrupted in this manner? > > Nope, it'll only prevent it from happening to n

Re: [ceph-users] why was osd pool default size changed from 2 to 3.

>but because there were only two copies it had no way to tell which one was >correct, and when I forced it to choose it often chose wrong. Yeah. This is a BIG problem with only running with two copies. Good luck if your pgs ever get inconsistent :) --Lincoln > On Oct 26, 2015, at 10:41 AM, Que

Re: [ceph-users] State of nfs-ganesha CEPH fsal

Hi Dennis, We're using NFS Ganesha here as well. I can send you my configuration which is working but we squash users and groups down to a particular uid/gid, so it may not be super helpful for you. I think files not being immediately visible is working as intended, due to directory caching. I

Re: [ceph-users] Troubleshoot blocked OSDs

Hi Brian, The first thing you can do is “ceph health detail”, which should give you some more information about which OSD(s) have blocked requests. If it’s isolated to one OSD in particular, perhaps use iostat to check utilization and/or smartctl to check health. —Lincoln > On Apr 28, 2016,

Re: [ceph-users] Troubleshoot blocked OSDs

an Andrus > ITACS/Research Computing > Naval Postgraduate School > Monterey, California > voice: 831-656-6238 > > >   <> > From: Lincoln Bryant [mailto:linco...@uchicago.edu > <mailto:linco...@uchicago.edu>] > Sent: Thursday, April 28, 2016 12:31 PM >

Re: [ceph-users] Issues with CephFS

Hi, Are there any messages in 'dmesg'? Are you running a recent kernel on your client? --Lincoln On 6/18/2016 6:25 PM, ServerPoint wrote: Hi Adam, Thank you ! That worked. So now I am testing another large cluster. This is the Ceph status : (I am using Public network so I have put * ) --

[ceph-users] some OSDs stuck down after 10.2.7 -> 10.2.9 update

Hi all, After updating to 10.2.9, some of our SSD-based OSDs get put into "down" state and die as in [1]. After bringing these OSDs back up, they sit at 100% CPU utilization and never become up/in. From the log I see (from [2]): heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f1cfad0d7

Re: [ceph-users] Long OSD restart after upgrade to 10.2.9

Hi Anton, We observe something similar on our OSDs going from 10.2.7 to 10.2.9 (see thread "some OSDs stuck down after 10.2.7 -> 10.2.9 update"). Some of our OSDs are not working at all on 10.2.9 or die with suicide timeouts. Those that come up/in take a very long time to boot up. Seems to no

Re: [ceph-users] Inconsistent pgs with size_mismatch_oi

b 36.2c85 36:a13626c6:::1002378e9a9.0001:head on disk size (0) does not match object info size (4173824) adjusted for ondisk to (4173824) 2017-08-08 13:26:34.310237 7fafac78a700 -1 log_channel(cluster) log [ERR] : 36.2c85 scrub 4 errors > On May 15, 2017, at 5:28 PM, Gregory Farnum wrote: >

Re: [ceph-users] upgrade Hammer>Jewel>Luminous OSD fail to start

Did you set the sortbitwise flag, fix OSD ownership (or use the "setuser match path" option) and such after upgrading from Hammer to Jewel? I am not sure if that matters here, but it might help if you elaborate on your upgrade process a bit. --Lincoln > On Sep 12, 2017, at 2:22 PM, kevin parri

Re: [ceph-users] Inconsistent PG won't repair

Hi Rich, Is the object inconsistent and 0-bytes on all OSDs? We ran into a similar issue on Jewel, where an object was empty across the board but had inconsistent metadata. Ultimately it was resolved by doing a "rados get" and then a "rados put" on the object. *However* that was a last ditch e

Re: [ceph-users] who is using nfs-ganesha and cephfs?

Hi Sage, We have been running the Ganesha FSAL for a while (as far back as Hammer / Ganesha 2.2.0), primarily for uid/gid squashing. Things are basically OK for our application, but we've seen the following weirdness*: - Sometimes there are duplicated entries when directories are listed

Re: [ceph-users] cephfs degraded on ceph luminous 12.2.2

Hi Alessandro, What is the state of your PGs? Inactive PGs have blocked CephFS recovery on our cluster before. I'd try to clear any blocked ops and see if the MDSes recover. --Lincoln On Mon, 2018-01-08 at 17:21 +0100, Alessandro De Salvo wrote: > Hi, > > I'm running on ceph luminous 12.2.2 and

Re: [ceph-users] OSD servers swapping despite having free memory capacity

Hi Sam, What happens if you just disable swap altogether? i.e., with `swapoff -a` --Lincoln On Tue, 2018-01-23 at 19:54 +, Samuel Taylor Liston wrote: > We have a 9 - node (16 - 8TB OSDs per node) running jewel on centos > 7.4.  The OSDs are configured with encryption.  The cluster is > acce

Re: [ceph-users] Ceph 0.94.8 Hammer released

Hi all, We are also interested in EL6 RPMs. My understanding was that EL6 would continue to be supported through Hammer. Is there anything we can do to help? Thanks, Lincoln > On Aug 29, 2016, at 11:14 AM, Alex Litvak > wrote: > > Hammer RPMs for 0.94.8 are still not available for EL6. C

Re: [ceph-users] CephFS and calculation of directory size

Are you running ‘ls’ or are you doing something like: 'getfattr -d -m ceph.dir.* /path/to/your/ceph/mount’ ? —Lincoln > On Sep 12, 2016, at 1:00 PM, Ilya Moldovan wrote: > > Thanks, John > > But why listing files in a directory with about a million files takes > about 30 minutes? > > Ilya Mo

Re: [ceph-users] Server Down?

Hi Ashwin, Seems the website is down. From another thread: http://www.dreamhoststatus.com/2016/10/11/dreamcompute-us-east-1-cluster-service-disruption/ I’ve been using the EU mirrors in the meanwhile

Re: [ceph-users] Ensure Hammer client compatibility

Hi Kees, What interfaces do your Hammer clients need? If you're looking at CephFS, we have had reasonable success moving our older clients (EL6) to NFS Ganesha with the Ceph FSAL. --Lincoln On Mon, 2018-08-20 at 12:22 +0200, Kees Meijs wrote: > Good afternoon Cephers, > > While I'm fixing our u

Re: [ceph-users] Unable to boot OS on cluster node

Hi Shain, As long as you don’t nuke the OSDs or the journals, you should be OK. I think the keyring and such are typically stored on the OSD itself. If you have lost track of what physical device maps to what OSD, you can always mount the OSDs in a temporary spot and cat the “whoami” file. —Li

[ceph-users] Inconsistent pgs with size_mismatch_oi

Hi all, I’ve run across a peculiar issue on 10.2.7. On my 3x replicated cache tiering cache pool, routine scrubbing suddenly found a bunch of PGs with size_mismatch_oi errors. From the “rados list-inconsistent-pg tool”[1], I see that all OSDs are reporting size 0 for a particular pg. I’ve check

Re: [ceph-users] Inconsistent pgs with size_mismatch_oi

osd": 173, "errors": [ "size_mismatch_oi" ], "size": 0 } ] } ] } Perhaps new data being written to this pg cleared things up? The only other data point that I can add is that, due to some tweaking o

Re: [ceph-users] CEPH FS - all_squash option equivalent

Also very interested in this if there are any docs available! --Lincoln > On Mar 3, 2016, at 1:04 PM, Fred Rolland wrote: > > Can you share a link describing the UID squashing feature? > > On Mar 3, 2016 9:02 PM, "Gregory Farnum" wrote: > On Wed, Mar 2, 2016 at 11:22 PM, Fred Rolland wrote:

Re: [ceph-users] CEPHFS file or directories disappear when ls (metadata problem)

Hi, If you are using the kernel client, I would suggest trying something newer than 3.10.x. I ran into this issue in the past, but it was fixed by updating my kernel to something newer. You may want to check the OS recommendations page as well: http://docs.ceph.com/docs/master/start/os-recomme