Re: [ceph-users] Ensure Hammer client compatibility

Hi Kees, What interfaces do your Hammer clients need? If you're looking at CephFS, we have had reasonable success moving our older clients (EL6) to NFS Ganesha with the Ceph FSAL. --Lincoln On Mon, 2018-08-20 at 12:22 +0200, Kees Meijs wrote: > Good afternoon Cephers, > > While I'm fixing our

Re: [ceph-users] OSD servers swapping despite having free memory capacity

Hi Sam, What happens if you just disable swap altogether? i.e., with `swapoff -a` --Lincoln On Tue, 2018-01-23 at 19:54 +, Samuel Taylor Liston wrote: > We have a 9 - node (16 - 8TB OSDs per node) running jewel on centos > 7.4.  The OSDs are configured with encryption.  The cluster is >

Re: [ceph-users] cephfs degraded on ceph luminous 12.2.2

Hi Alessandro, What is the state of your PGs? Inactive PGs have blocked CephFS recovery on our cluster before. I'd try to clear any blocked ops and see if the MDSes recover. --Lincoln On Mon, 2018-01-08 at 17:21 +0100, Alessandro De Salvo wrote: > Hi, > > I'm running on ceph luminous 12.2.2

Re: [ceph-users] who is using nfs-ganesha and cephfs?

Hi Sage, We have been running the Ganesha FSAL for a while (as far back as Hammer / Ganesha 2.2.0), primarily for uid/gid squashing. Things are basically OK for our application, but we've seen the following weirdness*: - Sometimes there are duplicated entries when directories are

Re: [ceph-users] Inconsistent PG won't repair

Hi Rich, Is the object inconsistent and 0-bytes on all OSDs? We ran into a similar issue on Jewel, where an object was empty across the board but had inconsistent metadata. Ultimately it was resolved by doing a "rados get" and then a "rados put" on the object. *However* that was a last ditch

Re: [ceph-users] upgrade Hammer>Jewel>Luminous OSD fail to start

Did you set the sortbitwise flag, fix OSD ownership (or use the "setuser match path" option) and such after upgrading from Hammer to Jewel? I am not sure if that matters here, but it might help if you elaborate on your upgrade process a bit. --Lincoln > On Sep 12, 2017, at 2:22 PM, kevin

Re: [ceph-users] Inconsistent pgs with size_mismatch_oi

626c6:::1002378e9a9.0001:head on disk size (0) does not match object info size (4173824) adjusted for ondisk to (4173824) 2017-08-08 13:26:34.310237 7fafac78a700 -1 log_channel(cluster) log [ERR] : 36.2c85 scrub 4 errors > On May 15, 2017, at 5:28 PM, Gregory Farnum <gfar...@redhat.com&g

Re: [ceph-users] Long OSD restart after upgrade to 10.2.9

Hi Anton, We observe something similar on our OSDs going from 10.2.7 to 10.2.9 (see thread "some OSDs stuck down after 10.2.7 -> 10.2.9 update"). Some of our OSDs are not working at all on 10.2.9 or die with suicide timeouts. Those that come up/in take a very long time to boot up. Seems to

[ceph-users] some OSDs stuck down after 10.2.7 -> 10.2.9 update

Hi all, After updating to 10.2.9, some of our SSD-based OSDs get put into "down" state and die as in [1]. After bringing these OSDs back up, they sit at 100% CPU utilization and never become up/in. From the log I see (from [2]): heartbeat_map is_healthy 'OSD::osd_op_tp thread

Re: [ceph-users] Inconsistent pgs with size_mismatch_oi

uot;: 173, "errors": [ "size_mismatch_oi" ], "size": 0 } ] } ] } Perhaps new data being written to this pg cleared things up? The only other data point that I can add is that, due to some tweaking of the

[ceph-users] Inconsistent pgs with size_mismatch_oi

Hi all, I’ve run across a peculiar issue on 10.2.7. On my 3x replicated cache tiering cache pool, routine scrubbing suddenly found a bunch of PGs with size_mismatch_oi errors. From the “rados list-inconsistent-pg tool”[1], I see that all OSDs are reporting size 0 for a particular pg. I’ve

Re: [ceph-users] Unable to boot OS on cluster node

Hi Shain, As long as you don’t nuke the OSDs or the journals, you should be OK. I think the keyring and such are typically stored on the OSD itself. If you have lost track of what physical device maps to what OSD, you can always mount the OSDs in a temporary spot and cat the “whoami” file.

Re: [ceph-users] Server Down?

Hi Ashwin, Seems the website is down. From another thread: http://www.dreamhoststatus.com/2016/10/11/dreamcompute-us-east-1-cluster-service-disruption/ I’ve been using the EU mirrors in the

Re: [ceph-users] CephFS and calculation of directory size

Are you running ‘ls’ or are you doing something like: 'getfattr -d -m ceph.dir.* /path/to/your/ceph/mount’ ? —Lincoln > On Sep 12, 2016, at 1:00 PM, Ilya Moldovan wrote: > > Thanks, John > > But why listing files in a directory with about a million files takes > about

Re: [ceph-users] Ceph 0.94.8 Hammer released

Hi all, We are also interested in EL6 RPMs. My understanding was that EL6 would continue to be supported through Hammer. Is there anything we can do to help? Thanks, Lincoln > On Aug 29, 2016, at 11:14 AM, Alex Litvak > wrote: > > Hammer RPMs for 0.94.8 are

Re: [ceph-users] Issues with CephFS

Hi, Are there any messages in 'dmesg'? Are you running a recent kernel on your client? --Lincoln On 6/18/2016 6:25 PM, ServerPoint wrote: Hi Adam, Thank you ! That worked. So now I am testing another large cluster. This is the Ceph status : (I am using Public network so I have put * )

Re: [ceph-users] Troubleshoot blocked OSDs

gt; OSDs. > > Brian Andrus > ITACS/Research Computing > Naval Postgraduate School > Monterey, California > voice: 831-656-6238 > > >   <> > From: Lincoln Bryant [mailto:linco...@uchicago.edu > <mailto:linco...@uchicago.edu>] > Sent: Thursday, Ap

Re: [ceph-users] Troubleshoot blocked OSDs

Hi Brian, The first thing you can do is “ceph health detail”, which should give you some more information about which OSD(s) have blocked requests. If it’s isolated to one OSD in particular, perhaps use iostat to check utilization and/or smartctl to check health. —Lincoln > On Apr 28, 2016,

Re: [ceph-users] CEPHFS file or directories disappear when ls (metadata problem)

Hi, If you are using the kernel client, I would suggest trying something newer than 3.10.x. I ran into this issue in the past, but it was fixed by updating my kernel to something newer. You may want to check the OS recommendations page as well:

Re: [ceph-users] CEPH FS - all_squash option equivalent

Also very interested in this if there are any docs available! --Lincoln > On Mar 3, 2016, at 1:04 PM, Fred Rolland wrote: > > Can you share a link describing the UID squashing feature? > > On Mar 3, 2016 9:02 PM, "Gregory Farnum" wrote: > On Wed, Mar

Re: [ceph-users] State of nfs-ganesha CEPH fsal

Hi Dennis, We're using NFS Ganesha here as well. I can send you my configuration which is working but we squash users and groups down to a particular uid/gid, so it may not be super helpful for you. I think files not being immediately visible is working as intended, due to directory caching.

Re: [ceph-users] why was osd pool default size changed from 2 to 3.

>but because there were only two copies it had no way to tell which one was >correct, and when I forced it to choose it often chose wrong. Yeah. This is a BIG problem with only running with two copies. Good luck if your pgs ever get inconsistent :) --Lincoln > On Oct 26, 2015, at 10:41 AM,

Re: [ceph-users] CephFS "corruption" -- Nulled bytes

Hi Sage, Will this patch be in 0.94.4? We've got the same problem here. -Lincoln > On Oct 8, 2015, at 12:11 AM, Sage Weil wrote: > > On Wed, 7 Oct 2015, Adam Tygart wrote: >> Does this patch fix files that have been corrupted in this manner? > > Nope, it'll only prevent it

Re: [ceph-users] Ceph cluster NO read / write performance :: Ops are blocked

n’t seem to help. Thanks for any insight! Cheers, Lincoln Bryant > On Sep 9, 2015, at 1:09 PM, Lincoln Bryant <linco...@uchicago.edu> wrote: > > Hi Jan, > > I’ll take a look at all of those things and report back (hopefully :)) > > I did try setting all of m

Re: [ceph-users] Ceph cluster NO read / write performance :: Ops are blocked

Message- >> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of >> Lincoln Bryant >> Sent: 17 September 2015 17:42 >> To: ceph-users@lists.ceph.com >> Subject: Re: [ceph-users] Ceph cluster NO read / write performance :: Ops >> are blocked

Re: [ceph-users] Ceph cluster NO read / write performance :: Ops are blocked

I set up a cron to run cache-try-flush-evict-all every night? :)) Another curious thing is that a rolling restart of all OSDs also seems to fix the problem — for a time. I’m not sure how that would fit in if this is the problem. —Lincoln > On Sep 17, 2015, at 12:07 PM, Lincoln Bryant <

Re: [ceph-users] Ceph cluster NO read / write performance :: Ops are blocked

on tiered test pool and do some > benchmarks on that to rule out any issue with the hardware and OSD's. > >> -Original Message- >> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of >> Lincoln Bryant >> Sent: 17 September 2015 17:54 >>

Re: [ceph-users] Ceph cluster NO read / write performance :: Ops are blocked

the test 3 way pool, to rule out any hardware/OS issues. > > >> -Original Message- >> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of >> Lincoln Bryant >> Sent: 17 September 2015 18:36 >> To: Nick Fisk <n...@fisk.me.uk> &

Re: [ceph-users] Straw2 kernel version?

Hi Robert, I believe kernel versions 4.1 and beyond support straw2. —Lincoln > On Sep 10, 2015, at 1:43 PM, Robert LeBlanc wrote: > > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Has straw2 landed in the kernel and if so which version? > > Thanks, > -

Re: [ceph-users] Hammer reduce recovery impact

On 9/10/2015 5:39 PM, Lionel Bouton wrote: For example deep-scrubs were a problem on our installation when at times there were several going on. We implemented a scheduler that enforces limits on simultaneous deep-scrubs and these problems are gone. Hi Lionel, Out of curiosity, how many was

Re: [ceph-users] Ceph cluster NO read / write performance :: Ops are blocked

pd and > similiar)? Not out of any type of memory (slab, min_free_kbytes)? Not network > link errors, no bad checksums (those are hard to spot, though)? > > Unless you find something I suggest you try disabling offloads on the NICs > and see if the problem goes away. > > J

Re: [ceph-users] Ceph cluster NO read / write performance :: Ops are blocked

For whatever it’s worth, my problem has returned and is very similar to yours. Still trying to figure out what’s going on over here. Performance is nice for a few seconds, then goes to 0. This is a similar setup to yours (12 OSDs per box, Scientific Linux 6, Ceph 0.94.3, etc) 384 16

Re: [ceph-users] Ceph cluster NO read / write performance :: Ops are blocked

Hi Vickey, I had this exact same problem last week, resolved by rebooting all of my OSD nodes. I have yet to figure out why it happened, though. I _suspect_ in my case it's due to a failing controller on a particular box I've had trouble with in the past. I tried setting 'noout', stopping

Re: [ceph-users] CephFS vs RBD

Hi Hadi, AFAIK, you can’t safely mount RBD as R/W on multiple machines. You could re-export the RBD as NFS, but that’ll introduce a bottleneck and probably tank your performance gains over CephFS. For what it’s worth, some of our RBDs are mapped to multiple machines, mounted read-write on one

Re: [ceph-users] kernel 3.18 io bottlenecks?

Hi German, Is this with CephFS, or RBD? Thanks, Lincoln On Jun 24, 2015, at 9:44 AM, German Anders gand...@despegar.com wrote: Hi all, Is there any IO botleneck reported on kernel 3.18.3-031803-generic? since I'm having a lot of iowait and the cluster is really getting slow, and

[ceph-users] Anyone using Ganesha with CephFS?

Hi Cephers, Is anyone successfully using Ganesha for re-exporting CephFS as NFS? I’ve seen some blog posts about setting it up and the basic functionality seems to be there. Just wondering if anyone in the community is actively using it, and could relate some experiences. —Lincoln

Re: [ceph-users] EC on 1.1PB?

Hi Sean, We have ~1PB of EC storage using Dell R730xd servers with 6TB OSDs. We've got our erasure coding profile set up to be k=10,m=3 which gives us a very reasonable chunk of the raw storage with nice resiliency. I found that CPU usage was significantly higher in EC, but not so much as to

Re: [ceph-users] EC on 1.1PB?

. On 06/19/2015 11:05 AM, Lincoln Bryant wrote: Hi Sean, We have ~1PB of EC storage using Dell R730xd servers with 6TB OSDs. We've got our erasure coding profile set up to be k=10,m=3 which gives us a very reasonable chunk of the raw storage with nice resiliency. I found that CPU usage

Re: [ceph-users] CephFS client issue

Hi Matteo, Are your clients using the FUSE client or the kernel client? If the latter, what kernel version? --Lincoln On 6/14/2015 10:26 AM, Matteo Dacrema wrote: ?Hi all, I'm using CephFS on Hammer and sometimes I need to reboot one or more clients because , as ceph -s tells me, it's

Re: [ceph-users] Erasure Coding + CephFS, objects not being deleted after rm

Thanks John, Greg. If I understand this correctly, then, doing this: rados -p hotpool cache-flush-evict-all should start appropriately deleting objects from the cache pool. I just started one up, and that seems to be working. Otherwise, the cache's confgured timeouts/limits should get

[ceph-users] Erasure Coding + CephFS, objects not being deleted after rm

is that after deleting the files via 'rm' through my CephFS kernel client, the cache is emptied but the objects that were evicted to the EC pool stick around. I've attached an image that demonstrates what I'm seeing. Is this intended behavior, or have I misconfigured something? Thanks, Lincoln

Re: [ceph-users] ceph mount error

Hi, Are you using cephx? If so, does your client have the appropriate key on it? It looks like you have an mds set up and running from your screenshot. Try mounting it like so: mount -t ceph -o name=admin,secret=[your secret] 192.168.1.105:6789:/ /mnt/mycephfs --Lincoln On Jun 7, 2015, at

Re: [ceph-users] Ceph MDS continually respawning (hammer)

Hi Adam, You can get the MDS to spit out more debug information like so: # ceph mds tell 0 injectargs '--debug-mds 20 --debug-ms 1' At least then you can see where it's at when it crashes. --Lincoln On May 22, 2015, at 9:33 AM, Adam Tygart wrote: Hello all, The ceph-mds servers

Re: [ceph-users] Ceph MDS continually respawning (hammer)

, I added 'debug ms = 1' for this one: https://drive.google.com/file/d/0B4XF1RWjuGh5bXFnRzE1SHF6blE/view?usp=sharing Based on these logs, it looks like heartbeat_map is_healthy 'MDS' just times out and then the mds gets respawned. -- Adam On Fri, May 22, 2015 at 9:42 AM, Lincoln Bryant

Re: [ceph-users] Ceph MDS continually respawning (hammer)

, Lincoln Bryant linco...@uchicago.edu wrote: I've experienced MDS issues in the past, but nothing sticks out to me in your logs. Are you using a single active MDS with failover, or multiple active MDS? --Lincoln On May 22, 2015, at 10:10 AM, Adam Tygart wrote: Thanks for the quick

Re: [ceph-users] Kernel Bug in 3.13.0-52

Hi Daniel, There are some kernel recommendations here, although it's unclear if they only apply to RBD or also to CephFS. http://ceph.com/docs/master/start/os-recommendations/ --Lincoln On May 13, 2015, at 3:03 PM, Daniel Takatori Ohara wrote: Thank Gregory for the answer. I will

Re: [ceph-users] Kernel Bug in 3.13.0-52

-Libanês de Ensino e Pesquisa Hospital Sírio-Libanês Phone: +55 11 3155-0200 (extension 1927) R: Cel. Nicolau dos Santos, 69 São Paulo-SP. 01308-060 http://www.bioinfo.mochsl.org.br On Wed, May 13, 2015 at 5:11 PM, Lincoln Bryant linco...@uchicago.edu wrote: Hi Daniel, There are some kernel

[ceph-users] Failing to respond to cache pressure?

Hello all, I'm seeing some warnings regarding trimming and cache pressure. We're running 0.94.1 on our cluster, with erasure coding + cache tiering backing our CephFS. health HEALTH_WARN mds0: Behind on trimming (250/30) mds0: Client 74135 failing to respond to

Re: [ceph-users] Ceph User Teething Problems

Hi David, I also see only the RBD pool getting created by default in 0.93. With regards to resizing placement groups, I believe you can use: ceph osd pool set [pool name] pg_num ceph osd pool set [pool name] pgp_num Be forewarned, this will trigger data migration. Cheers,

Re: [ceph-users] PG to pool mapping?

On Feb 4, 2015, at 3:27 PM, Gregory Farnum wrote: On Wed, Feb 4, 2015 at 1:20 PM, Chad William Seys cws...@physics.wisc.edu wrote: Hi all, How do I determine which pool a PG belongs to? (Also, is it the case that all objects in a PG belong to one pool?) PGs are of the form 1.a2b3c4.

Re: [ceph-users] Ceph Testing

Hi Raj, Sébastien Han has done some excellent Ceph benchmarking on his blog here: http://www.sebastien-han.fr/blog/2012/08/26/ceph-benchmarks/ Maybe that's a good place to start for your own testing? Cheers, Lincoln On Jan 28, 2015, at 12:59 PM, Jeripotula, Shashiraj wrote: Resending, Guys,

Re: [ceph-users] Building Ceph

Hi Pankaj, You can search for the lib using the 'yum provides' command, which accepts wildcards. [root@sl7 ~]# yum provides */lib64/libkeyutils* Loaded plugins: langpacks keyutils-libs-1.5.8-3.el7.x86_64 : Key utilities library Repo: sl Matched from: Filename:

Re: [ceph-users] Ceph status

Hi Ajitha, For one, it looks like you don't have enough OSDs for the number of replicas you have specified in the config file. What is the value of your 'osd pool default size' in ceph.conf? If it's 3, for example, then you need to have at least 3 hosts with 1 OSD each (with the default

Re: [ceph-users] Slow/Hung IOs

Hi BIll, From your log excerpt, it looks like your slow requests are happening on OSDs 14 and 18. Is it always these two OSDs? If you don't have a long recovery time (e.g., the cluster is just full of test data), maybe you could try setting OSDs 14 and 18 out and re-benching? Alternatively I

[ceph-users] mds continuously crashing on Firefly

1' below [1]. Can anyone help me interpret this error? Thanks for your time, Lincoln Bryant [1] -7 2014-11-13 10:52:15.064784 7fc49d8ab700 7 mds.0.locker rdlock_start on (ifile sync-mix) on [inode 1000258c3c8 [2,head] /stash/sys/etc/grid-mapfile auth v754009 ap=27+0 s=17384 n(v0 b17384

Re: [ceph-users] mds continuously crashing on Firefly

On Nov 13, 2014, at 11:01 AM, Lincoln Bryant wrote: Hi Cephers, Over night, our MDS crashed, failing over to the standby which also crashed! Upon trying to restart them this morning, I find that they no longer start and always seem to crash on the same file in the logs. I've pasted part

Re: [ceph-users] Troubleshooting Incomplete PGs

Hi Greg, Loic, I think we have seen this as well (sent a mail to the list a week or so ago about incomplete pgs). I ended up giving up on the data and doing a force_create_pgs after doing a find on my OSDs and deleting the relevant pg dirs. If there are any logs etc you'd like to see for

[ceph-users] pgs stuck in 'incomplete' state, blocked ops, query command hangs

Hi cephers, We have two pgs that are stuck in 'incomplete' state across two different pools: pg 2.525 is stuck inactive since forever, current state incomplete, last acting [55,89] pg 0.527 is stuck inactive since forever, current state incomplete, last acting [55,89] pg 0.527 is stuck

Re: [ceph-users] pgs stuck in 'incomplete' state, blocked ops, query command hangs

DNE # ceph osd rm osd.85 osd.85 does not exist. # ceph osd lost 85 --yes-i-really-mean-it osd.85 is not down or doesn't exist Any help would be greatly appreciated. Thanks, Lincoln On Oct 21, 2014, at 9:39 AM, Lincoln Bryant wrote: Hi cephers, We have two pgs that are stuck in 'incomplete

Re: [ceph-users] pgs stuck in 'incomplete' state, blocked ops, query command hangs

, Lincoln On Oct 21, 2014, at 11:59 AM, Lincoln Bryant wrote: A small update on this, I rebooted all of the Ceph nodes and was able to then query one of the misbehaving pgs. I've attached the query for pg 2.525. incomplete-pg-query-2.525.rtf There are some things like this in the peer info

Re: [ceph-users] Mon won't start, possibly due to corrupt disk?

Thanks Greg. Just for posterity, ceph-kvstore-tool /var/lib/ceph/mon/store.db set auth last_committed ver 0 did the trick and we're back to HEALTH_OK. Cheers, Lincoln Bryant On Jul 18, 2014, at 4:15 PM, Gregory Farnum wrote: Hmm, this log is just leaving me with more questions. Could you tar

Re: [ceph-users] issues with ceph

Hi Erik, What happens if you try to stat one of the missing files (assuming you know the name of the file before you remount raw)? I had a problem where files would disappear and reappear in CephFS, which I believe was fixed in kernel 3.12. Cheers, Lincoln On May 9, 2014, at 9:30 AM,

Re: [ceph-users] issues with ceph

FWIW, I believe the particular/similar bug I was thinking of was fixed by: commit 590fb51f1c (vfs: call d_op-d_prune() before unhashing dentry) --Lincoln On May 9, 2014, at 12:37 PM, Gregory Farnum wrote: I'm less current on the kernel client, so maybe there are some since-fixed bugs

Re: [ceph-users] Ceph User Committee

Seems interesting to me. I've added my name to the pot :) --Lincoln On Nov 6, 2013, at 11:56 AM, Loic Dachary wrote: On 07/11/2013 01:53, ja...@peacon.co.uk wrote: It's a great idea... are there any requirements, to be considered? Being a Ceph user seems to be the only requirement to

Re: [ceph-users] ceph creating stuck inactive and unclean

Hi Juan, Are the two OSDs that you started with on the same host? I've seen the same problem, which fixed itself after I added more OSDs on a separate host. Cheers, Lincoln On 11/3/2013 12:09 PM, Juan Vega wrote: Ceph Users, I'm trying to create a cluster with 9 OSDs manually

Re: [ceph-users] xfsprogs not found in RHEL

Hi, xfsprogs should be included in the EL6 base. Perhaps run yum clean all and try again? Cheers, Lincoln On Aug 27, 2013, at 9:16 PM, sriram wrote: I am trying to install CEPH and I get the following error - --- Package ceph.x86_64 0:0.67.2-0.el6 will be installed -- Processing

[ceph-users] Trying to identify performance bottlenecks

Hi all, I'm trying to identify the performance bottlenecks in my experimental Ceph cluster. A little background on my setup: 10 storage servers, each configured with: -(2) dual-core opterons -8 GB of RAM -(6) 750GB disks (1 OSD per disk,