[ceph-users] Reading a crushtool compare output

Hi all, I'd like to update the tunables on our older ceph cluster, created with firefly and now on luminous. I need to update two tunables, chooseleaf_vary_r from 2 to 1, and chooseleaf_stable from 0 to 1. I'm going to do 1 tunable update at a time. With the first one, I've dumped the current

Re: [ceph-users] Changing the release cadence

I think 12 months cycle is much better from the cluster operations perspective. I also like March as a release month as well. From: ceph-users on behalf of Sage Weil Sent: Thursday, 6 June 2019 1:57 AM To: ceph-us...@ceph.com; ceph-de...@vger.kernel.org;

Re: [ceph-users] Best practice for increasing number of pg and pgp

We use https://github.com/cernceph/ceph-scripts ceph-gentle-split script to slowly increase by 16 pgs at a time until we hit the target. From: ceph-users on behalf of Albert Yue Sent: Wednesday, 30 January 2019 1:39:40 PM To: ceph-users Subject: [ceph-users]

Re: [ceph-users] cephfs kernel, hang with libceph: osdx X.X.X.X socket closed (con state OPEN)

If you're using kernel client for cephfs, I strongly advise to have the client on the same subnet as the ceph public one i.e all traffic should be on the same subnet/VLAN. Even if your firewall situation is good, if you have to cross subnets or VLANs, you will run into weird problems later.

Re: [ceph-users] CephFS kernel client versions - pg-upmap

Kernel 4.13+ (i tested up to 4.18) missed some non-essential feature (explained by a Ceph dev on this ML) that was in Luminous, so they show up as Jewel, but otherwise they're fully compatible with upmap. We have a few hundred nodes on the kernel client with CephFS, and we also run balancer

Re: [ceph-users] cephfs kernel client stability

Might be a networking problem. Are your client nodes on the same subnet as ceph client network (i.e public_network in ceph.conf)? In my experience, the kernel client only likes being on the same public_network subnet as the MDS, Mons and OSDs. Else you get tons of weird issues. The fuse client

Re: [ceph-users] Ceph and NVMe

We have P3700s and Optane 900P (similar to P4800 but the workstation version and a lot cheaper) on R730xds, for WAL, DB and metadata pools for cephfs and radosgw. They perform great! From: ceph-users on behalf of Jeff Bailey Sent: Friday, 7 September 2018

Re: [ceph-users] No announce for 12.2.8 / available in repositories

With more testing and checking, we realised that this had nothing to do with Ceph. One part of the upgrade accidentally changed the MTU of our VMs tap interface from 9000 to 1500... Sorry for the false warning everyone! From: ceph-users on behalf of Linh Vu

Re: [ceph-users] No announce for 12.2.8 / available in repositories

We're going to reproduce this again in testing (12.2.8 drops right between our previous testing and going production) and compare it to 12.2.7. Will update with our findings soon. :) From: Dan van der Ster Sent: Tuesday, 4 September 2018 3:41:01 PM To: Linh Vu

Re: [ceph-users] No announce for 12.2.8 / available in repositories

Version 12.2.8 seems broken. Someone earlier on the ML had a MDS issue. We accidentally upgraded an openstack compute node from 12.2.7 to 12.2.8 (librbd) and it caused all kinds of issues writing to the VM disks. From: ceph-users on behalf of Nicolas Huillard

Re: [ceph-users] Cephfs meta data pool to ssd and measuring performance difference

Try IOR mdtest for metadata performance. From: ceph-users on behalf of Marc Roos Sent: Friday, 3 August 2018 7:49:13 PM To: dcsysengineer Cc: ceph-users Subject: Re: [ceph-users] Cephfs meta data pool to ssd and measuring performance difference I have moved

Re: [ceph-users] Mgr cephx caps to run `ceph fs status`?

Thanks John, that works! Also works with multiple commands, e.g I granted my user access to both `ceph fs status` and `ceph status`: mgr 'allow command "fs status", allow command "status"' From: John Spray Sent: Tuesday, 31 July 2018 8:12:

[ceph-users] Mgr cephx caps to run `ceph fs status`?

Hi all, I want a non-admin client to be able to run `ceph fs status`, either via the ceph CLI or a python script. Adding `mgr "allow *"` to this client's cephx caps works, but I'd like to be more specific if possible. I can't find the complete list of mgr cephx caps anywhere, so if you could

Re: [ceph-users] 12.2.7 - Available space decreasing when adding disks

Something funny going on with your new disks: 138 ssd 0.90970 1.0 931G 820G 111G 88.08 2.71 216 Added 139 ssd 0.90970 1.0 931G 771G 159G 82.85 2.55 207 Added 140 ssd 0.90970 1.0 931G 709G 222G 76.12 2.34 197 Added 141 ssd 0.90970 1.0 931G 664G 267G 71.31

Re: [ceph-users] Crush Rules with multiple Device Classes

Since the new NVMes are meant to replace the existing SSDs, why don't you assign class "ssd" to the new NVMe OSDs? That way you don't need to change the existing OSDs nor the existing crush rule. And the new NVMe OSDs won't lose any performance, "ssd" or "nvme" is just a name. When you deploy

Re: [ceph-users] [Ceph-maintainers] v12.2.7 Luminous released

Awesome, thank you Sage! With that explanation, it's actually a lot easier and less impacting than I thought. :) Cheers, Linh From: Sage Weil Sent: Thursday, 19 July 2018 9:35:33 AM To: Linh Vu Cc: Stefan Kooman; ceph-de...@vger.kernel.org; ceph-us

Re: [ceph-users] CephFS with erasure coding, do I need a cache-pool?

, and allow us to get more out of our flash drives. From: Oliver Schulz Sent: Wednesday, 18 July 2018 12:00:14 PM To: Linh Vu; ceph-users Subject: Re: [ceph-users] CephFS with erasure coding, do I need a cache-pool? Thanks, Linh! A question regarding choice of NVMe - do

Re: [ceph-users] CephFS with erasure coding, do I need a cache-pool?

To: Linh Vu; ceph-users Subject: Re: [ceph-users] CephFS with erasure coding, do I need a cache-pool? Dear Linh, another question, if I may: How do you handle Bluestore WAL and DB, and how much SSD space do you allocate for them? Cheers, Oliver On 17.07.2018 08:55, Linh Vu wrote: > Hi Oli

Re: [ceph-users] v12.2.7 Luminous released

Thanks for all your hard work in putting out the fixes so quickly! :) We have a cluster on 12.2.5 with Bluestore and EC pool but for CephFS, not RGW. In the release notes, it says RGW is a risk especially the garbage collection, and the recommendation is to either pause IO or disable RGW

Re: [ceph-users] CephFS with erasure coding, do I need a cache-pool?

Hi Oliver, We have several CephFS on EC pool deployments, one been in production for a while, the others about to pending all the Bluestore+EC fixes in 12.2.7  Firstly as John and Greg have said, you don't need SSD cache pool at all. Secondly, regarding k/m, it depends on how many hosts or

Re: [ceph-users] Luminous 12.2.6 release date?

Going by http://tracker.ceph.com/issues/24597, does this only affect FileStore OSDs or are BlueStore ones affected too? Cheers, Linh From: ceph-users on behalf of Sage Weil Sent: Thursday, 12 July 2018 3:48:10 AM To: Ken Dreyer Cc: ceph-users;

Re: [ceph-users] CephFS - How to handle "loaded dup inode" errors

the previously mentioned steps. Cheers, Linh From: John Spray Sent: Wednesday, 11 July 2018 8:00:29 PM To: Linh Vu Cc: Wido den Hollander; ceph-users@lists.ceph.com Subject: Re: [ceph-users] CephFS - How to handle "loaded dup inode" errors On Wed, Jul 11, 2018

Re: [ceph-users] CephFS - How to handle "loaded dup inode" errors

Sent: Wednesday, 11 July 2018 5:23:30 PM To: Linh Vu; John Spray Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] CephFS - How to handle "loaded dup inode" errors On 07/11/2018 01:47 AM, Linh Vu wrote: > Thanks John :) Has it - asserting out on dupe inode - already been >

Re: [ceph-users] CephFS - How to handle "loaded dup inode" errors

n take_inos start mds up again Is that correct? Many thanks, I've learned a lot more about this process. Cheers, Linh From: John Spray Sent: Tuesday, 10 July 2018 7:24 PM To: Linh Vu Cc: Wido den Hollander; ceph-users@lists.ceph.com Subject: Re: [ceph-users] CephFS - H

Re: [ceph-users] CephFS - How to handle "loaded dup inode" errors

Thanks John :) Has it - asserting out on dupe inode - already been logged as a bug yet? I could put one in if needed. Cheers, Linh From: John Spray Sent: Tuesday, 10 July 2018 7:11 PM To: Linh Vu Cc: Wido den Hollander; ceph-users@lists.ceph.com Subject: Re

Re: [ceph-users] CephFS - How to handle "loaded dup inode" errors

While we're on this topic, could someone please explain to me what `cephfs-table-tool all reset inode` does? Does it only reset what the MDS has in its cache, and after starting up again, the MDS will read in new inode range from the metadata pool? If so, does it mean *before* we run

Re: [ceph-users] CephFS - How to handle "loaded dup inode" errors

We're affected by something like this right now (the dup inode causing MDS to crash via assert(!p) with add_inode(CInode) function). In terms of behaviours, shouldn't the MDS simply skip to the next available free inode in the event of a dup, than crashing the entire FS because of one file?

Re: [ceph-users] Help! Luminous 12.2.5 CephFS - MDS crashed and now won't start (failing at MDCache::add_inode)

on behalf of Linh Vu Sent: Monday, 25 June 2018 7:06:45 PM To: ceph-users Subject: [ceph-users] Help! Luminous 12.2.5 CephFS - MDS crashed and now won't start (failing at MDCache::add_inode) Hi all, We have a Luminous 12.2.5 cluster, running entirely just CephFS with 1 active and 1 standby MDS

[ceph-users] Help! Luminous 12.2.5 CephFS - MDS crashed and now won't start (failing at MDCache::add_inode)

Hi all, We have a Luminous 12.2.5 cluster, running entirely just CephFS with 1 active and 1 standby MDS. The active MDS crashed and now won't start again with this same error: ### 0> 2018-06-25 16:11:21.136203 7f01c2749700 -1

Re: [ceph-users] Filestore -> Bluestore

ceph-volume lvm zap --destroy $DEVICE From: ceph-users on behalf of Vadim Bulst Sent: Tuesday, 12 June 2018 4:46:44 PM To: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Filestore -> Bluestore Thanks Sergey. Could you specify your answer a bit more?

Re: [ceph-users] Luminous 12.2.4: CephFS kernel client (4.15/4.16) shows up as jewel

I see, thanks a lot Ilya :) Will test that out. From: Ilya Dryomov Sent: Thursday, 31 May 2018 10:50:48 PM To: Heðin Ejdesgaard Møller Cc: Linh Vu; ceph-users Subject: Re: [ceph-users] Luminous 12.2.4: CephFS kernel client (4.15/4.16) shows up as jewel On Thu

[ceph-users] Luminous 12.2.4: CephFS kernel client (4.15/4.16) shows up as jewel

Hi all, On my test Luminous 12.2.4 cluster, with this set (initially so I could use upmap in the mgr balancer module): # ceph osd set-require-min-compat-client luminous # ceph osd dump | grep client require_min_compat_client luminous min_compat_client jewel Not quite sure why

Re: [ceph-users] Ceph-fuse getting stuck with "currently failed to authpin local pins"

That could be it. Every time it happens for me, it is indeed from a non-auth MDS. From: Yan, Zheng Sent: Wednesday, 30 May 2018 11:25:59 AM To: Linh Vu Cc: Oliver Freyermuth; Ceph Users; Peter Wienemann Subject: Re: [ceph-users] Ceph-fuse getting stuck

Re: [ceph-users] Ceph-fuse getting stuck with "currently failed to authpin local pins"

In my case, I have multiple active MDS (with directory pinning at the very top level), and there would be "Client xxx failing to respond to capability release" health warning every single time that happens. From: ceph-users on behalf of Yan, Zheng Sent:

Re: [ceph-users] Luminous cluster - how to find out which clients are still jewel?

) and kernel client (4.15.x). I suspect the latter is only supporting jewel features but I'd like to confirm. From: Massimo Sgaravatto Sent: Tuesday, 29 May 2018 4:51:56 PM To: Linh Vu Cc: ceph-users Subject: Re: [ceph-users] Luminous cluster - how to find out which

[ceph-users] Luminous cluster - how to find out which clients are still jewel?

Hi all, I have a Luminous 12.2.4 cluster. This is what `ceph features` tells me: ... "client": { "group": { "features": "0x7010fb86aa42ada", "release": "jewel", "num": 257 }, "group": { "features":

Re: [ceph-users] Ceph-fuse getting stuck with "currently failed to authpin local pins"

I get the exact opposite to the same error message "currently failed to authpin local pins". Had a few clients on ceph-fuse 12.2.2 and they ran into those issues a lot (evicting works). Upgrading to ceph-fuse 12.2.5 fixed it. The main cluster is on 12.2.4. The cause is user's HPC jobs or even

Re: [ceph-users] Can't get ceph mgr balancer to work (Luminous 12.2.4)

! From: ceph-users <ceph-users-boun...@lists.ceph.com> on behalf of Linh Vu <v...@unimelb.edu.au> Sent: Monday, 28 May 2018 2:26:41 PM To: ceph-users Subject: Re: [ceph-users] Can't get ceph mgr balancer to work (Luminous 12.2.4) I turned debug_mgr to 4/5 and found this whi

Re: [ceph-users] Can't get ceph mgr balancer to work (Luminous 12.2.4)

1 mgr[restful] Unknown request 'foo' 2018-05-28 14:24:02.574008 7fc3f67f8700 1 mgr[restful] Unknown request 'foo' 2018-05-28 14:24:02.574047 7fc3f67f8700 1 mgr[restful] Unknown request 'foo' 2018-05-28 14:24:02.574142 7fc3f67f8700 1 mgr[restful] Unknown request 'foo' ______

[ceph-users] Can't get ceph mgr balancer to work (Luminous 12.2.4)

Hi all, I'm testing out ceph mgr balancer as per http://docs.ceph.com/docs/master/mgr/balancer/ on our test cluster on Luminous 12.2.4, but can't seem to get it to work. Everything looks good in the prep, the new plan shows that it will take some actions, but it doesn't execute at all. Am I

Re: [ceph-users] Can Bluestore work with 2 replicas or still need 3 for data integrity?

You can use erasure code for your SSDs in Luminous if you're worried about cost per TB. From: ceph-users on behalf of Pardhiv Karri Sent: Friday, 25 May 2018 11:16:07 AM To: ceph-users Subject:

Re: [ceph-users] cephfs luminous 12.2.4 - multi-active MDSes with manual pinning

Thanks Patrick! Good to know that it's nothing and will be fixed soon :) From: Patrick Donnelly <pdonn...@redhat.com> Sent: Wednesday, 25 April 2018 5:17:57 AM To: Linh Vu Cc: ceph-users Subject: Re: [ceph-users] cephfs luminous 12.2.4 - multi-active

Re: [ceph-users] cephfs luminous 12.2.4 - multi-active MDSes with manual pinning

ed the warning list. Only restarting mds.0 so that the standby mds replaces it restored cluster health. Cheers, Linh From: Dan van der Ster <d...@vanderster.com> Sent: Tuesday, 24 April 2018 6:20:18 PM To: Linh Vu Cc: ceph-users Subject: Re: [ceph-users] ceph

[ceph-users] cephfs luminous 12.2.4 - multi-active MDSes with manual pinning

Hi all, I have a cluster running cephfs on Luminous 12.2.4, using 2 active MDSes + 1 standby. I have 3 shares: /projects, /home and /scratch, and I've decided to try manual pinning as described here: http://docs.ceph.com/docs/master/cephfs/multimds/ /projects is pinned to mds.0 (rank 0)

Re: [ceph-users] CephFS very unstable with many small files

Sounds like you just need more RAM on your MDS. Ours have 256GB each, and the OSD nodes have 128GB each. Networking is 2x25Gbe. We are on luminous 12.2.1, bluestore, and use CephFS for HPC, with about 500-ish compute nodes. We have done stress testing with small files up to 2M per directory

Re: [ceph-users] Luminous: Help with Bluestore WAL

Yeah that is the expected behaviour. From: ceph-users on behalf of Balakumar Munusawmy Sent: Wednesday, 21 February 2018 1:41:36 PM To: ceph-users@lists.ceph.com Subject: [ceph-users]

Re: [ceph-users] Automated Failover of CephFS Clients

You're welcome :) From: Paul Kunicki <pkuni...@sproutloud.com> Sent: Wednesday, 21 February 2018 1:16:32 PM To: Linh Vu Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Automated Failover of CephFS Clients Thanks for the hint Linh. I had neglected t

Re: [ceph-users] Automated Failover of CephFS Clients

Why are you mounting with a single monitor? What is your mount command or /etc/fstab? Ceph-fuse should use the available mons you have on the client's /etc/ceph/ceph.conf. e.g our /etc/fstab entry: none/home fuse.ceph

Re: [ceph-users] Help rebalancing OSD usage, Luminous 12.2.2

Your PG count per OSD looks really low, that might be why. I think in Luminous, you should aim for about 200. I'd use the pgcalc on ceph.com to verify. From: ceph-users on behalf of Bryan Banister

Re: [ceph-users] OSDs going down/up at random

Have you checked your firewall? From: ceph-users on behalf of Mike O'Connor Sent: Wednesday, 10 January 2018 3:40:30 PM To: ceph-users@lists.ceph.com Subject: [ceph-users] OSDs going down/up at random Hi All

Re: [ceph-users] ceph luminous - SSD partitions disssapeared

was checking whether the devices are owned by ceph:ceph. I did not have time to hunt that down though. From: ceph-users <ceph-users-boun...@lists.ceph.com> on behalf of Linh Vu <v...@unimelb.edu.au> Sent: Thursday, 4 January 2018 11:46:40 AM To: Sergey Mal

Re: [ceph-users] ceph luminous - SSD partitions disssapeared

Seen this issue when I first created our Luminous cluster. I use a custom systemd service to chown the DB and WAL partitions before ceph osd services get started. The script in /usr/local/sbin just does the chowning. ceph-nvme.service: # This is a workaround to chown the rocksdb and wal

Re: [ceph-users] Is the 12.2.1 really stable? Anybody have production cluster with Luminous Bluestore?

We have a small prod cluster with 12.2.1 and bluestore, running just cephfs, for HPC use. It's been in prod for about 7 weeks now, and pretty stable.  From: ceph-users on behalf of Eric Nelson Sent:

Re: [ceph-users] Ceph Luminous Directory

Luminous supports this now http://docs.ceph.com/docs/master/cephfs/dirfrags/ and in my testing it has handled 2M files per directory with no problem. Configuring Directory fragmentation — Ceph Documentation docs.ceph.com Configuring Directory

Re: [ceph-users] luminous vs jewel rbd performance

Noticed that you're on 12.2.0 Raf. 12.2.1 fixed a lot of performance issues from 12.2.0 for us on Luminous/Bluestore. Have you tried upgrading to it? From: ceph-users on behalf of Rafael Lopez Sent:

Re: [ceph-users] mount failed since failed to load ceph kernel module

Odd, you only got 2 mons and 0 osds? Your cluster build looks incomplete. Get Outlook for Android<https://aka.ms/ghei36> From: Dai Xiang <xiang@sky-data.cn> Sent: Tuesday, November 14, 2017 6:12:27 PM To: Linh Vu Cc: ceph-users@lists.ceph.com

Re: [ceph-users] mount failed since failed to load ceph kernel module

Your kernel is way too old for CephFS Luminous. I'd use one of the newer kernels from elrepo.org. :) We're on 4.12 here on RHEL 7.4. From: ceph-users on behalf of xiang@sky-data.cn Sent: Tuesday, 14

Re: [ceph-users] 答复: Re: Luminous LTS: `ceph osd crush class create` isgone?

h class create` isgone? 2017-11-03 7:59 GMT+01:00 Brad Hubbard <bhubb...@redhat.com<mailto:bhubb...@redhat.com>>: On Fri, Nov 3, 2017 at 4:04 PM, Linh Vu <v...@unimelb.edu.au<mailto:v...@unimelb.edu.au>> wrote: > Hi all, > > > Back in Luminous Dev and RC, I

[ceph-users] Luminous LTS: `ceph osd crush class create` is gone?

Hi all, Back in Luminous Dev and RC, I was able to do this: `ceph osd crush class create myclass` so I could utilise the new CRUSH device classes feature as described here: http://ceph.com/community/new-luminous-crush-device-classes/ and in use here:

Re: [ceph-users] cephfs: some metadata operations take seconds to complete

We're using cephfs here as well for HPC scratch, but we're on Luminous 12.2.1. This issue seems to have been fixed between Jewel and Luminous, we don't have such problems. :) Any reason you guys aren't evaluating the latest LTS? From: ceph-users

Re: [ceph-users] Can't start ceph-mon through systemctl start ceph-mon@.service after upgrading from Hammer to Jewel

Permissions of your mon data directory under /var/lib/ceph/mon/ might have changed as part of Hammer -> Jewel upgrade. Have you had a look there? From: ceph-users on behalf of 许雪寒 Sent: Thursday, 22 June 2017

Re: [ceph-users] Packages for Luminous RC 12.1.0?

No worries, thanks a lot, look forward to testing it :) From: Abhishek Lekshmanan <alekshma...@suse.de> Sent: Monday, 19 June 2017 10:03:15 PM To: Linh Vu; ceph-users Subject: Re: [ceph-users] Packages for Luminous RC 12.1.0? Linh Vu <v...@unimelb.edu.a

[ceph-users] Packages for Luminous RC 12.1.0?

Hi all, I saw that Luminous RC 12.1.0 has been mentioned in the latest release notes here: http://docs.ceph.com/docs/master/release-notes/ However, I can't see any 12.1.0 package yet on http://download.ceph.com Does anyone have any idea when the packages will be available? Thanks 