Re: [ceph-users] ceph-disk is now deprecated

Hello, Thanks for the heads-up. As someone who's currently maintaining a Jewel cluster and are in the process of setting up a shiny new Luminous cluster and writing Ansible roles in the process to make setup reproducible. I immediately proceeded to look into ceph-volume and I've some

Re: [ceph-users] I/O stalls when doing fstrim on large RBD

Correct -- the optimization is in Luminous and not in Jewel. On Mon, Nov 27, 2017 at 8:53 PM, Brendan Moloney wrote: > Thanks Jason. > > I am running Jewel currently. I do have object-map enabled, but it sounds > like the no-op discard optimization is specific to the newer

Re: [ceph-users] I/O stalls when doing fstrim on large RBD

Thanks Jason. I am running Jewel currently. I do have object-map enabled, but it sounds like the no-op discard optimization is specific to the newer librbd? Thanks again, Brendan From: Jason Dillaman [jdill...@redhat.com] Sent: Monday, November 27,

Re: [ceph-users] I/O stalls when doing fstrim on large RBD

My only possible suggestion would be to try a Luminous librbd client and enable the object-map feature on the image. When the object map is enabled, librbd will optimize away all the no-op discard requests. If that doesn't work, it could be an issue in the OS / controller interaction. On Mon, Nov

Re: [ceph-users] install ceph-osd failed in docker

Allocate disk space from where? If it's a physical device, then migration still requires moving hardware. If it isn't a physical device, why are you setting up OSDs without a physical device? Who is the user in your scenario? If I'm the user, then I use a physical device to create the OSD. If

Re: [ceph-users] luminous ceph-fuse crashes with "failed to remount for kernel dentry trimming"

Hi Patrick, We are using the CentOS 7.4 kernel (3.10.0-693.5.2.el7.x86_64).  The nodes in question have a fairly large amount of RAM (512GB), I don't see any evidence in any logs that the nodes ran out of memory (no OOM killer, and we have a small amount of swap that is used to catch memory

Re: [ceph-users] luminous ceph-fuse crashes with "failed to remount for kernel dentry trimming"

Hello Andras, On Mon, Nov 27, 2017 at 2:31 PM, Andras Pataki wrote: > After upgrading to the Luminous 12.2.1 ceph-fuse client, we've seen clients > on various nodes randomly crash at the assert > FAILED assert(0 == "failed to remount for kernel dentry

Re: [ceph-users] I/O stalls when doing fstrim on large RBD

Hi, Anyone have input on this? I am surprised there are not more people running into this issue. I guess most people don't have multi-TB RBD images? I think ext4 might also fair better since it does keep track of blocks that have been discarded in the past and not modified so that they

[ceph-users] luminous ceph-fuse crashes with "failed to remount for kernel dentry trimming"

Dear ceph users, After upgrading to the Luminous 12.2.1 ceph-fuse client, we've seen clients on various nodes randomly crash at the assert     FAILED assert(0 == "failed to remount for kernel dentry trimming") with the stack:  ceph version 12.2.1

[ceph-users] Upgrade from Jewel to Luminous. REQUIRE_JEWEL OSDMap

Hello, Could someone please help me complete my botched upgrade from Jewel 10.2.3-r1 to Luminous 12.2.1. I have 9 Gentoo servers, 4 of which have 2 OSDs each. My OSD servers were accidentally rebooted before the monitor servers causing them to be running Luminous before the monitors. All

Re: [ceph-users] Cephfs Hadoop Plugin and CEPH integration

On Mon, Nov 27, 2017 at 12:55 PM Aristeu Gil Alves Jr wrote: > Hi. > > It's my first post on the list. First of all I have to say I'm new on > hadoop. > > We are here a small lab and we have being running cephfs for almost two > years, loading it with large files (4GB to

Re: [ceph-users] Cephfs Hadoop Plugin and CEPH integration

Why not use swift. The intergration has been around for a while, and may be a better fit. https://hadoop.apache.org/docs/stable/hadoop-openstack/index.html On Mon, Nov 27, 2017 at 12:55 PM, Aristeu Gil Alves Jr wrote: > Hi. > > It's my first post on the list. First of

[ceph-users] Cephfs Hadoop Plugin and CEPH integration

Hi. It's my first post on the list. First of all I have to say I'm new on hadoop. We are here a small lab and we have being running cephfs for almost two years, loading it with large files (4GB to 4TB in size). Our cluster is with approximately with 400TB with ~75% of usage, and we are planning

Re: [ceph-users] ceph all-nvme mysql performance tuning

Hi Nick, yeah, we are using the same nvme disk with an additional partition to use as journal/wal. We double check the c-state and it was not configure to use c1, so we change that on all the osd nodes and mon nodes and we're going to make some new tests, and see how it goes. I'll get back as

Re: [ceph-users] ceph all-nvme mysql performance tuning

Hi David, Thanks a lot for the response. In fact, we first try to not use any scheduler at all, but then we try kyber iosched and we notice a slightly improve in terms of performance, that's why we actually keep it. *German* 2017-11-27 13:48 GMT-03:00 David Byte : > From the

Re: [ceph-users] ceph all-nvme mysql performance tuning

From the benchmarks I have seen and done myself, I’m not sure why you are using an i/o scheduler at all with NVMe. While there are a few cases where it may provide a slight benefit, simply having mq enabled with no scheduler seems to provide the best performance for an all flash, especially

Re: [ceph-users] ceph all-nvme mysql performance tuning

Also what tuned profile are you using? There is something to be gained by using a matching tuned profile for your workload. On Mon, Nov 27, 2017 at 11:16 AM, Donny Davis wrote: > Why not ask Red Hat? All the rest of the storage vendors you are looking > at are not free. >

Re: [ceph-users] ceph all-nvme mysql performance tuning

Why not ask Red Hat? All the rest of the storage vendors you are looking at are not free. Full disclosure, I am an employee at Red Hat. On Mon, Nov 27, 2017 at 10:16 AM, Nick Fisk wrote: > *From:* ceph-users [mailto:ceph-users-boun...@lists.ceph.com] *On Behalf > Of *German

Re: [ceph-users] ceph all-nvme mysql performance tuning

Hi German, We have similar config: proxmox-ve: 5.1-27 (running kernel: 4.13.8-1-pve) pve-manager: 5.1-36 (running version: 5.1-36/131401db) pve-kernel-4.13.8-1-pve: 4.13.8-27 ceph: 12.2.1-pve3 system(4 nodes): Supermicro 2028U-TN24R4T+ 2 port Mellanox connect x3pro 56Gbit 4 port intel 10GigE

Re: [ceph-users] ceph all-nvme mysql performance tuning

From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of German Anders Sent: 27 November 2017 14:44 To: Maged Mokhtar Cc: ceph-users Subject: Re: [ceph-users] ceph all-nvme mysql performance tuning Hi Maged, Thanks a lot

Re: [ceph-users] iSCSI gateway

Unfortunately, the latest EL 7.4 kernel does not have the necessary bug fixes. We are trying to get the fixes included in a future 7.4 release, but in the meantime we are planning for EL 7.5. We were also hoping to get the fixes included in upstream 4.14, but there was a maintainer shuffle, so

Re: [ceph-users] "failed to open ino"

On 27 Nov 2017 1:06 p.m., "Jens-U. Mozdzen" wrote: Hi David, Zitat von David C : Hi Jens > > We also see these messages quite frequently, mainly the "replicating > dir...". Only seen "failed to open ino" a few times so didn't do any real >

Re: [ceph-users] ceph all-nvme mysql performance tuning

Hi Maged, Thanks a lot for the response. We try with different number of threads and we're getting almost the same kind of difference between the storage types. Going to try with different rbd stripe size, object size values and see if we get more competitive numbers. Will get back with more

Re: [ceph-users] ceph all-nvme mysql performance tuning

On 2017-11-27 15:02, German Anders wrote: > Hi All, > > I've a performance question, we recently install a brand new Ceph cluster > with all-nvme disks, using ceph version 12.2.0 with bluestore configured. The > back-end of the cluster is using a bond IPoIB (active/passive) , and for the >

Re: [ceph-users] iSCSI gateway

(moving to ceph-users mailing list since this isn't development related) Have you tried following the documentation located here [1]? Any particular step you are having trouble with? There are no repos for the Open-iSCSI packages of rtslib-fb and tcmu-runner since those will eventually trickle

Re: [ceph-users] ceph all-nvme mysql performance tuning

Hi Wido, thanks a lot for the quick response, regarding the questions: Have you tried to attach multiple RBD volumes: - Root for OS (the root partition has local SSDs) - MySQL data dir (the idea is to have all the storage tests with the same scheme, the first test is using one volume and put the

[ceph-users] ceph-disk is now deprecated

For the upcoming Luminous release (12.2.2), ceph-disk will be officially in 'deprecated' mode (bug fixes only). A large banner with deprecation information has been added, which will try to raise awareness. We are strongly suggesting using ceph-volume for new (and old) OSD deployments. The only

Re: [ceph-users] ceph all-nvme mysql performance tuning

> Op 27 november 2017 om 14:14 schreef German Anders : > > > Hi Jason, > > We are using librbd (librbd1-0.80.5-9.el6.x86_64), ok I will change those > parameters and see if that changes something > 0.80? Is that a typo? You should really use 12.2.1 on the client. Wido

Re: [ceph-users] ceph all-nvme mysql performance tuning

> Op 27 november 2017 om 14:02 schreef German Anders : > > > Hi All, > > I've a performance question, we recently install a brand new Ceph cluster > with all-nvme disks, using ceph version 12.2.0 with bluestore configured. > The back-end of the cluster is using a bond

Re: [ceph-users] ceph all-nvme mysql performance tuning

Hi Jason, We are using librbd (librbd1-0.80.5-9.el6.x86_64), ok I will change those parameters and see if that changes something thanks a lot best, *German* 2017-11-27 10:09 GMT-03:00 Jason Dillaman : > Are you using krbd or librbd? You might want to consider "debug_ms

Re: [ceph-users] ceph all-nvme mysql performance tuning

Are you using krbd or librbd? You might want to consider "debug_ms = 0/0" as well since per-message log gathering takes a large hit on small IO performance. On Mon, Nov 27, 2017 at 8:02 AM, German Anders wrote: > Hi All, > > I've a performance question, we recently install

Re: [ceph-users] "failed to open ino"

Hi David, Zitat von David C : Hi Jens We also see these messages quite frequently, mainly the "replicating dir...". Only seen "failed to open ino" a few times so didn't do any real investigation. Our set up is very similar to yours, 12.2.1, active/standby MDS and

[ceph-users] ceph all-nvme mysql performance tuning

Hi All, I've a performance question, we recently install a brand new Ceph cluster with all-nvme disks, using ceph version 12.2.0 with bluestore configured. The back-end of the cluster is using a bond IPoIB (active/passive) , and for the front-end we are using a bonding config with active/active

Re: [ceph-users] "failed to open ino"

Hi Jens We also see these messages quite frequently, mainly the "replicating dir...". Only seen "failed to open ino" a few times so didn't do any real investigation. Our set up is very similar to yours, 12.2.1, active/standby MDS and exporting cephfs through KNFS (hoping to replace with Ganesha

Re: [ceph-users] CephFS 12.2.0 -> 12.2.1 change in inode caching behaviour

Yep, that did it! Thanks, Zheng. I should read release notes more carefully! On Fri, Nov 24, 2017 at 7:09 AM, Yan, Zheng wrote: > On Thu, Nov 23, 2017 at 9:17 PM, David C wrote: > > Hi All > > > > I upgraded my 12.2.0 cluster to 12.2.1 a month or two

Re: [ceph-users] CephFS desync

No, it doesn't. But I'll keep this commit in mind, thanks! On 24 November 2017 at 03:33, Yan, Zheng wrote: > On Thu, Nov 23, 2017 at 5:49 PM, Andrey Klimentyev > wrote: > > The workload is... really common. It's just a bunch of PHP scripts being

Re: [ceph-users] "failed to open ino"

Hi, Zitat von "Yan, Zheng" : On Sat, Nov 25, 2017 at 2:27 AM, Jens-U. Mozdzen wrote: [...] In the log of the active MDS, we currently see the following two inodes reported over and over again, about every 30 seconds: --- cut here --- 2017-11-24

Re: [ceph-users] "failed to open ino"

On Sat, Nov 25, 2017 at 2:27 AM, Jens-U. Mozdzen wrote: > Hi all, > > with our Ceph Luminous CephFS, we're plaqued with "failed to open ino" > messages. These don't seem to affect daily business (in terms of "file > access"). (There's a backup performance issue that may