Re: [ceph-users] reproducible rbd-nbd crashes

2019-08-13 Thread Marc Schöchlin
Hello Jason, thanks for your response. See my inline comments. Am 31.07.19 um 14:43 schrieb Jason Dillaman: > On Wed, Jul 31, 2019 at 6:20 AM Marc Schöchlin wrote: > > > The problem not seems to be related to kernel releases, filesystem types or > the ceph and network setup. > Release 12.2.5

Re: [ceph-users] Cephfs cannot mount with kernel client

2019-08-13 Thread Serkan Çoban
I am out of office right now, but I am pretty sure it was the same stack trace as in tracker. I will confirm tomorrow. Any workarounds? On Tue, Aug 13, 2019 at 5:16 PM Ilya Dryomov wrote: > > On Tue, Aug 13, 2019 at 3:57 PM Serkan Çoban wrote: > > > > I checked /var/log/messages and see there

[ceph-users] Ceph capacity versus pool replicated size discrepancy?

2019-08-13 Thread Kenneth Van Alstyne
Hey guys, this is probably a really silly question, but I’m trying to reconcile where all of my space has gone in one cluster that I am responsible for. The cluster is made up of 36 2TB SSDs across 3 nodes (12 OSDs per node), all using FileStore on XFS. We are running Ceph Luminous 12.2.8 on

[ceph-users] Time of response of "rbd ls" command

2019-08-13 Thread Gesiel Galvão Bernardes
HI, I recently noticed that in two of my pools the command "rbd ls" has take several minutes to return the values. These pools have between 100 and 120 images each. Where should I look to check why this slowness? The cluster is apparently fine, without any warning. Thank you very much in

[ceph-users] CephFS "denied reconnect attempt" after updating Ceph

2019-08-13 Thread William Edwards
Hello, I've been using CephFS for quite a while now, and am very happy with it. However, I'm experiencing an issue that's quite hard to debug. On almost every server where CephFS is mounted, the CephFS mount becomes unusable after updating Ceph (has happened 3 times now, after Ceph update).

Re: [ceph-users] More than 100% in a dashboard PG Status

2019-08-13 Thread DHilsbos
All; I also noticed this behavior. It may have started after inducing a failure in the cluster in order to observe the self-healing behavior. In the "PG Status" section of the dashboard, I have "Clean (200%)." This has not seemed to affect the functioning of the cluster. Cluster is a new

Re: [ceph-users] Cephfs cannot mount with kernel client

2019-08-13 Thread Ilya Dryomov
On Tue, Aug 13, 2019 at 4:30 PM Serkan Çoban wrote: > > I am out of office right now, but I am pretty sure it was the same > stack trace as in tracker. > I will confirm tomorrow. > Any workarounds? Compaction # echo 1 >/proc/sys/vm/compact_memory might help if the memory in question is

[ceph-users] WAL/DB size

2019-08-13 Thread Hemant Sonawane
Hi All, I have 4 6TB of HDD and 2 450GB SSD and I am going to partition each disk to 220GB for rock.db. So my question is does it make sense to use wal for my configuration? if yes then what could be the size of it? help will be really appreciated. -- Thanks and Regards, Hemant Sonawane

Re: [ceph-users] More than 100% in a dashboard PG Status

2019-08-13 Thread Fyodor Ustinov
Hi! I create Bug #41234 thanx for your advice! - Original Message - > From: "Lenz Grimmer" > To: "ceph-users" > Cc: "Alfonso Martinez Hidalgo" > Sent: Tuesday, 13 August, 2019 16:13:18 > Subject: Re: [ceph-users] More than 100% in a dashboard PG Status > Hi Fyodor, > > (Cc:ing

[ceph-users] add writeback to Bluestore thanks to lvm-writecache

2019-08-13 Thread Olivier Bonvalet
Hi, we use OSDs with data on HDD and db/wal on NVMe. But for now, BlueStore.DB and BlueStore.WAL only store medadata NOT data. Right ? So, when we migrated from : A) Filestore + HDD with hardware writecache + journal on SSD to : B) Bluestore + HDD without hardware writecache + DB/WAL on NVMe

Re: [ceph-users] Time of response of "rbd ls" command

2019-08-13 Thread Ilya Dryomov
On Tue, Aug 13, 2019 at 6:37 PM Gesiel Galvão Bernardes wrote: > > HI, > > I recently noticed that in two of my pools the command "rbd ls" has take > several minutes to return the values. These pools have between 100 and 120 > images each. > > Where should I look to check why this slowness? The

Re: [ceph-users] WAL/DB size

2019-08-13 Thread DHilsbos
Wildo / Hemant; Current recommendations (since at least luminous) say that a block.db device should be at least 4% of the block device. For a 6 TB drive, this would be 240 GB, not 60 GB. Thank you, Dominic L. Hilsbos, MBA Director – Information Technology Perform Air International Inc.

[ceph-users] Canonical Livepatch broke CephFS client

2019-08-13 Thread Tim Bishop
Hi, This email is mostly a heads up for others who might be using Canonical's livepatch on Ubuntu on a CephFS client. I have an Ubuntu 18.04 client with the standard kernel currently at version linux-image-4.15.0-54-generic 4.15.0-54.58. CephFS is mounted with the kernel client. Cluster is

Re: [ceph-users] WAL/DB size

2019-08-13 Thread Paul Emmerich
On Tue, Aug 13, 2019 at 10:04 PM Wido den Hollander wrote: > I just checked an RGW-only setup. 6TB drive, 58% full, 11.2GB of DB in > use. No slow db in use. random rgw-only setup here: 12TB drive, 77% full, 48GB metadata and 10GB omap for index and whatever. That's 0.5% + 0.1%. And that's a

Re: [ceph-users] WAL/DB size

2019-08-13 Thread Wido den Hollander
On 8/13/19 5:54 PM, Hemant Sonawane wrote: > Hi All, > I have 4 6TB of HDD and 2 450GB SSD and I am going to partition each > disk to 220GB for rock.db. So my question is does it make sense to use > wal for my configuration? if yes then what could be the size of it? help > will be really

Re: [ceph-users] WAL/DB size

2019-08-13 Thread Wido den Hollander
On 8/13/19 10:00 PM, dhils...@performair.com wrote: > Wildo / Hemant; > > Current recommendations (since at least luminous) say that a block.db device > should be at least 4% of the block device. For a 6 TB drive, this would be > 240 GB, not 60 GB. I know and I don't agree with that. I'm

Re: [ceph-users] WAL/DB size

2019-08-13 Thread Mark Nelson
On 8/13/19 3:51 PM, Paul Emmerich wrote: On Tue, Aug 13, 2019 at 10:04 PM Wido den Hollander wrote: I just checked an RGW-only setup. 6TB drive, 58% full, 11.2GB of DB in use. No slow db in use. random rgw-only setup here: 12TB drive, 77% full, 48GB metadata and 10GB omap for index and

Re: [ceph-users] reproducible rbd-nbd crashes

2019-08-13 Thread Mike Christie
On 08/13/2019 07:04 PM, Mike Christie wrote: > On 07/31/2019 05:20 AM, Marc Schöchlin wrote: >> Hello Jason, >> >> it seems that there is something wrong in the rbd-nbd implementation. >> (added this information also at https://tracker.ceph.com/issues/40822) >> >> The problem not seems to be

Re: [ceph-users] reproducible rbd-nbd crashes

2019-08-13 Thread Mike Christie
On 07/31/2019 05:20 AM, Marc Schöchlin wrote: > Hello Jason, > > it seems that there is something wrong in the rbd-nbd implementation. > (added this information also at https://tracker.ceph.com/issues/40822) > > The problem not seems to be related to kernel releases, filesystem types or > the

Re: [ceph-users] Ceph capacity versus pool replicated size discrepancy?

2019-08-13 Thread Konstantin Shalygin
Hey guys, this is probably a really silly question, but I’m trying to reconcile where all of my space has gone in one cluster that I am responsible for. The cluster is made up of 36 2TB SSDs across 3 nodes (12 OSDs per node), all using FileStore on XFS. We are running Ceph Luminous 12.2.8 on

[ceph-users] Sudden loss of all SSD OSDs in a cluster, immedaite abort on restart [Mimic 13.2.6]

2019-08-13 Thread Troy Ablan
I've opened a tracker issue at https://tracker.ceph.com/issues/41240 Background: Cluster of 13 hosts, 5 of which contain 14 SSD OSDs between them. 409 HDDs in as well. The SSDs contain the RGW index and log pools, and some smaller pools The HDDs ccontain all other pools, including the RGW

Re: [ceph-users] MDS corruption

2019-08-13 Thread ☣Adam
Pierre Dittes helped me with adding --rank=yourfsname:all and I ran the following steps from the disaster recovery page: journal export, dentry recovery, journal truncation, mds table wipes (session, snap and inode), scan_extents, scan_inodes, scan_links, and cleanup. Now all three of my MDS

[ceph-users] CephFS meltdown fallout: mds assert failure, kernel oopses

2019-08-13 Thread Hector Martin
I just had a minor CephFS meltdown caused by underprovisioned RAM on the MDS servers. This is a CephFS with two ranks; I manually failed over the first rank and the new MDS server ran out of RAM in the rejoin phase (ceph-mds didn't get OOM-killed, but I think things slowed down enough due to

Re: [ceph-users] optane + 4x SSDs for VM disk images?

2019-08-13 Thread vitalif
Could performance of Optane + 4x SSDs per node ever exceed that of pure Optane disks? No. With Ceph, the results for Optane and just for good server SSDs are almost the same. One thing is that you can run more OSDs per an Optane than per a usual SSD. However, the latency you get from both is

Re: [ceph-users] RGW how to delete orphans

2019-08-13 Thread Andrei Mikhailovsky
Hello I was hoping to follow up on this email and if Florian manage to get to the bottom of this. I have a case where I believe my RGW bucket is using too much space. For me, the ceph df command shows over 16TB usage, whereas the bucket stats shows the total of about 6TB. So, It seems that

[ceph-users] More than 100% in a dashboard PG Status

2019-08-13 Thread Fyodor Ustinov
Hi! I have ceph nautilus (upgraded from mimic, if it is important) and in dashboard in "PG Status" section I see "Clean (2397%)" It's a bug? WBR, Fyodor. ___ ceph-users mailing list ceph-users@lists.ceph.com

[ceph-users] Cephfs cannot mount with kernel client

2019-08-13 Thread Serkan Çoban
Hi, Just installed nautilus 14.2.2 and setup cephfs on it. OS is all centos 7.6. >From a client I can mount the cephfs with ceph-fuse, but I cannot mount with ceph kernel client. It gives "mount error 110 connection timeout" and I can see "libceph: corrupt full osdmap (-12) epoch 2759 off 656" in

Re: [ceph-users] Cephfs cannot mount with kernel client

2019-08-13 Thread Serkan Çoban
I checked /var/log/messages and see there are page allocation failures. But I don't understand why? The client has 768GB memory and most of it is not used, cluster has 1500OSDs. Do I need to increase vm.min_free_kytes? It is set to 1GB now. Also huge_page is disabled in clients. Thanks, Serkan

Re: [ceph-users] Cephfs cannot mount with kernel client

2019-08-13 Thread Ilya Dryomov
On Tue, Aug 13, 2019 at 3:57 PM Serkan Çoban wrote: > > I checked /var/log/messages and see there are page allocation > failures. But I don't understand why? > The client has 768GB memory and most of it is not used, cluster has > 1500OSDs. Do I need to increase vm.min_free_kytes? It is set to 1GB

Re: [ceph-users] More than 100% in a dashboard PG Status

2019-08-13 Thread Lenz Grimmer
Hi Fyodor, (Cc:ing Alfonso) On 8/13/19 12:47 PM, Fyodor Ustinov wrote: > I have ceph nautilus (upgraded from mimic, if it is important) and in > dashboard in "PG Status" section I see "Clean (2397%)" > > It's a bug? Huh, That might be possible - sorry about that. We'd be grateful if you could

Re: [ceph-users] MDS corruption

2019-08-13 Thread Yan, Zheng
nautilus version (14.2.2) of ‘cephfs-data-scan scan_links’ can fix snaptable. hopefully it will fix your issue. you don't need to upgrade whole cluster. Just install nautilus in a temp machine or compile ceph from source. On Tue, Aug 13, 2019 at 2:35 PM Adam wrote: > > Pierre Dittes helped

Re: [ceph-users] Cephfs cannot mount with kernel client

2019-08-13 Thread Ilya Dryomov
On Tue, Aug 13, 2019 at 12:36 PM Serkan Çoban wrote: > > Hi, > > Just installed nautilus 14.2.2 and setup cephfs on it. OS is all centos 7.6. > From a client I can mount the cephfs with ceph-fuse, but I cannot > mount with ceph kernel client. > It gives "mount error 110 connection timeout" and I