Re: [ceph-users] slow_used_bytes - SlowDB being used despite lots of space free in BlockDB on SSD?

2018-10-19 Thread Nick Fisk
> -Original Message- > From: Nick Fisk [mailto:n...@fisk.me.uk] > Sent: 19 October 2018 08:15 > To: 'Igor Fedotov' ; ceph-users@lists.ceph.com > Subject: RE: [ceph-users] slow_used_bytes - SlowDB being used despite lots of > space free in BlockDB on SSD? > >

Re: [ceph-users] slow_used_bytes - SlowDB being used despite lots of space free in BlockDB on SSD?

2018-10-19 Thread Nick Fisk
> > On 10/18/2018 7:49 PM, Nick Fisk wrote: > > Hi, > > > > Ceph Version = 12.2.8 > > 8TB spinner with 20G SSD partition > > > > Perf dump shows the following: > > > > "bluefs": { > > "gift_bytes": 0,

[ceph-users] slow_used_bytes - SlowDB being used despite lots of space free in BlockDB on SSD?

2018-10-18 Thread Nick Fisk
Hi, Ceph Version = 12.2.8 8TB spinner with 20G SSD partition Perf dump shows the following: "bluefs": { "gift_bytes": 0, "reclaim_bytes": 0, "db_total_bytes": 21472731136, "db_used_bytes": 3467640832, "wal_total_bytes": 0, "wal_used_bytes": 0,

Re: [ceph-users] Bluestore DB size and onode count

2018-09-10 Thread Nick Fisk
2 PM, Igor Fedotov wrote: > > > Hi Nick. > > > > > > On 9/10/2018 1:30 PM, Nick Fisk wrote: > >> If anybody has 5 minutes could they just clarify a couple of things > >> for me > >> > >> 1. onode count, should this be equal to the nu

[ceph-users] Bluestore DB size and onode count

2018-09-10 Thread Nick Fisk
If anybody has 5 minutes could they just clarify a couple of things for me 1. onode count, should this be equal to the number of objects stored on the OSD? Through reading several posts, there seems to be a general indication that this is the case, but looking at my OSD's the maths don't work.

[ceph-users] Tiering stats are blank on Bluestore OSD's

2018-09-10 Thread Nick Fisk
After upgrading a number of OSD's to Bluestore I have noticed that the cache tier OSD's which have so far been upgraded are no longer logging tier_* stats "tier_promote": 0, "tier_flush": 0, "tier_flush_fail": 0, "tier_try_flush": 0, "tier_try_flush_fail":

Re: [ceph-users] help needed

2018-09-06 Thread Nick Fisk
If it helps, I’m seeing about a 3GB DB usage for a 3TB OSD about 60% full. This is with a pure RBD workload, I believe this can vary depending on what your Ceph use case is. From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of David Turner Sent: 06 September 2018 14:09

Re: [ceph-users] CephFS+NFS For VMWare

2018-07-02 Thread Nick Fisk
Quoting Ilya Dryomov : On Fri, Jun 29, 2018 at 8:08 PM Nick Fisk wrote: This is for us peeps using Ceph with VMWare. My current favoured solution for consuming Ceph in VMWare is via RBD’s formatted with XFS and exported via NFS to ESXi. This seems to perform better than iSCSI+VMFS

Re: [ceph-users] CephFS+NFS For VMWare

2018-06-30 Thread Nick Fisk
is of greater concern. Thanks, Nick From: Paul Emmerich [mailto:paul.emmer...@croit.io] Sent: 29 June 2018 17:57 To: Nick Fisk Cc: ceph-users Subject: Re: [ceph-users] CephFS+NFS For VMWare VMWare can be quite picky about NFS servers. Some things that you should test before deploying

[ceph-users] CephFS+NFS For VMWare

2018-06-29 Thread Nick Fisk
This is for us peeps using Ceph with VMWare. My current favoured solution for consuming Ceph in VMWare is via RBD's formatted with XFS and exported via NFS to ESXi. This seems to perform better than iSCSI+VMFS which seems to not play nicely with Ceph's PG contention issues particularly if

Re: [ceph-users] FAILED assert(p != recovery_info.ss.clone_snaps.end())

2018-06-14 Thread Nick Fisk
] On Behalf Of Nick Fisk Sent: 07 June 2018 14:01 To: 'ceph-users' Subject: Re: [ceph-users] FAILED assert(p != recovery_info.ss.clone_snaps.end()) So I've recompiled a 12.2.5 ceph-osd binary with the fix included in https://github.com/ceph/ceph/pull/22396 The OSD has restarted as expected and the PG

Re: [ceph-users] How to fix a Ceph PG in unkown state with no OSDs?

2018-06-14 Thread Nick Fisk
I’ve seen similar things like this happen if you tend to end up with extreme weighting towards a small set of OSD’s. Crush tries a slightly different combination of OSD’s at each attempt, but in an extremely lop sided weighting, it can run out of attempts before it finds a set of OSD’s which

Re: [ceph-users] Why the change from ceph-disk to ceph-volume and lvm? (and just not stick with direct disk access)

2018-06-08 Thread Nick Fisk
http://docs.ceph.com/docs/master/ceph-volume/simple/ ? From: ceph-users On Behalf Of Konstantin Shalygin Sent: 08 June 2018 11:11 To: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Why the change from ceph-disk to ceph-volume and lvm? (and just not stick with direct disk access)

Re: [ceph-users] FAILED assert(p != recovery_info.ss.clone_snaps.end())

2018-06-07 Thread Nick Fisk
-store-tool, but not sure if I want to clean the clone metadata or try and remove the actual snapshot object. -Original Message- From: ceph-users On Behalf Of Nick Fisk Sent: 05 June 2018 17:22 To: 'ceph-users' Subject: Re: [ceph-users] FAILED assert(p != recovery_info.ss.clone_snaps.end

Re: [ceph-users] FAILED assert(p != recovery_info.ss.clone_snaps.end())

2018-06-05 Thread Nick Fisk
uot; snapshot object and then allow thigs to backfill? -Original Message- From: ceph-users On Behalf Of Nick Fisk Sent: 05 June 2018 16:43 To: 'ceph-users' Subject: [ceph-users] FAILED assert(p != recovery_info.ss.clone_snaps.end()) Hi, After a RBD snapshot was removed, I seem to be havi

Re: [ceph-users] FAILED assert(p != recovery_info.ss.clone_snaps.end())

2018-06-05 Thread Nick Fisk
From: ceph-users On Behalf Of Paul Emmerich Sent: 05 June 2018 17:02 To: n...@fisk.me.uk Cc: ceph-users Subject: Re: [ceph-users] FAILED assert(p != recovery_info.ss.clone_snaps.end()) 2018-06-05 17:42 GMT+02:00 Nick Fisk mailto:n...@fisk.me.uk> >: Hi, After a RBD sn

[ceph-users] FAILED assert(p != recovery_info.ss.clone_snaps.end())

2018-06-05 Thread Nick Fisk
Hi, After a RBD snapshot was removed, I seem to be having OSD's assert when they try and recover pg 1.2ca. The issue seems to follow the PG around as OSD's fail. I've seen this bug tracker and associated mailing list post, but would appreciate if anyone can give any pointers.

Re: [ceph-users] Intel Xeon Scalable and CPU frequency scaling on NVMe/SSD Ceph OSDs

2018-05-14 Thread Nick Fisk
l.com> Cc: 'ceph-users' <ceph-users@lists.ceph.com> Subject: Re: [ceph-users] Intel Xeon Scalable and CPU frequency scaling on NVMe/SSD Ceph OSDs On 05/01/2018 10:19 PM, Nick Fisk wrote: > 4.16 required? > https://www.phoronix.com/scan.php?page=news_item=Skylake-X-P-State- >

[ceph-users] Scrubbing impacting write latency since Luminous

2018-05-10 Thread Nick Fisk
Hi All, I've just upgraded our main cluster to Luminous and have noticed that where before the cluster 64k write latency was always hovering around 2ms regardless of what scrubbing was going on, since the upgrade to Luminous, scrubbing takes the average latency up to around 5-10ms and deep

Re: [ceph-users] Bluestore on HDD+SSD sync write latency experiences

2018-05-03 Thread Nick Fisk
s case writing the IO's through the NVME first seems to help by quite a large margin. I'm curious what was the original rationale for 32kB? Cheers, Dan On Tue, May 1, 2018 at 10:50 PM, Nick Fisk <n...@fisk.me.uk> wrote: Hi all, Slowly getting round to migrating clusters to Bluest

Re: [ceph-users] Bluestore on HDD+SSD sync write latency experiences

2018-05-03 Thread Nick Fisk
Hi Nick, On 5/1/2018 11:50 PM, Nick Fisk wrote: Hi all, Slowly getting round to migrating clusters to Bluestore but I am interested in how people are handling the potential change in write latency coming from Filestore? Or maybe nobody is really seeing much difference? As we all know

Re: [ceph-users] Bluestore on HDD+SSD sync write latency experiences

2018-05-03 Thread Nick Fisk
-Original Message- From: Alex Gorbachev <a...@iss-integration.com> Sent: 02 May 2018 22:05 To: Nick Fisk <n...@fisk.me.uk> Cc: ceph-users <ceph-users@lists.ceph.com> Subject: Re: [ceph-users] Bluestore on HDD+SSD sync write latency experiences Hi Nick, On Tue, May

[ceph-users] Bluestore on HDD+SSD sync write latency experiences

2018-05-01 Thread Nick Fisk
Hi all, Slowly getting round to migrating clusters to Bluestore but I am interested in how people are handling the potential change in write latency coming from Filestore? Or maybe nobody is really seeing much difference? As we all know, in Bluestore, writes are not double written and in

Re: [ceph-users] Intel Xeon Scalable and CPU frequency scaling on NVMe/SSD Ceph OSDs

2018-05-01 Thread Nick Fisk
<ceph-users@lists.ceph.com>; Nick Fisk <n...@fisk.me.uk> Subject: Re: [ceph-users] Intel Xeon Scalable and CPU frequency scaling on NVMe/SSD Ceph OSDs Also curious about this over here. We've got a rack's worth of R740XDs with Xeon 4114's running RHEL 7.4 and intel-pstate isn't even active

Re: [ceph-users] pgs down after adding 260 OSDs & increasing PGs

2018-01-29 Thread Nick Fisk
Hi Jake, I suspect you have hit an issue that me and a few others have hit in Luminous. By increasing the number of PG's before all the data has re-balanced, you have probably exceeded hard PG per OSD limit. See this thread https://www.spinics.net/lists/ceph-users/msg41231.html Nick >

Re: [ceph-users] BlueStore.cc: 9363: FAILED assert(0 == "unexpected error")

2018-01-26 Thread Nick Fisk
I can see this in the logs: 2018-01-25 06:05:56.292124 7f37fa6ea700 -1 log_channel(cluster) log [ERR] : full status failsafe engaged, dropping updates, now 101% full 2018-01-25 06:05:56.325404 7f3803f9c700 -1 bluestore(/var/lib/ceph/osd/ceph-9) _do_alloc_write failed to reserve 0x4000

Re: [ceph-users] OSD servers swapping despite having free memory capacity

2018-01-24 Thread Nick Fisk
I know this may be a bit vague, but also suggests the "try a newer kernel" approach. We had constant problems with hosts mounting a number of RBD volumes formatted with XFS. The servers would start aggressively swapping even though the actual memory in use was nowhere near even 50% and

Re: [ceph-users] What is the should be the expected latency of 10Gbit network connections

2018-01-22 Thread Nick Fisk
Anyone with 25G ethernet willing to do the test? Would love to see what the latency figures are for that. From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Maged Mokhtar Sent: 22 January 2018 11:28 To: ceph-users@lists.ceph.com Subject: Re: [ceph-users] What is the

Re: [ceph-users] Ubuntu 17.10 or Debian 9.3 + Luminous = random OS hang ?

2018-01-21 Thread Nick Fisk
How up to date is your VM environment? We saw something very similar last year with Linux VM’s running newish kernels. It turns out newer kernels supported a new feature of the vmxnet3 adapters which had a bug in ESXi. The fix was release last year some time in ESXi6.5 U1, or a workaround was

Re: [ceph-users] Cluster crash - FAILED assert(interval.last > last)

2018-01-11 Thread Nick Fisk
I take my hat off to you, well done for solving that!!! > -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Zdenek Janda > Sent: 11 January 2018 13:01 > To: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] Cluster crash - FAILED

[ceph-users] Linux Meltdown (KPTI) fix and how it affects performance?

2018-01-04 Thread Nick Fisk
Hi All, As the KPTI fix largely only affects the performance where there are a large number of syscalls made, which Ceph does a lot of, I was wondering if anybody has had a chance to perform any initial tests. I suspect small write latencies will the worse affected? Although I'm thinking the

Re: [ceph-users] Cache tiering on Erasure coded pools

2017-12-27 Thread Nick Fisk
Also carefully read the word of caution section on David's link (which is absent in the jewel version of the docs), a cache tier in front of an ersure coded data pool for RBD is almost always a bad idea. I would say that statement is incorrect if using Bluestore. If using Bluestore,

Re: [ceph-users] Bluestore Compression not inheriting pool option

2017-12-13 Thread Nick Fisk
Thanks for confirming, logged http://tracker.ceph.com/issues/22419 > -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Stefan Kooman > Sent: 12 December 2017 20:35 > To: Nick Fisk <n...@fisk.me.uk> > Cc: ceph-users@

Re: [ceph-users] Odd object blocking IO on PG

2017-12-13 Thread Nick Fisk
alf Of Nick Fisk Sent: 13 December 2017 11:14 To: 'Gregory Farnum' <gfar...@redhat.com> Cc: 'ceph-users' <ceph-users@lists.ceph.com> Subject: Re: [ceph-users] Odd object blocking IO on PG On Tue, Dec 12, 2017 at 12:33 PM Nick Fisk <n...@fisk.me.uk <mailto:n..

Re: [ceph-users] Health Error : Request Stuck

2017-12-13 Thread Nick Fisk
ter to rebalance. And it worked! To be honest, not exactly sure its the correct way. P.S : I had upgraded to Luminous 12.2.2 yesterday. Karun Josy On Wed, Dec 13, 2017 at 4:31 PM, Nick Fisk <n...@fisk.me.uk <mailto:n...@fisk.me.uk> > wrote: Hi Karun, I too am experie

Re: [ceph-users] Odd object blocking IO on PG

2017-12-13 Thread Nick Fisk
On Tue, Dec 12, 2017 at 12:33 PM Nick Fisk <n...@fisk.me.uk <mailto:n...@fisk.me.uk> > wrote: > That doesn't look like an RBD object -- any idea who is > "client.34720596.1:212637720"? So I think these might be proxy ops from the cache tier, as there are also b

Re: [ceph-users] Health Error : Request Stuck

2017-12-13 Thread Nick Fisk
Hi Karun, I too am experiencing something very similar with a PG stuck in activating+remapped state after re-introducing a OSD back into the cluster as Bluestore. Although this new OSD is not the one listed against the PG’s stuck activating. I also see the same thing as you where the up set

Re: [ceph-users] Odd object blocking IO on PG

2017-12-12 Thread Nick Fisk
g in the main status that it has been blocked from peering or that there are any missing objects. I've tried restarting all OSD's I can see relating to the PG in case they needed a bit of a nudge. > > On Tue, Dec 12, 2017 at 12:36 PM, Nick Fisk <n...@fisk.me.uk> wrote: > >

[ceph-users] Bluestore Compression not inheriting pool option

2017-12-12 Thread Nick Fisk
Hi All, Has anyone been testing the bluestore pool compression option? I have set compression=snappy on a RBD pool. When I add a new bluestore OSD, data is not being compressed when backfilling, confirmed by looking at the perf dump results. If I then set again the compression type on the pool

[ceph-users] Odd object blocking IO on PG

2017-12-12 Thread Nick Fisk
Does anyone know what this object (0.ae78c1cf) might be, it's not your normal run of the mill RBD object and I can't seem to find it in the pool using rados --all ls . It seems to be leaving the 0.1cf PG stuck in an activating+remapped state and blocking IO. Pool 0 is just a pure RBD pool with a

Re: [ceph-users] what's the maximum number of OSDs per OSD server?

2017-12-10 Thread Nick Fisk
make sure you size the nodes to a point that if one has to be taken offline for any reason, that you are happy with the resulting state of the cluster, including the peering when suddenly taking ~200 OSD’s offline/online. Nick On Sun, Dec 10, 2017 at 11:17 AM, Nick Fisk <n...@fisk.me

Re: [ceph-users] what's the maximum number of OSDs per OSD server?

2017-12-10 Thread Nick Fisk
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Igor Mendelev Sent: 10 December 2017 15:39 To: ceph-users@lists.ceph.com Subject: [ceph-users] what's the maximum number of OSDs per OSD server? Given that servers with 64 CPU cores (128 threads @ 2.7GHz) and up to 2TB

Re: [ceph-users] ceph all-nvme mysql performance tuning

2017-11-27 Thread Nick Fisk
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of German Anders Sent: 27 November 2017 14:44 To: Maged Mokhtar Cc: ceph-users Subject: Re: [ceph-users] ceph all-nvme mysql performance tuning Hi Maged, Thanks a lot

Re: [ceph-users] Bluestore performance 50% of filestore

2017-11-18 Thread Nick Fisk
ely to see for > real > on your cluster. > > Doing writes over a much larger file, say 3-4x over the total amount of RAM > in all of the nodes, helps you get a better idea of what the behavior is like > when those tricks are less effective. I think that's probably a more likely > s

Re: [ceph-users] bluestore - wal,db on faster devices?

2017-11-08 Thread Nick Fisk
evices? > > > > On 11/08/2017 03:16 PM, Nick Fisk wrote: > >> -Original Message- > >> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf > >> Of Mark Nelson > >> Sent: 08 November 2017 19:46 > >> To: Wolfgang Lendl

Re: [ceph-users] Blog post: storage server power consumption

2017-11-08 Thread Nick Fisk
Also look at the new WD 10TB Red's if you want very low use archive storage. Because they spin at 5400, they only use 2.8W at idle. > -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Jack > Sent: 06 November 2017 22:31 > To:

Re: [ceph-users] Recovery operations and ioprio options

2017-11-08 Thread Nick Fisk
> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > ??? ??? > Sent: 08 November 2017 16:21 > To: ceph-users@lists.ceph.com > Subject: [ceph-users] Recovery operations and ioprio options > > Hello, > Today we use ceph jewel with: >

Re: [ceph-users] bluestore - wal,db on faster devices?

2017-11-08 Thread Nick Fisk
> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Mark Nelson > Sent: 08 November 2017 19:46 > To: Wolfgang Lendl > Cc: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] bluestore - wal,db on faster

Re: [ceph-users] VMware + Ceph using NFS sync/async ?

2017-08-16 Thread Nick Fisk
Hi Matt, Well behaved applications are the problem here. ESXi sends all writes as sync writes. So although OS’s will still do their own buffering, any ESXi level operation is all done as sync. This is probably seen the greatest when migrating vm’s between datastores, everything gets done as

Re: [ceph-users] VMware + Ceph using NFS sync/async ?

2017-08-14 Thread Nick Fisk
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Osama Hasebou Sent: 14 August 2017 12:27 To: ceph-users Subject: [ceph-users] VMware + Ceph using NFS sync/async ? Hi Everyone, We started testing the idea of using Ceph storage with

Re: [ceph-users] luminous/bluetsore osd memory requirements

2017-08-14 Thread Nick Fisk
> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Ronny Aasen > Sent: 14 August 2017 18:55 > To: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] luminous/bluetsore osd memory requirements > > On 10.08.2017 17:30, Gregory Farnum wrote:

Re: [ceph-users] luminous/bluetsore osd memory requirements

2017-08-13 Thread Nick Fisk
. On Sat, Aug 12, 2017, 2:40 PM Nick Fisk <n...@fisk.me.uk <mailto:n...@fisk.me.uk> > wrote: I was under the impression the memory requirements for Bluestore would be around 2-3GB per OSD regardless of capacity. CPU wise, I would lean towards working out how much total Ghz you require

Re: [ceph-users] luminous/bluetsore osd memory requirements

2017-08-12 Thread Nick Fisk
I was under the impression the memory requirements for Bluestore would be around 2-3GB per OSD regardless of capacity. CPU wise, I would lean towards working out how much total Ghz you require and then get whatever CPU you need to get there, but with a preference of Ghz over cores. Yes, there

Re: [ceph-users] ceph cluster experiencing major performance issues

2017-08-08 Thread Nick Fisk
> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Mclean, Patrick > Sent: 08 August 2017 20:13 > To: David Turner ; ceph-us...@ceph.com > Cc: Colenbrander, Roelof ; Payno, > Victor

Re: [ceph-users] Kernel mounted RBD's hanging

2017-07-31 Thread Nick Fisk
> -Original Message- > From: Ilya Dryomov [mailto:idryo...@gmail.com] > Sent: 31 July 2017 11:36 > To: Nick Fisk <n...@fisk.me.uk> > Cc: Ceph Users <ceph-users@lists.ceph.com> > Subject: Re: [ceph-users] Kernel mounted RBD's hanging > > On Thu, Ju

Re: [ceph-users] RBD cache being filled up in small increases instead of 4MB

2017-07-15 Thread Nick Fisk
> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Gregory Farnum > Sent: 15 July 2017 00:09 > To: Ruben Rodriguez > Cc: ceph-users > Subject: Re: [ceph-users] RBD cache being filled up in small

Re: [ceph-users] Ceph mount rbd

2017-07-14 Thread Nick Fisk
> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Jason Dillaman > Sent: 14 July 2017 16:40 > To: li...@marcelofrota.info > Cc: ceph-users > Subject: Re: [ceph-users] Ceph mount rbd > > On Fri, Jul 14, 2017 at

Re: [ceph-users] Kernel mounted RBD's hanging

2017-07-12 Thread Nick Fisk
> -Original Message- > From: Nick Fisk [mailto:n...@fisk.me.uk] > Sent: 12 July 2017 13:47 > To: 'Ilya Dryomov' <idryo...@gmail.com> > Cc: 'Ceph Users' <ceph-users@lists.ceph.com> > Subject: RE: [ceph-users] Kernel mounted RBD's hanging > > > --

Re: [ceph-users] Kernel mounted RBD's hanging

2017-07-08 Thread Nick Fisk
> -Original Message- > From: Ilya Dryomov [mailto:idryo...@gmail.com] > Sent: 07 July 2017 11:32 > To: Nick Fisk <n...@fisk.me.uk> > Cc: Ceph Users <ceph-users@lists.ceph.com> > Subject: Re: [ceph-users] Kernel mounted RBD's hanging > > On Fri, J

Re: [ceph-users] Kernel mounted RBD's hanging

2017-07-07 Thread Nick Fisk
> -Original Message- > From: Ilya Dryomov [mailto:idryo...@gmail.com] > Sent: 01 July 2017 13:19 > To: Nick Fisk <n...@fisk.me.uk> > Cc: Ceph Users <ceph-users@lists.ceph.com> > Subject: Re: [ceph-users] Kernel mounted RBD's hanging > > On Sat,

Re: [ceph-users] Kernel mounted RBD's hanging

2017-07-01 Thread Nick Fisk
> -Original Message- > From: Ilya Dryomov [mailto:idryo...@gmail.com] > Sent: 30 June 2017 14:06 > To: Nick Fisk <n...@fisk.me.uk> > Cc: Ceph Users <ceph-users@lists.ceph.com> > Subject: Re: [ceph-users] Kernel mounted RBD's hanging > > On Fri, J

Re: [ceph-users] Kernel mounted RBD's hanging

2017-06-30 Thread Nick Fisk
> -Original Message- > From: Ilya Dryomov [mailto:idryo...@gmail.com] > Sent: 29 June 2017 18:54 > To: Nick Fisk <n...@fisk.me.uk> > Cc: Ceph Users <ceph-users@lists.ceph.com> > Subject: Re: [ceph-users] Kernel mounted RBD's hanging > > On Thu, J

Re: [ceph-users] Kernel mounted RBD's hanging

2017-06-30 Thread Nick Fisk
From: Alex Gorbachev [mailto:a...@iss-integration.com] Sent: 30 June 2017 03:54 To: Ceph Users <ceph-users@lists.ceph.com>; n...@fisk.me.uk Subject: Re: [ceph-users] Kernel mounted RBD's hanging On Thu, Jun 29, 2017 at 10:30 AM Nick Fisk <n...@fisk.me.uk <mailto:n...@fisk.me

Re: [ceph-users] Kernel mounted RBD's hanging

2017-06-29 Thread Nick Fisk
> -Original Message- > From: Ilya Dryomov [mailto:idryo...@gmail.com] > Sent: 29 June 2017 16:58 > To: Nick Fisk <n...@fisk.me.uk> > Cc: Ceph Users <ceph-users@lists.ceph.com> > Subject: Re: [ceph-users] Kernel mounted RBD's hanging > > On Thu, J

[ceph-users] Kernel mounted RBD's hanging

2017-06-29 Thread Nick Fisk
Hi All, Putting out a call for help to see if anyone can shed some light on this. Configuration: Ceph cluster presenting RBD's->XFS->NFS->ESXi Running 10.2.7 on the OSD's and 4.11 kernel on the NFS gateways in a pacemaker cluster Both OSD's and clients are go into a pair of switches, single L2

Re: [ceph-users] Ceph random read IOPS

2017-06-26 Thread Nick Fisk
> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Willem Jan Withagen > Sent: 26 June 2017 14:35 > To: Christian Wuerdig > Cc: Ceph Users > Subject: Re: [ceph-users] Ceph random read

Re: [ceph-users] Ceph random read IOPS

2017-06-24 Thread Nick Fisk
Apologies for the top post, I can't seem to break indents on my phone. Anyway the point of that test was as maged suggests to show the effect of serial CPU speed on latency. IO is effectively serialised by the pg lock, and so trying to reduce the time spent in this area is key. Fast cpu, fast

Re: [ceph-users] VMware + CEPH Integration

2017-06-22 Thread Nick Fisk
> -Original Message- > From: Adrian Saul [mailto:adrian.s...@tpgtelecom.com.au] > Sent: 19 June 2017 06:54 > To: n...@fisk.me.uk; 'Alex Gorbachev' > Cc: 'ceph-users' > Subject: RE: [ceph-users] VMware + CEPH Integration > > > Hi Alex,

Re: [ceph-users] VMware + CEPH Integration

2017-06-17 Thread Nick Fisk
> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Alex Gorbachev > Sent: 16 June 2017 01:48 > To: Osama Hasebou > Cc: ceph-users > Subject: Re: [ceph-users] VMware + CEPH Integration > > On

Re: [ceph-users] 2x replica with NVMe

2017-06-08 Thread Nick Fisk
Bluestore will make 2x Replica’s “safer” to use in theory. Until Bluestore is in use in the wild, I don’t think anyone can give any guarantees. From: i...@witeq.com [mailto:i...@witeq.com] Sent: 08 June 2017 14:32 To: nick Cc: Vy Nguyen Tan ;

Re: [ceph-users] 2x replica with NVMe

2017-06-08 Thread Nick Fisk
There are two main concerns with using 2x replicas, recovery speed and coming across inconsistent objects. With spinning disks their size to access speed means recovery can take a long time and increases the chance that additional failures may happen during the recovery process. NVME will

Re: [ceph-users] Changing SSD Landscape

2017-05-18 Thread Nick Fisk
he 2 mission critical production clusters, they are (will be) > > frozen most likely. > > > > Christian > > > >> -Ben > >> > >> On Wed, May 17, 2017 at 5:30 PM, Christian Balzer <ch...@gol.com> wrote: > >> > >> > > >&

Re: [ceph-users] Changing SSD Landscape

2017-05-17 Thread Nick Fisk
Hi Dan, > -Original Message- > From: Dan van der Ster [mailto:d...@vanderster.com] > Sent: 17 May 2017 10:29 > To: Nick Fisk <n...@fisk.me.uk> > Cc: ceph-users <ceph-users@lists.ceph.com> > Subject: Re: [ceph-users] Changing SSD Landscape > > I am

[ceph-users] Changing SSD Landscape

2017-05-17 Thread Nick Fisk
Hi All, There seems to be a shift in enterprise SSD products to larger less write intensive products and generally costing more than what the existing P/S 3600/3700 ranges were. For example the new Intel NVME P4600 range seems to start at 2TB. Although I mention Intel products, this seems to be

Re: [ceph-users] Intel power tuning - 30% throughput performance increase

2017-05-03 Thread Nick Fisk
> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Blair Bethwaite > Sent: 03 May 2017 09:53 > To: Dan van der Ster > Cc: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] Intel power tuning - 30% throughput

Re: [ceph-users] Maintaining write performance under a steady intake of small objects

2017-05-01 Thread Nick Fisk
Hi Patrick, Is there any chance that you can graph the XFS stats to see if there is an increase in inode/dentry cache misses as the ingest performance drops off? At least that might confirm the issue. Only other thing I can think of would be to try running the OSD’s on top of something

Re: [ceph-users] slow requests and short OSD failures in small cluster

2017-04-20 Thread Nick Fisk
> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Jogi > Hofmüller > Sent: 20 April 2017 13:51 > To: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] slow requests and short OSD failures in small > cluster > > Hi, > > Am Dienstag, den

Re: [ceph-users] Mon not starting after upgrading to 10.2.7

2017-04-12 Thread Nick Fisk
an der Ster [mailto:d...@vanderster.com] > Sent: 12 April 2017 10:53 > To: Nick Fisk <n...@fisk.me.uk> > Cc: ceph-users <ceph-users@lists.ceph.com> > Subject: Re: [ceph-users] Mon not starting after upgrading to 10.2.7 > > Can't help, but just wanted to say that the upgra

[ceph-users] Mon not starting after upgrading to 10.2.7

2017-04-12 Thread Nick Fisk
Hi, I just upgraded one of my mons to 10.2.7 and it is now failing to start properly. What's really odd is all the mon specific commands are now missing from the admin socket. ceph --admin-daemon /var/run/ceph/ceph-mon.gp-ceph-mon2.asok help { "config diff": "dump diff of current config and

Re: [ceph-users] rbd iscsi gateway question

2017-04-06 Thread Nick Fisk
> -Original Message- > From: David Disseldorp [mailto:dd...@suse.de] > Sent: 06 April 2017 14:06 > To: Nick Fisk <n...@fisk.me.uk> > Cc: 'Maged Mokhtar' <mmokh...@petasan.org>; 'Brady Deetz' > <bde...@gmail.com>; 'ceph-users' <ceph-us...@ceph.com

Re: [ceph-users] rbd iscsi gateway question

2017-04-06 Thread Nick Fisk
> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Maged Mokhtar > Sent: 06 April 2017 12:21 > To: Brady Deetz ; ceph-users > Subject: Re: [ceph-users] rbd iscsi gateway question > > The io hang (it is

Re: [ceph-users] rbd iscsi gateway question

2017-04-06 Thread Nick Fisk
I assume Brady is referring to the death spiral LIO gets into with some initiators, including vmware, if an IO takes longer than about 10s. I haven’t heard of anything, and can’t see any changes, so I would assume this issue still remains. I would look at either SCST or NFS for now.

Re: [ceph-users] Question about unfound objects

2017-03-30 Thread Nick Fisk
TB OSDs. I'm not sure how that would cause such a problem, but it's an interesting data point. On Thu, 2017-03-30 at 17:33 +0100, Nick Fisk wrote: Hi Steve, If you can recreate or if you can remember the object name, it might be worth trying to run “ceph osd map” on the objects and see

Re: [ceph-users] Question about unfound objects

2017-03-30 Thread Nick Fisk
Hi Steve, If you can recreate or if you can remember the object name, it might be worth trying to run "ceph osd map" on the objects and see where it thinks they map to. And/or maybe pg query might show something? Nick From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On

Re: [ceph-users] New hardware for OSDs

2017-03-28 Thread Nick Fisk
Hi Christian, > -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Christian Balzer > Sent: 28 March 2017 00:59 > To: ceph-users@lists.ceph.com > Cc: Nick Fisk <n...@fisk.me.uk> > Subject: Re: [ceph-users] New hard

Re: [ceph-users] New hardware for OSDs

2017-03-27 Thread Nick Fisk
> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Wido den Hollander > Sent: 27 March 2017 12:35 > To: ceph-users@lists.ceph.com; Christian Balzer > Subject: Re: [ceph-users] New hardware for OSDs > > > > Op 27 maart 2017 om

Re: [ceph-users] Preconditioning an RBD image

2017-03-25 Thread Nick Fisk
. What method are you using to create OSDs? 4. As mentioned above any stats around percentage of MB/s that is hitting your cache device vs journal (assuming journal is 100% of IO). This is to calculate extra wear Thanks, Nick > > Peter > > On 03/23/17 21:18, Nick Fisk wrote: &g

Re: [ceph-users] cephfs cache tiering - hitset

2017-03-23 Thread Nick Fisk
ing - hitset On Mon, Mar 20, 2017 at 4:20 PM, Nick Fisk <n...@fisk.me.uk <mailto:n...@fisk.me.uk> > wrote: Just a few corrections, hope you don't mind > -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Mike Lovell > Sent:

Re: [ceph-users] Preconditioning an RBD image

2017-03-23 Thread Nick Fisk
Hi Peter, Interesting graph. Out of interest, when you use bcache, do you then just leave the journal collocated on the combined bcache device and rely on the writeback to provide journal performance, or do you still create a separate partition on whatever SSD/NVME you use, effectively giving

Re: [ceph-users] cephfs cache tiering - hitset

2017-03-20 Thread Nick Fisk
Just a few corrections, hope you don't mind > -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Mike Lovell > Sent: 20 March 2017 20:30 > To: Webert de Souza Lima > Cc: ceph-users > Subject:

Re: [ceph-users] Snapshot Costs

2017-03-19 Thread Nick Fisk
> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Simon Leinen > Sent: 19 March 2017 17:23 > To: Gregory Farnum > Cc: ceph-users > Subject: Re: [ceph-users] Snapshot Costs > > Gregory Farnum

Re: [ceph-users] osd_disk_thread_ioprio_priority help

2017-03-13 Thread Nick Fisk
> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Florian Haas > Sent: 13 March 2017 10:09 > To: Dan van der Ster > Cc: ceph-users > Subject: Re: [ceph-users] osd_disk_thread_ioprio_priority

Re: [ceph-users] osd_disk_thread_ioprio_priority help

2017-03-11 Thread Nick Fisk
> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Laszlo Budai > Sent: 11 March 2017 13:51 > To: ceph-users > Subject: [ceph-users] osd_disk_thread_ioprio_priority help > > Hello, > > > Can someone explain the

Re: [ceph-users] A Jewel in the rough? (cache tier bugs and documentation omissions)

2017-03-07 Thread Nick Fisk
> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of John > Spray > Sent: 07 March 2017 01:45 > To: Christian Balzer > Cc: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] A Jewel in the rough? (cache tier bugs and >

Re: [ceph-users] Current CPU recommendations for storage nodes with multiple HDDs

2017-03-06 Thread Nick Fisk
> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Andreas Gerstmayr > Sent: 06 March 2017 12:58 > To: Ceph Users > Subject: [ceph-users] Current CPU recommendations for storage nodes with > multiple HDDs > > Hi, >

Re: [ceph-users] Passing LUA script via python rados execute

2017-02-21 Thread Nick Fisk
> > On 02/19/2017 12:15 PM, Patrick Donnelly wrote: > > On Sat, Feb 18, 2017 at 2:55 PM, Noah Watkins > wrote: > >> The least intrusive solution is to simply change the sandbox to allow > >> the standard file system module loading function as expected. Then > >> any user

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-02-21 Thread Nick Fisk
Yep sure, will try and present some figures at tomorrow’s meeting again. From: Samuel Just [mailto:sj...@redhat.com] Sent: 21 February 2017 18:14 To: Nick Fisk <n...@fisk.me.uk> Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

Re: [ceph-users] How safe is ceph pg repair these days?

2017-02-21 Thread Nick Fisk
> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Gregory Farnum > Sent: 20 February 2017 22:13 > To: Nick Fisk <n...@fisk.me.uk>; David Zafman <dzaf...@redhat.com> > Cc: ceph-users <ceph-us...@ceph.com>

Re: [ceph-users] Passing LUA script via python rados execute

2017-02-18 Thread Nick Fisk
n the same pool as the objects to be worked on, then in theory could someone modify the script object to do something nasty, intentional or not. Nick > -Original Message- > From: Noah Watkins [mailto:noahwatk...@gmail.com] > Sent: 18 February 2017 19:56 > To: Nick Fisk <n.

Re: [ceph-users] How safe is ceph pg repair these days?

2017-02-18 Thread Nick Fisk
>From what I understand in Jewel+ Ceph has the concept of an authorative shard, so in the case of a 3x replica pools, it will notice that 2 replicas match and one doesn't and use one of the good replicas. However, in a 2x pool your out of luck. However, if someone could confirm my suspicions that

  1   2   3   4   5   6   7   >