Re: [ceph-users] Clarification on sequence of recovery and client ops after OSDs rejoin cluster (also, slow requests)

On Mon, Sep 18, 2017 at 2:02 PM, Christian Theune wrote: > Hi, > > and here’s another update which others might find quite interesting. > > Florian and I spend some time discussing the issue further, face to face. I > had one switch that I brought up again

Re: [ceph-users] Clarification on sequence of recovery and client ops after OSDs rejoin cluster (also, slow requests)

Hi, > On Sep 18, 2017, at 10:06 AM, Christian Theune wrote: > > We’re doing the typical SSD/non-SSD pool separation. Currently we effectively > only use 2 pools: rbd.hdd and rbd.ssd. The ~4TB OSDs in the rbd.hdd pool are > “capacity endurance” SSDs (Micron S610DC). We

Re: [ceph-users] Clarification on sequence of recovery and client ops after OSDs rejoin cluster (also, slow requests)

Hi, > On Sep 18, 2017, at 9:51 AM, Florian Haas wrote: > > For Josh's and others' benefit, I think you might want to share how > many nodes you operate, as that would be quite relevant to the > discussion. Sure. See the OSD tree at the end. We’re doing the typical

Re: [ceph-users] Clarification on sequence of recovery and client ops after OSDs rejoin cluster (also, slow requests)

On Mon, Sep 18, 2017 at 8:48 AM, Christian Theune wrote: > Hi Josh, > >> On Sep 16, 2017, at 3:13 AM, Josh Durgin wrote: >> >> (Sorry for top posting, this email client isn't great at editing) > > Thanks for taking the time to respond. :) > >> The

Re: [ceph-users] Clarification on sequence of recovery and client ops after OSDs rejoin cluster (also, slow requests)

Hi Josh, > On Sep 16, 2017, at 3:13 AM, Josh Durgin wrote: > > (Sorry for top posting, this email client isn't great at editing) Thanks for taking the time to respond. :) > The mitigation strategy I mentioned before of forcing backfill could be > backported to jewel, but

Re: [ceph-users] Clarification on sequence of recovery and client ops after OSDs rejoin cluster (also, slow requests)

(Sorry for top posting, this email client isn't great at editing) The mitigation strategy I mentioned before of forcing backfill could be backported to jewel, but I don't think it's a very good option for RBD users without SSDs. In luminous there is a command (something like 'ceph pg

Re: [ceph-users] Clarification on sequence of recovery and client ops after OSDs rejoin cluster (also, slow requests)

On Fri, Sep 15, 2017 at 10:37 PM, Josh Durgin wrote: >> So this affects just writes. Then I'm really not following the >> reasoning behind the current behavior. Why would you want to wait for >> the recovery of an object that you're about to clobber anyway? Naïvely >> thinking

Re: [ceph-users] Clarification on sequence of recovery and client ops after OSDs rejoin cluster (also, slow requests)

On 09/15/2017 01:57 AM, Florian Haas wrote: On Fri, Sep 15, 2017 at 8:58 AM, Josh Durgin wrote: This is more of an issue with write-intensive RGW buckets, since the bucket index object is a single bottleneck if it needs recovery, and all further writes to a shard of a

Re: [ceph-users] Clarification on sequence of recovery and client ops after OSDs rejoin cluster (also, slow requests)

On Fri, Sep 15, 2017 at 8:58 AM, Josh Durgin wrote: >> OK, maybe the "also" can be removed to reduce potential confusion? > > > Sure That'd be great. :) >> - We have a bunch of objects that need to be recovered onto the >> just-returned OSD(s). >> - Clients access some of

Re: [ceph-users] Clarification on sequence of recovery and client ops after OSDs rejoin cluster (also, slow requests)

On 09/14/2017 12:44 AM, Florian Haas wrote: On Thu, Sep 14, 2017 at 2:47 AM, Josh Durgin wrote: On 09/13/2017 03:40 AM, Florian Haas wrote: So we have a client that is talking to OSD 30. OSD 30 was never down; OSD 17 was. OSD 30 is also the preferred primary for this PG

Re: [ceph-users] Clarification on sequence of recovery and client ops after OSDs rejoin cluster (also, slow requests)

On Thu, Sep 14, 2017 at 5:42 PM, Florian Haas wrote: > On Thu, Sep 14, 2017 at 3:15 AM, Brad Hubbard wrote: >> On Wed, Sep 13, 2017 at 8:40 PM, Florian Haas wrote: >>> Hi everyone, >>> >>> >>> disclaimer upfront: this was seen in

Re: [ceph-users] Clarification on sequence of recovery and client ops after OSDs rejoin cluster (also, slow requests)

On Thu, Sep 14, 2017 at 2:47 AM, Josh Durgin wrote: > On 09/13/2017 03:40 AM, Florian Haas wrote: >> >> So we have a client that is talking to OSD 30. OSD 30 was never down; >> OSD 17 was. OSD 30 is also the preferred primary for this PG (via >> primary affinity). The OSD now

Re: [ceph-users] Clarification on sequence of recovery and client ops after OSDs rejoin cluster (also, slow requests)

On Thu, Sep 14, 2017 at 3:15 AM, Brad Hubbard wrote: > On Wed, Sep 13, 2017 at 8:40 PM, Florian Haas wrote: >> Hi everyone, >> >> >> disclaimer upfront: this was seen in the wild on Hammer, and on 0.94.7 >> no less. Reproducing this on 0.94.10 is a

Re: [ceph-users] Clarification on sequence of recovery and client ops after OSDs rejoin cluster (also, slow requests)

Hi Brad, > On Sep 14, 2017, at 3:15 AM, Brad Hubbard wrote: > > On Wed, Sep 13, 2017 at 8:40 PM, Florian Haas wrote: >> Hi everyone, >> >> >> disclaimer upfront: this was seen in the wild on Hammer, and on 0.94.7 >> no less. Reproducing this on

Re: [ceph-users] Clarification on sequence of recovery and client ops after OSDs rejoin cluster (also, slow requests)

On Wed, Sep 13, 2017 at 8:40 PM, Florian Haas wrote: > Hi everyone, > > > disclaimer upfront: this was seen in the wild on Hammer, and on 0.94.7 > no less. Reproducing this on 0.94.10 is a pending process, and we'll > update here with findings, but my goal with this post is

Re: [ceph-users] Clarification on sequence of recovery and client ops after OSDs rejoin cluster (also, slow requests)

On 09/13/2017 03:40 AM, Florian Haas wrote: So we have a client that is talking to OSD 30. OSD 30 was never down; OSD 17 was. OSD 30 is also the preferred primary for this PG (via primary affinity). The OSD now says that - it does itself have a copy of the object, - so does OSD 94, - but that

[ceph-users] Clarification on sequence of recovery and client ops after OSDs rejoin cluster (also, slow requests)

Hi everyone, disclaimer upfront: this was seen in the wild on Hammer, and on 0.94.7 no less. Reproducing this on 0.94.10 is a pending process, and we'll update here with findings, but my goal with this post is really to establish whether the behavior as seen is expected, and if so, what the

Re: [ceph-users] Clarification on sequence of recovery and client ops after OSDs rejoin cluster (also, slow requests)

Hi, (thanks to Florian who’s helping us getting this sorted out) > On Sep 13, 2017, at 12:40 PM, Florian Haas wrote: > > Hi everyone, > > > disclaimer upfront: this was seen in the wild on Hammer, and on 0.94.7 > no less. Reproducing this on 0.94.10 is a pending process,