Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-04-27 Thread Gregory Farnum
t >> is at https://github.com/ceph/ceph/pull/14597) >> -Greg > >> On Tue, Feb 21, 2017 at 3:32 PM, Nick Fisk wrote: >> > Yep sure, will try and present some figures at tomorrow’s meeting again. > > > >> > From: Samuel Just [mailto:sj...@redhat.com] >> &g

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-04-26 Thread Frédéric Nass
non-blocking implementation) to that branch. Care to take it for a whirl? > > -Sam > > On Thu, Feb 9, 2017 at 11:36 AM, Nick Fisk wrote: > > Building now > > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > > Samuel Just > > S

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-04-26 Thread Gregory Farnum
uary 2017 18:14 > > > To: Nick Fisk > Cc: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep? > > > > Ok, I've added explicit support for osd_snap_trim_sleep (same param, new > non-blocking implementation) to

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-02-21 Thread Nick Fisk
Yep sure, will try and present some figures at tomorrow’s meeting again. From: Samuel Just [mailto:sj...@redhat.com] Sent: 21 February 2017 18:14 To: Nick Fisk Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep? Ok, I've added exp

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-02-21 Thread Samuel Just
@lists.ceph.com] *On Behalf > Of *Samuel Just > *Sent:* 09 February 2017 19:22 > *To:* Nick Fisk > *Cc:* ceph-users@lists.ceph.com > > *Subject:* Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during > sleep? > > > > Ok, https://github.com/athanatos/c

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-02-09 Thread Nick Fisk
Building now From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Samuel Just Sent: 09 February 2017 19:22 To: Nick Fisk Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep? Ok, https://github.com/athanatos/ceph/tree

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-02-09 Thread Samuel Just
t possibly see up > to a 20% hit on performance, in exchange for more consistent client > latency. > > > > *From:* Steve Taylor [mailto:steve.tay...@storagecraft.com] > *Sent:* 07 February 2017 20:35 > *To:* n...@fisk.me.uk; ceph-users@lists.ceph.com > > *Subject:* RE: R

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-02-07 Thread Nick Fisk
: Steve Taylor [mailto:steve.tay...@storagecraft.com] Sent: 07 February 2017 20:35 To: n...@fisk.me.uk; ceph-users@lists.ceph.com Subject: RE: Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep? Thanks, Nick. One other data point that has come up is that nearly all of the

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-02-07 Thread Steve Taylor
.@fisk.me.uk] Sent: Tuesday, February 7, 2017 10:25 AM To: Steve Taylor ; ceph-users@lists.ceph.com Subject: RE: Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep? Hi Steve, From what I understand, the issue is not with the queueing in Ceph, which is correctly moving client IO to the

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-02-07 Thread Nick Fisk
a trade off of peak throughput vs queue latency though. Nick From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Steve Taylor Sent: 07 February 2017 17:01 To: ceph-users@lists.ceph.com Subject: Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep? As I

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-02-07 Thread Steve Taylor
s prohibited. From: Steve Taylor Sent: Tuesday, February 7, 2017 9:13 AM To: 'ceph-users@lists.ceph.com' Subject: Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep? Sorry, I lost the previous thread on this. I apologize for the resulting incomp

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-02-07 Thread Steve Taylor
Sorry, I lost the previous thread on this. I apologize for the resulting incomplete reply. The issue that we’re having with Jewel, as David Turner mentioned, is that we can’t seem to throttle snap trimming sufficiently to prevent it from blocking I/O requests. On further investigation, I encoun

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-02-05 Thread David Turner
I accidentally responded from the wrong email address. We will be able to build off of master and test that. Sent from my iPhone On Feb 5, 2017, at 3:27 PM, Peter Maloney mailto:peter.malo...@brockmann-consult.de>> wrote: On 02/03/17 19:54, David Turner wrote: Our current solution in Hammer i

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-02-05 Thread David Turner
lease notify the sender and delete it, together with any > attachments, and be advised that any dissemination or copying of this > message is prohibited. > > -- > > -- > *From:* Samuel Just [sj...@redhat.com] > *Sent:* Thur

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-02-05 Thread Peter Maloney
On 02/03/17 19:54, David Turner wrote: > Our current solution in Hammer involves a daemon monitoring the > cluster load and setting the osd_snap_trim_sleep accordingly between 0 > and 0.35 which does a good job of preventing IO blocking and help us > to clear out the snap_trim_q each day. These se

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-02-03 Thread David Turner
variable settings throughout the day. From: Samuel Just [sj...@redhat.com] Sent: Friday, February 03, 2017 11:24 AM To: David Turner Cc: Nick Fisk; ceph-users Subject: Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep? They do seem to exist in Jewel.

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-02-03 Thread Samuel Just
>> If you are not the intended recipient of this message or received it >> erroneously, please notify the sender and delete it, together with any >> attachments, and be advised that any dissemination or copying of this >> message is prohibited. >> >>

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-02-03 Thread David Turner
message is prohibited. From: Samuel Just [sj...@redhat.com] Sent: Friday, February 03, 2017 11:24 AM To: David Turner Cc: Nick Fisk; ceph-users Subject: Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep? They do seem to exist in Jewel. -Sam On

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-02-03 Thread Nick Fisk
...@redhat.com] Sent: 03 February 2017 18:24 To: David Turner Cc: Nick Fisk ; ceph-users Subject: Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep? They do seem to exist in Jewel. -Sam On Fri, Feb 3, 2017 at 10:12 AM, David Turner mailto:david.tur...@storagecraft.com>

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-02-03 Thread Samuel Just
ete it, together with any > attachments, and be advised that any dissemination or copying of this > message is prohibited. > > -- > > -- > *From:* Samuel Just [sj...@redhat.com] > *Sent:* Thursday, January 26, 2017 1:14 P

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-02-03 Thread David Turner
@redhat.com] Sent: Thursday, January 26, 2017 1:14 PM To: Nick Fisk Cc: David Turner; ceph-users Subject: Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep? Just an update. I think the real goal with the sleep configs in general was to reduce the number of concurrent snap trims ha

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-01-26 Thread Samuel Just
ent:* 19 January 2017 23:23 > *To:* David Turner > > *Cc:* Nick Fisk ; ceph-users > *Subject:* Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during > sleep? > > > > I could probably put together a wip branch if you have a test cluster you > could try it out on

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-01-20 Thread Nick Fisk
com <mailto:ceph-users-boun...@lists.ceph.com> ] on behalf of David Turner [david.tur...@storagecraft.com <mailto:david.tur...@storagecraft.com> ] Sent: Thursday, January 19, 2017 3:25 PM To: Samuel Just; Nick Fisk Cc: ceph-users Subject: Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-01-19 Thread Samuel Just
> -- > > -- > *From:* ceph-users [ceph-users-boun...@lists.ceph.com] on behalf of David > Turner [david.tur...@storagecraft.com] > *Sent:* Thursday, January 19, 2017 3:25 PM > *To:* Samuel Just; Nick Fisk > > *Cc:* cep

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-01-19 Thread David Turner
the disk. > >> -Original Message- >> From: Samuel Just [mailto:sj...@redhat.com] >> Sent: 19 January 2017 18:58 >> To: Nick Fisk >> Cc: Dan van der Ster ; ceph-users >> >> Subject: Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-01-19 Thread David Turner
s prohibited. From: ceph-users [ceph-users-boun...@lists.ceph.com] on behalf of Samuel Just [sj...@redhat.com] Sent: Thursday, January 19, 2017 2:45 PM To: Nick Fisk Cc: ceph-users Subject: Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep? Yeah, I think you're probably righ

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-01-19 Thread Samuel Just
essage- >> From: Samuel Just [mailto:sj...@redhat.com] >> Sent: 19 January 2017 18:58 >> To: Nick Fisk >> Cc: Dan van der Ster ; ceph-users >> >> Subject: Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep? >> >> Have you also t

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-01-19 Thread Nick Fisk
uel Just [mailto:sj...@redhat.com] > Sent: 19 January 2017 18:58 > To: Nick Fisk > Cc: Dan van der Ster ; ceph-users > > Subject: Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep? > > Have you also tried setting osd_snap_trim_cost to

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-01-19 Thread Samuel Just
fixing a bug with that. > > Nick > >> -Original Message- >> From: Samuel Just [mailto:sj...@redhat.com] >> Sent: 19 January 2017 15:47 >> To: Dan van der Ster >> Cc: Nick Fisk ; ceph-users >> Subject: Re: [ceph-users] osd_snap_trim_sleep k

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-01-19 Thread Nick Fisk
I know you are working on fixing a bug with that. Nick > -Original Message- > From: Samuel Just [mailto:sj...@redhat.com] > Sent: 19 January 2017 15:47 > To: Dan van der Ster > Cc: Nick Fisk ; ceph-users > Subject: Re: [ceph-users] osd_snap_trim_sleep keeps locks PG d

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-01-19 Thread Samuel Just
RBD cluster is still running 0.94.9. > > Cheers, Dan > > >> >>> -Original Message- >>> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of >>> Nick Fisk >>> Sent: 13 January 2017 20:38 >>> To: 'Dan va

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-01-19 Thread Dan van der Ster
o:ceph-users-boun...@lists.ceph.com] On Behalf Of >> Nick Fisk >> Sent: 13 January 2017 20:38 >> To: 'Dan van der Ster' >> Cc: 'ceph-users' >> Subject: Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep? >> >> We're o

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-01-19 Thread Nick Fisk
ticable. It might be that this option shouldn't be used with Jewel+? > -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Nick > Fisk > Sent: 13 January 2017 20:38 > To: 'Dan van der Ster' > Cc: 'ceph-users

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-01-13 Thread Nick Fisk
1] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-March/008652.html > -Original Message- > From: Dan van der Ster [mailto:d...@vanderster.com] > Sent: 13 January 2017 10:28 > To: Nick Fisk > Cc: ceph-users > Subject: Re: [ceph-users] osd_snap_trim_sleep keeps lo

Re: [ceph-users] osd_snap_trim_sleep keeps locks PG during sleep?

2017-01-13 Thread Dan van der Ster
Hammer or jewel? I've forgotten which thread pool is handling the snap trim nowadays -- is it the op thread yet? If so, perhaps all the op threads are stuck sleeping? Just a wild guess. (Maybe increasing # op threads would help?). -- Dan On Thu, Jan 12, 2017 at 3:11 PM, Nick Fisk wrote: > Hi, >