Re: [ceph-users] Some long running ops may lock osd

2015-03-03 Thread Erdem Agaoglu
From: bhi...@gmail.com Date: Mon, 2 Mar 2015 18:13:25 -0800 To: erdem.agao...@gmail.com CC: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Some long running ops may lock osd We're seeing a lot of this as well. (as i mentioned to sage at SCALE

Re: [ceph-users] Some long running ops may lock osd

2015-03-02 Thread Ben Hines
CC: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Some long running ops may lock osd We're seeing a lot of this as well. (as i mentioned to sage at SCALE..) Is there a rule of thumb at all for how big is safe to let a RGW bucket get? Also, is this theoretically resolved by the new

Re: [ceph-users] Some long running ops may lock osd

2015-03-02 Thread GuangYang
Subject: Re: [ceph-users] Some long running ops may lock osd We're seeing a lot of this as well. (as i mentioned to sage at SCALE..) Is there a rule of thumb at all for how big is safe to let a RGW bucket get? Also, is this theoretically resolved by the new bucket-sharding feature

Re: [ceph-users] Some long running ops may lock osd

2015-03-02 Thread Ben Hines
We're seeing a lot of this as well. (as i mentioned to sage at SCALE..) Is there a rule of thumb at all for how big is safe to let a RGW bucket get? Also, is this theoretically resolved by the new bucket-sharding feature in the latest dev release? -Ben On Mon, Mar 2, 2015 at 11:08 AM, Erdem

Re: [ceph-users] Some long running ops may lock osd

2015-03-02 Thread Erdem Agaoglu
... Thanks, Guang From: bhi...@gmail.com Date: Mon, 2 Mar 2015 18:13:25 -0800 To: erdem.agao...@gmail.com CC: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Some long running ops may lock osd We're seeing a lot of this as well

[ceph-users] Some long running ops may lock osd

2015-03-02 Thread Erdem Agaoglu
Hi all, especially devs, We have recently pinpointed one of the causes of slow requests in our cluster. It seems deep-scrubs on pg's that contain the index file for a large radosgw bucket lock the osds. Incresing op threads and/or disk threads helps a little bit, but we need to increase them

Re: [ceph-users] Some long running ops may lock osd

2015-03-02 Thread Gregory Farnum
On Mon, Mar 2, 2015 at 7:56 AM, Erdem Agaoglu erdem.agao...@gmail.com wrote: Hi all, especially devs, We have recently pinpointed one of the causes of slow requests in our cluster. It seems deep-scrubs on pg's that contain the index file for a large radosgw bucket lock the osds. Incresing op

Re: [ceph-users] Some long running ops may lock osd

2015-03-02 Thread Erdem Agaoglu
Hi Gregory, We are not using listomapkeys that way or in any way to be precise. I used it here just to reproduce the behavior/issue. What i am really interested in is if scrubbing-deep actually mitigates the problem and/or is there something that can be further improved. Or i guess we should go