Re: dump_historic_ops, slow requests

2015-10-13 Thread Gregory Farnum
large (40M objects) with --no-cleanup, > the rados bench commands seem to finish OK but I often get health warnings > like > HEALTH_WARN 4 requests are blocked > 32 sec; > 2 osds have slow requests 3 ops are blocked > 32.768 sec on > osd.9 >

dump_historic_ops, slow requests

2015-10-12 Thread Deneau, Tom
health warnings like HEALTH_WARN 4 requests are blocked > 32 sec; 2 osds have slow requests 3 ops are blocked > 32.768 sec on osd.9 1 ops are blocked > 32.768 sec on osd.10 2 osds have slow requests After a couple of minutes, health goes to

Re: slow requests, hunting for new mon

2013-02-14 Thread Chris Dunlop
On 2013-02-12, Chris Dunlop ch...@onthe.net.au wrote: Hi, What are likely causes for slow requests and monclient: hunting for new mon messages? E.g.: 2013-02-12 16:27:07.318943 7f9c0bc16700 0 monclient: hunting for new mon ... 2013-02-12 16:27:45.892314 7f9c13c26700 0 log [WRN] : 6 slow

Re: slow requests, hunting for new mon

2013-02-14 Thread Sage Weil
On Fri, 15 Feb 2013, Chris Dunlop wrote: On 2013-02-12, Chris Dunlop ch...@onthe.net.au wrote: Hi, What are likely causes for slow requests and monclient: hunting for new mon messages? E.g.: 2013-02-12 16:27:07.318943 7f9c0bc16700 0 monclient: hunting for new mon ... 2013-02-12

Re: slow requests, hunting for new mon

2013-02-12 Thread Chris Dunlop
On Tue, Feb 12, 2013 at 06:28:15PM +1100, Chris Dunlop wrote: Hi, What are likely causes for slow requests and monclient: hunting for new mon messages? E.g.: 2013-02-12 16:27:07.318943 7f9c0bc16700 0 monclient: hunting for new mon ... 2013-02-12 16:27:45.892314 7f9c13c26700 0 log [WRN

slow requests, hunting for new mon

2013-02-11 Thread Chris Dunlop
Hi, What are likely causes for slow requests and monclient: hunting for new mon messages? E.g.: 2013-02-12 16:27:07.318943 7f9c0bc16700 0 monclient: hunting for new mon ... 2013-02-12 16:27:45.892314 7f9c13c26700 0 log [WRN] : 6 slow requests, 6 included below; oldest blocked for 30.383883

RE: Slow requests

2013-01-15 Thread Chen, Xiaoxi
.6b8b4567.07a7 [write 4046848~147456] 2.54bcf797) currently waiting for sub ops 2013-01-15 15:54:06.990355 7ff97dd0c700 0 log [WRN] : 2 slow requests, 2 included below; oldest blocked for 33.225773 secs 2013-01-15 15:54:06.990366 7ff97dd0c700 0 log [WRN] : slow request 33.225773 seconds old

Re: Slow requests

2012-12-18 Thread Jens Kristian Søgaard
Hi Dino, Seems that the RPM packager likes to keep the latest and greatest versions in http://ceph.com/rpm-testing/ but this path isn't defined in the ceph yum repository. Thanks for the link! Perhaps the documentation should be updated with this URL? The release notes link to:

Re: Slow requests

2012-12-17 Thread Dino Yancey
Hi Jens, Seems that the RPM packager likes to keep the latest and greatest versions in http://ceph.com/rpm-testing/ but this path isn't defined in the ceph yum repository. Dino On Sun, Dec 16, 2012 at 5:11 PM, Jens Kristian Søgaard j...@mermaidconsulting.dk wrote: Hi Sage, v0.52 is also

RE: Slow requests

2012-12-16 Thread Holcombe, Christopher
@vger.kernel.org Subject: Slow requests Hi, My log is filling up with warnings about a single slow request that has been around for a very long time: osd.1 10.0.0.2:6800/900 162926 : [WRN] 1 slow requests, 1 included below; oldest blocked for 84446.312051 secs osd.1 10.0.0.2:6800/900 162927 : [WRN] slow

Re: Slow requests

2012-12-16 Thread Andrey Korolyov
On Sun, Dec 16, 2012 at 5:59 PM, Jens Kristian Søgaard j...@mermaidconsulting.dk wrote: Hi, My log is filling up with warnings about a single slow request that has been around for a very long time: osd.1 10.0.0.2:6800/900 162926 : [WRN] 1 slow requests, 1 included below; oldest blocked

Re: Slow requests

2012-12-16 Thread Jens Kristian Søgaard
Hi Andrey, Thanks for your reply! Please take a look to this thread: http://comments.gmane.org/gmane.comp.file-systems.ceph.devel/10843 I took your advice and restarted each of my three osd's individually. The first two restarted within a minute or two. The last one took 20 minutes to

Re: Slow requests

2012-12-16 Thread Andrey Korolyov
On Mon, Dec 17, 2012 at 2:42 AM, Jens Kristian Søgaard j...@mermaidconsulting.dk wrote: Hi Andrey, Thanks for your reply! Please take a look to this thread: http://comments.gmane.org/gmane.comp.file-systems.ceph.devel/10843 I took your advice and restarted each of my three osd's

RE: Slow requests

2012-12-16 Thread Sage Weil
Message- From: ceph-devel-ow...@vger.kernel.org [mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Jens Kristian S?gaard Sent: Sunday, December 16, 2012 9:00 AM To: ceph-devel@vger.kernel.org Subject: Slow requests Hi, My log is filling up with warnings about a single slow request

Re: Slow requests

2012-12-16 Thread Jens Kristian Søgaard
Hi Andrey, Oh, sorry, I maybe misguided you - solution is the patch from Sam, restarts may help only on the short distance and you`re not able to check some pgs for consistency until patch have been applied - they`ll hang on scrub every time. I'll need to update, yes! I don't have any pgs in

Re: endless flying slow requests

2012-12-02 Thread Samuel Just
:54:45.770792, event: sub_op_applied}]}]} sage Stefan On Wed, 14 Nov 2012, Stefan Priebe - Profihost AG wrote: Hello list, i see this several times. Endless flying slow requests. And they never stop until i restart

Re: endless flying slow requests

2012-12-01 Thread Sage Weil
On Wed, 14 Nov 2012, Stefan Priebe - Profihost AG wrote: Hello list, i see this several times. Endless flying slow requests. And they never stop until i restart the mentioned osd. 2012-11-14 10:11:57.513395 osd.24 [WRN] 1 slow requests, 1

Re: endless flying slow requests

2012-11-30 Thread Samuel Just
times. Endless flying slow requests. And they never stop until i restart the mentioned osd. 2012-11-14 10:11:57.513395 osd.24 [WRN] 1 slow requests, 1 included below; oldest blocked for 31789.858457 secs 2012-11-14 10:11:57.513399 osd.24 [WRN] slow request

Re: endless flying slow requests

2012-11-28 Thread Samuel Just
: Hello list, i see this several times. Endless flying slow requests. And they never stop until i restart the mentioned osd. 2012-11-14 10:11:57.513395 osd.24 [WRN] 1 slow requests, 1 included below; oldest blocked for 31789.858457 secs 2012-11-14 10:11

Re: endless flying slow requests

2012-11-28 Thread Samuel Just
requests. And they never stop until i restart the mentioned osd. 2012-11-14 10:11:57.513395 osd.24 [WRN] 1 slow requests, 1 included below; oldest blocked for 31789.858457 secs 2012-11-14 10:11:57.513399 osd.24 [WRN] slow request 31789.858457 seconds old

RE: endless flying slow requests

2012-11-28 Thread Moore, Shawn M
I actually have this exact issue with hung io during these slow requests. The odd thing is when I restarted the osd that was being mentioned via ceph -w it didn't resume io. A bit later (unsure on exact time) another one started being listed in ceph -w and when I restarted that different osd

Re: endless flying slow requests

2012-11-28 Thread Andrey Korolyov
Nov 2012, Stefan Priebe - Profihost AG wrote: Hello list, i see this several times. Endless flying slow requests. And they never stop until i restart the mentioned osd. 2012-11-14 10:11:57.513395 osd.24 [WRN] 1 slow requests, 1 included below; oldest

Re: endless flying slow requests

2012-11-27 Thread Sage Weil
(i think) Were you able to reproduce this? Thanks! sage sage Stefan On Wed, 14 Nov 2012, Stefan Priebe - Profihost AG wrote: Hello list, i see this several times. Endless flying slow requests. And they never stop until i restart the mentioned osd

Re: endless flying slow requests

2012-11-27 Thread Stefan Priebe - Profihost AG
Am 28.11.2012 02:51, schrieb Sage Weil: Hi Stefan, Yes it is. So i have to specify admin socket at the KVM host? Right. IIRC the disk line is a ; (or \;) separated list of key/value pairs. How do i query the admin socket for requests? ceph --admin-daemon /path/to/socket help ceph

Re: endless flying slow requests

2012-11-27 Thread Andrey Korolyov
-11-28 03:54:45.770792, event: sub_op_applied}]}]} sage Stefan On Wed, 14 Nov 2012, Stefan Priebe - Profihost AG wrote: Hello list, i see this several times. Endless flying slow requests. And they never stop until i restart the mentioned osd

Re: endless flying slow requests

2012-11-15 Thread Stefan Priebe - Profihost AG
Nov 2012, Stefan Priebe - Profihost AG wrote: Hello list, i see this several times. Endless flying slow requests. And they never stop until i restart the mentioned osd. 2012-11-14 10:11:57.513395 osd.24 [WRN] 1 slow requests, 1 included below; oldest blocked for 31789.858457 secs 2012-11-14 10

Re: endless flying slow requests

2012-11-15 Thread Josh Durgin
On 11/15/2012 12:09 AM, Stefan Priebe - Profihost AG wrote: Am 14.11.2012 15:59, schrieb Sage Weil: Hi Stefan, I would be nice to confirm that no clients are waiting on replies for these requests; currently we suspect that the OSD request tracking is the buggy part. If you query the OSD admin

Re: endless flying slow requests

2012-11-15 Thread Sage Weil
- Profihost AG wrote: Hello list, i see this several times. Endless flying slow requests. And they never stop until i restart the mentioned osd. 2012-11-14 10:11:57.513395 osd.24 [WRN] 1 slow requests, 1 included below; oldest blocked for 31789.858457 secs 2012-11-14 10

endless flying slow requests

2012-11-14 Thread Stefan Priebe - Profihost AG
Hello list, i see this several times. Endless flying slow requests. And they never stop until i restart the mentioned osd. 2012-11-14 10:11:57.513395 osd.24 [WRN] 1 slow requests, 1 included below; oldest blocked for 31789.858457 secs 2012-11-14 10:11:57.513399 osd.24 [WRN] slow request

Re: endless flying slow requests

2012-11-14 Thread Sage Weil
. Is it librbd? In that case you likely need to change the config so that it is listening on an admin socket ('admin socket = path'). Thanks! sage On Wed, 14 Nov 2012, Stefan Priebe - Profihost AG wrote: Hello list, i see this several times. Endless flying slow requests. And they never stop