large (40M objects) with --no-cleanup,
> the rados bench commands seem to finish OK but I often get health warnings
> like
> HEALTH_WARN 4 requests are blocked > 32 sec;
> 2 osds have slow requests 3 ops are blocked > 32.768 sec on
> osd.9
>
health warnings like
HEALTH_WARN 4 requests are blocked > 32 sec;
2 osds have slow requests 3 ops are blocked > 32.768 sec on
osd.9
1 ops are blocked > 32.768 sec on osd.10
2 osds have slow requests
After a couple of minutes, health goes to
On 2013-02-12, Chris Dunlop ch...@onthe.net.au wrote:
Hi,
What are likely causes for slow requests and monclient: hunting for new
mon messages? E.g.:
2013-02-12 16:27:07.318943 7f9c0bc16700 0 monclient: hunting for new mon
...
2013-02-12 16:27:45.892314 7f9c13c26700 0 log [WRN] : 6 slow
On Fri, 15 Feb 2013, Chris Dunlop wrote:
On 2013-02-12, Chris Dunlop ch...@onthe.net.au wrote:
Hi,
What are likely causes for slow requests and monclient: hunting for new
mon messages? E.g.:
2013-02-12 16:27:07.318943 7f9c0bc16700 0 monclient: hunting for new mon
...
2013-02-12
On Tue, Feb 12, 2013 at 06:28:15PM +1100, Chris Dunlop wrote:
Hi,
What are likely causes for slow requests and monclient: hunting for new
mon messages? E.g.:
2013-02-12 16:27:07.318943 7f9c0bc16700 0 monclient: hunting for new mon
...
2013-02-12 16:27:45.892314 7f9c13c26700 0 log [WRN
Hi,
What are likely causes for slow requests and monclient: hunting for new
mon messages? E.g.:
2013-02-12 16:27:07.318943 7f9c0bc16700 0 monclient: hunting for new mon
...
2013-02-12 16:27:45.892314 7f9c13c26700 0 log [WRN] : 6 slow requests, 6
included below; oldest blocked for 30.383883
.6b8b4567.07a7 [write 4046848~147456] 2.54bcf797) currently
waiting for sub ops
2013-01-15 15:54:06.990355 7ff97dd0c700 0 log [WRN] : 2 slow requests, 2
included below; oldest blocked for 33.225773 secs
2013-01-15 15:54:06.990366 7ff97dd0c700 0 log [WRN] : slow request 33.225773
seconds old
Hi Dino,
Seems that the RPM packager likes to keep the latest and greatest
versions in http://ceph.com/rpm-testing/ but this path isn't defined
in the ceph yum repository.
Thanks for the link!
Perhaps the documentation should be updated with this URL?
The release notes link to:
Hi Jens,
Seems that the RPM packager likes to keep the latest and greatest
versions in http://ceph.com/rpm-testing/ but this path isn't defined
in the ceph yum repository.
Dino
On Sun, Dec 16, 2012 at 5:11 PM, Jens Kristian Søgaard
j...@mermaidconsulting.dk wrote:
Hi Sage,
v0.52 is also
@vger.kernel.org
Subject: Slow requests
Hi,
My log is filling up with warnings about a single slow request that has been
around for a very long time:
osd.1 10.0.0.2:6800/900 162926 : [WRN] 1 slow requests, 1 included below;
oldest blocked for 84446.312051 secs
osd.1 10.0.0.2:6800/900 162927 : [WRN] slow
On Sun, Dec 16, 2012 at 5:59 PM, Jens Kristian Søgaard
j...@mermaidconsulting.dk wrote:
Hi,
My log is filling up with warnings about a single slow request that has been
around for a very long time:
osd.1 10.0.0.2:6800/900 162926 : [WRN] 1 slow requests, 1 included below;
oldest blocked
Hi Andrey,
Thanks for your reply!
Please take a look to this thread:
http://comments.gmane.org/gmane.comp.file-systems.ceph.devel/10843
I took your advice and restarted each of my three osd's individually.
The first two restarted within a minute or two. The last one took 20
minutes to
On Mon, Dec 17, 2012 at 2:42 AM, Jens Kristian Søgaard
j...@mermaidconsulting.dk wrote:
Hi Andrey,
Thanks for your reply!
Please take a look to this thread:
http://comments.gmane.org/gmane.comp.file-systems.ceph.devel/10843
I took your advice and restarted each of my three osd's
Message-
From: ceph-devel-ow...@vger.kernel.org
[mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Jens Kristian S?gaard
Sent: Sunday, December 16, 2012 9:00 AM
To: ceph-devel@vger.kernel.org
Subject: Slow requests
Hi,
My log is filling up with warnings about a single slow request
Hi Andrey,
Oh, sorry, I maybe misguided you - solution is the patch from Sam,
restarts may help only on the short distance and you`re not able to
check some pgs for consistency until patch have been applied - they`ll
hang on scrub every time.
I'll need to update, yes!
I don't have any pgs in
:54:45.770792,
event: sub_op_applied}]}]}
sage
Stefan
On Wed, 14 Nov 2012, Stefan Priebe - Profihost AG wrote:
Hello list,
i see this several times. Endless flying slow requests. And
they never
stop
until i restart
On Wed, 14 Nov 2012, Stefan Priebe - Profihost AG wrote:
Hello list,
i see this several times. Endless flying slow requests. And they
never
stop
until i restart the mentioned osd.
2012-11-14 10:11:57.513395 osd.24 [WRN] 1 slow requests, 1
times. Endless flying slow requests. And they
never
stop
until i restart the mentioned osd.
2012-11-14 10:11:57.513395 osd.24 [WRN] 1 slow requests, 1
included below;
oldest blocked for 31789.858457 secs
2012-11-14 10:11:57.513399 osd.24 [WRN] slow request
:
Hello list,
i see this several times. Endless flying slow requests. And they never
stop
until i restart the mentioned osd.
2012-11-14 10:11:57.513395 osd.24 [WRN] 1 slow requests, 1 included
below;
oldest blocked for 31789.858457 secs
2012-11-14 10:11
requests. And they
never
stop
until i restart the mentioned osd.
2012-11-14 10:11:57.513395 osd.24 [WRN] 1 slow requests, 1 included
below;
oldest blocked for 31789.858457 secs
2012-11-14 10:11:57.513399 osd.24 [WRN] slow request 31789.858457
seconds
old
I actually have this exact issue with hung io during these slow requests. The
odd thing is when I restarted the osd that was being mentioned via ceph -w it
didn't resume io. A bit later (unsure on exact time) another one started being
listed in ceph -w and when I restarted that different osd
Nov 2012, Stefan Priebe - Profihost AG wrote:
Hello list,
i see this several times. Endless flying slow requests. And they
never
stop
until i restart the mentioned osd.
2012-11-14 10:11:57.513395 osd.24 [WRN] 1 slow requests, 1 included
below;
oldest
(i think)
Were you able to reproduce this?
Thanks!
sage
sage
Stefan
On Wed, 14 Nov 2012, Stefan Priebe - Profihost AG wrote:
Hello list,
i see this several times. Endless flying slow requests. And they never
stop
until i restart the mentioned osd
Am 28.11.2012 02:51, schrieb Sage Weil:
Hi Stefan,
Yes it is. So i have to specify admin socket at the KVM host?
Right. IIRC the disk line is a ; (or \;) separated list of key/value
pairs.
How do i query the admin socket for requests?
ceph --admin-daemon /path/to/socket help
ceph
-11-28 03:54:45.770792,
event: sub_op_applied}]}]}
sage
Stefan
On Wed, 14 Nov 2012, Stefan Priebe - Profihost AG wrote:
Hello list,
i see this several times. Endless flying slow requests. And they never
stop
until i restart the mentioned osd
Nov 2012, Stefan Priebe - Profihost AG wrote:
Hello list,
i see this several times. Endless flying slow requests. And they never stop
until i restart the mentioned osd.
2012-11-14 10:11:57.513395 osd.24 [WRN] 1 slow requests, 1 included below;
oldest blocked for 31789.858457 secs
2012-11-14 10
On 11/15/2012 12:09 AM, Stefan Priebe - Profihost AG wrote:
Am 14.11.2012 15:59, schrieb Sage Weil:
Hi Stefan,
I would be nice to confirm that no clients are waiting on replies for
these requests; currently we suspect that the OSD request tracking is the
buggy part. If you query the OSD admin
- Profihost AG wrote:
Hello list,
i see this several times. Endless flying slow requests. And they never
stop
until i restart the mentioned osd.
2012-11-14 10:11:57.513395 osd.24 [WRN] 1 slow requests, 1 included below;
oldest blocked for 31789.858457 secs
2012-11-14 10
Hello list,
i see this several times. Endless flying slow requests. And they never
stop until i restart the mentioned osd.
2012-11-14 10:11:57.513395 osd.24 [WRN] 1 slow requests, 1 included
below; oldest blocked for 31789.858457 secs
2012-11-14 10:11:57.513399 osd.24 [WRN] slow request
.
Is it librbd? In that case you likely need to change the config so that
it is listening on an admin socket ('admin socket = path').
Thanks!
sage
On Wed, 14 Nov 2012, Stefan Priebe - Profihost AG wrote:
Hello list,
i see this several times. Endless flying slow requests. And they never stop
30 matches
Mail list logo