RE: Ceph slow request unstable issue
On Thu, 17 Jan 2013, Chen, Xiaoxi wrote:
Some update summary for tested case till now:
Ceph is v0.56.1
1.RBD:Ubuntu 13.04 + 3.7Kernel
OSD:Ubuntu 13.04 + 3.7Kernel
XFS
Result: Kernel Panic on both RBD and OSD sides
We're very interested in the RBD client-side kernel panic! I don't think
there are known issues with 3.7.
2.RBD:Ubuntu 13.04 +3.2Kernel
OSD:Ubuntu 13.04 +3.2Kernel
XFS
Result:Kernel Panic on RBD( ~15Minus)
This less so; we've only backported fixes as far as 3.4.
3.RBD:Ubuntu 13.04 + 3.6.7 Kernel (Suggested by Ceph.com)
OSD:Ubuntu 13.04 + 3.2 Kernel
XFS
Result: Auto-reset on OSD ( ~ 30 mins after the test started)
4.RBD:Ubuntu 13.04+3.6.7 Kernel (Suggested by Ceph.com)
OSD:Ubuntu 12.04 + 3.2.0-36 Kernel (Suggested by Ceph.com)
XFS
Result:auto-reset on OSD ( ~ 30 mins after the test started)
These the the weird exit_mm traces shown below?
5.RBD:Ubuntu 13.04+3.6.7 Kernel (Suggested by Ceph.com)
OSD:Ubuntu 13.04 +3.6.7 (Suggested by Sage)
XFS
Result: seems stable for last 1 hour, still running till now
Eager to hear how this goes.
Thanks!
sage
Test 34 are repeatable.
My test setup
OSD side:
3 nodes, 60 Disks(20 per nodes,1 per OSD),10Gb E, 4 *Intel 520 SSD per node
as journal,XFS
For each node,2 * Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GH + 128GB RAM were
used.
RBD side:
8 nodes,for each node:10Gb E,2 * Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GH ,
128GB RAM
Method:
Create 240 RBD and mounted to 8 nodes ( 30 RBD per nodes), doing DD
concurrently on all 240 RBDs.
After ~ 30 minutes, it's likely to have one of the OSD node reset.
Ceph OSD logs, syslog and dmesg from reseted node are available if you
needed.(It looks to me that no valuable information except a lot of
slow-request in OSD's log)
Xiaoxi
-Original Message-
From: Sage Weil [mailto:s...@inktank.com]
Sent: 2013?1?17? 10:35
To: Chen, Xiaoxi
Subject: RE: Ceph slow request unstable issue
On Thu, 17 Jan 2013, Chen, Xiaoxi wrote:
No, on the OSD node, not the same node. OSD node with 3.2 kernel while
client node with 3.6 kernel
We did suffer kernel panic on rbd client nodes but after upgrade
client kernel to 3.6.6 it seems solved .
Is it easy to try the 3.6 kernel on the osd nodes too?
-Original Message-
From: Sage Weil [mailto:s...@inktank.com]
Sent: 2013?1?17? 10:17
To: Chen, Xiaoxi
Subject: RE: Ceph slow request unstable issue
On Thu, 17 Jan 2013, Chen, Xiaoxi wrote:
It is easily to reproduce in my setup...
Once I have enough high load on it and waiting for tens of minutes? I can
see such log.
As a forecast, slow requests more than 30~60s are frequently present
in ceph osd's log.
Just replied to your other email. Do I understand correctly that you are
seeing this problem on the *rbd client* nodes? Or also on the OSDs? Are
they the same nodes?
sage
-Original Message-
From: Sage Weil [mailto:s...@inktank.com]
Sent: 2013?1?17? 0:59
To: Andrey Korolyov
Cc: Chen, Xiaoxi; ceph-devel@vger.kernel.org
Subject: Re: Ceph slow request unstable issue
Hi,
On Wed, 16 Jan 2013, Andrey Korolyov wrote:
On Wed, Jan 16, 2013 at 4:58 AM, Chen, Xiaoxi xiaoxi.c...@intel.com
wrote:
Hi list,
We are suffering from OSD or OS down when there is continuing
high pressure on the Ceph rack.
Basically we are on Ubuntu 12.04+ Ceph 0.56.1, 6 nodes, in
each nodes with 20 * spindles + 4* SSDs as journal.(120 spindles in
total)
We create a lots of RBD volumes (say 240),mounting to 16
different client machines ( 15 RBD Volumes/ client) and running DD
concurrently on top of each RBD.
The issues are:
1. Slow requests
??From the list-archive it seems solved in 0.56.1 but we still
notice such warning 2. OSD Down or even host down Like the
message below.Seems some OSD has been blocking there for quite a long
time.
Suggestions are highly appreciate.Thanks
Xiaoxi
_
Bad news:
I have back all my Ceph machine?s OS to kernel 3.2.0-23, which
Ubuntu 12.04 use.
I run dd command (dd if=/dev/zero bs=1M count=6 of=/dev/rbd${i}
)on Ceph client to create data prepare test at last night.
Now, I have one machine down (can?t be reached by ping
Re: Ceph slow request unstable issue
Hi,
On Wed, 16 Jan 2013, Andrey Korolyov wrote:
On Wed, Jan 16, 2013 at 4:58 AM, Chen, Xiaoxi xiaoxi.c...@intel.com wrote:
Hi list,
We are suffering from OSD or OS down when there is continuing high
pressure on the Ceph rack.
Basically we are on Ubuntu 12.04+ Ceph 0.56.1, 6 nodes, in each
nodes with 20 * spindles + 4* SSDs as journal.(120 spindles in total)
We create a lots of RBD volumes (say 240),mounting to 16 different
client machines ( 15 RBD Volumes/ client) and running DD concurrently on
top of each RBD.
The issues are:
1. Slow requests
??From the list-archive it seems solved in 0.56.1 but we still notice such
warning
2. OSD Down or even host down
Like the message below.Seems some OSD has been blocking there for quite a
long time.
Suggestions are highly appreciate.Thanks
Xiaoxi
_
Bad news:
I have back all my Ceph machine?s OS to kernel 3.2.0-23, which Ubuntu
12.04 use.
I run dd command (dd if=/dev/zero bs=1M count=6 of=/dev/rbd${i} )on
Ceph client to create data prepare test at last night.
Now, I have one machine down (can?t be reached by ping), another two
machine has all OSD daemon down, while the three left has some daemon down.
I have many warnings in OSD log like this:
no flag points reached
2013-01-15 19:14:22.769898 7f20a2d57700 0 log [WRN] : slow request
52.218106 seconds old, received at 2013-01-15 19:13:30.551718:
osd_op(client.10674.1:1002417 rb.0.27a8.6b8b4567.0eba [write
3145728~524288] 2.c61810ee RETRY) currently waiting for sub ops
2013-01-15 19:14:23.770077 7f20a2d57700 0 log [WRN] : 21 slow requests, 6
included below; oldest blocked for 1132.138983 secs
2013-01-15 19:14:23.770086 7f20a2d57700 0 log [WRN] : slow request
53.216404 seconds old, received at 2013-01-15 19:13:30.553616:
osd_op(client.10671.1:1066860 rb.0.282c.6b8b4567.1057 [write
2621440~524288] 2.ea7acebc) currently waiting for sub ops
2013-01-15 19:14:23.770096 7f20a2d57700 0 log [WRN] : slow request
51.442032 seconds old, received at 2013-01-15 19:13:32.327988:
osd_op(client.10674.1:1002418
Similar info in dmesg we have saw pervious:
[21199.036476] INFO: task ceph-osd:7788 blocked for more than 120 seconds.
[21199.037493] echo 0 /proc/sys/kernel/hung_task_timeout_secs disables
this message.
[21199.038841] ceph-osdD 0006 0 7788 1
0x
[21199.038844] 880fefdafcc8 0086
ffe0
[21199.038848] 880fefdaffd8 880fefdaffd8 880fefdaffd8
00013780
[21199.038852] 88081aa58000 880f68f52de0 880f68f52de0
882017556200
[21199.038856] Call Trace:
[21199.038858] [8165a55f] schedule+0x3f/0x60
[21199.038861] [8106b7e5] exit_mm+0x85/0x130
[21199.038864] [8106b9fe] do_exit+0x16e/0x420
[21199.038866] [8109d88f] ? __unqueue_futex+0x3f/0x80
[21199.038869] [8107a19a] ? __dequeue_signal+0x6a/0xb0
[21199.038872] [8106be54] do_group_exit+0x44/0xa0
[21199.038874] [8107ccdc] get_signal_to_deliver+0x21c/0x420
[21199.038877] [81013865] do_signal+0x45/0x130
[21199.038880] [810a091c] ? do_futex+0x7c/0x1b0
[21199.038882] [810a0b5a] ? sys_futex+0x10a/0x1a0
[21199.038885] [81013b15] do_notify_resume+0x65/0x80
[21199.038887] [81664d50] int_signal+0x12/0x17
We have seen this stack trace several times over the past 6 months, but
are not sure what the trigger is. In principle, the ceph server-side
daemons shouldn't be capable of locking up like this, but clearly
something is amiss between what they are doing in userland and how the
kernel is tolerating that. Low memory, perhaps? In each case where we
tried to track it down, the problem seemed to go away on its own. Is this
easily reproducible in your case?
my 0.02$:
http://www.mail-archive.com/ceph-devel@vger.kernel.org/msg11531.html
and kernel panic from two different hosts from yesterday during ceph
startup(on 3.8-rc3, images from console available at
http://imgur.com/wIRVn,k0QCS#0) leads to suggestion that Ceph may have
been introduced lockup-alike behavior not a long ago, causing, in my
case, excessive amount of context switches on the host leading to osd
flaps and panic at the ip-ib stack due to same issue.
For the stack trace my first guess would be a problem with the IB driver
that is triggered by memory pressure. Can you characterize what the
system utilization (CPU, memory) looks like leading up to the lockup?
sage
--
To unsubscribe from this list: send the line unsubscribe ceph-devel in
the body of a
Re: Ceph slow request unstable issue
On Wed, Jan 16, 2013 at 10:35 PM, Andrey Korolyov and...@xdel.ru wrote:
On Wed, Jan 16, 2013 at 8:58 PM, Sage Weil s...@inktank.com wrote:
Hi,
On Wed, 16 Jan 2013, Andrey Korolyov wrote:
On Wed, Jan 16, 2013 at 4:58 AM, Chen, Xiaoxi xiaoxi.c...@intel.com wrote:
Hi list,
We are suffering from OSD or OS down when there is continuing
high pressure on the Ceph rack.
Basically we are on Ubuntu 12.04+ Ceph 0.56.1, 6 nodes, in each
nodes with 20 * spindles + 4* SSDs as journal.(120 spindles in total)
We create a lots of RBD volumes (say 240),mounting to 16
different client machines ( 15 RBD Volumes/ client) and running DD
concurrently on top of each RBD.
The issues are:
1. Slow requests
??From the list-archive it seems solved in 0.56.1 but we still notice
such warning
2. OSD Down or even host down
Like the message below.Seems some OSD has been blocking there for quite a
long time.
Suggestions are highly appreciate.Thanks
Xiaoxi
_
Bad news:
I have back all my Ceph machine?s OS to kernel 3.2.0-23, which Ubuntu
12.04 use.
I run dd command (dd if=/dev/zero bs=1M count=6 of=/dev/rbd${i} )on
Ceph client to create data prepare test at last night.
Now, I have one machine down (can?t be reached by ping), another two
machine has all OSD daemon down, while the three left has some daemon
down.
I have many warnings in OSD log like this:
no flag points reached
2013-01-15 19:14:22.769898 7f20a2d57700 0 log [WRN] : slow request
52.218106 seconds old, received at 2013-01-15 19:13:30.551718:
osd_op(client.10674.1:1002417 rb.0.27a8.6b8b4567.0eba [write
3145728~524288] 2.c61810ee RETRY) currently waiting for sub ops
2013-01-15 19:14:23.770077 7f20a2d57700 0 log [WRN] : 21 slow requests,
6 included below; oldest blocked for 1132.138983 secs
2013-01-15 19:14:23.770086 7f20a2d57700 0 log [WRN] : slow request
53.216404 seconds old, received at 2013-01-15 19:13:30.553616:
osd_op(client.10671.1:1066860 rb.0.282c.6b8b4567.1057 [write
2621440~524288] 2.ea7acebc) currently waiting for sub ops
2013-01-15 19:14:23.770096 7f20a2d57700 0 log [WRN] : slow request
51.442032 seconds old, received at 2013-01-15 19:13:32.327988:
osd_op(client.10674.1:1002418
Similar info in dmesg we have saw pervious:
[21199.036476] INFO: task ceph-osd:7788 blocked for more than 120 seconds.
[21199.037493] echo 0 /proc/sys/kernel/hung_task_timeout_secs
disables this message.
[21199.038841] ceph-osdD 0006 0 7788 1
0x
[21199.038844] 880fefdafcc8 0086
ffe0
[21199.038848] 880fefdaffd8 880fefdaffd8 880fefdaffd8
00013780
[21199.038852] 88081aa58000 880f68f52de0 880f68f52de0
882017556200
[21199.038856] Call Trace:
[21199.038858] [8165a55f] schedule+0x3f/0x60
[21199.038861] [8106b7e5] exit_mm+0x85/0x130
[21199.038864] [8106b9fe] do_exit+0x16e/0x420
[21199.038866] [8109d88f] ? __unqueue_futex+0x3f/0x80
[21199.038869] [8107a19a] ? __dequeue_signal+0x6a/0xb0
[21199.038872] [8106be54] do_group_exit+0x44/0xa0
[21199.038874] [8107ccdc] get_signal_to_deliver+0x21c/0x420
[21199.038877] [81013865] do_signal+0x45/0x130
[21199.038880] [810a091c] ? do_futex+0x7c/0x1b0
[21199.038882] [810a0b5a] ? sys_futex+0x10a/0x1a0
[21199.038885] [81013b15] do_notify_resume+0x65/0x80
[21199.038887] [81664d50] int_signal+0x12/0x17
We have seen this stack trace several times over the past 6 months, but
are not sure what the trigger is. In principle, the ceph server-side
daemons shouldn't be capable of locking up like this, but clearly
something is amiss between what they are doing in userland and how the
kernel is tolerating that. Low memory, perhaps? In each case where we
tried to track it down, the problem seemed to go away on its own. Is this
easily reproducible in your case?
my 0.02$:
http://www.mail-archive.com/ceph-devel@vger.kernel.org/msg11531.html
and kernel panic from two different hosts from yesterday during ceph
startup(on 3.8-rc3, images from console available at
http://imgur.com/wIRVn,k0QCS#0) leads to suggestion that Ceph may have
been introduced lockup-alike behavior not a long ago, causing, in my
case, excessive amount of context switches on the host leading to osd
flaps and panic at the ip-ib stack due to same issue.
For the stack trace my first guess would be a problem with the IB driver
that is triggered by memory pressure. Can you characterize what the
system utilization
RE: Ceph slow request unstable issue
On Thu, 17 Jan 2013, Chen, Xiaoxi wrote:
Hi Sage?
Both CPU and Memory utilization are very low. CPU is ~ 20% (with
60% IOWAIT), Memory is far more less . I have 32 Core Sandybridege
CPU(64 Core for HT), together with 128GB RAM per node.
Hmm!
-Original Message-
From: Sage Weil [mailto:s...@inktank.com]
Sent: 2013?1?17? 0:59
To: Andrey Korolyov
Cc: Chen, Xiaoxi; ceph-devel@vger.kernel.org
Subject: Re: Ceph slow request unstable issue
Hi,
On Wed, 16 Jan 2013, Andrey Korolyov wrote:
On Wed, Jan 16, 2013 at 4:58 AM, Chen, Xiaoxi xiaoxi.c...@intel.com wrote:
Hi list,
We are suffering from OSD or OS down when there is continuing
high pressure on the Ceph rack.
Basically we are on Ubuntu 12.04+ Ceph 0.56.1, 6 nodes, in each
nodes with 20 * spindles + 4* SSDs as journal.(120 spindles in total)
We create a lots of RBD volumes (say 240),mounting to 16
different client machines ( 15 RBD Volumes/ client) and running DD
concurrently on top of each RBD.
The issues are:
1. Slow requests
??From the list-archive it seems solved in 0.56.1 but we still
notice such warning 2. OSD Down or even host down Like the message
below.Seems some OSD has been blocking there for quite a long time.
There is still an issue with throttling recovery/migration traffic
leading to the slow requests that should be fixed shortly.
Suggestions are highly appreciate.Thanks
Xiaoxi
_
Bad news:
I have back all my Ceph machine?s OS to kernel 3.2.0-23, which Ubuntu
12.04 use.
I run dd command (dd if=/dev/zero bs=1M count=6 of=/dev/rbd${i} )on
Ceph client to create data prepare test at last night.
Oooh, you are running the kernel RBD client on a 3.2 kernel. There have
been a long series of fixes since then, but we've only backported as far
back as 3.4. Can you try a newer kernel version for the client?
Something a recnet 3.4 or 3.7 series, like 3.7.2 or 3.4.25...
Thanks!
Now, I have one machine down (can?t be reached by ping), another two
machine has all OSD daemon down, while the three left has some daemon
down.
I have many warnings in OSD log like this:
no flag points reached
2013-01-15 19:14:22.769898 7f20a2d57700 0 log [WRN] : slow request
52.218106 seconds old, received at 2013-01-15 19:13:30.551718:
osd_op(client.10674.1:1002417 rb.0.27a8.6b8b4567.0eba [write
3145728~524288] 2.c61810ee RETRY) currently waiting for sub ops
2013-01-15 19:14:23.770077 7f20a2d57700 0 log [WRN] : 21 slow
requests, 6 included below; oldest blocked for 1132.138983 secs
2013-01-15 19:14:23.770086 7f20a2d57700 0 log [WRN] : slow request
53.216404 seconds old, received at 2013-01-15 19:13:30.553616:
osd_op(client.10671.1:1066860 rb.0.282c.6b8b4567.1057 [write
2621440~524288] 2.ea7acebc) currently waiting for sub ops
2013-01-15 19:14:23.770096 7f20a2d57700 0 log [WRN] : slow request
51.442032 seconds old, received at 2013-01-15 19:13:32.327988:
osd_op(client.10674.1:1002418
Similar info in dmesg we have saw pervious:
[21199.036476] INFO: task ceph-osd:7788 blocked for more than 120 seconds.
[21199.037493] echo 0 /proc/sys/kernel/hung_task_timeout_secs
disables this message.
[21199.038841] ceph-osdD 0006 0 7788 1
0x
[21199.038844] 880fefdafcc8 0086
ffe0 [21199.038848] 880fefdaffd8 880fefdaffd8
880fefdaffd8 00013780 [21199.038852] 88081aa58000
880f68f52de0 880f68f52de0 882017556200 [21199.038856] Call
Trace:
[21199.038858] [8165a55f] schedule+0x3f/0x60
[21199.038861] [8106b7e5] exit_mm+0x85/0x130
[21199.038864] [8106b9fe] do_exit+0x16e/0x420
[21199.038866] [8109d88f] ? __unqueue_futex+0x3f/0x80
[21199.038869] [8107a19a] ? __dequeue_signal+0x6a/0xb0
[21199.038872] [8106be54] do_group_exit+0x44/0xa0
[21199.038874] [8107ccdc]
get_signal_to_deliver+0x21c/0x420 [21199.038877]
[81013865] do_signal+0x45/0x130 [21199.038880]
[810a091c] ? do_futex+0x7c/0x1b0 [21199.038882]
[810a0b5a] ? sys_futex+0x10a/0x1a0 [21199.038885]
[81013b15] do_notify_resume+0x65/0x80 [21199.038887]
[81664d50] int_signal+0x12/0x17
We have seen this stack trace several times over the past 6 months, but are
not sure what the trigger is. In principle, the ceph server-side daemons
shouldn't be capable of locking up like this, but clearly something is amiss
between what they are doing in userland
RE: Ceph slow request unstable issue
Some update summary for tested case till now:
Ceph is v0.56.1
1. RBD:Ubuntu 13.04 + 3.7Kernel
OSD:Ubuntu 13.04 + 3.7Kernel
XFS
Result: Kernel Panic on both RBD and OSD sides
2. RBD:Ubuntu 13.04 +3.2Kernel
OSD:Ubuntu 13.04 +3.2Kernel
XFS
Result:Kernel Panic on RBD( ~15Minus)
3. RBD:Ubuntu 13.04 + 3.6.7 Kernel (Suggested by Ceph.com)
OSD:Ubuntu 13.04 + 3.2 Kernel
XFS
Result: Auto-reset on OSD ( ~ 30 mins after the test started)
4. RBD:Ubuntu 13.04+3.6.7 Kernel (Suggested by Ceph.com)
OSD:Ubuntu 12.04 + 3.2.0-36 Kernel (Suggested by Ceph.com)
XFS
Result:auto-reset on OSD ( ~ 30 mins after the test started)
5. RBD:Ubuntu 13.04+3.6.7 Kernel (Suggested by Ceph.com)
OSD:Ubuntu 13.04 +3.6.7 (Suggested by Sage)
XFS
Result: seems stable for last 1 hour, still running till now
Test 34 are repeatable.
My test setup
OSD side:
3 nodes, 60 Disks(20 per nodes,1 per OSD),10Gb E, 4 *Intel 520 SSD per node
as journal,XFS
For each node,2 * Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GH + 128GB RAM were
used.
RBD side:
8 nodes,for each node:10Gb E,2 * Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GH ,
128GB RAM
Method:
Create 240 RBD and mounted to 8 nodes ( 30 RBD per nodes), doing DD
concurrently on all 240 RBDs.
After ~ 30 minutes, it's likely to have one of the OSD node reset.
Ceph OSD logs, syslog and dmesg from reseted node are available if you
needed.(It looks to me that no valuable information except a lot of
slow-request in OSD's log)
Xiaoxi
-Original Message-
From: Sage Weil [mailto:s...@inktank.com]
Sent: 2013年1月17日 10:35
To: Chen, Xiaoxi
Subject: RE: Ceph slow request unstable issue
On Thu, 17 Jan 2013, Chen, Xiaoxi wrote:
No, on the OSD node, not the same node. OSD node with 3.2 kernel while
client node with 3.6 kernel
We did suffer kernel panic on rbd client nodes but after upgrade
client kernel to 3.6.6 it seems solved .
Is it easy to try the 3.6 kernel on the osd nodes too?
-Original Message-
From: Sage Weil [mailto:s...@inktank.com]
Sent: 2013?1?17? 10:17
To: Chen, Xiaoxi
Subject: RE: Ceph slow request unstable issue
On Thu, 17 Jan 2013, Chen, Xiaoxi wrote:
It is easily to reproduce in my setup...
Once I have enough high load on it and waiting for tens of minutes? I can
see such log.
As a forecast, slow requests more than 30~60s are frequently present in
ceph osd's log.
Just replied to your other email. Do I understand correctly that you are
seeing this problem on the *rbd client* nodes? Or also on the OSDs? Are
they the same nodes?
sage
-Original Message-
From: Sage Weil [mailto:s...@inktank.com]
Sent: 2013?1?17? 0:59
To: Andrey Korolyov
Cc: Chen, Xiaoxi; ceph-devel@vger.kernel.org
Subject: Re: Ceph slow request unstable issue
Hi,
On Wed, 16 Jan 2013, Andrey Korolyov wrote:
On Wed, Jan 16, 2013 at 4:58 AM, Chen, Xiaoxi xiaoxi.c...@intel.com
wrote:
Hi list,
We are suffering from OSD or OS down when there is continuing
high pressure on the Ceph rack.
Basically we are on Ubuntu 12.04+ Ceph 0.56.1, 6 nodes, in each
nodes with 20 * spindles + 4* SSDs as journal.(120 spindles in total)
We create a lots of RBD volumes (say 240),mounting to 16
different client machines ( 15 RBD Volumes/ client) and running DD
concurrently on top of each RBD.
The issues are:
1. Slow requests
??From the list-archive it seems solved in 0.56.1 but we still
notice such warning 2. OSD Down or even host down Like the
message below.Seems some OSD has been blocking there for quite a long
time.
Suggestions are highly appreciate.Thanks
Xiaoxi
_
Bad news:
I have back all my Ceph machine?s OS to kernel 3.2.0-23, which Ubuntu
12.04 use.
I run dd command (dd if=/dev/zero bs=1M count=6 of=/dev/rbd${i}
)on Ceph client to create data prepare test at last night.
Now, I have one machine down (can?t be reached by ping), another two
machine has all OSD daemon down, while the three left has some daemon
down.
I have many warnings in OSD log like this:
no flag points reached
2013-01-15 19:14:22.769898 7f20a2d57700 0 log [WRN] : slow
request
52.218106 seconds old, received at 2013-01-15 19:13:30.551718:
osd_op(client.10674.1:1002417 rb.0.27a8.6b8b4567.0eba
Ceph slow request unstable issue
Hi list,
We are suffering from OSD or OS down when there is continuing high
pressure on the Ceph rack.
Basically we are on Ubuntu 12.04+ Ceph 0.56.1, 6 nodes, in each nodes
with 20 * spindles + 4* SSDs as journal.(120 spindles in total)
We create a lots of RBD volumes (say 240),mounting to 16 different
client machines ( 15 RBD Volumes/ client) and running DD concurrently on top of
each RBD.
The issues are:
1. Slow requests
From the list-archive it seems solved in 0.56.1 but we still notice such
warning
2. OSD Down or even host down
Like the message below.Seems some OSD has been blocking there for quite a long
time.
Suggestions are highly appreciate.Thanks
Xiaoxi
_
Bad news:
I have back all my Ceph machine’s OS to kernel 3.2.0-23, which Ubuntu 12.04
use.
I run dd command (dd if=/dev/zero bs=1M count=6 of=/dev/rbd${i} )on Ceph
client to create data prepare test at last night.
Now, I have one machine down (can’t be reached by ping), another two machine
has all OSD daemon down, while the three left has some daemon down.
I have many warnings in OSD log like this:
no flag points reached
2013-01-15 19:14:22.769898 7f20a2d57700 0 log [WRN] : slow request 52.218106
seconds old, received at 2013-01-15 19:13:30.551718:
osd_op(client.10674.1:1002417 rb.0.27a8.6b8b4567.0eba [write
3145728~524288] 2.c61810ee RETRY) currently waiting for sub ops
2013-01-15 19:14:23.770077 7f20a2d57700 0 log [WRN] : 21 slow requests, 6
included below; oldest blocked for 1132.138983 secs
2013-01-15 19:14:23.770086 7f20a2d57700 0 log [WRN] : slow request 53.216404
seconds old, received at 2013-01-15 19:13:30.553616:
osd_op(client.10671.1:1066860 rb.0.282c.6b8b4567.1057 [write
2621440~524288] 2.ea7acebc) currently waiting for sub ops
2013-01-15 19:14:23.770096 7f20a2d57700 0 log [WRN] : slow request 51.442032
seconds old, received at 2013-01-15 19:13:32.327988:
osd_op(client.10674.1:1002418
Similar info in dmesg we have saw pervious:
[21199.036476] INFO: task ceph-osd:7788 blocked for more than 120 seconds.
[21199.037493] echo 0 /proc/sys/kernel/hung_task_timeout_secs disables this
message.
[21199.038841] ceph-osdD 0006 0 7788 1 0x
[21199.038844] 880fefdafcc8 0086
ffe0
[21199.038848] 880fefdaffd8 880fefdaffd8 880fefdaffd8
00013780
[21199.038852] 88081aa58000 880f68f52de0 880f68f52de0
882017556200
[21199.038856] Call Trace:
[21199.038858] [8165a55f] schedule+0x3f/0x60
[21199.038861] [8106b7e5] exit_mm+0x85/0x130
[21199.038864] [8106b9fe] do_exit+0x16e/0x420
[21199.038866] [8109d88f] ? __unqueue_futex+0x3f/0x80
[21199.038869] [8107a19a] ? __dequeue_signal+0x6a/0xb0
[21199.038872] [8106be54] do_group_exit+0x44/0xa0
[21199.038874] [8107ccdc] get_signal_to_deliver+0x21c/0x420
[21199.038877] [81013865] do_signal+0x45/0x130
[21199.038880] [810a091c] ? do_futex+0x7c/0x1b0
[21199.038882] [810a0b5a] ? sys_futex+0x10a/0x1a0
[21199.038885] [81013b15] do_notify_resume+0x65/0x80
[21199.038887] [81664d50] int_signal+0x12/0x17
Thanks.
-chen
N�Р骒r��yb�X�肚�v�^�)藓{.n�+���z�]z鳐�{ay� �,j��f＂�h���z� �wア�
⒎�j:+v���w�j�m��赙zZ+�茛j��!�i
