Hi,
On Wed, 16 Jan 2013, Andrey Korolyov wrote:
> On Wed, Jan 16, 2013 at 4:58 AM, Chen, Xiaoxi <xiaoxi.c...@intel.com> wrote:
> > Hi list,
> > We are suffering from OSD or OS down when there is continuing high
> > pressure on the Ceph rack.
> > Basically we are on Ubuntu 12.04+ Ceph 0.56.1, 6 nodes, in each
> > nodes with 20 * spindles + 4* SSDs as journal.(120 spindles in total)
> > We create a lots of RBD volumes (say 240),mounting to 16 different
> > client machines ( 15 RBD Volumes/ client) and running DD concurrently on
> > top of each RBD.
> >
> > The issues are:
> > 1. Slow requests
> > ??From the list-archive it seems solved in 0.56.1 but we still notice such
> > warning
> > 2. OSD Down or even host down
> > Like the message below.Seems some OSD has been blocking there for quite a
> > long time.
> >
> > Suggestions are highly appreciate.Thanks
> >
> >
> > Xiaoxi
> >
> > _____________________________________________
> >
> > Bad news:
> >
> > I have back all my Ceph machine?s OS to kernel 3.2.0-23, which Ubuntu
> > 12.04 use.
> > I run dd command (dd if=/dev/zero bs=1M count=60000 of=/dev/rbd${i} & )on
> > Ceph client to create data prepare test at last night.
> > Now, I have one machine down (can?t be reached by ping), another two
> > machine has all OSD daemon down, while the three left has some daemon down.
> >
> > I have many warnings in OSD log like this:
> >
> > no flag points reached
> > 2013-01-15 19:14:22.769898 7f20a2d57700 0 log [WRN] : slow request
> > 52.218106 seconds old, received at 2013-01-15 19:13:30.551718:
> > osd_op(client.10674.1:1002417 rb.0.27a8.6b8b4567.000000000eba [write
> > 3145728~524288] 2.c61810ee RETRY) currently waiting for sub ops
> > 2013-01-15 19:14:23.770077 7f20a2d57700 0 log [WRN] : 21 slow requests, 6
> > included below; oldest blocked for > 1132.138983 secs
> > 2013-01-15 19:14:23.770086 7f20a2d57700 0 log [WRN] : slow request
> > 53.216404 seconds old, received at 2013-01-15 19:13:30.553616:
> > osd_op(client.10671.1:1066860 rb.0.282c.6b8b4567.000000001057 [write
> > 2621440~524288] 2.ea7acebc) currently waiting for sub ops
> > 2013-01-15 19:14:23.770096 7f20a2d57700 0 log [WRN] : slow request
> > 51.442032 seconds old, received at 2013-01-15 19:13:32.327988:
> > osd_op(client.10674.1:1002418
> >
> > Similar info in dmesg we have saw pervious:
> >
> > [21199.036476] INFO: task ceph-osd:7788 blocked for more than 120 seconds.
> > [21199.037493] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> > this message.
> > [21199.038841] ceph-osd D 0000000000000006 0 7788 1
> > 0x00000000
> > [21199.038844] ffff880fefdafcc8 0000000000000086 0000000000000000
> > ffffffffffffffe0
> > [21199.038848] ffff880fefdaffd8 ffff880fefdaffd8 ffff880fefdaffd8
> > 0000000000013780
> > [21199.038852] ffff88081aa58000 ffff880f68f52de0 ffff880f68f52de0
> > ffff882017556200
> > [21199.038856] Call Trace:
> > [21199.038858] [<ffffffff8165a55f>] schedule+0x3f/0x60
> > [21199.038861] [<ffffffff8106b7e5>] exit_mm+0x85/0x130
> > [21199.038864] [<ffffffff8106b9fe>] do_exit+0x16e/0x420
> > [21199.038866] [<ffffffff8109d88f>] ? __unqueue_futex+0x3f/0x80
> > [21199.038869] [<ffffffff8107a19a>] ? __dequeue_signal+0x6a/0xb0
> > [21199.038872] [<ffffffff8106be54>] do_group_exit+0x44/0xa0
> > [21199.038874] [<ffffffff8107ccdc>] get_signal_to_deliver+0x21c/0x420
> > [21199.038877] [<ffffffff81013865>] do_signal+0x45/0x130
> > [21199.038880] [<ffffffff810a091c>] ? do_futex+0x7c/0x1b0
> > [21199.038882] [<ffffffff810a0b5a>] ? sys_futex+0x10a/0x1a0
> > [21199.038885] [<ffffffff81013b15>] do_notify_resume+0x65/0x80
> > [21199.038887] [<ffffffff81664d50>] int_signal+0x12/0x17
We have seen this stack trace several times over the past 6 months, but
are not sure what the trigger is. In principle, the ceph server-side
daemons shouldn't be capable of locking up like this, but clearly
something is amiss between what they are doing in userland and how the
kernel is tolerating that. Low memory, perhaps? In each case where we
tried to track it down, the problem seemed to go away on its own. Is this
easily reproducible in your case?
> my 0.02$:
> http://www.mail-archive.com/ceph-devel@vger.kernel.org/msg11531.html
> and kernel panic from two different hosts from yesterday during ceph
> startup(on 3.8-rc3, images from console available at
> http://imgur.com/wIRVn,k0QCS#0) leads to suggestion that Ceph may have
> been introduced lockup-alike behavior not a long ago, causing, in my
> case, excessive amount of context switches on the host leading to osd
> flaps and panic at the ip-ib stack due to same issue.
For the stack trace my first guess would be a problem with the IB driver
that is triggered by memory pressure. Can you characterize what the
system utilization (CPU, memory) looks like leading up to the lockup?
sage
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
