Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-20 Thread Jens Axboe
On 11/20/2013 01:07 AM, Christoph Hellwig wrote: >> Just stumbled on that too. You need one more, btw, for the sg failure >> case: >> >> >> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c >> index 588479d58f52..6a680d4de7f1 100644 >> --- a/drivers/block/virtio_blk.c >> +++

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-20 Thread Jens Axboe
On 11/20/2013 01:04 AM, Christoph Hellwig wrote: > On Tue, Nov 19, 2013 at 02:43:51PM -0700, Jens Axboe wrote: >> No, the nr_requests isn't actually relevant in the blk-mq context, the >> driver sets its own depth. For the above, it's 64 normal commands, and 2 >> reserved. The reserved would be

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-20 Thread Christoph Hellwig
> Just stumbled on that too. You need one more, btw, for the sg failure > case: > > > diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c > index 588479d58f52..6a680d4de7f1 100644 > --- a/drivers/block/virtio_blk.c > +++ b/drivers/block/virtio_blk.c > @@ -199,15 +199,16 @@

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-20 Thread Christoph Hellwig
On Tue, Nov 19, 2013 at 02:43:51PM -0700, Jens Axboe wrote: > No, the nr_requests isn't actually relevant in the blk-mq context, the > driver sets its own depth. For the above, it's 64 normal commands, and 2 > reserved. The reserved would be for a flush, for instance. If someone > attempts to

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-20 Thread Christoph Hellwig
On Tue, Nov 19, 2013 at 02:43:51PM -0700, Jens Axboe wrote: No, the nr_requests isn't actually relevant in the blk-mq context, the driver sets its own depth. For the above, it's 64 normal commands, and 2 reserved. The reserved would be for a flush, for instance. If someone attempts to queue

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-20 Thread Christoph Hellwig
Just stumbled on that too. You need one more, btw, for the sg failure case: diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c index 588479d58f52..6a680d4de7f1 100644 --- a/drivers/block/virtio_blk.c +++ b/drivers/block/virtio_blk.c @@ -199,15 +199,16 @@ static int

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-20 Thread Jens Axboe
On 11/20/2013 01:04 AM, Christoph Hellwig wrote: On Tue, Nov 19, 2013 at 02:43:51PM -0700, Jens Axboe wrote: No, the nr_requests isn't actually relevant in the blk-mq context, the driver sets its own depth. For the above, it's 64 normal commands, and 2 reserved. The reserved would be for a

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-20 Thread Jens Axboe
On 11/20/2013 01:07 AM, Christoph Hellwig wrote: Just stumbled on that too. You need one more, btw, for the sg failure case: diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c index 588479d58f52..6a680d4de7f1 100644 --- a/drivers/block/virtio_blk.c +++

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Jens Axboe
On Wed, Nov 20 2013, Dave Chinner wrote: > On Tue, Nov 19, 2013 at 07:02:30PM -0700, Jens Axboe wrote: > > On Tue, Nov 19 2013, Jens Axboe wrote: > > > > Looks like a race condition, below works for me, please try. > > > > > > > > > > > > Subject: virtio_blk: fix race condition > > > > > > > >

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Dave Chinner
On Tue, Nov 19, 2013 at 07:02:30PM -0700, Jens Axboe wrote: > On Tue, Nov 19 2013, Jens Axboe wrote: > > > Looks like a race condition, below works for me, please try. > > > > > > > > > Subject: virtio_blk: fix race condition > > > > > > virtqueue_kick() isn't multi-thread safe. > > > > > >

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Jens Axboe
On Tue, Nov 19 2013, Jens Axboe wrote: > > Looks like a race condition, below works for me, please try. > > > > > > Subject: virtio_blk: fix race condition > > > > virtqueue_kick() isn't multi-thread safe. > > > > Signed-off-by: Shaohua Li > > > > diff --git a/drivers/block/virtio_blk.c

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Jens Axboe
On Wed, Nov 20 2013, Shaohua Li wrote: > On Tue, Nov 19, 2013 at 05:08:58PM -0700, Jens Axboe wrote: > > On Tue, Nov 19 2013, Jens Axboe wrote: > > > On Wed, Nov 20 2013, Dave Chinner wrote: > > > > On Tue, Nov 19, 2013 at 03:51:27PM -0700, Jens Axboe wrote: > > > > > On 11/19/2013 03:42 PM, Jens

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Shaohua Li
On Tue, Nov 19, 2013 at 05:08:58PM -0700, Jens Axboe wrote: > On Tue, Nov 19 2013, Jens Axboe wrote: > > On Wed, Nov 20 2013, Dave Chinner wrote: > > > On Tue, Nov 19, 2013 at 03:51:27PM -0700, Jens Axboe wrote: > > > > On 11/19/2013 03:42 PM, Jens Axboe wrote: > > > > > On 11/19/2013 02:43 PM,

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Jens Axboe
On Tue, Nov 19 2013, Jens Axboe wrote: > On Wed, Nov 20 2013, Dave Chinner wrote: > > On Tue, Nov 19, 2013 at 03:51:27PM -0700, Jens Axboe wrote: > > > On 11/19/2013 03:42 PM, Jens Axboe wrote: > > > > On 11/19/2013 02:43 PM, Jens Axboe wrote: > > > >> On 11/19/2013 02:34 PM, Dave Chinner wrote: >

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Jens Axboe
On Wed, Nov 20 2013, Dave Chinner wrote: > On Tue, Nov 19, 2013 at 03:51:27PM -0700, Jens Axboe wrote: > > On 11/19/2013 03:42 PM, Jens Axboe wrote: > > > On 11/19/2013 02:43 PM, Jens Axboe wrote: > > >> On 11/19/2013 02:34 PM, Dave Chinner wrote: > > >>> On Tue, Nov 19, 2013 at 02:20:42PM -0700,

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Dave Chinner
On Tue, Nov 19, 2013 at 03:51:27PM -0700, Jens Axboe wrote: > On 11/19/2013 03:42 PM, Jens Axboe wrote: > > On 11/19/2013 02:43 PM, Jens Axboe wrote: > >> On 11/19/2013 02:34 PM, Dave Chinner wrote: > >>> On Tue, Nov 19, 2013 at 02:20:42PM -0700, Jens Axboe wrote: > On Tue, Nov 19 2013, Jens

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Jens Axboe
On 11/19/2013 03:42 PM, Jens Axboe wrote: > On 11/19/2013 02:43 PM, Jens Axboe wrote: >> On 11/19/2013 02:34 PM, Dave Chinner wrote: >>> On Tue, Nov 19, 2013 at 02:20:42PM -0700, Jens Axboe wrote: On Tue, Nov 19 2013, Jens Axboe wrote: > On Tue, Nov 19 2013, Dave Chinner wrote: >> Hi

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Jens Axboe
On 11/19/2013 02:43 PM, Jens Axboe wrote: > On 11/19/2013 02:34 PM, Dave Chinner wrote: >> On Tue, Nov 19, 2013 at 02:20:42PM -0700, Jens Axboe wrote: >>> On Tue, Nov 19 2013, Jens Axboe wrote: On Tue, Nov 19 2013, Dave Chinner wrote: > Hi Jens, > > I was just running xfstests on

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Jens Axboe
On 11/19/2013 02:34 PM, Dave Chinner wrote: > On Tue, Nov 19, 2013 at 02:20:42PM -0700, Jens Axboe wrote: >> On Tue, Nov 19 2013, Jens Axboe wrote: >>> On Tue, Nov 19 2013, Dave Chinner wrote: Hi Jens, I was just running xfstests on a 3.13 kernel that has had the block layer

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Jens Axboe
On 11/19/2013 02:30 PM, Dave Chinner wrote: > On Tue, Nov 19, 2013 at 09:05:09AM -0700, Jens Axboe wrote: >> On Tue, Nov 19 2013, Christoph Hellwig wrote: >>> On Tue, Nov 19, 2013 at 07:02:18PM +1100, Dave Chinner wrote: I have no idea if it's related to the above hang, but either way

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Dave Chinner
On Tue, Nov 19, 2013 at 02:20:42PM -0700, Jens Axboe wrote: > On Tue, Nov 19 2013, Jens Axboe wrote: > > On Tue, Nov 19 2013, Dave Chinner wrote: > > > Hi Jens, > > > > > > I was just running xfstests on a 3.13 kernel that has had the block > > > layer changed merged into it. generic/269 on XFS

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Dave Chinner
On Tue, Nov 19, 2013 at 09:05:09AM -0700, Jens Axboe wrote: > On Tue, Nov 19 2013, Christoph Hellwig wrote: > > On Tue, Nov 19, 2013 at 07:02:18PM +1100, Dave Chinner wrote: > > > I have no idea if it's related to the above hang, but either way > > > breaking iostat is a major regression > >

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Jens Axboe
On Tue, Nov 19 2013, Jens Axboe wrote: > On Tue, Nov 19 2013, Dave Chinner wrote: > > Hi Jens, > > > > I was just running xfstests on a 3.13 kernel that has had the block > > layer changed merged into it. generic/269 on XFS is hanging on a 2 > > CPU VM using virtio,cache=none for the block

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Jens Axboe
On Tue, Nov 19 2013, Dave Chinner wrote: > Hi Jens, > > I was just running xfstests on a 3.13 kernel that has had the block > layer changed merged into it. generic/269 on XFS is hanging on a 2 > CPU VM using virtio,cache=none for the block devices under test, > with many (130+) threads stuck

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Jens Axboe
On 11/19/2013 09:09 AM, Christoph Hellwig wrote: > On Tue, Nov 19, 2013 at 09:05:09AM -0700, Jens Axboe wrote: >> I have queued up the below. > > I don't think that will work. We only set QUEUE_FLAG_IO_STAT in > blk_init_allocated_queue, which doesn't get called from > blk_mq_init_queue. That

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Christoph Hellwig
On Tue, Nov 19, 2013 at 09:05:09AM -0700, Jens Axboe wrote: > I have queued up the below. I don't think that will work. We only set QUEUE_FLAG_IO_STAT in blk_init_allocated_queue, which doesn't get called from blk_mq_init_queue. That being said I really don't understand why we need to have the

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Jens Axboe
On Tue, Nov 19 2013, Christoph Hellwig wrote: > On Tue, Nov 19, 2013 at 07:02:18PM +1100, Dave Chinner wrote: > > I have no idea if it's related to the above hang, but either way > > breaking iostat is a major regression > > Both of them are most likely due to the conversion of virtio_blk >

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Christoph Hellwig
On Tue, Nov 19, 2013 at 07:02:18PM +1100, Dave Chinner wrote: > I have no idea if it's related to the above hang, but either way > breaking iostat is a major regression Both of them are most likely due to the conversion of virtio_blk to the blk-mq code. I've not seen the hang in my heavy

[Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Dave Chinner
Hi Jens, I was just running xfstests on a 3.13 kernel that has had the block layer changed merged into it. generic/269 on XFS is hanging on a 2 CPU VM using virtio,cache=none for the block devices under test, with many (130+) threads stuck below submit_bio() like this: Call Trace: []

[Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Dave Chinner
Hi Jens, I was just running xfstests on a 3.13 kernel that has had the block layer changed merged into it. generic/269 on XFS is hanging on a 2 CPU VM using virtio,cache=none for the block devices under test, with many (130+) threads stuck below submit_bio() like this: Call Trace:

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Christoph Hellwig
On Tue, Nov 19, 2013 at 07:02:18PM +1100, Dave Chinner wrote: I have no idea if it's related to the above hang, but either way breaking iostat is a major regression Both of them are most likely due to the conversion of virtio_blk to the blk-mq code. I've not seen the hang in my heavy

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Jens Axboe
On Tue, Nov 19 2013, Christoph Hellwig wrote: On Tue, Nov 19, 2013 at 07:02:18PM +1100, Dave Chinner wrote: I have no idea if it's related to the above hang, but either way breaking iostat is a major regression Both of them are most likely due to the conversion of virtio_blk to the

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Christoph Hellwig
On Tue, Nov 19, 2013 at 09:05:09AM -0700, Jens Axboe wrote: I have queued up the below. I don't think that will work. We only set QUEUE_FLAG_IO_STAT in blk_init_allocated_queue, which doesn't get called from blk_mq_init_queue. That being said I really don't understand why we need to have the

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Jens Axboe
On 11/19/2013 09:09 AM, Christoph Hellwig wrote: On Tue, Nov 19, 2013 at 09:05:09AM -0700, Jens Axboe wrote: I have queued up the below. I don't think that will work. We only set QUEUE_FLAG_IO_STAT in blk_init_allocated_queue, which doesn't get called from blk_mq_init_queue. That being

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Jens Axboe
On Tue, Nov 19 2013, Dave Chinner wrote: Hi Jens, I was just running xfstests on a 3.13 kernel that has had the block layer changed merged into it. generic/269 on XFS is hanging on a 2 CPU VM using virtio,cache=none for the block devices under test, with many (130+) threads stuck below

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Jens Axboe
On Tue, Nov 19 2013, Jens Axboe wrote: On Tue, Nov 19 2013, Dave Chinner wrote: Hi Jens, I was just running xfstests on a 3.13 kernel that has had the block layer changed merged into it. generic/269 on XFS is hanging on a 2 CPU VM using virtio,cache=none for the block devices under

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Dave Chinner
On Tue, Nov 19, 2013 at 09:05:09AM -0700, Jens Axboe wrote: On Tue, Nov 19 2013, Christoph Hellwig wrote: On Tue, Nov 19, 2013 at 07:02:18PM +1100, Dave Chinner wrote: I have no idea if it's related to the above hang, but either way breaking iostat is a major regression Both of

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Dave Chinner
On Tue, Nov 19, 2013 at 02:20:42PM -0700, Jens Axboe wrote: On Tue, Nov 19 2013, Jens Axboe wrote: On Tue, Nov 19 2013, Dave Chinner wrote: Hi Jens, I was just running xfstests on a 3.13 kernel that has had the block layer changed merged into it. generic/269 on XFS is hanging on a

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Jens Axboe
On 11/19/2013 02:30 PM, Dave Chinner wrote: On Tue, Nov 19, 2013 at 09:05:09AM -0700, Jens Axboe wrote: On Tue, Nov 19 2013, Christoph Hellwig wrote: On Tue, Nov 19, 2013 at 07:02:18PM +1100, Dave Chinner wrote: I have no idea if it's related to the above hang, but either way breaking iostat

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Jens Axboe
On 11/19/2013 02:34 PM, Dave Chinner wrote: On Tue, Nov 19, 2013 at 02:20:42PM -0700, Jens Axboe wrote: On Tue, Nov 19 2013, Jens Axboe wrote: On Tue, Nov 19 2013, Dave Chinner wrote: Hi Jens, I was just running xfstests on a 3.13 kernel that has had the block layer changed merged into it.

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Jens Axboe
On 11/19/2013 02:43 PM, Jens Axboe wrote: On 11/19/2013 02:34 PM, Dave Chinner wrote: On Tue, Nov 19, 2013 at 02:20:42PM -0700, Jens Axboe wrote: On Tue, Nov 19 2013, Jens Axboe wrote: On Tue, Nov 19 2013, Dave Chinner wrote: Hi Jens, I was just running xfstests on a 3.13 kernel that has

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Jens Axboe
On 11/19/2013 03:42 PM, Jens Axboe wrote: On 11/19/2013 02:43 PM, Jens Axboe wrote: On 11/19/2013 02:34 PM, Dave Chinner wrote: On Tue, Nov 19, 2013 at 02:20:42PM -0700, Jens Axboe wrote: On Tue, Nov 19 2013, Jens Axboe wrote: On Tue, Nov 19 2013, Dave Chinner wrote: Hi Jens, I was just

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Dave Chinner
On Tue, Nov 19, 2013 at 03:51:27PM -0700, Jens Axboe wrote: On 11/19/2013 03:42 PM, Jens Axboe wrote: On 11/19/2013 02:43 PM, Jens Axboe wrote: On 11/19/2013 02:34 PM, Dave Chinner wrote: On Tue, Nov 19, 2013 at 02:20:42PM -0700, Jens Axboe wrote: On Tue, Nov 19 2013, Jens Axboe wrote:

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Jens Axboe
On Wed, Nov 20 2013, Dave Chinner wrote: On Tue, Nov 19, 2013 at 03:51:27PM -0700, Jens Axboe wrote: On 11/19/2013 03:42 PM, Jens Axboe wrote: On 11/19/2013 02:43 PM, Jens Axboe wrote: On 11/19/2013 02:34 PM, Dave Chinner wrote: On Tue, Nov 19, 2013 at 02:20:42PM -0700, Jens Axboe

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Jens Axboe
On Tue, Nov 19 2013, Jens Axboe wrote: On Wed, Nov 20 2013, Dave Chinner wrote: On Tue, Nov 19, 2013 at 03:51:27PM -0700, Jens Axboe wrote: On 11/19/2013 03:42 PM, Jens Axboe wrote: On 11/19/2013 02:43 PM, Jens Axboe wrote: On 11/19/2013 02:34 PM, Dave Chinner wrote: On Tue, Nov

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Shaohua Li
On Tue, Nov 19, 2013 at 05:08:58PM -0700, Jens Axboe wrote: On Tue, Nov 19 2013, Jens Axboe wrote: On Wed, Nov 20 2013, Dave Chinner wrote: On Tue, Nov 19, 2013 at 03:51:27PM -0700, Jens Axboe wrote: On 11/19/2013 03:42 PM, Jens Axboe wrote: On 11/19/2013 02:43 PM, Jens Axboe wrote:

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Jens Axboe
On Wed, Nov 20 2013, Shaohua Li wrote: On Tue, Nov 19, 2013 at 05:08:58PM -0700, Jens Axboe wrote: On Tue, Nov 19 2013, Jens Axboe wrote: On Wed, Nov 20 2013, Dave Chinner wrote: On Tue, Nov 19, 2013 at 03:51:27PM -0700, Jens Axboe wrote: On 11/19/2013 03:42 PM, Jens Axboe wrote:

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Jens Axboe
On Tue, Nov 19 2013, Jens Axboe wrote: Looks like a race condition, below works for me, please try. Subject: virtio_blk: fix race condition virtqueue_kick() isn't multi-thread safe. Signed-off-by: Shaohua Li s...@fusionio.com diff --git a/drivers/block/virtio_blk.c

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Dave Chinner
On Tue, Nov 19, 2013 at 07:02:30PM -0700, Jens Axboe wrote: On Tue, Nov 19 2013, Jens Axboe wrote: Looks like a race condition, below works for me, please try. Subject: virtio_blk: fix race condition virtqueue_kick() isn't multi-thread safe. Signed-off-by: Shaohua Li

Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices

2013-11-19 Thread Jens Axboe
On Wed, Nov 20 2013, Dave Chinner wrote: On Tue, Nov 19, 2013 at 07:02:30PM -0700, Jens Axboe wrote: On Tue, Nov 19 2013, Jens Axboe wrote: Looks like a race condition, below works for me, please try. Subject: virtio_blk: fix race condition virtqueue_kick() isn't