Re: [Cluster-devel] [dm-devel] [PATCH V15 00/18] block: support multi-page bvec
On Sun, 2019-02-17 at 21:11 +0800, Ming Lei wrote:
> The following patch should fix this issue:
>
>
> diff --git a/block/blk-merge.c b/block/blk-merge.c
> index bed065904677..066b66430523 100644
> --- a/block/blk-merge.c
> +++ b/block/blk-merge.c
> @@ -363,13 +363,15 @@ static unsigned int __blk_recalc_rq_segments(struct
> request_queue *q,
> struct bio_vec bv, bvprv = { NULL };
> int prev = 0;
> unsigned int seg_size, nr_phys_segs;
> - unsigned front_seg_size = bio->bi_seg_front_size;
> + unsigned front_seg_size;
> struct bio *fbio, *bbio;
> struct bvec_iter iter;
>
> if (!bio)
> return 0;
>
> + front_seg_size = bio->bi_seg_front_size;
> +
> switch (bio_op(bio)) {
> case REQ_OP_DISCARD:
> case REQ_OP_SECURE_ERASE:
Hi Ming,
With this patch applied test nvmeof-mp/002 fails as follows:
[ 694.700400] kernel BUG at lib/sg_pool.c:103!
[ 694.705932] invalid opcode: [#1] PREEMPT SMP KASAN
[ 694.708297] CPU: 2 PID: 349 Comm: kworker/2:1H Tainted: GB
5.0.0-rc6-dbg+ #2
[ 694.711730] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
1.10.2-1 04/01/2014
[ 694.715113] Workqueue: kblockd blk_mq_run_work_fn
[ 694.716894] RIP: 0010:sg_alloc_table_chained+0xe5/0xf0
[ 694.758222] Call Trace:
[ 694.759645] nvme_rdma_queue_rq+0x2aa/0xcc0 [nvme_rdma]
[ 694.764915] blk_mq_try_issue_directly+0x2a5/0x4b0
[ 694.771779] blk_insert_cloned_request+0x11e/0x1c0
[ 694.778417] dm_mq_queue_rq+0x3d1/0x770
[ 694.793400] blk_mq_dispatch_rq_list+0x5fc/0xb10
[ 694.798386] blk_mq_sched_dispatch_requests+0x2f7/0x300
[ 694.803180] __blk_mq_run_hw_queue+0xd6/0x180
[ 694.808933] blk_mq_run_work_fn+0x27/0x30
[ 694.810315] process_one_work+0x4f1/0xa40
[ 694.813178] worker_thread+0x67/0x5b0
[ 694.814487] kthread+0x1cf/0x1f0
[ 694.819134] ret_from_fork+0x24/0x30
The code in sg_pool.c that triggers the BUG() statement is as follows:
int sg_alloc_table_chained(struct sg_table *table, int nents,
struct scatterlist *first_chunk)
{
int ret;
BUG_ON(!nents);
[ ... ]
Bart.
Re: [Cluster-devel] [dm-devel] [PATCH V15 00/18] block: support multi-page bvec
On 2/19/19 5:17 PM, Ming Lei wrote:
On Tue, Feb 19, 2019 at 08:28:19AM -0800, Bart Van Assche wrote:
With this patch applied test nvmeof-mp/002 fails as follows:
[ 694.700400] kernel BUG at lib/sg_pool.c:103!
[ 694.705932] invalid opcode: [#1] PREEMPT SMP KASAN
[ 694.708297] CPU: 2 PID: 349 Comm: kworker/2:1H Tainted: GB
5.0.0-rc6-dbg+ #2
[ 694.711730] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
1.10.2-1 04/01/2014
[ 694.715113] Workqueue: kblockd blk_mq_run_work_fn
[ 694.716894] RIP: 0010:sg_alloc_table_chained+0xe5/0xf0
[ 694.758222] Call Trace:
[ 694.759645] nvme_rdma_queue_rq+0x2aa/0xcc0 [nvme_rdma]
[ 694.764915] blk_mq_try_issue_directly+0x2a5/0x4b0
[ 694.771779] blk_insert_cloned_request+0x11e/0x1c0
[ 694.778417] dm_mq_queue_rq+0x3d1/0x770
[ 694.793400] blk_mq_dispatch_rq_list+0x5fc/0xb10
[ 694.798386] blk_mq_sched_dispatch_requests+0x2f7/0x300
[ 694.803180] __blk_mq_run_hw_queue+0xd6/0x180
[ 694.808933] blk_mq_run_work_fn+0x27/0x30
[ 694.810315] process_one_work+0x4f1/0xa40
[ 694.813178] worker_thread+0x67/0x5b0
[ 694.814487] kthread+0x1cf/0x1f0
[ 694.819134] ret_from_fork+0x24/0x30
The code in sg_pool.c that triggers the BUG() statement is as follows:
int sg_alloc_table_chained(struct sg_table *table, int nents,
struct scatterlist *first_chunk)
{
int ret;
BUG_ON(!nents);
[ ... ]
Bart.
I can reproduce this issue("kernel BUG at lib/sg_pool.c:103") without mp-bvec
patches,
so looks it isn't the fault of this patchset.
Thanks Ming for your feedback.
Jens, I don't see that issue with kernel v5.0-rc6. Does that mean that
the sg_pool BUG() is a regression in your for-next branch that predates
Ming's multi-page bvec patch series?
Thanks,
Bart.
Re: [Cluster-devel] [dm-devel] [PATCH V15 00/18] block: support multi-page bvec
On Tue, Feb 19, 2019 at 08:28:19AM -0800, Bart Van Assche wrote:
> On Sun, 2019-02-17 at 21:11 +0800, Ming Lei wrote:
> > The following patch should fix this issue:
> >
> >
> > diff --git a/block/blk-merge.c b/block/blk-merge.c
> > index bed065904677..066b66430523 100644
> > --- a/block/blk-merge.c
> > +++ b/block/blk-merge.c
> > @@ -363,13 +363,15 @@ static unsigned int __blk_recalc_rq_segments(struct
> > request_queue *q,
> > struct bio_vec bv, bvprv = { NULL };
> > int prev = 0;
> > unsigned int seg_size, nr_phys_segs;
> > - unsigned front_seg_size = bio->bi_seg_front_size;
> > + unsigned front_seg_size;
> > struct bio *fbio, *bbio;
> > struct bvec_iter iter;
> >
> > if (!bio)
> > return 0;
> >
> > + front_seg_size = bio->bi_seg_front_size;
> > +
> > switch (bio_op(bio)) {
> > case REQ_OP_DISCARD:
> > case REQ_OP_SECURE_ERASE:
>
> Hi Ming,
>
> With this patch applied test nvmeof-mp/002 fails as follows:
>
> [ 694.700400] kernel BUG at lib/sg_pool.c:103!
> [ 694.705932] invalid opcode: [#1] PREEMPT SMP KASAN
> [ 694.708297] CPU: 2 PID: 349 Comm: kworker/2:1H Tainted: GB
> 5.0.0-rc6-dbg+ #2
> [ 694.711730] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
> 1.10.2-1 04/01/2014
> [ 694.715113] Workqueue: kblockd blk_mq_run_work_fn
> [ 694.716894] RIP: 0010:sg_alloc_table_chained+0xe5/0xf0
> [ 694.758222] Call Trace:
> [ 694.759645] nvme_rdma_queue_rq+0x2aa/0xcc0 [nvme_rdma]
> [ 694.764915] blk_mq_try_issue_directly+0x2a5/0x4b0
> [ 694.771779] blk_insert_cloned_request+0x11e/0x1c0
> [ 694.778417] dm_mq_queue_rq+0x3d1/0x770
> [ 694.793400] blk_mq_dispatch_rq_list+0x5fc/0xb10
> [ 694.798386] blk_mq_sched_dispatch_requests+0x2f7/0x300
> [ 694.803180] __blk_mq_run_hw_queue+0xd6/0x180
> [ 694.808933] blk_mq_run_work_fn+0x27/0x30
> [ 694.810315] process_one_work+0x4f1/0xa40
> [ 694.813178] worker_thread+0x67/0x5b0
> [ 694.814487] kthread+0x1cf/0x1f0
> [ 694.819134] ret_from_fork+0x24/0x30
>
> The code in sg_pool.c that triggers the BUG() statement is as follows:
>
> int sg_alloc_table_chained(struct sg_table *table, int nents,
> struct scatterlist *first_chunk)
> {
> int ret;
>
> BUG_ON(!nents);
> [ ... ]
>
> Bart.
I can reproduce this issue("kernel BUG at lib/sg_pool.c:103") without mp-bvec
patches,
so looks it isn't the fault of this patchset.
Thanks,
Ming
Re: [Cluster-devel] [dm-devel] [PATCH V15 00/18] block: support multi-page bvec
On Sun, Feb 17, 2019 at 09:13:32PM +0800, Ming Lei wrote:
> On Fri, Feb 15, 2019 at 10:59:47AM -0700, Jens Axboe wrote:
> > On 2/15/19 10:14 AM, Bart Van Assche wrote:
> > > On Fri, 2019-02-15 at 08:49 -0700, Jens Axboe wrote:
> > >> On 2/15/19 4:13 AM, Ming Lei wrote:
> > >>> This patchset brings multi-page bvec into block layer:
> > >>
> > >> Applied, thanks Ming. Let's hope it sticks!
> > >
> > > Hi Jens and Ming,
> > >
> > > Test nvmeof-mp/002 fails with Jens' for-next branch from this morning.
> > > I have not yet tried to figure out which patch introduced the failure.
> > > Anyway, this is what I see in the kernel log for test nvmeof-mp/002:
> > >
> > > [ 475.611363] BUG: unable to handle kernel NULL pointer dereference at
> > > 0020
> > > [ 475.621188] #PF error: [normal kernel read fault]
> > > [ 475.623148] PGD 0 P4D 0
> > > [ 475.624737] Oops: [#1] PREEMPT SMP KASAN
> > > [ 475.626628] CPU: 1 PID: 277 Comm: kworker/1:1H Tainted: GB
> > > 5.0.0-rc6-dbg+ #1
> > > [ 475.630232] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
> > > BIOS 1.10.2-1 04/01/2014
> > > [ 475.633855] Workqueue: kblockd blk_mq_requeue_work
> > > [ 475.635777] RIP: 0010:__blk_recalc_rq_segments+0xbe/0x590
> > > [ 475.670948] Call Trace:
> > > [ 475.693515] blk_recalc_rq_segments+0x2f/0x50
> > > [ 475.695081] blk_insert_cloned_request+0xbb/0x1c0
> > > [ 475.701142] dm_mq_queue_rq+0x3d1/0x770
> > > [ 475.707225] blk_mq_dispatch_rq_list+0x5fc/0xb10
> > > [ 475.717137] blk_mq_sched_dispatch_requests+0x256/0x300
> > > [ 475.721767] __blk_mq_run_hw_queue+0xd6/0x180
> > > [ 475.725920] __blk_mq_delay_run_hw_queue+0x25c/0x290
> > > [ 475.727480] blk_mq_run_hw_queue+0x119/0x1b0
> > > [ 475.732019] blk_mq_run_hw_queues+0x7b/0xa0
> > > [ 475.733468] blk_mq_requeue_work+0x2cb/0x300
> > > [ 475.736473] process_one_work+0x4f1/0xa40
> > > [ 475.739424] worker_thread+0x67/0x5b0
> > > [ 475.741751] kthread+0x1cf/0x1f0
> > > [ 475.746034] ret_from_fork+0x24/0x30
> > >
> > > (gdb) list *(__blk_recalc_rq_segments+0xbe)
> > > 0x816a152e is in __blk_recalc_rq_segments (block/blk-merge.c:366).
> > > 361 struct bio *bio)
> > > 362 {
> > > 363 struct bio_vec bv, bvprv = { NULL };
> > > 364 int prev = 0;
> > > 365 unsigned int seg_size, nr_phys_segs;
> > > 366 unsigned front_seg_size = bio->bi_seg_front_size;
> > > 367 struct bio *fbio, *bbio;
> > > 368 struct bvec_iter iter;
> > > 369
> > > 370 if (!bio)
> >
> > Just ran a few tests, and it also seems to cause about a 5% regression
> > in per-core IOPS throughput. Prior to this work, I could get 1620K 4k
> > rand read IOPS out of core, now I'm at ~1535K. The cycler stealer seems
> > to be blk_queue_split() and blk_rq_map_sg().
>
> Could you share us your test setting?
>
> I will run null_blk first and see if it can be reproduced.
Looks this performance drop isn't reproduced on null_blk with the following
setting by me:
- modprobe null_blk nr_devices=4 submit_queues=48
- test machine : dual socket, two NUMA nodes, 24cores/socket
- fio script:
fio --direct=1 --size=128G --bsrange=4k-4k --runtime=40 --numjobs=48
--ioengine=libaio --iodepth=64 --group_reporting=1 --filename=/dev/nullb0
--name=randread --rw=randread
result: 10.7M IOPS(base kernel), 10.6M IOPS(patched kernel)
And if 'bs' is increased to 256k, 512k, 1024k, IOPS improvement can be ~8%
with multi-page bvec patches in above test.
BTW, there isn't cost added to bio_for_each_bvec(), so blk_queue_split() and
blk_rq_map_sg() should be fine. However, bio_for_each_segment_all()
may not be quick as before.
Thanks,
Ming
Re: [Cluster-devel] [dm-devel] [PATCH V15 00/18] block: support multi-page bvec
On Fri, Feb 15, 2019 at 10:59:47AM -0700, Jens Axboe wrote:
> On 2/15/19 10:14 AM, Bart Van Assche wrote:
> > On Fri, 2019-02-15 at 08:49 -0700, Jens Axboe wrote:
> >> On 2/15/19 4:13 AM, Ming Lei wrote:
> >>> This patchset brings multi-page bvec into block layer:
> >>
> >> Applied, thanks Ming. Let's hope it sticks!
> >
> > Hi Jens and Ming,
> >
> > Test nvmeof-mp/002 fails with Jens' for-next branch from this morning.
> > I have not yet tried to figure out which patch introduced the failure.
> > Anyway, this is what I see in the kernel log for test nvmeof-mp/002:
> >
> > [ 475.611363] BUG: unable to handle kernel NULL pointer dereference at
> > 0020
> > [ 475.621188] #PF error: [normal kernel read fault]
> > [ 475.623148] PGD 0 P4D 0
> > [ 475.624737] Oops: [#1] PREEMPT SMP KASAN
> > [ 475.626628] CPU: 1 PID: 277 Comm: kworker/1:1H Tainted: GB
> > 5.0.0-rc6-dbg+ #1
> > [ 475.630232] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
> > 1.10.2-1 04/01/2014
> > [ 475.633855] Workqueue: kblockd blk_mq_requeue_work
> > [ 475.635777] RIP: 0010:__blk_recalc_rq_segments+0xbe/0x590
> > [ 475.670948] Call Trace:
> > [ 475.693515] blk_recalc_rq_segments+0x2f/0x50
> > [ 475.695081] blk_insert_cloned_request+0xbb/0x1c0
> > [ 475.701142] dm_mq_queue_rq+0x3d1/0x770
> > [ 475.707225] blk_mq_dispatch_rq_list+0x5fc/0xb10
> > [ 475.717137] blk_mq_sched_dispatch_requests+0x256/0x300
> > [ 475.721767] __blk_mq_run_hw_queue+0xd6/0x180
> > [ 475.725920] __blk_mq_delay_run_hw_queue+0x25c/0x290
> > [ 475.727480] blk_mq_run_hw_queue+0x119/0x1b0
> > [ 475.732019] blk_mq_run_hw_queues+0x7b/0xa0
> > [ 475.733468] blk_mq_requeue_work+0x2cb/0x300
> > [ 475.736473] process_one_work+0x4f1/0xa40
> > [ 475.739424] worker_thread+0x67/0x5b0
> > [ 475.741751] kthread+0x1cf/0x1f0
> > [ 475.746034] ret_from_fork+0x24/0x30
> >
> > (gdb) list *(__blk_recalc_rq_segments+0xbe)
> > 0x816a152e is in __blk_recalc_rq_segments (block/blk-merge.c:366).
> > 361 struct bio *bio)
> > 362 {
> > 363 struct bio_vec bv, bvprv = { NULL };
> > 364 int prev = 0;
> > 365 unsigned int seg_size, nr_phys_segs;
> > 366 unsigned front_seg_size = bio->bi_seg_front_size;
> > 367 struct bio *fbio, *bbio;
> > 368 struct bvec_iter iter;
> > 369
> > 370 if (!bio)
>
> Just ran a few tests, and it also seems to cause about a 5% regression
> in per-core IOPS throughput. Prior to this work, I could get 1620K 4k
> rand read IOPS out of core, now I'm at ~1535K. The cycler stealer seems
> to be blk_queue_split() and blk_rq_map_sg().
Could you share us your test setting?
I will run null_blk first and see if it can be reproduced.
Thanks,
Ming
Re: [Cluster-devel] [dm-devel] [PATCH V15 00/18] block: support multi-page bvec
On Fri, Feb 15, 2019 at 09:14:15AM -0800, Bart Van Assche wrote:
> On Fri, 2019-02-15 at 08:49 -0700, Jens Axboe wrote:
> > On 2/15/19 4:13 AM, Ming Lei wrote:
> > > This patchset brings multi-page bvec into block layer:
> >
> > Applied, thanks Ming. Let's hope it sticks!
>
> Hi Jens and Ming,
>
> Test nvmeof-mp/002 fails with Jens' for-next branch from this morning.
> I have not yet tried to figure out which patch introduced the failure.
> Anyway, this is what I see in the kernel log for test nvmeof-mp/002:
>
> [ 475.611363] BUG: unable to handle kernel NULL pointer dereference at
> 0020
> [ 475.621188] #PF error: [normal kernel read fault]
> [ 475.623148] PGD 0 P4D 0
> [ 475.624737] Oops: [#1] PREEMPT SMP KASAN
> [ 475.626628] CPU: 1 PID: 277 Comm: kworker/1:1H Tainted: GB
> 5.0.0-rc6-dbg+ #1
> [ 475.630232] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
> 1.10.2-1 04/01/2014
> [ 475.633855] Workqueue: kblockd blk_mq_requeue_work
> [ 475.635777] RIP: 0010:__blk_recalc_rq_segments+0xbe/0x590
> [ 475.670948] Call Trace:
> [ 475.693515] blk_recalc_rq_segments+0x2f/0x50
> [ 475.695081] blk_insert_cloned_request+0xbb/0x1c0
> [ 475.701142] dm_mq_queue_rq+0x3d1/0x770
> [ 475.707225] blk_mq_dispatch_rq_list+0x5fc/0xb10
> [ 475.717137] blk_mq_sched_dispatch_requests+0x256/0x300
> [ 475.721767] __blk_mq_run_hw_queue+0xd6/0x180
> [ 475.725920] __blk_mq_delay_run_hw_queue+0x25c/0x290
> [ 475.727480] blk_mq_run_hw_queue+0x119/0x1b0
> [ 475.732019] blk_mq_run_hw_queues+0x7b/0xa0
> [ 475.733468] blk_mq_requeue_work+0x2cb/0x300
> [ 475.736473] process_one_work+0x4f1/0xa40
> [ 475.739424] worker_thread+0x67/0x5b0
> [ 475.741751] kthread+0x1cf/0x1f0
> [ 475.746034] ret_from_fork+0x24/0x30
>
> (gdb) list *(__blk_recalc_rq_segments+0xbe)
> 0x816a152e is in __blk_recalc_rq_segments (block/blk-merge.c:366).
> 361 struct bio *bio)
> 362 {
> 363 struct bio_vec bv, bvprv = { NULL };
> 364 int prev = 0;
> 365 unsigned int seg_size, nr_phys_segs;
> 366 unsigned front_seg_size = bio->bi_seg_front_size;
> 367 struct bio *fbio, *bbio;
> 368 struct bvec_iter iter;
> 369
> 370 if (!bio)
>
> Bart.
Thanks for your test!
The following patch should fix this issue:
diff --git a/block/blk-merge.c b/block/blk-merge.c
index bed065904677..066b66430523 100644
--- a/block/blk-merge.c
+++ b/block/blk-merge.c
@@ -363,13 +363,15 @@ static unsigned int __blk_recalc_rq_segments(struct
request_queue *q,
struct bio_vec bv, bvprv = { NULL };
int prev = 0;
unsigned int seg_size, nr_phys_segs;
- unsigned front_seg_size = bio->bi_seg_front_size;
+ unsigned front_seg_size;
struct bio *fbio, *bbio;
struct bvec_iter iter;
if (!bio)
return 0;
+ front_seg_size = bio->bi_seg_front_size;
+
switch (bio_op(bio)) {
case REQ_OP_DISCARD:
case REQ_OP_SECURE_ERASE:
Thanks,
Ming
Re: [Cluster-devel] [dm-devel] [PATCH V15 00/18] block: support multi-page bvec
On 2/15/19 10:14 AM, Bart Van Assche wrote:
> On Fri, 2019-02-15 at 08:49 -0700, Jens Axboe wrote:
>> On 2/15/19 4:13 AM, Ming Lei wrote:
>>> This patchset brings multi-page bvec into block layer:
>>
>> Applied, thanks Ming. Let's hope it sticks!
>
> Hi Jens and Ming,
>
> Test nvmeof-mp/002 fails with Jens' for-next branch from this morning.
> I have not yet tried to figure out which patch introduced the failure.
> Anyway, this is what I see in the kernel log for test nvmeof-mp/002:
>
> [ 475.611363] BUG: unable to handle kernel NULL pointer dereference at
> 0020
> [ 475.621188] #PF error: [normal kernel read fault]
> [ 475.623148] PGD 0 P4D 0
> [ 475.624737] Oops: [#1] PREEMPT SMP KASAN
> [ 475.626628] CPU: 1 PID: 277 Comm: kworker/1:1H Tainted: GB
> 5.0.0-rc6-dbg+ #1
> [ 475.630232] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
> 1.10.2-1 04/01/2014
> [ 475.633855] Workqueue: kblockd blk_mq_requeue_work
> [ 475.635777] RIP: 0010:__blk_recalc_rq_segments+0xbe/0x590
> [ 475.670948] Call Trace:
> [ 475.693515] blk_recalc_rq_segments+0x2f/0x50
> [ 475.695081] blk_insert_cloned_request+0xbb/0x1c0
> [ 475.701142] dm_mq_queue_rq+0x3d1/0x770
> [ 475.707225] blk_mq_dispatch_rq_list+0x5fc/0xb10
> [ 475.717137] blk_mq_sched_dispatch_requests+0x256/0x300
> [ 475.721767] __blk_mq_run_hw_queue+0xd6/0x180
> [ 475.725920] __blk_mq_delay_run_hw_queue+0x25c/0x290
> [ 475.727480] blk_mq_run_hw_queue+0x119/0x1b0
> [ 475.732019] blk_mq_run_hw_queues+0x7b/0xa0
> [ 475.733468] blk_mq_requeue_work+0x2cb/0x300
> [ 475.736473] process_one_work+0x4f1/0xa40
> [ 475.739424] worker_thread+0x67/0x5b0
> [ 475.741751] kthread+0x1cf/0x1f0
> [ 475.746034] ret_from_fork+0x24/0x30
>
> (gdb) list *(__blk_recalc_rq_segments+0xbe)
> 0x816a152e is in __blk_recalc_rq_segments (block/blk-merge.c:366).
> 361 struct bio *bio)
> 362 {
> 363 struct bio_vec bv, bvprv = { NULL };
> 364 int prev = 0;
> 365 unsigned int seg_size, nr_phys_segs;
> 366 unsigned front_seg_size = bio->bi_seg_front_size;
> 367 struct bio *fbio, *bbio;
> 368 struct bvec_iter iter;
> 369
> 370 if (!bio)
Just ran a few tests, and it also seems to cause about a 5% regression
in per-core IOPS throughput. Prior to this work, I could get 1620K 4k
rand read IOPS out of core, now I'm at ~1535K. The cycler stealer seems
to be blk_queue_split() and blk_rq_map_sg().
--
Jens Axboe
Re: [Cluster-devel] [dm-devel] [PATCH V15 00/18] block: support multi-page bvec
On Fri, 2019-02-15 at 08:49 -0700, Jens Axboe wrote:
> On 2/15/19 4:13 AM, Ming Lei wrote:
> > This patchset brings multi-page bvec into block layer:
>
> Applied, thanks Ming. Let's hope it sticks!
Hi Jens and Ming,
Test nvmeof-mp/002 fails with Jens' for-next branch from this morning.
I have not yet tried to figure out which patch introduced the failure.
Anyway, this is what I see in the kernel log for test nvmeof-mp/002:
[ 475.611363] BUG: unable to handle kernel NULL pointer dereference at
0020
[ 475.621188] #PF error: [normal kernel read fault]
[ 475.623148] PGD 0 P4D 0
[ 475.624737] Oops: [#1] PREEMPT SMP KASAN
[ 475.626628] CPU: 1 PID: 277 Comm: kworker/1:1H Tainted: GB
5.0.0-rc6-dbg+ #1
[ 475.630232] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
1.10.2-1 04/01/2014
[ 475.633855] Workqueue: kblockd blk_mq_requeue_work
[ 475.635777] RIP: 0010:__blk_recalc_rq_segments+0xbe/0x590
[ 475.670948] Call Trace:
[ 475.693515] blk_recalc_rq_segments+0x2f/0x50
[ 475.695081] blk_insert_cloned_request+0xbb/0x1c0
[ 475.701142] dm_mq_queue_rq+0x3d1/0x770
[ 475.707225] blk_mq_dispatch_rq_list+0x5fc/0xb10
[ 475.717137] blk_mq_sched_dispatch_requests+0x256/0x300
[ 475.721767] __blk_mq_run_hw_queue+0xd6/0x180
[ 475.725920] __blk_mq_delay_run_hw_queue+0x25c/0x290
[ 475.727480] blk_mq_run_hw_queue+0x119/0x1b0
[ 475.732019] blk_mq_run_hw_queues+0x7b/0xa0
[ 475.733468] blk_mq_requeue_work+0x2cb/0x300
[ 475.736473] process_one_work+0x4f1/0xa40
[ 475.739424] worker_thread+0x67/0x5b0
[ 475.741751] kthread+0x1cf/0x1f0
[ 475.746034] ret_from_fork+0x24/0x30
(gdb) list *(__blk_recalc_rq_segments+0xbe)
0x816a152e is in __blk_recalc_rq_segments (block/blk-merge.c:366).
361 struct bio *bio)
362 {
363 struct bio_vec bv, bvprv = { NULL };
364 int prev = 0;
365 unsigned int seg_size, nr_phys_segs;
366 unsigned front_seg_size = bio->bi_seg_front_size;
367 struct bio *fbio, *bbio;
368 struct bvec_iter iter;
369
370 if (!bio)
Bart.
