Re: [PATCH v5] conversion to blk-mq
On 06/04/2014 08:52 PM, Keith Busch wrote: > On Wed, 4 Jun 2014, Jens Axboe wrote: >> On 06/04/2014 12:28 PM, Keith Busch wrote: >> Are you testing against 3.13? You really need the current tree for this, >> otherwise I'm sure you'll run into issues (as you appear to be :-) > > I'm using Matias' current tree: > > git://github.com/MatiasBjorling/linux-collab nvmeq_wip_6 > > Is this the right one to use? Looks like a 3.15rc1+. > > Also ... obviously that's the wrong diff from before. I didn't save my > file before creating the diff. Should have looked like this: > > diff --git a/block/blk-mq.c b/block/blk-mq.c > index d10013b..d1b986c 100644 > --- a/block/blk-mq.c > +++ b/block/blk-mq.c > @@ -105,7 +105,8 @@ static int blk_mq_queue_enter(struct request_queue *q) > __percpu_counter_add(>mq_usage_counter, 1, 100); > smp_wmb(); > /* we have problems to freeze the queue if it's initializing */ > -if (!blk_queue_bypass(q) || !blk_queue_init_done(q)) > +if (!blk_queue_dying(q) && (!blk_queue_bypass(q) || > +!blk_queue_init_done(q))) > return 0; > > __percpu_counter_add(>mq_usage_counter, -1, 100); Thanks! I've put your patch into the nvmemq_wip_v6 branch and rebased it on top of Jens' for-linus. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v5] conversion to blk-mq
On 06/04/2014 12:52 PM, Keith Busch wrote: > On Wed, 4 Jun 2014, Jens Axboe wrote: >> On 06/04/2014 12:28 PM, Keith Busch wrote: >> Are you testing against 3.13? You really need the current tree for this, >> otherwise I'm sure you'll run into issues (as you appear to be :-) > > I'm using Matias' current tree: > > git://github.com/MatiasBjorling/linux-collab nvmeq_wip_6 > > Is this the right one to use? Looks like a 3.15rc1+. It is, I misread that diff wrong. > Also ... obviously that's the wrong diff from before. I didn't save my > file before creating the diff. Should have looked like this: > > diff --git a/block/blk-mq.c b/block/blk-mq.c > index d10013b..d1b986c 100644 > --- a/block/blk-mq.c > +++ b/block/blk-mq.c > @@ -105,7 +105,8 @@ static int blk_mq_queue_enter(struct request_queue *q) > __percpu_counter_add(>mq_usage_counter, 1, 100); > smp_wmb(); > /* we have problems to freeze the queue if it's initializing */ > -if (!blk_queue_bypass(q) || !blk_queue_init_done(q)) > +if (!blk_queue_dying(q) && (!blk_queue_bypass(q) || > +!blk_queue_init_done(q))) > return 0; > > __percpu_counter_add(>mq_usage_counter, -1, 100); This looks better. -- Jens Axboe -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v5] conversion to blk-mq
On Wed, 4 Jun 2014, Jens Axboe wrote: On 06/04/2014 12:28 PM, Keith Busch wrote: Are you testing against 3.13? You really need the current tree for this, otherwise I'm sure you'll run into issues (as you appear to be :-) I'm using Matias' current tree: git://github.com/MatiasBjorling/linux-collab nvmeq_wip_6 Is this the right one to use? Looks like a 3.15rc1+. Also ... obviously that's the wrong diff from before. I didn't save my file before creating the diff. Should have looked like this: diff --git a/block/blk-mq.c b/block/blk-mq.c index d10013b..d1b986c 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -105,7 +105,8 @@ static int blk_mq_queue_enter(struct request_queue *q) __percpu_counter_add(>mq_usage_counter, 1, 100); smp_wmb(); /* we have problems to freeze the queue if it's initializing */ - if (!blk_queue_bypass(q) || !blk_queue_init_done(q)) + if (!blk_queue_dying(q) && (!blk_queue_bypass(q) || + !blk_queue_init_done(q))) return 0; __percpu_counter_add(>mq_usage_counter, -1, 100); -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v5] conversion to blk-mq
On 06/04/2014 12:28 PM, Keith Busch wrote:
> On Wed, 4 Jun 2014, Matias Bjørling wrote:
>> On 06/04/2014 12:27 AM, Keith Busch wrote:
On Tue, 3 Jun 2014, Matias Bjorling wrote:
>
> Keith, will you take the nvmemq_wip_v6 branch for a spin? Thanks!
>>>
>>> BTW, if you want to test this out yourself, it's pretty simple to
>>> recreate. I just run a simple user admin program sending nvme
>>> passthrough
>>> commands in a tight loop, then run:
>>>
>>> # echo 1 > /sys/bus/pci/devices//remove
>>
>> I can't recreate- I use the nvme_get_feature program to continuously
>> hit the ioctl path, testing using your nvme qemu branch.
>
> Okay, I'll try to fix it.
>
> I think there are multiple problems, but the first is that since there
> is no gendisk associated with the admin_q, the QUEUE_FLAG_INIT_DONE flag
> is never set, and blk_mq_queue_enter returns successful whenever this
> flag is not set even though this queue is dying, so we enter with all
> its invalid pointers.
>
> Here's a couple diff's. The first fixes the kernel oops by not entering a
> dying queue. The second is just a few unrelated clean-ups in nvme-core.c.
>
> I still can't complete my current hot-removal test, though; something
> appears hung, but haven't nailed that down yet.
>
> Please let me know what you think! Thanks.
>
> diff --git a/block/blk-mq.c b/block/blk-mq.c
> index d10013b..5a9ae8a 100644
> --- a/block/blk-mq.c
> +++ b/block/blk-mq.c
> @@ -105,6 +105,10 @@ static int blk_mq_queue_enter(struct request_queue *q)
> __percpu_counter_add(>mq_usage_counter, 1, 100);
> smp_wmb();
> /* we have problems to freeze the queue if it's initializing */
> +if (blk_queue_dying(q)) {
> +__percpu_counter_add(>mq_usage_counter, -1, 100);
> +ret = -ENODEV;
> +}
> if (!blk_queue_bypass(q) || !blk_queue_init_done(q))
> return 0;
Are you testing against 3.13? You really need the current tree for this,
otherwise I'm sure you'll run into issues (as you appear to be :-)
--
Jens Axboe
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v5] conversion to blk-mq
On Wed, 4 Jun 2014, Matias Bjørling wrote:
On 06/04/2014 12:27 AM, Keith Busch wrote:
On Tue, 3 Jun 2014, Matias Bjorling wrote:
Keith, will you take the nvmemq_wip_v6 branch for a spin? Thanks!
BTW, if you want to test this out yourself, it's pretty simple to
recreate. I just run a simple user admin program sending nvme passthrough
commands in a tight loop, then run:
# echo 1 > /sys/bus/pci/devices//remove
I can't recreate- I use the nvme_get_feature program to continuously hit the
ioctl path, testing using your nvme qemu branch.
Okay, I'll try to fix it.
I think there are multiple problems, but the first is that since there
is no gendisk associated with the admin_q, the QUEUE_FLAG_INIT_DONE flag
is never set, and blk_mq_queue_enter returns successful whenever this
flag is not set even though this queue is dying, so we enter with all
its invalid pointers.
Here's a couple diff's. The first fixes the kernel oops by not entering a
dying queue. The second is just a few unrelated clean-ups in nvme-core.c.
I still can't complete my current hot-removal test, though; something
appears hung, but haven't nailed that down yet.
Please let me know what you think! Thanks.
diff --git a/block/blk-mq.c b/block/blk-mq.c
index d10013b..5a9ae8a 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -105,6 +105,10 @@ static int blk_mq_queue_enter(struct request_queue *q)
__percpu_counter_add(>mq_usage_counter, 1, 100);
smp_wmb();
/* we have problems to freeze the queue if it's initializing */
+ if (blk_queue_dying(q)) {
+ __percpu_counter_add(>mq_usage_counter, -1, 100);
+ ret = -ENODEV;
+ }
if (!blk_queue_bypass(q) || !blk_queue_init_done(q))
return 0;
diff --git a/drivers/block/nvme-core.c b/drivers/block/nvme-core.c
index 243a5e6..22e9c82 100644
--- a/drivers/block/nvme-core.c
+++ b/drivers/block/nvme-core.c
@@ -98,7 +98,6 @@ struct nvme_queue {
u8 cq_phase;
u8 cqe_seen;
u8 q_suspended;
- cpumask_var_t cpu_mask;
struct async_cmd_info cmdinfo;
struct blk_mq_hw_ctx *hctx;
};
@@ -1055,8 +1054,6 @@ static void nvme_free_queue(struct nvme_queue *nvmeq)
(void *)nvmeq->cqes, nvmeq->cq_dma_addr);
dma_free_coherent(nvmeq->q_dmadev, SQ_SIZE(nvmeq->q_depth),
nvmeq->sq_cmds, nvmeq->sq_dma_addr);
- if (nvmeq->qid)
- free_cpumask_var(nvmeq->cpu_mask);
kfree(nvmeq);
}
@@ -1066,9 +1063,9 @@ static void nvme_free_queues(struct nvme_dev *dev, int
lowest)
for (i = dev->queue_count - 1; i >= lowest; i--) {
struct nvme_queue *nvmeq = dev->queues[i];
- nvme_free_queue(nvmeq);
dev->queue_count--;
dev->queues[i] = NULL;
+ nvme_free_queue(nvmeq);
}
}
@@ -1142,9 +1139,6 @@ static struct nvme_queue *nvme_alloc_queue(struct
nvme_dev *dev, int qid,
if (!nvmeq->sq_cmds)
goto free_cqdma;
- if (qid && !zalloc_cpumask_var(>cpu_mask, GFP_KERNEL))
- goto free_sqdma;
-
nvmeq->q_dmadev = dmadev;
nvmeq->dev = dev;
snprintf(nvmeq->irqname, sizeof(nvmeq->irqname), "nvme%dq%d",
@@ -1162,9 +1156,6 @@ static struct nvme_queue *nvme_alloc_queue(struct
nvme_dev *dev, int qid,
return nvmeq;
- free_sqdma:
- dma_free_coherent(dmadev, SQ_SIZE(depth), (void *)nvmeq->sq_cmds,
- nvmeq->sq_dma_addr);
free_cqdma:
dma_free_coherent(dmadev, CQ_SIZE(depth), (void *)nvmeq->cqes,
nvmeq->cq_dma_addr);
Re: [PATCH v5] conversion to blk-mq
On 06/04/2014 12:27 AM, Keith Busch wrote: On Tue, 3 Jun 2014, Matias Bjorling wrote: Keith, will you take the nvmemq_wip_v6 branch for a spin? Thanks! BTW, if you want to test this out yourself, it's pretty simple to recreate. I just run a simple user admin program sending nvme passthrough commands in a tight loop, then run: # echo 1 > /sys/bus/pci/devices//remove I can't recreate- I use the nvme_get_feature program to continuously hit the ioctl path, testing using your nvme qemu branch. Still fails as before: [ 88.933881] BUG: unable to handle kernel NULL pointer dereference at 0014 [ 88.942900] IP: [] blk_mq_map_queue+0xf/0x1e [ 88.949605] PGD 427be0067 PUD 425495067 PMD 0 [ 88.954915] Oops: [#1] SMP [ 88.958787] Modules linked in: nvme parport_pc ppdev lp parport dlm sctp libcrc32c configfs nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc md4 hmac cifs bridge stp llc joydev jfs hid_generic usbhid hid loop md_mod x86_pkg_temp_thermal coretemp kvm_intel kvm iTCO_wdt iTCO_vendor_support crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd microcode ehci_pci ehci_hcd pcspkr usbcore lpc_ich ioatdma usb_common mfd_core evdev i2c_i801 wmi acpi_cpufreq tpm_tis tpm ipmi_si ipmi_msghandler processor thermal_sys button ext4 crc16 jbd2 mbcache dm_mod nbd sg sd_mod sr_mod crc_t10dif cdrom crct10dif_common isci ahci libsas igb libahci scsi_transport_sas ptp pps_core i2c_algo_bit libata i2c_core scsi_mod dca [ 89.042529] CPU: 5 PID: 4544 Comm: nvme_id_ctrl Not tainted 3.15.0-rc1+ #3 [ 89.050295] Hardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013 [ 89.061856] task: 88042bbdb0d0 ti: 88042c24c000 task.ti: 88042c24c000 [ 89.070305] RIP: 0010:[] [] blk_mq_map_queue+0xf/0x1e [ 89.079747] RSP: 0018:88042c24dda0 EFLAGS: 00010202 [ 89.085795] RAX: RBX: e8fbffaa1b00 RCX: 88042e8ec4b0 [ 89.093868] RDX: 8be6 RSI: 0005 RDI: 88042abdf048 [ 89.101950] RBP: 88042c2b81c0 R08: 88042c24c000 R09: 880035c58410 [ 89.110033] R10: 88043f6b2dc0 R11: 88043f6b2dc0 R12: 88042c24de94 [ 89.118119] R13: 007d R14: 7fff0cd892b0 R15: [ 89.126210] FS: 7f39866c5700() GS:88043f6a() knlGS: [ 89.135387] CS: 0010 DS: ES: CR0: 80050033 [ 89.141916] CR2: 0014 CR3: 00042b387000 CR4: 000407e0 [ 89.149997] Stack: [ 89.152353] 811c6334 fffc 88042c2b81c0 88042c24de10 [ 89.161096] a054dcbb 0246 fffc 8800b5e05cc0 [ 89.169839] fff4 8800b5e05cc0 88042bbc3000 1000 [ 89.178583] Call Trace: [ 89.181429] [] ? blk_mq_free_request+0x37/0x48 [ 89.188360] [] ? __nvme_submit_admin_cmd+0x52/0x68 [nvme] [ 89.196349] [] ? nvme_user_admin_cmd+0x144/0x1b1 [nvme] [ 89.204150] [] ? nvme_dev_ioctl+0x1d/0x2b [nvme] [ 89.211278] [] ? do_vfs_ioctl+0x3f2/0x43b [ 89.217710] [] ? vfs_write+0xde/0xfc [ 89.223657] [] ? SyS_ioctl+0x4e/0x7d [ 89.229622] [] ? system_call_fastpath+0x16/0x1b [ 89.236636] Code: 8b 4a 38 48 39 4e 38 72 12 74 06 b8 01 00 00 00 c3 48 8b 4a 60 48 39 4e 60 73 f0 c3 66 66 66 66 90 48 8b 87 e0 00 00 00 48 63 f6 <8b> 14 b0 48 8b 87 f8 00 00 00 48 8b 04 d0 c3 89 ff f0 48 0f ab [ 89.263435] RIP [] blk_mq_map_queue+0xf/0x1e [ 89.270237] RSP [ 89.274237] CR2: 0014 [ 89.278095] ---[ end trace 54c0e8cbb1fe2ec3 ]--- -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v5] conversion to blk-mq
On 06/04/2014 12:27 AM, Keith Busch wrote: On Tue, 3 Jun 2014, Matias Bjorling wrote: Keith, will you take the nvmemq_wip_v6 branch for a spin? Thanks! BTW, if you want to test this out yourself, it's pretty simple to recreate. I just run a simple user admin program sending nvme passthrough commands in a tight loop, then run: # echo 1 /sys/bus/pci/devices/bdf/remove I can't recreate- I use the nvme_get_feature program to continuously hit the ioctl path, testing using your nvme qemu branch. Still fails as before: [ 88.933881] BUG: unable to handle kernel NULL pointer dereference at 0014 [ 88.942900] IP: [811c51b8] blk_mq_map_queue+0xf/0x1e [ 88.949605] PGD 427be0067 PUD 425495067 PMD 0 [ 88.954915] Oops: [#1] SMP [ 88.958787] Modules linked in: nvme parport_pc ppdev lp parport dlm sctp libcrc32c configfs nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc md4 hmac cifs bridge stp llc joydev jfs hid_generic usbhid hid loop md_mod x86_pkg_temp_thermal coretemp kvm_intel kvm iTCO_wdt iTCO_vendor_support crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd microcode ehci_pci ehci_hcd pcspkr usbcore lpc_ich ioatdma usb_common mfd_core evdev i2c_i801 wmi acpi_cpufreq tpm_tis tpm ipmi_si ipmi_msghandler processor thermal_sys button ext4 crc16 jbd2 mbcache dm_mod nbd sg sd_mod sr_mod crc_t10dif cdrom crct10dif_common isci ahci libsas igb libahci scsi_transport_sas ptp pps_core i2c_algo_bit libata i2c_core scsi_mod dca [ 89.042529] CPU: 5 PID: 4544 Comm: nvme_id_ctrl Not tainted 3.15.0-rc1+ #3 [ 89.050295] Hardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013 [ 89.061856] task: 88042bbdb0d0 ti: 88042c24c000 task.ti: 88042c24c000 [ 89.070305] RIP: 0010:[811c51b8] [811c51b8] blk_mq_map_queue+0xf/0x1e [ 89.079747] RSP: 0018:88042c24dda0 EFLAGS: 00010202 [ 89.085795] RAX: RBX: e8fbffaa1b00 RCX: 88042e8ec4b0 [ 89.093868] RDX: 8be6 RSI: 0005 RDI: 88042abdf048 [ 89.101950] RBP: 88042c2b81c0 R08: 88042c24c000 R09: 880035c58410 [ 89.110033] R10: 88043f6b2dc0 R11: 88043f6b2dc0 R12: 88042c24de94 [ 89.118119] R13: 007d R14: 7fff0cd892b0 R15: [ 89.126210] FS: 7f39866c5700() GS:88043f6a() knlGS: [ 89.135387] CS: 0010 DS: ES: CR0: 80050033 [ 89.141916] CR2: 0014 CR3: 00042b387000 CR4: 000407e0 [ 89.149997] Stack: [ 89.152353] 811c6334 fffc 88042c2b81c0 88042c24de10 [ 89.161096] a054dcbb 0246 fffc 8800b5e05cc0 [ 89.169839] fff4 8800b5e05cc0 88042bbc3000 1000 [ 89.178583] Call Trace: [ 89.181429] [811c6334] ? blk_mq_free_request+0x37/0x48 [ 89.188360] [a054dcbb] ? __nvme_submit_admin_cmd+0x52/0x68 [nvme] [ 89.196349] [a054f761] ? nvme_user_admin_cmd+0x144/0x1b1 [nvme] [ 89.204150] [a054f7eb] ? nvme_dev_ioctl+0x1d/0x2b [nvme] [ 89.211278] [81125916] ? do_vfs_ioctl+0x3f2/0x43b [ 89.217710] [81117e35] ? vfs_write+0xde/0xfc [ 89.223657] [811259ad] ? SyS_ioctl+0x4e/0x7d [ 89.229622] [8139c6d2] ? system_call_fastpath+0x16/0x1b [ 89.236636] Code: 8b 4a 38 48 39 4e 38 72 12 74 06 b8 01 00 00 00 c3 48 8b 4a 60 48 39 4e 60 73 f0 c3 66 66 66 66 90 48 8b 87 e0 00 00 00 48 63 f6 8b 14 b0 48 8b 87 f8 00 00 00 48 8b 04 d0 c3 89 ff f0 48 0f ab [ 89.263435] RIP [811c51b8] blk_mq_map_queue+0xf/0x1e [ 89.270237] RSP 88042c24dda0 [ 89.274237] CR2: 0014 [ 89.278095] ---[ end trace 54c0e8cbb1fe2ec3 ]--- -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v5] conversion to blk-mq
On Wed, 4 Jun 2014, Matias Bjørling wrote:
On 06/04/2014 12:27 AM, Keith Busch wrote:
On Tue, 3 Jun 2014, Matias Bjorling wrote:
Keith, will you take the nvmemq_wip_v6 branch for a spin? Thanks!
BTW, if you want to test this out yourself, it's pretty simple to
recreate. I just run a simple user admin program sending nvme passthrough
commands in a tight loop, then run:
# echo 1 /sys/bus/pci/devices/bdf/remove
I can't recreate- I use the nvme_get_feature program to continuously hit the
ioctl path, testing using your nvme qemu branch.
Okay, I'll try to fix it.
I think there are multiple problems, but the first is that since there
is no gendisk associated with the admin_q, the QUEUE_FLAG_INIT_DONE flag
is never set, and blk_mq_queue_enter returns successful whenever this
flag is not set even though this queue is dying, so we enter with all
its invalid pointers.
Here's a couple diff's. The first fixes the kernel oops by not entering a
dying queue. The second is just a few unrelated clean-ups in nvme-core.c.
I still can't complete my current hot-removal test, though; something
appears hung, but haven't nailed that down yet.
Please let me know what you think! Thanks.
diff --git a/block/blk-mq.c b/block/blk-mq.c
index d10013b..5a9ae8a 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -105,6 +105,10 @@ static int blk_mq_queue_enter(struct request_queue *q)
__percpu_counter_add(q-mq_usage_counter, 1, 100);
smp_wmb();
/* we have problems to freeze the queue if it's initializing */
+ if (blk_queue_dying(q)) {
+ __percpu_counter_add(q-mq_usage_counter, -1, 100);
+ ret = -ENODEV;
+ }
if (!blk_queue_bypass(q) || !blk_queue_init_done(q))
return 0;
diff --git a/drivers/block/nvme-core.c b/drivers/block/nvme-core.c
index 243a5e6..22e9c82 100644
--- a/drivers/block/nvme-core.c
+++ b/drivers/block/nvme-core.c
@@ -98,7 +98,6 @@ struct nvme_queue {
u8 cq_phase;
u8 cqe_seen;
u8 q_suspended;
- cpumask_var_t cpu_mask;
struct async_cmd_info cmdinfo;
struct blk_mq_hw_ctx *hctx;
};
@@ -1055,8 +1054,6 @@ static void nvme_free_queue(struct nvme_queue *nvmeq)
(void *)nvmeq-cqes, nvmeq-cq_dma_addr);
dma_free_coherent(nvmeq-q_dmadev, SQ_SIZE(nvmeq-q_depth),
nvmeq-sq_cmds, nvmeq-sq_dma_addr);
- if (nvmeq-qid)
- free_cpumask_var(nvmeq-cpu_mask);
kfree(nvmeq);
}
@@ -1066,9 +1063,9 @@ static void nvme_free_queues(struct nvme_dev *dev, int
lowest)
for (i = dev-queue_count - 1; i = lowest; i--) {
struct nvme_queue *nvmeq = dev-queues[i];
- nvme_free_queue(nvmeq);
dev-queue_count--;
dev-queues[i] = NULL;
+ nvme_free_queue(nvmeq);
}
}
@@ -1142,9 +1139,6 @@ static struct nvme_queue *nvme_alloc_queue(struct
nvme_dev *dev, int qid,
if (!nvmeq-sq_cmds)
goto free_cqdma;
- if (qid !zalloc_cpumask_var(nvmeq-cpu_mask, GFP_KERNEL))
- goto free_sqdma;
-
nvmeq-q_dmadev = dmadev;
nvmeq-dev = dev;
snprintf(nvmeq-irqname, sizeof(nvmeq-irqname), nvme%dq%d,
@@ -1162,9 +1156,6 @@ static struct nvme_queue *nvme_alloc_queue(struct
nvme_dev *dev, int qid,
return nvmeq;
- free_sqdma:
- dma_free_coherent(dmadev, SQ_SIZE(depth), (void *)nvmeq-sq_cmds,
- nvmeq-sq_dma_addr);
free_cqdma:
dma_free_coherent(dmadev, CQ_SIZE(depth), (void *)nvmeq-cqes,
nvmeq-cq_dma_addr);
Re: [PATCH v5] conversion to blk-mq
On 06/04/2014 12:28 PM, Keith Busch wrote:
On Wed, 4 Jun 2014, Matias Bjørling wrote:
On 06/04/2014 12:27 AM, Keith Busch wrote:
On Tue, 3 Jun 2014, Matias Bjorling wrote:
Keith, will you take the nvmemq_wip_v6 branch for a spin? Thanks!
BTW, if you want to test this out yourself, it's pretty simple to
recreate. I just run a simple user admin program sending nvme
passthrough
commands in a tight loop, then run:
# echo 1 /sys/bus/pci/devices/bdf/remove
I can't recreate- I use the nvme_get_feature program to continuously
hit the ioctl path, testing using your nvme qemu branch.
Okay, I'll try to fix it.
I think there are multiple problems, but the first is that since there
is no gendisk associated with the admin_q, the QUEUE_FLAG_INIT_DONE flag
is never set, and blk_mq_queue_enter returns successful whenever this
flag is not set even though this queue is dying, so we enter with all
its invalid pointers.
Here's a couple diff's. The first fixes the kernel oops by not entering a
dying queue. The second is just a few unrelated clean-ups in nvme-core.c.
I still can't complete my current hot-removal test, though; something
appears hung, but haven't nailed that down yet.
Please let me know what you think! Thanks.
diff --git a/block/blk-mq.c b/block/blk-mq.c
index d10013b..5a9ae8a 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -105,6 +105,10 @@ static int blk_mq_queue_enter(struct request_queue *q)
__percpu_counter_add(q-mq_usage_counter, 1, 100);
smp_wmb();
/* we have problems to freeze the queue if it's initializing */
+if (blk_queue_dying(q)) {
+__percpu_counter_add(q-mq_usage_counter, -1, 100);
+ret = -ENODEV;
+}
if (!blk_queue_bypass(q) || !blk_queue_init_done(q))
return 0;
Are you testing against 3.13? You really need the current tree for this,
otherwise I'm sure you'll run into issues (as you appear to be :-)
--
Jens Axboe
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v5] conversion to blk-mq
On Wed, 4 Jun 2014, Jens Axboe wrote: On 06/04/2014 12:28 PM, Keith Busch wrote: Are you testing against 3.13? You really need the current tree for this, otherwise I'm sure you'll run into issues (as you appear to be :-) I'm using Matias' current tree: git://github.com/MatiasBjorling/linux-collab nvmeq_wip_6 Is this the right one to use? Looks like a 3.15rc1+. Also ... obviously that's the wrong diff from before. I didn't save my file before creating the diff. Should have looked like this: diff --git a/block/blk-mq.c b/block/blk-mq.c index d10013b..d1b986c 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -105,7 +105,8 @@ static int blk_mq_queue_enter(struct request_queue *q) __percpu_counter_add(q-mq_usage_counter, 1, 100); smp_wmb(); /* we have problems to freeze the queue if it's initializing */ - if (!blk_queue_bypass(q) || !blk_queue_init_done(q)) + if (!blk_queue_dying(q) (!blk_queue_bypass(q) || + !blk_queue_init_done(q))) return 0; __percpu_counter_add(q-mq_usage_counter, -1, 100); -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v5] conversion to blk-mq
On 06/04/2014 12:52 PM, Keith Busch wrote: On Wed, 4 Jun 2014, Jens Axboe wrote: On 06/04/2014 12:28 PM, Keith Busch wrote: Are you testing against 3.13? You really need the current tree for this, otherwise I'm sure you'll run into issues (as you appear to be :-) I'm using Matias' current tree: git://github.com/MatiasBjorling/linux-collab nvmeq_wip_6 Is this the right one to use? Looks like a 3.15rc1+. It is, I misread that diff wrong. Also ... obviously that's the wrong diff from before. I didn't save my file before creating the diff. Should have looked like this: diff --git a/block/blk-mq.c b/block/blk-mq.c index d10013b..d1b986c 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -105,7 +105,8 @@ static int blk_mq_queue_enter(struct request_queue *q) __percpu_counter_add(q-mq_usage_counter, 1, 100); smp_wmb(); /* we have problems to freeze the queue if it's initializing */ -if (!blk_queue_bypass(q) || !blk_queue_init_done(q)) +if (!blk_queue_dying(q) (!blk_queue_bypass(q) || +!blk_queue_init_done(q))) return 0; __percpu_counter_add(q-mq_usage_counter, -1, 100); This looks better. -- Jens Axboe -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v5] conversion to blk-mq
On 06/04/2014 08:52 PM, Keith Busch wrote: On Wed, 4 Jun 2014, Jens Axboe wrote: On 06/04/2014 12:28 PM, Keith Busch wrote: Are you testing against 3.13? You really need the current tree for this, otherwise I'm sure you'll run into issues (as you appear to be :-) I'm using Matias' current tree: git://github.com/MatiasBjorling/linux-collab nvmeq_wip_6 Is this the right one to use? Looks like a 3.15rc1+. Also ... obviously that's the wrong diff from before. I didn't save my file before creating the diff. Should have looked like this: diff --git a/block/blk-mq.c b/block/blk-mq.c index d10013b..d1b986c 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -105,7 +105,8 @@ static int blk_mq_queue_enter(struct request_queue *q) __percpu_counter_add(q-mq_usage_counter, 1, 100); smp_wmb(); /* we have problems to freeze the queue if it's initializing */ -if (!blk_queue_bypass(q) || !blk_queue_init_done(q)) +if (!blk_queue_dying(q) (!blk_queue_bypass(q) || +!blk_queue_init_done(q))) return 0; __percpu_counter_add(q-mq_usage_counter, -1, 100); Thanks! I've put your patch into the nvmemq_wip_v6 branch and rebased it on top of Jens' for-linus. -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v5] conversion to blk-mq
On Tue, 3 Jun 2014, Matias Bjorling wrote: Keith, will you take the nvmemq_wip_v6 branch for a spin? Thanks! BTW, if you want to test this out yourself, it's pretty simple to recreate. I just run a simple user admin program sending nvme passthrough commands in a tight loop, then run: # echo 1 > /sys/bus/pci/devices//remove Still fails as before: [ 88.933881] BUG: unable to handle kernel NULL pointer dereference at 0014 [ 88.942900] IP: [] blk_mq_map_queue+0xf/0x1e [ 88.949605] PGD 427be0067 PUD 425495067 PMD 0 [ 88.954915] Oops: [#1] SMP [ 88.958787] Modules linked in: nvme parport_pc ppdev lp parport dlm sctp libcrc32c configfs nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc md4 hmac cifs bridge stp llc joydev jfs hid_generic usbhid hid loop md_mod x86_pkg_temp_thermal coretemp kvm_intel kvm iTCO_wdt iTCO_vendor_support crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd microcode ehci_pci ehci_hcd pcspkr usbcore lpc_ich ioatdma usb_common mfd_core evdev i2c_i801 wmi acpi_cpufreq tpm_tis tpm ipmi_si ipmi_msghandler processor thermal_sys button ext4 crc16 jbd2 mbcache dm_mod nbd sg sd_mod sr_mod crc_t10dif cdrom crct10dif_common isci ahci libsas igb libahci scsi_transport_sas ptp pps_core i2c_algo_bit libata i2c_core scsi_mod dca [ 89.042529] CPU: 5 PID: 4544 Comm: nvme_id_ctrl Not tainted 3.15.0-rc1+ #3 [ 89.050295] Hardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013 [ 89.061856] task: 88042bbdb0d0 ti: 88042c24c000 task.ti: 88042c24c000 [ 89.070305] RIP: 0010:[] [] blk_mq_map_queue+0xf/0x1e [ 89.079747] RSP: 0018:88042c24dda0 EFLAGS: 00010202 [ 89.085795] RAX: RBX: e8fbffaa1b00 RCX: 88042e8ec4b0 [ 89.093868] RDX: 8be6 RSI: 0005 RDI: 88042abdf048 [ 89.101950] RBP: 88042c2b81c0 R08: 88042c24c000 R09: 880035c58410 [ 89.110033] R10: 88043f6b2dc0 R11: 88043f6b2dc0 R12: 88042c24de94 [ 89.118119] R13: 007d R14: 7fff0cd892b0 R15: [ 89.126210] FS: 7f39866c5700() GS:88043f6a() knlGS: [ 89.135387] CS: 0010 DS: ES: CR0: 80050033 [ 89.141916] CR2: 0014 CR3: 00042b387000 CR4: 000407e0 [ 89.149997] Stack: [ 89.152353] 811c6334 fffc 88042c2b81c0 88042c24de10 [ 89.161096] a054dcbb 0246 fffc 8800b5e05cc0 [ 89.169839] fff4 8800b5e05cc0 88042bbc3000 1000 [ 89.178583] Call Trace: [ 89.181429] [] ? blk_mq_free_request+0x37/0x48 [ 89.188360] [] ? __nvme_submit_admin_cmd+0x52/0x68 [nvme] [ 89.196349] [] ? nvme_user_admin_cmd+0x144/0x1b1 [nvme] [ 89.204150] [] ? nvme_dev_ioctl+0x1d/0x2b [nvme] [ 89.211278] [] ? do_vfs_ioctl+0x3f2/0x43b [ 89.217710] [] ? vfs_write+0xde/0xfc [ 89.223657] [] ? SyS_ioctl+0x4e/0x7d [ 89.229622] [] ? system_call_fastpath+0x16/0x1b [ 89.236636] Code: 8b 4a 38 48 39 4e 38 72 12 74 06 b8 01 00 00 00 c3 48 8b 4a 60 48 39 4e 60 73 f0 c3 66 66 66 66 90 48 8b 87 e0 00 00 00 48 63 f6 <8b> 14 b0 48 8b 87 f8 00 00 00 48 8b 04 d0 c3 89 ff f0 48 0f ab [ 89.263435] RIP [] blk_mq_map_queue+0xf/0x1e [ 89.270237] RSP [ 89.274237] CR2: 0014 [ 89.278095] ---[ end trace 54c0e8cbb1fe2ec3 ]--- -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v5] conversion to blk-mq
On Tue, 3 Jun 2014, Matias Bjorling wrote: Keith, will you take the nvmemq_wip_v6 branch for a spin? Thanks! Still fails as before: [ 88.933881] BUG: unable to handle kernel NULL pointer dereference at 0014 [ 88.942900] IP: [] blk_mq_map_queue+0xf/0x1e [ 88.949605] PGD 427be0067 PUD 425495067 PMD 0 [ 88.954915] Oops: [#1] SMP [ 88.958787] Modules linked in: nvme parport_pc ppdev lp parport dlm sctp libcrc32c configfs nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc md4 hmac cifs bridge stp llc joydev jfs hid_generic usbhid hid loop md_mod x86_pkg_temp_thermal coretemp kvm_intel kvm iTCO_wdt iTCO_vendor_support crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd microcode ehci_pci ehci_hcd pcspkr usbcore lpc_ich ioatdma usb_common mfd_core evdev i2c_i801 wmi acpi_cpufreq tpm_tis tpm ipmi_si ipmi_msghandler processor thermal_sys button ext4 crc16 jbd2 mbcache dm_mod nbd sg sd_mod sr_mod crc_t10dif cdrom crct10dif_common isci ahci libsas igb libahci scsi_transport_sas ptp pps_core i2c_algo_bit libata i2c_core scsi_mod dca [ 89.042529] CPU: 5 PID: 4544 Comm: nvme_id_ctrl Not tainted 3.15.0-rc1+ #3 [ 89.050295] Hardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013 [ 89.061856] task: 88042bbdb0d0 ti: 88042c24c000 task.ti: 88042c24c000 [ 89.070305] RIP: 0010:[] [] blk_mq_map_queue+0xf/0x1e [ 89.079747] RSP: 0018:88042c24dda0 EFLAGS: 00010202 [ 89.085795] RAX: RBX: e8fbffaa1b00 RCX: 88042e8ec4b0 [ 89.093868] RDX: 8be6 RSI: 0005 RDI: 88042abdf048 [ 89.101950] RBP: 88042c2b81c0 R08: 88042c24c000 R09: 880035c58410 [ 89.110033] R10: 88043f6b2dc0 R11: 88043f6b2dc0 R12: 88042c24de94 [ 89.118119] R13: 007d R14: 7fff0cd892b0 R15: [ 89.126210] FS: 7f39866c5700() GS:88043f6a() knlGS: [ 89.135387] CS: 0010 DS: ES: CR0: 80050033 [ 89.141916] CR2: 0014 CR3: 00042b387000 CR4: 000407e0 [ 89.149997] Stack: [ 89.152353] 811c6334 fffc 88042c2b81c0 88042c24de10 [ 89.161096] a054dcbb 0246 fffc 8800b5e05cc0 [ 89.169839] fff4 8800b5e05cc0 88042bbc3000 1000 [ 89.178583] Call Trace: [ 89.181429] [] ? blk_mq_free_request+0x37/0x48 [ 89.188360] [] ? __nvme_submit_admin_cmd+0x52/0x68 [nvme] [ 89.196349] [] ? nvme_user_admin_cmd+0x144/0x1b1 [nvme] [ 89.204150] [] ? nvme_dev_ioctl+0x1d/0x2b [nvme] [ 89.211278] [] ? do_vfs_ioctl+0x3f2/0x43b [ 89.217710] [] ? vfs_write+0xde/0xfc [ 89.223657] [] ? SyS_ioctl+0x4e/0x7d [ 89.229622] [] ? system_call_fastpath+0x16/0x1b [ 89.236636] Code: 8b 4a 38 48 39 4e 38 72 12 74 06 b8 01 00 00 00 c3 48 8b 4a 60 48 39 4e 60 73 f0 c3 66 66 66 66 90 48 8b 87 e0 00 00 00 48 63 f6 <8b> 14 b0 48 8b 87 f8 00 00 00 48 8b 04 d0 c3 89 ff f0 48 0f ab [ 89.263435] RIP [] blk_mq_map_queue+0xf/0x1e [ 89.270237] RSP [ 89.274237] CR2: 0014 [ 89.278095] ---[ end trace 54c0e8cbb1fe2ec3 ]--- -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v5] conversion to blk-mq
On 06/03/2014 01:06 AM, Keith Busch wrote: > Depending on the timing, it might fail in alloc instead of free: > > Jun 2 16:45:40 kbgrz1 kernel: [ 265.421243] NULL pointer dereference > at (null) > Jun 2 16:45:40 kbgrz1 kernel: [ 265.434284] PGD 429acf067 PUD > 42ce28067 PMD 0 > Jun 2 16:45:40 kbgrz1 kernel: [ 265.439565] Oops: [#1] SMP > Jun 2 16:45:40 kbgrz1 kernel: [ 265.443413] Modules linked in: nvme > parport_pc ppdev lp parport dlm sctp libcrc32c configfs nfsd auth_rpcgss > oid_registry nfs_acl nfs lockd fscache sunrpc md4 hmac cifs bridge stp > llc jfs joydev hid_generic usbhid hid loop md_mod x86_pkg_temp_thermal > coretemp kvm_intel kvm iTCO_wdt iTCO_vendor_support crc32c_intel > ghash_clmulni_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul > ablk_helper cryptd ehci_pci ehci_hcd microcode usbcore pcspkr ioatdma > lpc_ich usb_common mfd_core i2c_i801 evdev wmi acpi_cpufreq tpm_tis > ipmi_si tpm ipmi_msghandler processor thermal_sys button ext4 crc16 jbd2 > mbcache dm_mod nbd sg sd_mod sr_mod crc_t10dif cdrom crct10dif_common > isci libsas igb ahci scsi_transport_sas libahci ptp libata pps_core > i2c_algo_bit i2c_core scsi_mod dca > Jun 2 16:45:40 kbgrz1 kernel: [ 265.526398] CPU: 4 PID: 5454 Comm: > nvme_id_ctrl Not tainted 3.15.0-rc1+ #2 > Jun 2 16:45:40 kbgrz1 kernel: [ 265.534181] Hardware name: Intel > Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 > 12/23/2013 > Jun 2 16:45:40 kbgrz1 kernel: [ 265.545789] task: 88042e418390 ti: > 8804283e6000 task.ti: 8804283e6000 > Jun 2 16:45:40 kbgrz1 kernel: [ 265.554270] RIP: > 0010:[] [] blk_mq_map_queue+0xf/0x1e > Jun 2 16:45:40 kbgrz1 kernel: [ 265.563720] RSP: > 0018:8804283e7d80 EFLAGS: 00010286 > Jun 2 16:45:40 kbgrz1 kernel: [ 265.569755] RAX: RBX: > 880035a06008 RCX: 000f4240 > Jun 2 16:45:40 kbgrz1 kernel: [ 265.577830] RDX: 0028a360 RSI: > RDI: 880035a06008 > Jun 2 16:45:40 kbgrz1 kernel: [ 265.585904] RBP: 88043f68 R08: > R09: 1000 > Jun 2 16:45:40 kbgrz1 kernel: [ 265.593979] R10: 1000 R11: > 0410 R12: 00d0 > Jun 2 16:45:40 kbgrz1 kernel: [ 265.602053] R13: 0001 R14: > R15: > Jun 2 16:45:40 kbgrz1 kernel: [ 265.610134] FS: > 7f74b8bc9700() GS:88043f68() knlGS: > Jun 2 16:45:40 kbgrz1 kernel: [ 265.619303] CS: 0010 DS: ES: > CR0: 80050033 > Jun 2 16:45:40 kbgrz1 kernel: [ 265.625824] CR2: CR3: > 0004292c4000 CR4: 000407e0 > Jun 2 16:45:40 kbgrz1 kernel: [ 265.633889] Stack: > Jun 2 16:45:40 kbgrz1 kernel: [ 265.636236] 811c689f > fff4 8804283e7e10 > Jun 2 16:45:40 kbgrz1 kernel: [ 265.644949] 8804283e7e94 > 007d 7fff4f8a73b0 > Jun 2 16:45:40 kbgrz1 kernel: [ 265.653653] a055acc7 > 0246 2d0aaec0 88042d0aaec0 > Jun 2 16:45:40 kbgrz1 kernel: [ 265.662358] Call Trace: > Jun 2 16:45:40 kbgrz1 kernel: [ 265.665194] [] ? > blk_mq_alloc_request+0x54/0xd5 > Jun 2 16:45:40 kbgrz1 kernel: [ 265.672217] [] ? > __nvme_submit_admin_cmd+0x2d/0x68 [nvme] > Jun 2 16:45:40 kbgrz1 kernel: [ 265.680196] [] ? > nvme_user_admin_cmd+0x144/0x1b1 [nvme] > Jun 2 16:45:40 kbgrz1 kernel: [ 265.687987] [] ? > nvme_dev_ioctl+0x1d/0x2b [nvme] > Jun 2 16:45:40 kbgrz1 kernel: [ 265.695107] [] ? > do_vfs_ioctl+0x3f2/0x43b > Jun 2 16:45:40 kbgrz1 kernel: [ 265.701547] [] ? > finish_task_switch+0x84/0xc4 > Jun 2 16:45:40 kbgrz1 kernel: [ 265.708382] [] ? > __schedule+0x45c/0x4f0 > Jun 2 16:45:40 kbgrz1 kernel: [ 265.714603] [] ? > SyS_ioctl+0x4e/0x7d > Jun 2 16:45:40 kbgrz1 kernel: [ 265.720555] [] ? > system_call_fastpath+0x16/0x1b > Jun 2 16:45:40 kbgrz1 kernel: [ 265.727560] Code: 8b 4a 38 48 39 4e 38 > 72 12 74 06 b8 01 00 00 00 c3 48 8b 4a 60 48 39 4e 60 73 f0 c3 66 66 66 > 66 90 48 8b 87 e0 00 00 00 48 63 f6 <8b> 14 b0 48 8b 87 f8 00 00 00 48 > 8b 04 d0 c3 89 ff f0 48 0f ab > Jun 2 16:45:40 kbgrz1 kernel: [ 265.760706] RSP > Jun 2 16:45:40 kbgrz1 kernel: [ 265.764705] CR2: > Jun 2 16:45:40 kbgrz1 kernel: [ 265.768531] ---[ end trace > 785048a51785f51e ]--- > > On Mon, 2 Jun 2014, Keith Busch wrote: >> On Mon, 2 Jun 2014, Matias Bjørling wrote: >>> Hi Matthew and Keith, >>> >>> Here is an updated patch with the feedback from the previous days. >>> It's against >>> Jens' for-3.16/core tree. You may use the nvmemq_wip_review branch at: >> >> I'm testing this on my normal hardware now. As I feared, hot removal >> doesn't work while the device is actively being managed. Here's the oops: >> >> [ 1267.018283] BUG: unable to handle kernel NULL pointer dereference >> at 0004 >> [ 1267.018288] IP: [] blk_mq_map_queue+0xf/0x1e >> [ 1267.018292]
Re: [PATCH v5] conversion to blk-mq
On 06/03/2014 01:06 AM, Keith Busch wrote: Depending on the timing, it might fail in alloc instead of free: Jun 2 16:45:40 kbgrz1 kernel: [ 265.421243] NULL pointer dereference at (null) Jun 2 16:45:40 kbgrz1 kernel: [ 265.434284] PGD 429acf067 PUD 42ce28067 PMD 0 Jun 2 16:45:40 kbgrz1 kernel: [ 265.439565] Oops: [#1] SMP Jun 2 16:45:40 kbgrz1 kernel: [ 265.443413] Modules linked in: nvme parport_pc ppdev lp parport dlm sctp libcrc32c configfs nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc md4 hmac cifs bridge stp llc jfs joydev hid_generic usbhid hid loop md_mod x86_pkg_temp_thermal coretemp kvm_intel kvm iTCO_wdt iTCO_vendor_support crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd ehci_pci ehci_hcd microcode usbcore pcspkr ioatdma lpc_ich usb_common mfd_core i2c_i801 evdev wmi acpi_cpufreq tpm_tis ipmi_si tpm ipmi_msghandler processor thermal_sys button ext4 crc16 jbd2 mbcache dm_mod nbd sg sd_mod sr_mod crc_t10dif cdrom crct10dif_common isci libsas igb ahci scsi_transport_sas libahci ptp libata pps_core i2c_algo_bit i2c_core scsi_mod dca Jun 2 16:45:40 kbgrz1 kernel: [ 265.526398] CPU: 4 PID: 5454 Comm: nvme_id_ctrl Not tainted 3.15.0-rc1+ #2 Jun 2 16:45:40 kbgrz1 kernel: [ 265.534181] Hardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013 Jun 2 16:45:40 kbgrz1 kernel: [ 265.545789] task: 88042e418390 ti: 8804283e6000 task.ti: 8804283e6000 Jun 2 16:45:40 kbgrz1 kernel: [ 265.554270] RIP: 0010:[] [] blk_mq_map_queue+0xf/0x1e Jun 2 16:45:40 kbgrz1 kernel: [ 265.563720] RSP: 0018:8804283e7d80 EFLAGS: 00010286 Jun 2 16:45:40 kbgrz1 kernel: [ 265.569755] RAX: RBX: 880035a06008 RCX: 000f4240 Jun 2 16:45:40 kbgrz1 kernel: [ 265.577830] RDX: 0028a360 RSI: RDI: 880035a06008 Jun 2 16:45:40 kbgrz1 kernel: [ 265.585904] RBP: 88043f68 R08: R09: 1000 Jun 2 16:45:40 kbgrz1 kernel: [ 265.593979] R10: 1000 R11: 0410 R12: 00d0 Jun 2 16:45:40 kbgrz1 kernel: [ 265.602053] R13: 0001 R14: R15: Jun 2 16:45:40 kbgrz1 kernel: [ 265.610134] FS: 7f74b8bc9700() GS:88043f68() knlGS: Jun 2 16:45:40 kbgrz1 kernel: [ 265.619303] CS: 0010 DS: ES: CR0: 80050033 Jun 2 16:45:40 kbgrz1 kernel: [ 265.625824] CR2: CR3: 0004292c4000 CR4: 000407e0 Jun 2 16:45:40 kbgrz1 kernel: [ 265.633889] Stack: Jun 2 16:45:40 kbgrz1 kernel: [ 265.636236] 811c689f fff4 8804283e7e10 Jun 2 16:45:40 kbgrz1 kernel: [ 265.644949] 8804283e7e94 007d 7fff4f8a73b0 Jun 2 16:45:40 kbgrz1 kernel: [ 265.653653] a055acc7 0246 2d0aaec0 88042d0aaec0 Jun 2 16:45:40 kbgrz1 kernel: [ 265.662358] Call Trace: Jun 2 16:45:40 kbgrz1 kernel: [ 265.665194] [] ? blk_mq_alloc_request+0x54/0xd5 Jun 2 16:45:40 kbgrz1 kernel: [ 265.672217] [] ? __nvme_submit_admin_cmd+0x2d/0x68 [nvme] Jun 2 16:45:40 kbgrz1 kernel: [ 265.680196] [] ? nvme_user_admin_cmd+0x144/0x1b1 [nvme] Jun 2 16:45:40 kbgrz1 kernel: [ 265.687987] [] ? nvme_dev_ioctl+0x1d/0x2b [nvme] Jun 2 16:45:40 kbgrz1 kernel: [ 265.695107] [] ? do_vfs_ioctl+0x3f2/0x43b Jun 2 16:45:40 kbgrz1 kernel: [ 265.701547] [] ? finish_task_switch+0x84/0xc4 Jun 2 16:45:40 kbgrz1 kernel: [ 265.708382] [] ? __schedule+0x45c/0x4f0 Jun 2 16:45:40 kbgrz1 kernel: [ 265.714603] [] ? SyS_ioctl+0x4e/0x7d Jun 2 16:45:40 kbgrz1 kernel: [ 265.720555] [] ? system_call_fastpath+0x16/0x1b Jun 2 16:45:40 kbgrz1 kernel: [ 265.727560] Code: 8b 4a 38 48 39 4e 38 72 12 74 06 b8 01 00 00 00 c3 48 8b 4a 60 48 39 4e 60 73 f0 c3 66 66 66 66 90 48 8b 87 e0 00 00 00 48 63 f6 <8b> 14 b0 48 8b 87 f8 00 00 00 48 8b 04 d0 c3 89 ff f0 48 0f ab Jun 2 16:45:40 kbgrz1 kernel: [ 265.760706] RSP Jun 2 16:45:40 kbgrz1 kernel: [ 265.764705] CR2: Jun 2 16:45:40 kbgrz1 kernel: [ 265.768531] ---[ end trace 785048a51785f51e ]--- On Mon, 2 Jun 2014, Keith Busch wrote: On Mon, 2 Jun 2014, Matias Bjørling wrote: Hi Matthew and Keith, Here is an updated patch with the feedback from the previous days. It's against Jens' for-3.16/core tree. You may use the nvmemq_wip_review branch at: I'm testing this on my normal hardware now. As I feared, hot removal doesn't work while the device is actively being managed. Here's the oops: [ 1267.018283] BUG: unable to handle kernel NULL pointer dereference at 0004 [ 1267.018288] IP: [] blk_mq_map_queue+0xf/0x1e [ 1267.018292] PGD b5ed5067 PUD b57e2067 PMD 0 [ 1267.018294] Oops: [#1] SMP [ 1267.018343] Modules linked in: nvme parport_pc ppdev lp parport dlm sctp libcrc32c configfs nfsd auth_rpcgss oid_registry nfs_acl nfs
Re: [PATCH v5] conversion to blk-mq
On 06/03/2014 01:06 AM, Keith Busch wrote: Depending on the timing, it might fail in alloc instead of free: Jun 2 16:45:40 kbgrz1 kernel: [ 265.421243] NULL pointer dereference at (null) Jun 2 16:45:40 kbgrz1 kernel: [ 265.434284] PGD 429acf067 PUD 42ce28067 PMD 0 Jun 2 16:45:40 kbgrz1 kernel: [ 265.439565] Oops: [#1] SMP Jun 2 16:45:40 kbgrz1 kernel: [ 265.443413] Modules linked in: nvme parport_pc ppdev lp parport dlm sctp libcrc32c configfs nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc md4 hmac cifs bridge stp llc jfs joydev hid_generic usbhid hid loop md_mod x86_pkg_temp_thermal coretemp kvm_intel kvm iTCO_wdt iTCO_vendor_support crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd ehci_pci ehci_hcd microcode usbcore pcspkr ioatdma lpc_ich usb_common mfd_core i2c_i801 evdev wmi acpi_cpufreq tpm_tis ipmi_si tpm ipmi_msghandler processor thermal_sys button ext4 crc16 jbd2 mbcache dm_mod nbd sg sd_mod sr_mod crc_t10dif cdrom crct10dif_common isci libsas igb ahci scsi_transport_sas libahci ptp libata pps_core i2c_algo_bit i2c_core scsi_mod dca Jun 2 16:45:40 kbgrz1 kernel: [ 265.526398] CPU: 4 PID: 5454 Comm: nvme_id_ctrl Not tainted 3.15.0-rc1+ #2 Jun 2 16:45:40 kbgrz1 kernel: [ 265.534181] Hardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013 Jun 2 16:45:40 kbgrz1 kernel: [ 265.545789] task: 88042e418390 ti: 8804283e6000 task.ti: 8804283e6000 Jun 2 16:45:40 kbgrz1 kernel: [ 265.554270] RIP: 0010:[811c51b8] [811c51b8] blk_mq_map_queue+0xf/0x1e Jun 2 16:45:40 kbgrz1 kernel: [ 265.563720] RSP: 0018:8804283e7d80 EFLAGS: 00010286 Jun 2 16:45:40 kbgrz1 kernel: [ 265.569755] RAX: RBX: 880035a06008 RCX: 000f4240 Jun 2 16:45:40 kbgrz1 kernel: [ 265.577830] RDX: 0028a360 RSI: RDI: 880035a06008 Jun 2 16:45:40 kbgrz1 kernel: [ 265.585904] RBP: 88043f68 R08: R09: 1000 Jun 2 16:45:40 kbgrz1 kernel: [ 265.593979] R10: 1000 R11: 0410 R12: 00d0 Jun 2 16:45:40 kbgrz1 kernel: [ 265.602053] R13: 0001 R14: R15: Jun 2 16:45:40 kbgrz1 kernel: [ 265.610134] FS: 7f74b8bc9700() GS:88043f68() knlGS: Jun 2 16:45:40 kbgrz1 kernel: [ 265.619303] CS: 0010 DS: ES: CR0: 80050033 Jun 2 16:45:40 kbgrz1 kernel: [ 265.625824] CR2: CR3: 0004292c4000 CR4: 000407e0 Jun 2 16:45:40 kbgrz1 kernel: [ 265.633889] Stack: Jun 2 16:45:40 kbgrz1 kernel: [ 265.636236] 811c689f fff4 8804283e7e10 Jun 2 16:45:40 kbgrz1 kernel: [ 265.644949] 8804283e7e94 007d 7fff4f8a73b0 Jun 2 16:45:40 kbgrz1 kernel: [ 265.653653] a055acc7 0246 2d0aaec0 88042d0aaec0 Jun 2 16:45:40 kbgrz1 kernel: [ 265.662358] Call Trace: Jun 2 16:45:40 kbgrz1 kernel: [ 265.665194] [811c689f] ? blk_mq_alloc_request+0x54/0xd5 Jun 2 16:45:40 kbgrz1 kernel: [ 265.672217] [a055acc7] ? __nvme_submit_admin_cmd+0x2d/0x68 [nvme] Jun 2 16:45:40 kbgrz1 kernel: [ 265.680196] [a055c879] ? nvme_user_admin_cmd+0x144/0x1b1 [nvme] Jun 2 16:45:40 kbgrz1 kernel: [ 265.687987] [a055c903] ? nvme_dev_ioctl+0x1d/0x2b [nvme] Jun 2 16:45:40 kbgrz1 kernel: [ 265.695107] [81125916] ? do_vfs_ioctl+0x3f2/0x43b Jun 2 16:45:40 kbgrz1 kernel: [ 265.701547] [8105ca6b] ? finish_task_switch+0x84/0xc4 Jun 2 16:45:40 kbgrz1 kernel: [ 265.708382] [81394a8c] ? __schedule+0x45c/0x4f0 Jun 2 16:45:40 kbgrz1 kernel: [ 265.714603] [811259ad] ? SyS_ioctl+0x4e/0x7d Jun 2 16:45:40 kbgrz1 kernel: [ 265.720555] [8139c6d2] ? system_call_fastpath+0x16/0x1b Jun 2 16:45:40 kbgrz1 kernel: [ 265.727560] Code: 8b 4a 38 48 39 4e 38 72 12 74 06 b8 01 00 00 00 c3 48 8b 4a 60 48 39 4e 60 73 f0 c3 66 66 66 66 90 48 8b 87 e0 00 00 00 48 63 f6 8b 14 b0 48 8b 87 f8 00 00 00 48 8b 04 d0 c3 89 ff f0 48 0f ab Jun 2 16:45:40 kbgrz1 kernel: [ 265.760706] RSP 8804283e7d80 Jun 2 16:45:40 kbgrz1 kernel: [ 265.764705] CR2: Jun 2 16:45:40 kbgrz1 kernel: [ 265.768531] ---[ end trace 785048a51785f51e ]--- On Mon, 2 Jun 2014, Keith Busch wrote: On Mon, 2 Jun 2014, Matias Bjørling wrote: Hi Matthew and Keith, Here is an updated patch with the feedback from the previous days. It's against Jens' for-3.16/core tree. You may use the nvmemq_wip_review branch at: I'm testing this on my normal hardware now. As I feared, hot removal doesn't work while the device is actively being managed. Here's the oops: [ 1267.018283] BUG: unable to handle kernel NULL pointer dereference at 0004 [ 1267.018288] IP: [811c51b8] blk_mq_map_queue+0xf/0x1e [
Re: [PATCH v5] conversion to blk-mq
On 06/03/2014 01:06 AM, Keith Busch wrote: Depending on the timing, it might fail in alloc instead of free: Jun 2 16:45:40 kbgrz1 kernel: [ 265.421243] NULL pointer dereference at (null) Jun 2 16:45:40 kbgrz1 kernel: [ 265.434284] PGD 429acf067 PUD 42ce28067 PMD 0 Jun 2 16:45:40 kbgrz1 kernel: [ 265.439565] Oops: [#1] SMP Jun 2 16:45:40 kbgrz1 kernel: [ 265.443413] Modules linked in: nvme parport_pc ppdev lp parport dlm sctp libcrc32c configfs nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc md4 hmac cifs bridge stp llc jfs joydev hid_generic usbhid hid loop md_mod x86_pkg_temp_thermal coretemp kvm_intel kvm iTCO_wdt iTCO_vendor_support crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd ehci_pci ehci_hcd microcode usbcore pcspkr ioatdma lpc_ich usb_common mfd_core i2c_i801 evdev wmi acpi_cpufreq tpm_tis ipmi_si tpm ipmi_msghandler processor thermal_sys button ext4 crc16 jbd2 mbcache dm_mod nbd sg sd_mod sr_mod crc_t10dif cdrom crct10dif_common isci libsas igb ahci scsi_transport_sas libahci ptp libata pps_core i2c_algo_bit i2c_core scsi_mod dca Jun 2 16:45:40 kbgrz1 kernel: [ 265.526398] CPU: 4 PID: 5454 Comm: nvme_id_ctrl Not tainted 3.15.0-rc1+ #2 Jun 2 16:45:40 kbgrz1 kernel: [ 265.534181] Hardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013 Jun 2 16:45:40 kbgrz1 kernel: [ 265.545789] task: 88042e418390 ti: 8804283e6000 task.ti: 8804283e6000 Jun 2 16:45:40 kbgrz1 kernel: [ 265.554270] RIP: 0010:[811c51b8] [811c51b8] blk_mq_map_queue+0xf/0x1e Jun 2 16:45:40 kbgrz1 kernel: [ 265.563720] RSP: 0018:8804283e7d80 EFLAGS: 00010286 Jun 2 16:45:40 kbgrz1 kernel: [ 265.569755] RAX: RBX: 880035a06008 RCX: 000f4240 Jun 2 16:45:40 kbgrz1 kernel: [ 265.577830] RDX: 0028a360 RSI: RDI: 880035a06008 Jun 2 16:45:40 kbgrz1 kernel: [ 265.585904] RBP: 88043f68 R08: R09: 1000 Jun 2 16:45:40 kbgrz1 kernel: [ 265.593979] R10: 1000 R11: 0410 R12: 00d0 Jun 2 16:45:40 kbgrz1 kernel: [ 265.602053] R13: 0001 R14: R15: Jun 2 16:45:40 kbgrz1 kernel: [ 265.610134] FS: 7f74b8bc9700() GS:88043f68() knlGS: Jun 2 16:45:40 kbgrz1 kernel: [ 265.619303] CS: 0010 DS: ES: CR0: 80050033 Jun 2 16:45:40 kbgrz1 kernel: [ 265.625824] CR2: CR3: 0004292c4000 CR4: 000407e0 Jun 2 16:45:40 kbgrz1 kernel: [ 265.633889] Stack: Jun 2 16:45:40 kbgrz1 kernel: [ 265.636236] 811c689f fff4 8804283e7e10 Jun 2 16:45:40 kbgrz1 kernel: [ 265.644949] 8804283e7e94 007d 7fff4f8a73b0 Jun 2 16:45:40 kbgrz1 kernel: [ 265.653653] a055acc7 0246 2d0aaec0 88042d0aaec0 Jun 2 16:45:40 kbgrz1 kernel: [ 265.662358] Call Trace: Jun 2 16:45:40 kbgrz1 kernel: [ 265.665194] [811c689f] ? blk_mq_alloc_request+0x54/0xd5 Jun 2 16:45:40 kbgrz1 kernel: [ 265.672217] [a055acc7] ? __nvme_submit_admin_cmd+0x2d/0x68 [nvme] Jun 2 16:45:40 kbgrz1 kernel: [ 265.680196] [a055c879] ? nvme_user_admin_cmd+0x144/0x1b1 [nvme] Jun 2 16:45:40 kbgrz1 kernel: [ 265.687987] [a055c903] ? nvme_dev_ioctl+0x1d/0x2b [nvme] Jun 2 16:45:40 kbgrz1 kernel: [ 265.695107] [81125916] ? do_vfs_ioctl+0x3f2/0x43b Jun 2 16:45:40 kbgrz1 kernel: [ 265.701547] [8105ca6b] ? finish_task_switch+0x84/0xc4 Jun 2 16:45:40 kbgrz1 kernel: [ 265.708382] [81394a8c] ? __schedule+0x45c/0x4f0 Jun 2 16:45:40 kbgrz1 kernel: [ 265.714603] [811259ad] ? SyS_ioctl+0x4e/0x7d Jun 2 16:45:40 kbgrz1 kernel: [ 265.720555] [8139c6d2] ? system_call_fastpath+0x16/0x1b Jun 2 16:45:40 kbgrz1 kernel: [ 265.727560] Code: 8b 4a 38 48 39 4e 38 72 12 74 06 b8 01 00 00 00 c3 48 8b 4a 60 48 39 4e 60 73 f0 c3 66 66 66 66 90 48 8b 87 e0 00 00 00 48 63 f6 8b 14 b0 48 8b 87 f8 00 00 00 48 8b 04 d0 c3 89 ff f0 48 0f ab Jun 2 16:45:40 kbgrz1 kernel: [ 265.760706] RSP 8804283e7d80 Jun 2 16:45:40 kbgrz1 kernel: [ 265.764705] CR2: Jun 2 16:45:40 kbgrz1 kernel: [ 265.768531] ---[ end trace 785048a51785f51e ]--- On Mon, 2 Jun 2014, Keith Busch wrote: On Mon, 2 Jun 2014, Matias Bjørling wrote: Hi Matthew and Keith, Here is an updated patch with the feedback from the previous days. It's against Jens' for-3.16/core tree. You may use the nvmemq_wip_review branch at: I'm testing this on my normal hardware now. As I feared, hot removal doesn't work while the device is actively being managed. Here's the oops: [ 1267.018283] BUG: unable to handle kernel NULL pointer dereference at
Re: [PATCH v5] conversion to blk-mq
On Tue, 3 Jun 2014, Matias Bjorling wrote: Keith, will you take the nvmemq_wip_v6 branch for a spin? Thanks! Still fails as before: [ 88.933881] BUG: unable to handle kernel NULL pointer dereference at 0014 [ 88.942900] IP: [811c51b8] blk_mq_map_queue+0xf/0x1e [ 88.949605] PGD 427be0067 PUD 425495067 PMD 0 [ 88.954915] Oops: [#1] SMP [ 88.958787] Modules linked in: nvme parport_pc ppdev lp parport dlm sctp libcrc32c configfs nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc md4 hmac cifs bridge stp llc joydev jfs hid_generic usbhid hid loop md_mod x86_pkg_temp_thermal coretemp kvm_intel kvm iTCO_wdt iTCO_vendor_support crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd microcode ehci_pci ehci_hcd pcspkr usbcore lpc_ich ioatdma usb_common mfd_core evdev i2c_i801 wmi acpi_cpufreq tpm_tis tpm ipmi_si ipmi_msghandler processor thermal_sys button ext4 crc16 jbd2 mbcache dm_mod nbd sg sd_mod sr_mod crc_t10dif cdrom crct10dif_common isci ahci libsas igb libahci scsi_transport_sas ptp pps_core i2c_algo_bit libata i2c_core scsi_mod dca [ 89.042529] CPU: 5 PID: 4544 Comm: nvme_id_ctrl Not tainted 3.15.0-rc1+ #3 [ 89.050295] Hardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013 [ 89.061856] task: 88042bbdb0d0 ti: 88042c24c000 task.ti: 88042c24c000 [ 89.070305] RIP: 0010:[811c51b8] [811c51b8] blk_mq_map_queue+0xf/0x1e [ 89.079747] RSP: 0018:88042c24dda0 EFLAGS: 00010202 [ 89.085795] RAX: RBX: e8fbffaa1b00 RCX: 88042e8ec4b0 [ 89.093868] RDX: 8be6 RSI: 0005 RDI: 88042abdf048 [ 89.101950] RBP: 88042c2b81c0 R08: 88042c24c000 R09: 880035c58410 [ 89.110033] R10: 88043f6b2dc0 R11: 88043f6b2dc0 R12: 88042c24de94 [ 89.118119] R13: 007d R14: 7fff0cd892b0 R15: [ 89.126210] FS: 7f39866c5700() GS:88043f6a() knlGS: [ 89.135387] CS: 0010 DS: ES: CR0: 80050033 [ 89.141916] CR2: 0014 CR3: 00042b387000 CR4: 000407e0 [ 89.149997] Stack: [ 89.152353] 811c6334 fffc 88042c2b81c0 88042c24de10 [ 89.161096] a054dcbb 0246 fffc 8800b5e05cc0 [ 89.169839] fff4 8800b5e05cc0 88042bbc3000 1000 [ 89.178583] Call Trace: [ 89.181429] [811c6334] ? blk_mq_free_request+0x37/0x48 [ 89.188360] [a054dcbb] ? __nvme_submit_admin_cmd+0x52/0x68 [nvme] [ 89.196349] [a054f761] ? nvme_user_admin_cmd+0x144/0x1b1 [nvme] [ 89.204150] [a054f7eb] ? nvme_dev_ioctl+0x1d/0x2b [nvme] [ 89.211278] [81125916] ? do_vfs_ioctl+0x3f2/0x43b [ 89.217710] [81117e35] ? vfs_write+0xde/0xfc [ 89.223657] [811259ad] ? SyS_ioctl+0x4e/0x7d [ 89.229622] [8139c6d2] ? system_call_fastpath+0x16/0x1b [ 89.236636] Code: 8b 4a 38 48 39 4e 38 72 12 74 06 b8 01 00 00 00 c3 48 8b 4a 60 48 39 4e 60 73 f0 c3 66 66 66 66 90 48 8b 87 e0 00 00 00 48 63 f6 8b 14 b0 48 8b 87 f8 00 00 00 48 8b 04 d0 c3 89 ff f0 48 0f ab [ 89.263435] RIP [811c51b8] blk_mq_map_queue+0xf/0x1e [ 89.270237] RSP 88042c24dda0 [ 89.274237] CR2: 0014 [ 89.278095] ---[ end trace 54c0e8cbb1fe2ec3 ]--- -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v5] conversion to blk-mq
On Tue, 3 Jun 2014, Matias Bjorling wrote: Keith, will you take the nvmemq_wip_v6 branch for a spin? Thanks! BTW, if you want to test this out yourself, it's pretty simple to recreate. I just run a simple user admin program sending nvme passthrough commands in a tight loop, then run: # echo 1 /sys/bus/pci/devices/bdf/remove Still fails as before: [ 88.933881] BUG: unable to handle kernel NULL pointer dereference at 0014 [ 88.942900] IP: [811c51b8] blk_mq_map_queue+0xf/0x1e [ 88.949605] PGD 427be0067 PUD 425495067 PMD 0 [ 88.954915] Oops: [#1] SMP [ 88.958787] Modules linked in: nvme parport_pc ppdev lp parport dlm sctp libcrc32c configfs nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc md4 hmac cifs bridge stp llc joydev jfs hid_generic usbhid hid loop md_mod x86_pkg_temp_thermal coretemp kvm_intel kvm iTCO_wdt iTCO_vendor_support crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd microcode ehci_pci ehci_hcd pcspkr usbcore lpc_ich ioatdma usb_common mfd_core evdev i2c_i801 wmi acpi_cpufreq tpm_tis tpm ipmi_si ipmi_msghandler processor thermal_sys button ext4 crc16 jbd2 mbcache dm_mod nbd sg sd_mod sr_mod crc_t10dif cdrom crct10dif_common isci ahci libsas igb libahci scsi_transport_sas ptp pps_core i2c_algo_bit libata i2c_core scsi_mod dca [ 89.042529] CPU: 5 PID: 4544 Comm: nvme_id_ctrl Not tainted 3.15.0-rc1+ #3 [ 89.050295] Hardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013 [ 89.061856] task: 88042bbdb0d0 ti: 88042c24c000 task.ti: 88042c24c000 [ 89.070305] RIP: 0010:[811c51b8] [811c51b8] blk_mq_map_queue+0xf/0x1e [ 89.079747] RSP: 0018:88042c24dda0 EFLAGS: 00010202 [ 89.085795] RAX: RBX: e8fbffaa1b00 RCX: 88042e8ec4b0 [ 89.093868] RDX: 8be6 RSI: 0005 RDI: 88042abdf048 [ 89.101950] RBP: 88042c2b81c0 R08: 88042c24c000 R09: 880035c58410 [ 89.110033] R10: 88043f6b2dc0 R11: 88043f6b2dc0 R12: 88042c24de94 [ 89.118119] R13: 007d R14: 7fff0cd892b0 R15: [ 89.126210] FS: 7f39866c5700() GS:88043f6a() knlGS: [ 89.135387] CS: 0010 DS: ES: CR0: 80050033 [ 89.141916] CR2: 0014 CR3: 00042b387000 CR4: 000407e0 [ 89.149997] Stack: [ 89.152353] 811c6334 fffc 88042c2b81c0 88042c24de10 [ 89.161096] a054dcbb 0246 fffc 8800b5e05cc0 [ 89.169839] fff4 8800b5e05cc0 88042bbc3000 1000 [ 89.178583] Call Trace: [ 89.181429] [811c6334] ? blk_mq_free_request+0x37/0x48 [ 89.188360] [a054dcbb] ? __nvme_submit_admin_cmd+0x52/0x68 [nvme] [ 89.196349] [a054f761] ? nvme_user_admin_cmd+0x144/0x1b1 [nvme] [ 89.204150] [a054f7eb] ? nvme_dev_ioctl+0x1d/0x2b [nvme] [ 89.211278] [81125916] ? do_vfs_ioctl+0x3f2/0x43b [ 89.217710] [81117e35] ? vfs_write+0xde/0xfc [ 89.223657] [811259ad] ? SyS_ioctl+0x4e/0x7d [ 89.229622] [8139c6d2] ? system_call_fastpath+0x16/0x1b [ 89.236636] Code: 8b 4a 38 48 39 4e 38 72 12 74 06 b8 01 00 00 00 c3 48 8b 4a 60 48 39 4e 60 73 f0 c3 66 66 66 66 90 48 8b 87 e0 00 00 00 48 63 f6 8b 14 b0 48 8b 87 f8 00 00 00 48 8b 04 d0 c3 89 ff f0 48 0f ab [ 89.263435] RIP [811c51b8] blk_mq_map_queue+0xf/0x1e [ 89.270237] RSP 88042c24dda0 [ 89.274237] CR2: 0014 [ 89.278095] ---[ end trace 54c0e8cbb1fe2ec3 ]--- -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v5] conversion to blk-mq
Depending on the timing, it might fail in alloc instead of free: Jun 2 16:45:40 kbgrz1 kernel: [ 265.421243] NULL pointer dereference at (null) Jun 2 16:45:40 kbgrz1 kernel: [ 265.434284] PGD 429acf067 PUD 42ce28067 PMD 0 Jun 2 16:45:40 kbgrz1 kernel: [ 265.439565] Oops: [#1] SMP Jun 2 16:45:40 kbgrz1 kernel: [ 265.443413] Modules linked in: nvme parport_pc ppdev lp parport dlm sctp libcrc32c configfs nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc md4 hmac cifs bridge stp llc jfs joydev hid_generic usbhid hid loop md_mod x86_pkg_temp_thermal coretemp kvm_intel kvm iTCO_wdt iTCO_vendor_support crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd ehci_pci ehci_hcd microcode usbcore pcspkr ioatdma lpc_ich usb_common mfd_core i2c_i801 evdev wmi acpi_cpufreq tpm_tis ipmi_si tpm ipmi_msghandler processor thermal_sys button ext4 crc16 jbd2 mbcache dm_mod nbd sg sd_mod sr_mod crc_t10dif cdrom crct10dif_common isci libsas igb ahci scsi_transport_sas libahci ptp libata pps_core i2c_algo_bit i2c_core scsi_mod dca Jun 2 16:45:40 kbgrz1 kernel: [ 265.526398] CPU: 4 PID: 5454 Comm: nvme_id_ctrl Not tainted 3.15.0-rc1+ #2 Jun 2 16:45:40 kbgrz1 kernel: [ 265.534181] Hardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013 Jun 2 16:45:40 kbgrz1 kernel: [ 265.545789] task: 88042e418390 ti: 8804283e6000 task.ti: 8804283e6000 Jun 2 16:45:40 kbgrz1 kernel: [ 265.554270] RIP: 0010:[] [] blk_mq_map_queue+0xf/0x1e Jun 2 16:45:40 kbgrz1 kernel: [ 265.563720] RSP: 0018:8804283e7d80 EFLAGS: 00010286 Jun 2 16:45:40 kbgrz1 kernel: [ 265.569755] RAX: RBX: 880035a06008 RCX: 000f4240 Jun 2 16:45:40 kbgrz1 kernel: [ 265.577830] RDX: 0028a360 RSI: RDI: 880035a06008 Jun 2 16:45:40 kbgrz1 kernel: [ 265.585904] RBP: 88043f68 R08: R09: 1000 Jun 2 16:45:40 kbgrz1 kernel: [ 265.593979] R10: 1000 R11: 0410 R12: 00d0 Jun 2 16:45:40 kbgrz1 kernel: [ 265.602053] R13: 0001 R14: R15: Jun 2 16:45:40 kbgrz1 kernel: [ 265.610134] FS: 7f74b8bc9700() GS:88043f68() knlGS: Jun 2 16:45:40 kbgrz1 kernel: [ 265.619303] CS: 0010 DS: ES: CR0: 80050033 Jun 2 16:45:40 kbgrz1 kernel: [ 265.625824] CR2: CR3: 0004292c4000 CR4: 000407e0 Jun 2 16:45:40 kbgrz1 kernel: [ 265.633889] Stack: Jun 2 16:45:40 kbgrz1 kernel: [ 265.636236] 811c689f fff4 8804283e7e10 Jun 2 16:45:40 kbgrz1 kernel: [ 265.644949] 8804283e7e94 007d 7fff4f8a73b0 Jun 2 16:45:40 kbgrz1 kernel: [ 265.653653] a055acc7 0246 2d0aaec0 88042d0aaec0 Jun 2 16:45:40 kbgrz1 kernel: [ 265.662358] Call Trace: Jun 2 16:45:40 kbgrz1 kernel: [ 265.665194] [] ? blk_mq_alloc_request+0x54/0xd5 Jun 2 16:45:40 kbgrz1 kernel: [ 265.672217] [] ? __nvme_submit_admin_cmd+0x2d/0x68 [nvme] Jun 2 16:45:40 kbgrz1 kernel: [ 265.680196] [] ? nvme_user_admin_cmd+0x144/0x1b1 [nvme] Jun 2 16:45:40 kbgrz1 kernel: [ 265.687987] [] ? nvme_dev_ioctl+0x1d/0x2b [nvme] Jun 2 16:45:40 kbgrz1 kernel: [ 265.695107] [] ? do_vfs_ioctl+0x3f2/0x43b Jun 2 16:45:40 kbgrz1 kernel: [ 265.701547] [] ? finish_task_switch+0x84/0xc4 Jun 2 16:45:40 kbgrz1 kernel: [ 265.708382] [] ? __schedule+0x45c/0x4f0 Jun 2 16:45:40 kbgrz1 kernel: [ 265.714603] [] ? SyS_ioctl+0x4e/0x7d Jun 2 16:45:40 kbgrz1 kernel: [ 265.720555] [] ? system_call_fastpath+0x16/0x1b Jun 2 16:45:40 kbgrz1 kernel: [ 265.727560] Code: 8b 4a 38 48 39 4e 38 72 12 74 06 b8 01 00 00 00 c3 48 8b 4a 60 48 39 4e 60 73 f0 c3 66 66 66 66 90 48 8b 87 e0 00 00 00 48 63 f6 <8b> 14 b0 48 8b 87 f8 00 00 00 48 8b 04 d0 c3 89 ff f0 48 0f ab Jun 2 16:45:40 kbgrz1 kernel: [ 265.760706] RSP Jun 2 16:45:40 kbgrz1 kernel: [ 265.764705] CR2: Jun 2 16:45:40 kbgrz1 kernel: [ 265.768531] ---[ end trace 785048a51785f51e ]--- On Mon, 2 Jun 2014, Keith Busch wrote: On Mon, 2 Jun 2014, Matias Bjørling wrote: Hi Matthew and Keith, Here is an updated patch with the feedback from the previous days. It's against Jens' for-3.16/core tree. You may use the nvmemq_wip_review branch at: I'm testing this on my normal hardware now. As I feared, hot removal doesn't work while the device is actively being managed. Here's the oops: [ 1267.018283] BUG: unable to handle kernel NULL pointer dereference at 0004 [ 1267.018288] IP: [] blk_mq_map_queue+0xf/0x1e [ 1267.018292] PGD b5ed5067 PUD b57e2067 PMD 0 [ 1267.018294] Oops: [#1] SMP [ 1267.018343] Modules linked in: nvme parport_pc ppdev lp parport dlm sctp libcrc32c configfs nfsd auth_rpcgss oid_registry nfs_acl nfs
Re: [PATCH v5] conversion to blk-mq
On Mon, 2 Jun 2014, Matias Bjørling wrote: Hi Matthew and Keith, Here is an updated patch with the feedback from the previous days. It's against Jens' for-3.16/core tree. You may use the nvmemq_wip_review branch at: I'm testing this on my normal hardware now. As I feared, hot removal doesn't work while the device is actively being managed. Here's the oops: [ 1267.018283] BUG: unable to handle kernel NULL pointer dereference at 0004 [ 1267.018288] IP: [] blk_mq_map_queue+0xf/0x1e [ 1267.018292] PGD b5ed5067 PUD b57e2067 PMD 0 [ 1267.018294] Oops: [#1] SMP [ 1267.018343] Modules linked in: nvme parport_pc ppdev lp parport dlm sctp libcrc32c configfs nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc md4 hmac cifs bridge stp llc joydev jfs hid_generic usbhid hid loop md_mod x86_pkg_temp_thermal coretemp kvm_intel kvm iTCO_wdt iTCO_vendor_support crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd microcode ehci_pci ehci_hcd usbcore pcspkr lpc_ich acpi_cpufreq ioatdma mfd_core usb_common i2c_i801 evdev wmi ipmi_si tpm_tis ipmi_msghandler tpm processor thermal_sys button ext4 crc16 jbd2 mbcache dm_mod nbd sg sr_mod cdrom sd_mod crc_t10dif crct10dif_common isci libsas ahci scsi_transport_sas libahci igb libata ptp pps_core i2c_algo_bit scsi_mod i2c_core dca [last unloaded: nvme] [ 1267.018346] CPU: 1 PID: 6618 Comm: nvme_id_ctrl Tainted: GW 3.15.0-rc1+ #2 [ 1267.018347] Hardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013 [ 1267.018349] task: 88042879eef0 ti: 8800b5f14000 task.ti: 8800b5f14000 [ 1267.018354] RIP: 0010:[] [] blk_mq_map_queue+0xf/0x1e [ 1267.018356] RSP: 0018:8800b5f15db0 EFLAGS: 00010206 [ 1267.018357] RAX: RBX: e8fbffa21e80 RCX: 88042f3952e0 [ 1267.018359] RDX: 1c10 RSI: 0001 RDI: 8800b5e9f008 [ 1267.018360] RBP: 88042d993480 R08: 8800b5e18cc0 R09: [ 1267.018362] R10: 0ca0 R11: 0ca0 R12: 8800b5f15e94 [ 1267.018363] R13: 3a98 R14: 7fffc7b44c40 R15: [ 1267.018365] FS: 7feb4cb05700() GS:88043f62() knlGS: [ 1267.018367] CS: 0010 DS: ES: CR0: 80050033 [ 1267.018369] CR2: 0004 CR3: b50f2000 CR4: 000407e0 [ 1267.018369] Stack: [ 1267.018372] 811c6334 fffc 88042d993480 fffc [ 1267.018375] a0532d73 0ca0 fff4 88042562b440 [ 1267.018378] 88042d853000 1000 a05348f3 c9001518b2a8 [ 1267.018378] Call Trace: [ 1267.018383] [] ? blk_mq_free_request+0x37/0x48 [ 1267.018388] [] ? __nvme_submit_admin_cmd+0x7f/0x8a [nvme] [ 1267.018392] [] ? nvme_user_admin_cmd+0x144/0x1b1 [nvme] [ 1267.018397] [] ? nvme_dev_ioctl+0x1d/0x2b [nvme] [ 1267.018399] [] ? do_vfs_ioctl+0x3f2/0x43b [ 1267.018402] [] ? finish_task_switch+0x84/0xc4 [ 1267.018406] [] ? __schedule+0x45c/0x4f0 [ 1267.018408] [] ? SyS_ioctl+0x4e/0x7d [ 1267.018411] [] ? system_call_fastpath+0x16/0x1b [ 1267.018435] Code: 8b 4a 38 48 39 4e 38 72 12 74 06 b8 01 00 00 00 c3 48 8b 4a 60 48 39 4e 60 73 f0 c3 66 66 66 66 90 48 8b 87 e0 00 00 00 48 63 f6 <8b> 14 b0 48 8b 87 f8 00 00 00 48 8b 04 d0 c3 89 ff f0 48 0f ab [ 1267.018438] RIP [] blk_mq_map_queue+0xf/0x1e [ 1267.018439] RSP [ 1267.018440] CR2: 0004 [ 1267.018443] ---[ end trace 1e244ae5f3ceb23b ]---
Re: [PATCH v5] conversion to blk-mq
On Mon, 2 Jun 2014, Matias Bjørling wrote: Hi Matthew and Keith, Here is an updated patch with the feedback from the previous days. It's against Jens' for-3.16/core tree. You may use the nvmemq_wip_review branch at: I'm testing this on my normal hardware now. As I feared, hot removal doesn't work while the device is actively being managed. Here's the oops: [ 1267.018283] BUG: unable to handle kernel NULL pointer dereference at 0004 [ 1267.018288] IP: [811c51b8] blk_mq_map_queue+0xf/0x1e [ 1267.018292] PGD b5ed5067 PUD b57e2067 PMD 0 [ 1267.018294] Oops: [#1] SMP [ 1267.018343] Modules linked in: nvme parport_pc ppdev lp parport dlm sctp libcrc32c configfs nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc md4 hmac cifs bridge stp llc joydev jfs hid_generic usbhid hid loop md_mod x86_pkg_temp_thermal coretemp kvm_intel kvm iTCO_wdt iTCO_vendor_support crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd microcode ehci_pci ehci_hcd usbcore pcspkr lpc_ich acpi_cpufreq ioatdma mfd_core usb_common i2c_i801 evdev wmi ipmi_si tpm_tis ipmi_msghandler tpm processor thermal_sys button ext4 crc16 jbd2 mbcache dm_mod nbd sg sr_mod cdrom sd_mod crc_t10dif crct10dif_common isci libsas ahci scsi_transport_sas libahci igb libata ptp pps_core i2c_algo_bit scsi_mod i2c_core dca [last unloaded: nvme] [ 1267.018346] CPU: 1 PID: 6618 Comm: nvme_id_ctrl Tainted: GW 3.15.0-rc1+ #2 [ 1267.018347] Hardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013 [ 1267.018349] task: 88042879eef0 ti: 8800b5f14000 task.ti: 8800b5f14000 [ 1267.018354] RIP: 0010:[811c51b8] [811c51b8] blk_mq_map_queue+0xf/0x1e [ 1267.018356] RSP: 0018:8800b5f15db0 EFLAGS: 00010206 [ 1267.018357] RAX: RBX: e8fbffa21e80 RCX: 88042f3952e0 [ 1267.018359] RDX: 1c10 RSI: 0001 RDI: 8800b5e9f008 [ 1267.018360] RBP: 88042d993480 R08: 8800b5e18cc0 R09: [ 1267.018362] R10: 0ca0 R11: 0ca0 R12: 8800b5f15e94 [ 1267.018363] R13: 3a98 R14: 7fffc7b44c40 R15: [ 1267.018365] FS: 7feb4cb05700() GS:88043f62() knlGS: [ 1267.018367] CS: 0010 DS: ES: CR0: 80050033 [ 1267.018369] CR2: 0004 CR3: b50f2000 CR4: 000407e0 [ 1267.018369] Stack: [ 1267.018372] 811c6334 fffc 88042d993480 fffc [ 1267.018375] a0532d73 0ca0 fff4 88042562b440 [ 1267.018378] 88042d853000 1000 a05348f3 c9001518b2a8 [ 1267.018378] Call Trace: [ 1267.018383] [811c6334] ? blk_mq_free_request+0x37/0x48 [ 1267.018388] [a0532d73] ? __nvme_submit_admin_cmd+0x7f/0x8a [nvme] [ 1267.018392] [a05348f3] ? nvme_user_admin_cmd+0x144/0x1b1 [nvme] [ 1267.018397] [a053497d] ? nvme_dev_ioctl+0x1d/0x2b [nvme] [ 1267.018399] [81125916] ? do_vfs_ioctl+0x3f2/0x43b [ 1267.018402] [8105ca6b] ? finish_task_switch+0x84/0xc4 [ 1267.018406] [81394a8c] ? __schedule+0x45c/0x4f0 [ 1267.018408] [811259ad] ? SyS_ioctl+0x4e/0x7d [ 1267.018411] [8139c6d2] ? system_call_fastpath+0x16/0x1b [ 1267.018435] Code: 8b 4a 38 48 39 4e 38 72 12 74 06 b8 01 00 00 00 c3 48 8b 4a 60 48 39 4e 60 73 f0 c3 66 66 66 66 90 48 8b 87 e0 00 00 00 48 63 f6 8b 14 b0 48 8b 87 f8 00 00 00 48 8b 04 d0 c3 89 ff f0 48 0f ab [ 1267.018438] RIP [811c51b8] blk_mq_map_queue+0xf/0x1e [ 1267.018439] RSP 8800b5f15db0 [ 1267.018440] CR2: 0004 [ 1267.018443] ---[ end trace 1e244ae5f3ceb23b ]---
Re: [PATCH v5] conversion to blk-mq
Depending on the timing, it might fail in alloc instead of free: Jun 2 16:45:40 kbgrz1 kernel: [ 265.421243] NULL pointer dereference at (null) Jun 2 16:45:40 kbgrz1 kernel: [ 265.434284] PGD 429acf067 PUD 42ce28067 PMD 0 Jun 2 16:45:40 kbgrz1 kernel: [ 265.439565] Oops: [#1] SMP Jun 2 16:45:40 kbgrz1 kernel: [ 265.443413] Modules linked in: nvme parport_pc ppdev lp parport dlm sctp libcrc32c configfs nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc md4 hmac cifs bridge stp llc jfs joydev hid_generic usbhid hid loop md_mod x86_pkg_temp_thermal coretemp kvm_intel kvm iTCO_wdt iTCO_vendor_support crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd ehci_pci ehci_hcd microcode usbcore pcspkr ioatdma lpc_ich usb_common mfd_core i2c_i801 evdev wmi acpi_cpufreq tpm_tis ipmi_si tpm ipmi_msghandler processor thermal_sys button ext4 crc16 jbd2 mbcache dm_mod nbd sg sd_mod sr_mod crc_t10dif cdrom crct10dif_common isci libsas igb ahci scsi_transport_sas libahci ptp libata pps_core i2c_algo_bit i2c_core scsi_mod dca Jun 2 16:45:40 kbgrz1 kernel: [ 265.526398] CPU: 4 PID: 5454 Comm: nvme_id_ctrl Not tainted 3.15.0-rc1+ #2 Jun 2 16:45:40 kbgrz1 kernel: [ 265.534181] Hardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013 Jun 2 16:45:40 kbgrz1 kernel: [ 265.545789] task: 88042e418390 ti: 8804283e6000 task.ti: 8804283e6000 Jun 2 16:45:40 kbgrz1 kernel: [ 265.554270] RIP: 0010:[811c51b8] [811c51b8] blk_mq_map_queue+0xf/0x1e Jun 2 16:45:40 kbgrz1 kernel: [ 265.563720] RSP: 0018:8804283e7d80 EFLAGS: 00010286 Jun 2 16:45:40 kbgrz1 kernel: [ 265.569755] RAX: RBX: 880035a06008 RCX: 000f4240 Jun 2 16:45:40 kbgrz1 kernel: [ 265.577830] RDX: 0028a360 RSI: RDI: 880035a06008 Jun 2 16:45:40 kbgrz1 kernel: [ 265.585904] RBP: 88043f68 R08: R09: 1000 Jun 2 16:45:40 kbgrz1 kernel: [ 265.593979] R10: 1000 R11: 0410 R12: 00d0 Jun 2 16:45:40 kbgrz1 kernel: [ 265.602053] R13: 0001 R14: R15: Jun 2 16:45:40 kbgrz1 kernel: [ 265.610134] FS: 7f74b8bc9700() GS:88043f68() knlGS: Jun 2 16:45:40 kbgrz1 kernel: [ 265.619303] CS: 0010 DS: ES: CR0: 80050033 Jun 2 16:45:40 kbgrz1 kernel: [ 265.625824] CR2: CR3: 0004292c4000 CR4: 000407e0 Jun 2 16:45:40 kbgrz1 kernel: [ 265.633889] Stack: Jun 2 16:45:40 kbgrz1 kernel: [ 265.636236] 811c689f fff4 8804283e7e10 Jun 2 16:45:40 kbgrz1 kernel: [ 265.644949] 8804283e7e94 007d 7fff4f8a73b0 Jun 2 16:45:40 kbgrz1 kernel: [ 265.653653] a055acc7 0246 2d0aaec0 88042d0aaec0 Jun 2 16:45:40 kbgrz1 kernel: [ 265.662358] Call Trace: Jun 2 16:45:40 kbgrz1 kernel: [ 265.665194] [811c689f] ? blk_mq_alloc_request+0x54/0xd5 Jun 2 16:45:40 kbgrz1 kernel: [ 265.672217] [a055acc7] ? __nvme_submit_admin_cmd+0x2d/0x68 [nvme] Jun 2 16:45:40 kbgrz1 kernel: [ 265.680196] [a055c879] ? nvme_user_admin_cmd+0x144/0x1b1 [nvme] Jun 2 16:45:40 kbgrz1 kernel: [ 265.687987] [a055c903] ? nvme_dev_ioctl+0x1d/0x2b [nvme] Jun 2 16:45:40 kbgrz1 kernel: [ 265.695107] [81125916] ? do_vfs_ioctl+0x3f2/0x43b Jun 2 16:45:40 kbgrz1 kernel: [ 265.701547] [8105ca6b] ? finish_task_switch+0x84/0xc4 Jun 2 16:45:40 kbgrz1 kernel: [ 265.708382] [81394a8c] ? __schedule+0x45c/0x4f0 Jun 2 16:45:40 kbgrz1 kernel: [ 265.714603] [811259ad] ? SyS_ioctl+0x4e/0x7d Jun 2 16:45:40 kbgrz1 kernel: [ 265.720555] [8139c6d2] ? system_call_fastpath+0x16/0x1b Jun 2 16:45:40 kbgrz1 kernel: [ 265.727560] Code: 8b 4a 38 48 39 4e 38 72 12 74 06 b8 01 00 00 00 c3 48 8b 4a 60 48 39 4e 60 73 f0 c3 66 66 66 66 90 48 8b 87 e0 00 00 00 48 63 f6 8b 14 b0 48 8b 87 f8 00 00 00 48 8b 04 d0 c3 89 ff f0 48 0f ab Jun 2 16:45:40 kbgrz1 kernel: [ 265.760706] RSP 8804283e7d80 Jun 2 16:45:40 kbgrz1 kernel: [ 265.764705] CR2: Jun 2 16:45:40 kbgrz1 kernel: [ 265.768531] ---[ end trace 785048a51785f51e ]--- On Mon, 2 Jun 2014, Keith Busch wrote: On Mon, 2 Jun 2014, Matias Bjørling wrote: Hi Matthew and Keith, Here is an updated patch with the feedback from the previous days. It's against Jens' for-3.16/core tree. You may use the nvmemq_wip_review branch at: I'm testing this on my normal hardware now. As I feared, hot removal doesn't work while the device is actively being managed. Here's the oops: [ 1267.018283] BUG: unable to handle kernel NULL pointer dereference at 0004 [ 1267.018288] IP: [811c51b8] blk_mq_map_queue+0xf/0x1e [
