RE: [PATCH 1/3] blk-mq: Allow PCI vector offset for mapping queues
> -Original Message- > From: linux-scsi-ow...@vger.kernel.org [mailto:linux-scsi- > ow...@vger.kernel.org] On Behalf Of Keith Busch > Sent: Tuesday, March 27, 2018 10:39 AM > To: Linux NVMe <linux-n...@lists.infradead.org>; Linux Block bl...@vger.kernel.org> > Cc: Christoph Hellwig <h...@lst.de>; Sagi Grimberg <s...@grimberg.me>; > Jianchao Wang <jianchao.w.w...@oracle.com>; Ming Lei > <ming@redhat.com>; Jens Axboe <ax...@kernel.dk>; Keith Busch > <keith.bu...@intel.com>; Don Brace <don.br...@microsemi.com>; qla2xxx- > upstr...@qlogic.com; linux-s...@vger.kernel.org > Subject: [PATCH 1/3] blk-mq: Allow PCI vector offset for mapping queues > > EXTERNAL EMAIL > > > The PCI interrupt vectors intended to be associated with a queue may > not start at 0; a driver may allocate pre_vectors for special use. This > patch adds an offset parameter so blk-mq may find the intended affinity > mask and updates all drivers using this API accordingly. > > Cc: Don Brace <don.br...@microsemi.com> > Cc: <qla2xxx-upstr...@qlogic.com> > Cc: <linux-s...@vger.kernel.org> > Signed-off-by: Keith Busch <keith.bu...@intel.com> > --- > v1 -> v2: > > Update blk-mq API directly instead of chaining a default parameter to > a new API, and update all drivers accordingly. > > block/blk-mq-pci.c| 6 -- > drivers/nvme/host/pci.c | 2 +- > drivers/scsi/qla2xxx/qla_os.c | 2 +- > drivers/scsi/smartpqi/smartpqi_init.c | 2 +- > include/linux/blk-mq-pci.h| 3 ++- > 5 files changed, 9 insertions(+), 6 deletions(-) > > diff --git a/drivers/scsi/smartpqi/smartpqi_init.c > b/drivers/scsi/smartpqi/smartpqi_init.c > index b2880c7709e6..10c94011c8a8 100644 > --- a/drivers/scsi/smartpqi/smartpqi_init.c > +++ b/drivers/scsi/smartpqi/smartpqi_init.c > @@ -5348,7 +5348,7 @@ static int pqi_map_queues(struct Scsi_Host *shost) > { > struct pqi_ctrl_info *ctrl_info = shost_to_hba(shost); > > - return blk_mq_pci_map_queues(>tag_set, ctrl_info->pci_dev); > + return blk_mq_pci_map_queues(>tag_set, ctrl_info->pci_dev, 0); > } > Acked-by: Don Brace <don.br...@microsemi.com>
Re: [PATCH 1/3] blk-mq: Allow PCI vector offset for mapping queues
On 3/27/18 9:39 AM, Keith Busch wrote: > The PCI interrupt vectors intended to be associated with a queue may > not start at 0; a driver may allocate pre_vectors for special use. This > patch adds an offset parameter so blk-mq may find the intended affinity > mask and updates all drivers using this API accordingly. Looks good to me, I'll queue it up for 4.17. -- Jens Axboe
Re: [PATCH 1/3] blk-mq: Allow PCI vector offset for mapping queues
On Tue, Mar 27, 2018 at 09:39:06AM -0600, Keith Busch wrote: > The PCI interrupt vectors intended to be associated with a queue may > not start at 0; a driver may allocate pre_vectors for special use. This > patch adds an offset parameter so blk-mq may find the intended affinity > mask and updates all drivers using this API accordingly. > > Cc: Don Brace> Cc: > Cc: > Signed-off-by: Keith Busch > --- > v1 -> v2: > > Update blk-mq API directly instead of chaining a default parameter to > a new API, and update all drivers accordingly. > > block/blk-mq-pci.c| 6 -- > drivers/nvme/host/pci.c | 2 +- > drivers/scsi/qla2xxx/qla_os.c | 2 +- > drivers/scsi/smartpqi/smartpqi_init.c | 2 +- > include/linux/blk-mq-pci.h| 3 ++- > 5 files changed, 9 insertions(+), 6 deletions(-) > > diff --git a/block/blk-mq-pci.c b/block/blk-mq-pci.c > index 76944e3271bf..e233996bb76f 100644 > --- a/block/blk-mq-pci.c > +++ b/block/blk-mq-pci.c > @@ -21,6 +21,7 @@ > * blk_mq_pci_map_queues - provide a default queue mapping for PCI device > * @set: tagset to provide the mapping for > * @pdev:PCI device associated with @set. > + * @offset: Offset to use for the pci irq vector > * > * This function assumes the PCI device @pdev has at least as many available > * interrupt vectors as @set has queues. It will then query the vector > @@ -28,13 +29,14 @@ > * that maps a queue to the CPUs that have irq affinity for the corresponding > * vector. > */ > -int blk_mq_pci_map_queues(struct blk_mq_tag_set *set, struct pci_dev *pdev) > +int blk_mq_pci_map_queues(struct blk_mq_tag_set *set, struct pci_dev *pdev, > + int offset) > { > const struct cpumask *mask; > unsigned int queue, cpu; > > for (queue = 0; queue < set->nr_hw_queues; queue++) { > - mask = pci_irq_get_affinity(pdev, queue); > + mask = pci_irq_get_affinity(pdev, queue + offset); > if (!mask) > goto fallback; > > diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c > index cef5ce851a92..e3b9efca0571 100644 > --- a/drivers/nvme/host/pci.c > +++ b/drivers/nvme/host/pci.c > @@ -414,7 +414,7 @@ static int nvme_pci_map_queues(struct blk_mq_tag_set *set) > { > struct nvme_dev *dev = set->driver_data; > > - return blk_mq_pci_map_queues(set, to_pci_dev(dev->dev)); > + return blk_mq_pci_map_queues(set, to_pci_dev(dev->dev), 0); > } > > /** > diff --git a/drivers/scsi/qla2xxx/qla_os.c b/drivers/scsi/qla2xxx/qla_os.c > index 12ee6e02d146..2c705f3dd265 100644 > --- a/drivers/scsi/qla2xxx/qla_os.c > +++ b/drivers/scsi/qla2xxx/qla_os.c > @@ -6805,7 +6805,7 @@ static int qla2xxx_map_queues(struct Scsi_Host *shost) > if (USER_CTRL_IRQ(vha->hw)) > rc = blk_mq_map_queues(>tag_set); > else > - rc = blk_mq_pci_map_queues(>tag_set, vha->hw->pdev); > + rc = blk_mq_pci_map_queues(>tag_set, vha->hw->pdev, 0); > return rc; > } > > diff --git a/drivers/scsi/smartpqi/smartpqi_init.c > b/drivers/scsi/smartpqi/smartpqi_init.c > index b2880c7709e6..10c94011c8a8 100644 > --- a/drivers/scsi/smartpqi/smartpqi_init.c > +++ b/drivers/scsi/smartpqi/smartpqi_init.c > @@ -5348,7 +5348,7 @@ static int pqi_map_queues(struct Scsi_Host *shost) > { > struct pqi_ctrl_info *ctrl_info = shost_to_hba(shost); > > - return blk_mq_pci_map_queues(>tag_set, ctrl_info->pci_dev); > + return blk_mq_pci_map_queues(>tag_set, ctrl_info->pci_dev, 0); > } > > static int pqi_getpciinfo_ioctl(struct pqi_ctrl_info *ctrl_info, > diff --git a/include/linux/blk-mq-pci.h b/include/linux/blk-mq-pci.h > index 6338551e0fb9..9f4c17f0d2d8 100644 > --- a/include/linux/blk-mq-pci.h > +++ b/include/linux/blk-mq-pci.h > @@ -5,6 +5,7 @@ > struct blk_mq_tag_set; > struct pci_dev; > > -int blk_mq_pci_map_queues(struct blk_mq_tag_set *set, struct pci_dev *pdev); > +int blk_mq_pci_map_queues(struct blk_mq_tag_set *set, struct pci_dev *pdev, > + int offset); > > #endif /* _LINUX_BLK_MQ_PCI_H */ > -- > 2.14.3 > Reviewed-by: Ming Lei -- Ming
[PATCH 1/3] blk-mq: Allow PCI vector offset for mapping queues
The PCI interrupt vectors intended to be associated with a queue may not start at 0; a driver may allocate pre_vectors for special use. This patch adds an offset parameter so blk-mq may find the intended affinity mask and updates all drivers using this API accordingly. Cc: Don BraceCc: Cc: Signed-off-by: Keith Busch --- v1 -> v2: Update blk-mq API directly instead of chaining a default parameter to a new API, and update all drivers accordingly. block/blk-mq-pci.c| 6 -- drivers/nvme/host/pci.c | 2 +- drivers/scsi/qla2xxx/qla_os.c | 2 +- drivers/scsi/smartpqi/smartpqi_init.c | 2 +- include/linux/blk-mq-pci.h| 3 ++- 5 files changed, 9 insertions(+), 6 deletions(-) diff --git a/block/blk-mq-pci.c b/block/blk-mq-pci.c index 76944e3271bf..e233996bb76f 100644 --- a/block/blk-mq-pci.c +++ b/block/blk-mq-pci.c @@ -21,6 +21,7 @@ * blk_mq_pci_map_queues - provide a default queue mapping for PCI device * @set: tagset to provide the mapping for * @pdev: PCI device associated with @set. + * @offset:Offset to use for the pci irq vector * * This function assumes the PCI device @pdev has at least as many available * interrupt vectors as @set has queues. It will then query the vector @@ -28,13 +29,14 @@ * that maps a queue to the CPUs that have irq affinity for the corresponding * vector. */ -int blk_mq_pci_map_queues(struct blk_mq_tag_set *set, struct pci_dev *pdev) +int blk_mq_pci_map_queues(struct blk_mq_tag_set *set, struct pci_dev *pdev, + int offset) { const struct cpumask *mask; unsigned int queue, cpu; for (queue = 0; queue < set->nr_hw_queues; queue++) { - mask = pci_irq_get_affinity(pdev, queue); + mask = pci_irq_get_affinity(pdev, queue + offset); if (!mask) goto fallback; diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index cef5ce851a92..e3b9efca0571 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -414,7 +414,7 @@ static int nvme_pci_map_queues(struct blk_mq_tag_set *set) { struct nvme_dev *dev = set->driver_data; - return blk_mq_pci_map_queues(set, to_pci_dev(dev->dev)); + return blk_mq_pci_map_queues(set, to_pci_dev(dev->dev), 0); } /** diff --git a/drivers/scsi/qla2xxx/qla_os.c b/drivers/scsi/qla2xxx/qla_os.c index 12ee6e02d146..2c705f3dd265 100644 --- a/drivers/scsi/qla2xxx/qla_os.c +++ b/drivers/scsi/qla2xxx/qla_os.c @@ -6805,7 +6805,7 @@ static int qla2xxx_map_queues(struct Scsi_Host *shost) if (USER_CTRL_IRQ(vha->hw)) rc = blk_mq_map_queues(>tag_set); else - rc = blk_mq_pci_map_queues(>tag_set, vha->hw->pdev); + rc = blk_mq_pci_map_queues(>tag_set, vha->hw->pdev, 0); return rc; } diff --git a/drivers/scsi/smartpqi/smartpqi_init.c b/drivers/scsi/smartpqi/smartpqi_init.c index b2880c7709e6..10c94011c8a8 100644 --- a/drivers/scsi/smartpqi/smartpqi_init.c +++ b/drivers/scsi/smartpqi/smartpqi_init.c @@ -5348,7 +5348,7 @@ static int pqi_map_queues(struct Scsi_Host *shost) { struct pqi_ctrl_info *ctrl_info = shost_to_hba(shost); - return blk_mq_pci_map_queues(>tag_set, ctrl_info->pci_dev); + return blk_mq_pci_map_queues(>tag_set, ctrl_info->pci_dev, 0); } static int pqi_getpciinfo_ioctl(struct pqi_ctrl_info *ctrl_info, diff --git a/include/linux/blk-mq-pci.h b/include/linux/blk-mq-pci.h index 6338551e0fb9..9f4c17f0d2d8 100644 --- a/include/linux/blk-mq-pci.h +++ b/include/linux/blk-mq-pci.h @@ -5,6 +5,7 @@ struct blk_mq_tag_set; struct pci_dev; -int blk_mq_pci_map_queues(struct blk_mq_tag_set *set, struct pci_dev *pdev); +int blk_mq_pci_map_queues(struct blk_mq_tag_set *set, struct pci_dev *pdev, + int offset); #endif /* _LINUX_BLK_MQ_PCI_H */ -- 2.14.3
Re: [PATCH 1/3] blk-mq: Allow PCI vector offset for mapping queues
On Fri, Mar 23, 2018 at 04:19:21PM -0600, Keith Busch wrote: > The PCI interrupt vectors intended to be associated with a queue may > not start at 0. This patch adds an offset parameter so blk-mq may find > the intended affinity mask. The default value is 0 so existing drivers > that don't care about this parameter don't need to change. > > Signed-off-by: Keith Busch> --- > block/blk-mq-pci.c | 12 ++-- > include/linux/blk-mq-pci.h | 2 ++ > 2 files changed, 12 insertions(+), 2 deletions(-) > > diff --git a/block/blk-mq-pci.c b/block/blk-mq-pci.c > index 76944e3271bf..1040a7705c13 100644 > --- a/block/blk-mq-pci.c > +++ b/block/blk-mq-pci.c > @@ -21,6 +21,7 @@ > * blk_mq_pci_map_queues - provide a default queue mapping for PCI device > * @set: tagset to provide the mapping for > * @pdev:PCI device associated with @set. > + * @offset: PCI irq starting vector offset > * > * This function assumes the PCI device @pdev has at least as many available > * interrupt vectors as @set has queues. It will then query the vector > @@ -28,13 +29,14 @@ > * that maps a queue to the CPUs that have irq affinity for the corresponding > * vector. > */ > -int blk_mq_pci_map_queues(struct blk_mq_tag_set *set, struct pci_dev *pdev) > +int __blk_mq_pci_map_queues(struct blk_mq_tag_set *set, struct pci_dev *pdev, > + int offset) Can you just change the blk_mq_pci_map_queues prototype instead?
Re: [PATCH 1/3] blk-mq: Allow PCI vector offset for mapping queues
On Mon, Mar 26, 2018 at 09:50:38AM +0800, Ming Lei wrote: > > Given no many callers of blk_mq_pci_map_queues(), I suggest to add the > parameter of 'offset' to this API directly, then people may keep the > '.pre_vectors' stuff in mind, and avoid to misuse it. Yeah, I think I have to agree. I was trying really hard to not touch other drivers, but the concept doesn't seem odd enough to justify hiding it behind a default parameter. I'll send v2 tomorrow if there's no other feedback.
Re: [PATCH 1/3] blk-mq: Allow PCI vector offset for mapping queues
On Sat, Mar 24, 2018 at 09:55:49PM +0800, jianchao.wang wrote: > Maybe we could provide a callback parameter for __blk_mq_pci_map_queues which > give the mapping from hctx queue num to device-relative interrupt vector > index. If a driver's mapping is so complicated as to require a special per-hctx callback, it'd be just as easy to implement the mapping in that driver's blk_mq_ops' 'map_queues' directly.
Re: [PATCH 1/3] blk-mq: Allow PCI vector offset for mapping queues
On Fri, Mar 23, 2018 at 04:19:21PM -0600, Keith Busch wrote: > The PCI interrupt vectors intended to be associated with a queue may > not start at 0. This patch adds an offset parameter so blk-mq may find > the intended affinity mask. The default value is 0 so existing drivers > that don't care about this parameter don't need to change. > > Signed-off-by: Keith Busch> --- > block/blk-mq-pci.c | 12 ++-- > include/linux/blk-mq-pci.h | 2 ++ > 2 files changed, 12 insertions(+), 2 deletions(-) > > diff --git a/block/blk-mq-pci.c b/block/blk-mq-pci.c > index 76944e3271bf..1040a7705c13 100644 > --- a/block/blk-mq-pci.c > +++ b/block/blk-mq-pci.c > @@ -21,6 +21,7 @@ > * blk_mq_pci_map_queues - provide a default queue mapping for PCI device > * @set: tagset to provide the mapping for > * @pdev:PCI device associated with @set. > + * @offset: PCI irq starting vector offset > * > * This function assumes the PCI device @pdev has at least as many available > * interrupt vectors as @set has queues. It will then query the vector > @@ -28,13 +29,14 @@ > * that maps a queue to the CPUs that have irq affinity for the corresponding > * vector. > */ > -int blk_mq_pci_map_queues(struct blk_mq_tag_set *set, struct pci_dev *pdev) > +int __blk_mq_pci_map_queues(struct blk_mq_tag_set *set, struct pci_dev *pdev, > + int offset) > { > const struct cpumask *mask; > unsigned int queue, cpu; > > for (queue = 0; queue < set->nr_hw_queues; queue++) { > - mask = pci_irq_get_affinity(pdev, queue); > + mask = pci_irq_get_affinity(pdev, queue + offset); > if (!mask) > goto fallback; > > @@ -50,4 +52,10 @@ int blk_mq_pci_map_queues(struct blk_mq_tag_set *set, > struct pci_dev *pdev) > set->mq_map[cpu] = 0; > return 0; > } > +EXPORT_SYMBOL_GPL(__blk_mq_pci_map_queues); > + > +int blk_mq_pci_map_queues(struct blk_mq_tag_set *set, struct pci_dev *pdev) > +{ > + return __blk_mq_pci_map_queues(set, pdev, 0); > +} > EXPORT_SYMBOL_GPL(blk_mq_pci_map_queues); > diff --git a/include/linux/blk-mq-pci.h b/include/linux/blk-mq-pci.h > index 6338551e0fb9..5a92ecdbd78e 100644 > --- a/include/linux/blk-mq-pci.h > +++ b/include/linux/blk-mq-pci.h > @@ -5,6 +5,8 @@ > struct blk_mq_tag_set; > struct pci_dev; > > +int __blk_mq_pci_map_queues(struct blk_mq_tag_set *set, struct pci_dev *pdev, > + int offset); > int blk_mq_pci_map_queues(struct blk_mq_tag_set *set, struct pci_dev *pdev); > > #endif /* _LINUX_BLK_MQ_PCI_H */ > -- > 2.14.3 > Given no many callers of blk_mq_pci_map_queues(), I suggest to add the parameter of 'offset' to this API directly, then people may keep the '.pre_vectors' stuff in mind, and avoid to misuse it. Thanks, Ming
Re: [PATCH 1/3] blk-mq: Allow PCI vector offset for mapping queues
Hi Keith Thanks for your time and patch for this. On 03/24/2018 06:19 AM, Keith Busch wrote: > The PCI interrupt vectors intended to be associated with a queue may > not start at 0. This patch adds an offset parameter so blk-mq may find > the intended affinity mask. The default value is 0 so existing drivers > that don't care about this parameter don't need to change. > > Signed-off-by: Keith Busch> --- > block/blk-mq-pci.c | 12 ++-- > include/linux/blk-mq-pci.h | 2 ++ > 2 files changed, 12 insertions(+), 2 deletions(-) > > diff --git a/block/blk-mq-pci.c b/block/blk-mq-pci.c > index 76944e3271bf..1040a7705c13 100644 > --- a/block/blk-mq-pci.c > +++ b/block/blk-mq-pci.c > @@ -21,6 +21,7 @@ > * blk_mq_pci_map_queues - provide a default queue mapping for PCI device > * @set: tagset to provide the mapping for > * @pdev:PCI device associated with @set. > + * @offset: PCI irq starting vector offset > * > * This function assumes the PCI device @pdev has at least as many available > * interrupt vectors as @set has queues. It will then query the vector > @@ -28,13 +29,14 @@ > * that maps a queue to the CPUs that have irq affinity for the corresponding > * vector. > */ > -int blk_mq_pci_map_queues(struct blk_mq_tag_set *set, struct pci_dev *pdev) > +int __blk_mq_pci_map_queues(struct blk_mq_tag_set *set, struct pci_dev *pdev, > + int offset) > { > const struct cpumask *mask; > unsigned int queue, cpu; > > for (queue = 0; queue < set->nr_hw_queues; queue++) { > - mask = pci_irq_get_affinity(pdev, queue); > + mask = pci_irq_get_affinity(pdev, queue + offset); > if (!mask) > goto fallback; > Maybe we could provide a callback parameter for __blk_mq_pci_map_queues which give the mapping from hctx queue num to device-relative interrupt vector index. Thanks Jianchao
[PATCH 1/3] blk-mq: Allow PCI vector offset for mapping queues
The PCI interrupt vectors intended to be associated with a queue may not start at 0. This patch adds an offset parameter so blk-mq may find the intended affinity mask. The default value is 0 so existing drivers that don't care about this parameter don't need to change. Signed-off-by: Keith Busch--- block/blk-mq-pci.c | 12 ++-- include/linux/blk-mq-pci.h | 2 ++ 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/block/blk-mq-pci.c b/block/blk-mq-pci.c index 76944e3271bf..1040a7705c13 100644 --- a/block/blk-mq-pci.c +++ b/block/blk-mq-pci.c @@ -21,6 +21,7 @@ * blk_mq_pci_map_queues - provide a default queue mapping for PCI device * @set: tagset to provide the mapping for * @pdev: PCI device associated with @set. + * @offset:PCI irq starting vector offset * * This function assumes the PCI device @pdev has at least as many available * interrupt vectors as @set has queues. It will then query the vector @@ -28,13 +29,14 @@ * that maps a queue to the CPUs that have irq affinity for the corresponding * vector. */ -int blk_mq_pci_map_queues(struct blk_mq_tag_set *set, struct pci_dev *pdev) +int __blk_mq_pci_map_queues(struct blk_mq_tag_set *set, struct pci_dev *pdev, + int offset) { const struct cpumask *mask; unsigned int queue, cpu; for (queue = 0; queue < set->nr_hw_queues; queue++) { - mask = pci_irq_get_affinity(pdev, queue); + mask = pci_irq_get_affinity(pdev, queue + offset); if (!mask) goto fallback; @@ -50,4 +52,10 @@ int blk_mq_pci_map_queues(struct blk_mq_tag_set *set, struct pci_dev *pdev) set->mq_map[cpu] = 0; return 0; } +EXPORT_SYMBOL_GPL(__blk_mq_pci_map_queues); + +int blk_mq_pci_map_queues(struct blk_mq_tag_set *set, struct pci_dev *pdev) +{ + return __blk_mq_pci_map_queues(set, pdev, 0); +} EXPORT_SYMBOL_GPL(blk_mq_pci_map_queues); diff --git a/include/linux/blk-mq-pci.h b/include/linux/blk-mq-pci.h index 6338551e0fb9..5a92ecdbd78e 100644 --- a/include/linux/blk-mq-pci.h +++ b/include/linux/blk-mq-pci.h @@ -5,6 +5,8 @@ struct blk_mq_tag_set; struct pci_dev; +int __blk_mq_pci_map_queues(struct blk_mq_tag_set *set, struct pci_dev *pdev, + int offset); int blk_mq_pci_map_queues(struct blk_mq_tag_set *set, struct pci_dev *pdev); #endif /* _LINUX_BLK_MQ_PCI_H */ -- 2.14.3