Re: [PATCH v2 1/4] MicroSemi Switchtec management interface driver
Hey Bjorn, Thanks for the thorough review. It definitely helped a lot to make the code as good as it can be. I've made all of the changes you suggested. I'm just going to do a bit more testing and then post a v4. See my responses to all of your feedback bellow. Logan On 23/02/17 05:35 PM, Bjorn Helgaas wrote: > Would it make any sense to integrate this with the perf tool? It > looks like switchtec-user measures bandwidth and latency, which sounds > sort of perf-ish. That would be cool but lets file that under potential future work. This driver also enables more than bandwidth and latency so it's still required for us. >> +MODULE_DESCRIPTION("Microsemi Switchtec(tm) PCI-E Management Driver"); > > Nit: s/PCI-E/PCIe/ > Fixed. >> +MODULE_VERSION("0.1"); >> +MODULE_LICENSE("GPL"); >> +MODULE_AUTHOR("Microsemi Corporation"); >> + >> +static int max_devices = 16; > > This static initialization is slightly misleading because > switchtec_init() immediately sets max_devices to at least 256. Oops copied that from dax without thinking. I've just removed the max() call in the init function. >> +atomic_t event_cnt; > > Why is this atomic_t? You only do an atomic_set() (in stdev_create()) > and atomic_reads() -- no writes other than the initial one. So I'm > not sure what value atomic_t brings. If you looked at a following patch in the series you'd see that there's an atomic_inc in the ISR. The splitting of the series wasn't as precise as it maybe should have been and thus this became a bit confusing. >> +memcpy_fromio(stuser->data, >mmio_mrpc->output_data, >> + sizeof(stuser->data)); > > Apparently you always get 1K of data back, no matter what? Yes, sort of unfortunately. Because these transactions can occur before the user actually does the read, we don't know how much data the user wants. This may be a performance improvement in the future (ie. if the user reads before the MRPC transaction comes back, then only read the requested amount). But we will leave it this way for now. >> +if (!stdev_is_alive(stdev)) >> +return -ENXIO; > > What exactly does this protect against? Is it OK if stdev_is_alive() > becomes false immediately after we check? Yup, you're right: that's racy. Turns out cleanup is hard and I've learned a lot even in the short time since I wrote this code. I've gotten rid of stdev_is_alive and moved the pci cleanup code into the device's release function so they won't occur until the last user closes the cdev. I think that should work better but please let me know if you see an issue with this. >> + >> +if (size < sizeof(stuser->cmd) || >> +size > sizeof(stuser->cmd) + SWITCHTEC_MRPC_PAYLOAD_SIZE) > > I think I would use "sizeof(stuser->data)" instead of > SWITCHTEC_MRPC_PAYLOAD_SIZE so it matches what mrpc_complete_cmd() > does. Same below in switchtec_dev_read(). Ok. > mrpc_mutex doesn't protect the stuser things, does it? If not, you > could do this without the gotos. And I think it's a little easier to > be confident there are no buffer overruns: I was using the mutex to protect stuser->state as well. But I've made your changes and just adjusted stuser->state to be atomic and I think this should be just as good. >> +static int switchtec_init_msix_isr(struct switchtec_dev *stdev) >> +{ >> +struct pci_dev *pdev = stdev->pdev; >> +int rc, i, msix_count, node; >> + >> +node = dev_to_node(>dev); > > Unused? Yup, I've removed it. >> +for (i = 0; i < ARRAY_SIZE(stdev->msix); ++i) >> +stdev->msix[i].entry = i; >> + >> +msix_count = pci_enable_msix_range(pdev, stdev->msix, 1, >> + ARRAY_SIZE(stdev->msix)); >> +if (msix_count < 0) >> +return msix_count; > > Maybe this could be simplified by using pci_alloc_irq_vectors()? Yup! I wasn't aware of that function. It's a _huge_ improvement. Thanks. Still I'd really appreciate a quick re-review after I post v4 just to make sure I got it correct. >> +stdev->event_irq = ioread32(>mmio_part_cfg->vep_vector_number); >> +if (stdev->event_irq < 0 || stdev->event_irq >= msix_count) { >> +rc = -EFAULT; >> +goto err_msix_request; >> +} > > Not sure what this is doing, but you immediately overwrite > stdev->event_irq below. If you're using it as just a temporary above > for doing some validation, I would just use a local variable to avoid > the connection with stdev->event_irq. Yes, I made this temporary. >> +stdev->event_irq = stdev->msix[stdev->event_irq].vector; > > Oh, I guess you allocate several MSI-X vectors, but you only actually > use one of them? Why is that? I was confused about why you allocate > several vectors, but there's only one devm_request_irq() call below. The event_irq is configurable in hardware and won't necessarily be the first irq available. (It should always be in the first four.) As I understand it, we need to allocate all of
Re: [PATCH v2 1/4] MicroSemi Switchtec management interface driver
[+cc Peter, Ingo, Arnaldo, Alexander, Christoph] On Thu, Feb 02, 2017 at 11:06:00AM -0700, Logan Gunthorpe wrote: > Microsemi's "Switchtec" line of PCI switch devices is already well > supported by the kernel with standard PCI switch drivers. However, the > Switchtec device advertises a special management endpoint with a separate > PCI function address and class code. This endpoint enables some additional > functionality which includes: > > * Packet and Byte Counters > * Switch Firmware Upgrades > * Event and Error logs > * Querying port link status > * Custom user firmware commands Would it make any sense to integrate this with the perf tool? It looks like switchtec-user measures bandwidth and latency, which sounds sort of perf-ish. > This patch introduces the switchtec kernel module which provides > PCI driver that exposes a char device. The char device provides > userspace access to this interface through read, write and (optionally) > poll calls. > > A userspace tool and library which utilizes this interface is available > at [1]. This tool takes inspiration (and borrows some code) from > nvme-cli [2]. The tool is largely complete at this time but additional > features may be added in the future. > > [1] https://github.com/sbates130272/switchtec-user > [2] https://github.com/linux-nvme/nvme-cli > > Signed-off-by: Logan Gunthorpe> Signed-off-by: Stephen Bates > --- > MAINTAINERS|8 + > drivers/pci/Kconfig|1 + > drivers/pci/Makefile |1 + > drivers/pci/switch/Kconfig | 13 + > drivers/pci/switch/Makefile|1 + > drivers/pci/switch/switchtec.c | 1028 > > 6 files changed, 1052 insertions(+) > create mode 100644 drivers/pci/switch/Kconfig > create mode 100644 drivers/pci/switch/Makefile > create mode 100644 drivers/pci/switch/switchtec.c > > diff --git a/MAINTAINERS b/MAINTAINERS > index 5f10c28..9c35198 100644 > --- a/MAINTAINERS > +++ b/MAINTAINERS > @@ -9507,6 +9507,14 @@ S: Maintained > F: Documentation/devicetree/bindings/pci/aardvark-pci.txt > F: drivers/pci/host/pci-aardvark.c > > +PCI DRIVER FOR MICROSEMI SWITCHTEC > +M: Kurt Schwemmer > +M: Stephen Bates > +M: Logan Gunthorpe > +L: linux-...@vger.kernel.org > +S: Maintained > +F: drivers/pci/switch/switchtec* > + > PCI DRIVER FOR NVIDIA TEGRA > M: Thierry Reding > L: linux-te...@vger.kernel.org > diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig > index 6555eb7..f72e8c5 100644 > --- a/drivers/pci/Kconfig > +++ b/drivers/pci/Kconfig > @@ -133,3 +133,4 @@ config PCI_HYPERV > > source "drivers/pci/hotplug/Kconfig" > source "drivers/pci/host/Kconfig" > +source "drivers/pci/switch/Kconfig" > diff --git a/drivers/pci/Makefile b/drivers/pci/Makefile > index 8db5079..15b46dd 100644 > --- a/drivers/pci/Makefile > +++ b/drivers/pci/Makefile > @@ -68,3 +68,4 @@ ccflags-$(CONFIG_PCI_DEBUG) := -DDEBUG > > # PCI host controller drivers > obj-y += host/ > +obj-y += switch/ > diff --git a/drivers/pci/switch/Kconfig b/drivers/pci/switch/Kconfig > new file mode 100644 > index 000..4c49648 > --- /dev/null > +++ b/drivers/pci/switch/Kconfig > @@ -0,0 +1,13 @@ > +menu "PCI switch controller drivers" > + depends on PCI > + > +config PCI_SW_SWITCHTEC > + tristate "MicroSemi Switchtec PCIe Switch Management Driver" > + help > + Enables support for the management interface for the MicroSemi > + Switchtec series of PCIe switches. Supports userspace access > + to submit MRPC commands to the switch via /dev/switchtecX > + devices. See for more > + information. > + > +endmenu > diff --git a/drivers/pci/switch/Makefile b/drivers/pci/switch/Makefile > new file mode 100644 > index 000..37d8cfb > --- /dev/null > +++ b/drivers/pci/switch/Makefile > @@ -0,0 +1 @@ > +obj-$(CONFIG_PCI_SW_SWITCHTEC) += switchtec.o > diff --git a/drivers/pci/switch/switchtec.c b/drivers/pci/switch/switchtec.c > new file mode 100644 > index 000..e4a4294 > --- /dev/null > +++ b/drivers/pci/switch/switchtec.c > @@ -0,0 +1,1028 @@ > +/* > + * Microsemi Switchtec(tm) PCIe Management Driver > + * Copyright (c) 2017, Microsemi Corporation > + * > + * This program is free software; you can redistribute it and/or modify it > + * under the terms and conditions of the GNU General Public License, > + * version 2, as published by the Free Software Foundation. > + * > + * This program is distributed in the hope it will be useful, but WITHOUT > + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or > + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for > + * more details. > + * > + */ > + > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +
Re: [PATCH v2 1/4] MicroSemi Switchtec management interface driver
On 10/02/17 10:09 AM, Greg Kroah-Hartman wrote: > Sure, or just wait for these to be applied to the PCI tree and then send > a follow-on patch. It's up to Bjorn really, as to what he wants. Ok, I sent a working follow-on patch to this thread. @Bjorn: I'm happy to send the patches however you like. Just let me know. If at all possible, we'd really like to see this in for the 4.11 merge window. Thanks, Logan -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 1/4] MicroSemi Switchtec management interface driver
On 10/02/17 09:55 AM, Greg Kroah-Hartman wrote: > Yes, but try it yourself to verify it really is correct :) Yes, of course... turns out it wasn't. I accidentally refereed to dev before I assigned it. I had mainly just wanted your feedback to ensure that switching to device_register made sense. > And it can just be an add-on patch, no need to respin a whole new > version for just that simple change, it doesn't hurt anything as-is, > it's just "not needed". Ok... How should I do that exactly? Attempt to reply to the thread with another patch? Thanks, Logan -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 1/4] MicroSemi Switchtec management interface driver
On Fri, Feb 10, 2017 at 10:03:10AM -0700, Logan Gunthorpe wrote: > > > On 10/02/17 09:55 AM, Greg Kroah-Hartman wrote: > > Yes, but try it yourself to verify it really is correct :) > > Yes, of course... turns out it wasn't. I accidentally refereed to dev > before I assigned it. I had mainly just wanted your feedback to ensure > that switching to device_register made sense. > > > And it can just be an add-on patch, no need to respin a whole new > > version for just that simple change, it doesn't hurt anything as-is, > > it's just "not needed". > > Ok... How should I do that exactly? Attempt to reply to the thread with > another patch? Sure, or just wait for these to be applied to the PCI tree and then send a follow-on patch. It's up to Bjorn really, as to what he wants. thanks, greg k-h -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 1/4] MicroSemi Switchtec management interface driver
On Fri, Feb 10, 2017 at 09:48:37AM -0700, Logan Gunthorpe wrote: > Hey Greg, > > Thanks so much for the review. > > On 10/02/17 07:51 AM, Greg Kroah-Hartman wrote: > > On Thu, Feb 02, 2017 at 11:06:00AM -0700, Logan Gunthorpe wrote: > >> + cdev = >cdev; > >> + cdev_init(cdev, _fops); > >> + cdev->owner = THIS_MODULE; > >> + cdev->kobj.parent = >kobj; > > > > Minor nit, the kobject in a cdev is unlike any other kobject you have > > ever seen, don't mess with it, it's not doing anything like you think it > > is doing. So no need to set the parent field. > > Ok, that makes sense. I'll do a v3 shortly. > > I copied this from drivers/dax/dax.c so when I have a spare moment I'll > submit a patch to remove it from there as well. > > Just to make sure I get this right without extra churn: does this look > correct? > > > cdev = >cdev; > cdev_init(cdev, _fops); > cdev->owner = THIS_MODULE; > > rc = cdev_add(>cdev, dev->devt, 1); > if (rc) > goto err_cdev; > > dev = >dev; > dev->devt = MKDEV(MAJOR(switchtec_devt), minor); > dev->class = switchtec_class; > dev->parent = >dev; > dev->groups = switchtec_device_groups; > dev->release = stdev_release; > dev_set_name(dev, "switchtec%d", minor); > > rc = device_register(dev); > if (rc) { > cdev_del(>cdev); > put_device(dev); > return ERR_PTR(rc); > } > Yes, but try it yourself to verify it really is correct :) And it can just be an add-on patch, no need to respin a whole new version for just that simple change, it doesn't hurt anything as-is, it's just "not needed". thanks, greg k-h -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 1/4] MicroSemi Switchtec management interface driver
Hey Greg, Thanks so much for the review. On 10/02/17 07:51 AM, Greg Kroah-Hartman wrote: > On Thu, Feb 02, 2017 at 11:06:00AM -0700, Logan Gunthorpe wrote: >> +cdev = >cdev; >> +cdev_init(cdev, _fops); >> +cdev->owner = THIS_MODULE; >> +cdev->kobj.parent = >kobj; > > Minor nit, the kobject in a cdev is unlike any other kobject you have > ever seen, don't mess with it, it's not doing anything like you think it > is doing. So no need to set the parent field. Ok, that makes sense. I'll do a v3 shortly. I copied this from drivers/dax/dax.c so when I have a spare moment I'll submit a patch to remove it from there as well. Just to make sure I get this right without extra churn: does this look correct? cdev = >cdev; cdev_init(cdev, _fops); cdev->owner = THIS_MODULE; rc = cdev_add(>cdev, dev->devt, 1); if (rc) goto err_cdev; dev = >dev; dev->devt = MKDEV(MAJOR(switchtec_devt), minor); dev->class = switchtec_class; dev->parent = >dev; dev->groups = switchtec_device_groups; dev->release = stdev_release; dev_set_name(dev, "switchtec%d", minor); rc = device_register(dev); if (rc) { cdev_del(>cdev); put_device(dev); return ERR_PTR(rc); } Thanks, Logan -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 1/4] MicroSemi Switchtec management interface driver
On Thu, Feb 02, 2017 at 11:06:00AM -0700, Logan Gunthorpe wrote: > Microsemi's "Switchtec" line of PCI switch devices is already well > supported by the kernel with standard PCI switch drivers. However, the > Switchtec device advertises a special management endpoint with a separate > PCI function address and class code. This endpoint enables some additional > functionality which includes: > > * Packet and Byte Counters > * Switch Firmware Upgrades > * Event and Error logs > * Querying port link status > * Custom user firmware commands > > This patch introduces the switchtec kernel module which provides > PCI driver that exposes a char device. The char device provides > userspace access to this interface through read, write and (optionally) > poll calls. > > A userspace tool and library which utilizes this interface is available > at [1]. This tool takes inspiration (and borrows some code) from > nvme-cli [2]. The tool is largely complete at this time but additional > features may be added in the future. > > [1] https://github.com/sbates130272/switchtec-user > [2] https://github.com/linux-nvme/nvme-cli > > Signed-off-by: Logan Gunthorpe> Signed-off-by: Stephen Bates Reviewed-by: Greg Kroah-Hartman -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 1/4] MicroSemi Switchtec management interface driver
On Thu, Feb 02, 2017 at 11:06:00AM -0700, Logan Gunthorpe wrote: > + cdev = >cdev; > + cdev_init(cdev, _fops); > + cdev->owner = THIS_MODULE; > + cdev->kobj.parent = >kobj; Minor nit, the kobject in a cdev is unlike any other kobject you have ever seen, don't mess with it, it's not doing anything like you think it is doing. So no need to set the parent field. thanks, greg k-h -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v2 1/4] MicroSemi Switchtec management interface driver
Microsemi's "Switchtec" line of PCI switch devices is already well supported by the kernel with standard PCI switch drivers. However, the Switchtec device advertises a special management endpoint with a separate PCI function address and class code. This endpoint enables some additional functionality which includes: * Packet and Byte Counters * Switch Firmware Upgrades * Event and Error logs * Querying port link status * Custom user firmware commands This patch introduces the switchtec kernel module which provides PCI driver that exposes a char device. The char device provides userspace access to this interface through read, write and (optionally) poll calls. A userspace tool and library which utilizes this interface is available at [1]. This tool takes inspiration (and borrows some code) from nvme-cli [2]. The tool is largely complete at this time but additional features may be added in the future. [1] https://github.com/sbates130272/switchtec-user [2] https://github.com/linux-nvme/nvme-cli Signed-off-by: Logan GunthorpeSigned-off-by: Stephen Bates --- MAINTAINERS|8 + drivers/pci/Kconfig|1 + drivers/pci/Makefile |1 + drivers/pci/switch/Kconfig | 13 + drivers/pci/switch/Makefile|1 + drivers/pci/switch/switchtec.c | 1028 6 files changed, 1052 insertions(+) create mode 100644 drivers/pci/switch/Kconfig create mode 100644 drivers/pci/switch/Makefile create mode 100644 drivers/pci/switch/switchtec.c diff --git a/MAINTAINERS b/MAINTAINERS index 5f10c28..9c35198 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -9507,6 +9507,14 @@ S: Maintained F: Documentation/devicetree/bindings/pci/aardvark-pci.txt F: drivers/pci/host/pci-aardvark.c +PCI DRIVER FOR MICROSEMI SWITCHTEC +M: Kurt Schwemmer +M: Stephen Bates +M: Logan Gunthorpe +L: linux-...@vger.kernel.org +S: Maintained +F: drivers/pci/switch/switchtec* + PCI DRIVER FOR NVIDIA TEGRA M: Thierry Reding L: linux-te...@vger.kernel.org diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig index 6555eb7..f72e8c5 100644 --- a/drivers/pci/Kconfig +++ b/drivers/pci/Kconfig @@ -133,3 +133,4 @@ config PCI_HYPERV source "drivers/pci/hotplug/Kconfig" source "drivers/pci/host/Kconfig" +source "drivers/pci/switch/Kconfig" diff --git a/drivers/pci/Makefile b/drivers/pci/Makefile index 8db5079..15b46dd 100644 --- a/drivers/pci/Makefile +++ b/drivers/pci/Makefile @@ -68,3 +68,4 @@ ccflags-$(CONFIG_PCI_DEBUG) := -DDEBUG # PCI host controller drivers obj-y += host/ +obj-y += switch/ diff --git a/drivers/pci/switch/Kconfig b/drivers/pci/switch/Kconfig new file mode 100644 index 000..4c49648 --- /dev/null +++ b/drivers/pci/switch/Kconfig @@ -0,0 +1,13 @@ +menu "PCI switch controller drivers" + depends on PCI + +config PCI_SW_SWITCHTEC + tristate "MicroSemi Switchtec PCIe Switch Management Driver" + help +Enables support for the management interface for the MicroSemi +Switchtec series of PCIe switches. Supports userspace access +to submit MRPC commands to the switch via /dev/switchtecX +devices. See for more +information. + +endmenu diff --git a/drivers/pci/switch/Makefile b/drivers/pci/switch/Makefile new file mode 100644 index 000..37d8cfb --- /dev/null +++ b/drivers/pci/switch/Makefile @@ -0,0 +1 @@ +obj-$(CONFIG_PCI_SW_SWITCHTEC) += switchtec.o diff --git a/drivers/pci/switch/switchtec.c b/drivers/pci/switch/switchtec.c new file mode 100644 index 000..e4a4294 --- /dev/null +++ b/drivers/pci/switch/switchtec.c @@ -0,0 +1,1028 @@ +/* + * Microsemi Switchtec(tm) PCIe Management Driver + * Copyright (c) 2017, Microsemi Corporation + * + * This program is free software; you can redistribute it and/or modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * This program is distributed in the hope it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + */ + +#include +#include +#include +#include +#include +#include +#include +#include + +MODULE_DESCRIPTION("Microsemi Switchtec(tm) PCI-E Management Driver"); +MODULE_VERSION("0.1"); +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Microsemi Corporation"); + +static int max_devices = 16; +module_param(max_devices, int, 0644); +MODULE_PARM_DESC(max_devices, "max number of switchtec device instances"); + +static dev_t switchtec_devt; +static struct class *switchtec_class; +static DEFINE_IDA(switchtec_minor_ida); + +#define MICROSEMI_VENDOR_ID 0x11f8 +#define MICROSEMI_NTB_CLASSCODE