Hi, > -----Original Message----- > From: Parav Pandit <pa...@mellanox.com> > Sent: Friday, July 17, 2020 4:49 PM > To: dev@dpdk.org; gr...@u256.net; ferruh.yi...@intel.com; Thomas > Monjalon <tho...@monjalon.net> > Cc: Raslan Darawsheh <rasl...@mellanox.com>; Ori Kam > <or...@mellanox.com>; Matan Azrad <ma...@mellanox.com>; > joyce.k...@arm.com; Parav Pandit <pa...@mellanox.com> > Subject: [PATCH v7 0/9] Improve mlx5 PMD driver framework for multiple > classes > > This series introduces mlx5 bus to support multiple class of devices > for a single PCI device. > > Motivation and example > ---------------------- > mlx5 PCI device supports multiple class of devices such as net, vdpa > and regex devices. > > Currently only one pmd (either net or vdpa) can bind to this device. > This design limits use of PCI device only for single device class. > > To support multiple classes simultaneously for a mlx5 PCI device, > a new mlx5 PCI bus is created. This bus allows binding multiple > class drivers (such as net, vdpa, regex(future)) to bind to the > mlx5 PCI bus driver. > > Change description > ------------------ > Patch-1 Introduces RTE_BIT() macro > Patch-2 Introduces new RTE constructor priority for common > initialization > Patch-3 Fixes compilation error > Patch-4 Define mlx5 class as bit fields > Patch-5 Uses new RTE common priority > Patch-6 Adds mlx5 PCI bus > Patch-7 Implements a mlx5 PCI bus driver > Patch-8 Migrates mlx5 net and vdpa driver to use mlx5 PCI bus > API instead of rte PCI bus API > Patch-9 Removed class check code as its already part of the bus now > > Design overview > --------------- > > ----------- ------------ ------------- > | mlx5 | | mlx5 | | mlx5 | > | net pmd | | vdpa pmd | | regex pmd | > ----------- ------------ ------------- > \ | / > \ | / > \ ------------- / > \______| mlx5 |_____ / > | pci bus | > ------------- > | > ----------- > | mlx5 | > | pci dev | > ----------- > > - mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI > ID table of all related mlx5 PCI devices. > - mlx5 class driver such as net, vdpa, regex PMD defines its > specific PCI ID table and mlx5 bus driver probes matching > class drivers. > - mlx5 pci bus driver is cental place that validates supported > class combinations. > - In future as code evolves, more device setup/cleanup and > resource creation code moves to mlx5 PCI bus driver. > > Alternatives considered > ----------------------- > 1. Instead of creating mlx5 pci bus, a common driver is > implemented which exposes class registration API. > However, bus model fits better with existing DPDK design > similar to ifpga driver. > Class registration API need to create a new callbacks > and ID signature; instead it is better to utilize current > well defined methods. > > 2. Enhance pci core to allow multiple driver binding to > single rte PCI device. > This approach is not taken, because peer drivers using > one PCI device won't be aware of other's presence. This > requires cross-driver syncronization of who initializes > common resources (such as irq, eq and more). > This also requires refcounting common objects etc among > peer drivers. > Instead of layered approach delivers and allows putting > common resource sharing, setup code in common bus driver. > It also eliminates peer blind zone problem as bottom pci > bus layer provides necessary setup without any reference > counting. > > 3. In future mlx5 prefers to use RDMA MR cache of the mbuf > used between net and regex pmd so that same mbuf use across > multiple device can be possible. > > Examples: > -------- > A user who wish to use a specific class(es) provides list > of classes at command line such as, > ./testpmd -w <PCI BDF>,class=net:vdpa > ./testpmd -w <PCI BDF>,class=vdpa > > In future, > ./testpmd -w <PCI BDF>,class=net:regex > > Changelog: > v6->v7: > - Updated release notes > v5->v6: > - Fixed compilation failure in parallel build for shared lib > v4->v5: > - Squash the maintainers update path with other patch which adds the > bus > - Addressed comments from Morten Brørup > - Renamed newly added macro to RTE_BIT64 > - Added doxygen comment section for the macro > v3->v4: > - Fixed dma_map error unwinding flow to follow same order for unmap > v2->v3: > - Added RTE priority for common driver initialization > - Addressed comments from Thomas and Asaf > - Fixed compilation error in glue wrapper > - Moved pci_driver structure instance as first in driver > - Removed white spaces at the end of line in diagram > - Address commnts from Matan > - Removed CONFIG_RTE_LIBRTE_MLX5_PCI_BUS from config files > - Renamed mlx5_valid_class_combo to mlx5_class_combinations > - Added cross check for class drivers to support only 3 flags for now > - Added full stop at the end of comment block > - Using full names in function names > - Added new line before function name in multiple functions > - Added example string to parse for multiple classes > - Dropped mlx5 prefix from static function > - Removed empty lines > - Fixed issue to remove multiple classes for a driver > - Using define for drv_flags at multiple places > - Deriving drv_flags based on the class drivers > - Fixed alignment for id_table > - Perform dma map on best effort basis for all supported drivers > - Dynamically build pci id table > - Using PCI to mlx5 device helper routines > v1->v2: > - Addressed most comments from Thomas and Gaetan. > - Symbols starting with prefix rte_bus_pci_mlx5 may be > confusing as it may appear as it belong to rte_bus_pci module. > Hence it is kept as rte_bus_mlx5_pci which matches with other > modules as mlx5_vdpa, mlx5_net. > - Dropped 2nd patch and replace with new 6th patch. > - Avoided new file, added macro to rte_bitops.h > - Inheriting ret_pci_driver instead of rte_driver > - Added design and description of the mlx5_pci bus > - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag > - Use anonymous structure for class search and code changes around it > - Define static for class comination array > - Use RTE_DIM to find array size > - Added OOM check for strdup() > - Renamed copy variable to nstr_orig > - Returning negagive error code > - Returning directly if match entry found > - Use compat condition check > - Avoided cutting error message string > - Use uint32_t datatype instead of enum mlx5_class > - Changed logic to parse device arguments only once during probe() > - Added check to fail driver probe if multiple classes register with > DMA ops > - Renamed function to parse_class_options > - Migreate API from rte_driver to rte_pci_driver > > Parav Pandit (9): > eal: introduce macros for getting value for bit > eal: introduce RTE common initialization level > common/mlx5: fix empty input style in glue wrappers > common/mlx5: change mlx5 class enum values as bits > common/mlx5: use common rte priority > bus/mlx5_pci: add mlx5 PCI bus > bus/mlx5_pci: register a PCI driver > bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver > common/mlx5: remove class checks from individual driver > > MAINTAINERS | 5 + > doc/guides/rel_notes/release_20_08.rst | 5 + > drivers/bus/Makefile | 4 + > drivers/bus/meson.build | 2 +- > drivers/bus/mlx5_pci/Makefile | 41 ++ > drivers/bus/mlx5_pci/meson.build | 19 + > drivers/bus/mlx5_pci/mlx5_pci_bus.c | 522 ++++++++++++++++++ > drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h | 84 +++ > .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map | 5 + > drivers/common/mlx5/linux/mlx5_glue.h | 4 +- > drivers/common/mlx5/mlx5_common.c | 39 +- > drivers/common/mlx5/mlx5_common.h | 7 +- > .../common/mlx5/rte_common_mlx5_version.map | 2 - > drivers/net/mlx5/Makefile | 3 +- > drivers/net/mlx5/linux/mlx5_os.c | 6 - > drivers/net/mlx5/meson.build | 2 +- > drivers/net/mlx5/mlx5.c | 26 +- > drivers/net/mlx5/mlx5.h | 1 - > drivers/vdpa/mlx5/Makefile | 3 +- > drivers/vdpa/mlx5/meson.build | 2 +- > drivers/vdpa/mlx5/mlx5_vdpa.c | 30 +- > lib/librte_eal/include/rte_bitops.h | 8 + > lib/librte_eal/include/rte_common.h | 1 + > mk/rte.app.mk | 1 + > 24 files changed, 737 insertions(+), 85 deletions(-) > create mode 100644 drivers/bus/mlx5_pci/Makefile > create mode 100644 drivers/bus/mlx5_pci/meson.build > create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c > create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h > create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map > > -- > 2.26.2
Series applied to next-net-mlx, Kindest regards, Raslan Darawsheh
