RE: [PATCH v4 00/16] Add Paravirtual RDMA Driver
Jason wrote, >I should be clearer here. I am *strongly* opposed to anything that changes the >license of the existing 4 core libraries away from the >GPLv2 or OpenIB.org situation we have today. (that includes to other varients >of the BSD license) >I just checked and we appear to be completely OK on this point today. Ok, thanks
Re: [PATCH v4 00/16] Add Paravirtual RDMA Driver
On Wed, Sep 14, 2016 at 04:59:10PM -0600, Jason Gunthorpe wrote: > > package follows that licensing model for accepting any new code into > > that combined repo ? > > As with the kernel we'd discourage 're-licensing' existing files. > > However, since this is not a OFA project, I, personally, would not > turn away a GPLv2 compatible contribution, but I am proposing that the > 'default' license for the project be OFA compatible. I should be clearer here. I am *strongly* opposed to anything that changes the license of the existing 4 core libraries away from the GPLv2 or OpenIB.org situation we have today. (that includes to other varients of the BSD license) I just checked and we appear to be completely OK on this point today. Jason
Re: [PATCH v4 00/16] Add Paravirtual RDMA Driver
On Wed, Sep 14, 2016 at 11:36:36AM -0600, Jason Gunthorpe wrote: > On Mon, Sep 12, 2016 at 10:43:00PM +, Adit Ranadive wrote: > > On Mon, Sep 12, 2016 at 11:03:39 -0700, Jason Gunthorpe wrote: > > > On Sun, Sep 11, 2016 at 09:49:10PM -0700, Adit Ranadive wrote: > > > > [2] Libpvrdma User-level library - > > > > http://git.openfabrics.org/?p=~aditr/libpvrdma.git;a=summary > > > > > > You will probably find that rdma-plumbing will be the best way to get > > > your userspace component into the distributors. > > > > Hi Jason, > > > > Sorry I haven't paying attention to that discussion. Do you know how soon > > distros will pick up the rdma-plumbing stuff? > > We desire to use this as the vehical for the userspace included with > the 4.9 kernel. > > I anticipate the tree will be running by Oct 1. +1 > > Jason > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html signature.asc Description: PGP signature
Re: [PATCH v4 00/16] Add Paravirtual RDMA Driver
On Wed, Sep 14, 2016 at 10:20:22PM +, Woodruff, Robert J wrote: > >this new scheme works with >kernel.org 4.8, then it is possible > >that it could go into that OFED-4.8 Release, but again, we are > >still looking at the new scheme and evaluating how it affects >the > >community OFED. > > One other question. Jason, the OFA (and its members) want to > maintain the dual-license (BSD/GPL) for the code, as is the case for > all the code that was in the OFA git trees on the OFA server that > you pulled from. The code in the OFA trees has a mixture of licenses. It is all GPLv2 compatible for sure, but there are at least three variations of the 'BSD' license. As far as I know, only the 2 clause OpenIB.org BSD license is approved for use in OFA projects. Someone from the board could correct me if I am wrong. This means the 3 projects with the BSD patent clause, hosted on the OFA servers, were already not conforming. As is Intels HFI1 driver which uses a three clause BSD license. > package follows that licensing model for accepting any new code into > that combined repo ? As with the kernel we'd discourage 're-licensing' existing files. However, since this is not a OFA project, I, personally, would not turn away a GPLv2 compatible contribution, but I am proposing that the 'default' license for the project be OFA compatible. I think license enforcement of its members falls to the OFA. Doug may feel differently. Jason
RE: [PATCH v4 00/16] Add Paravirtual RDMA Driver
Woody Wrote, >We are still discussing this within the OFA EWG for the OFED releases. I >suspect that if all of the maintainers of the user-space packages agree to >merge into >and support this new merged repo-model, then OFED would eventually >base their OFED user-space packages on that repo, rather than the individual >repo's >that are used today. The question is, when. Work has now just started >on OFED-4.8, that is based on the kernel.org 4.8 kernel. If this new scheme >works with >kernel.org 4.8, then it is possible that it could go into that >OFED-4.8 Release, but again, we are still looking at the new scheme and >evaluating how it affects >the community OFED. One other question. Jason, the OFA (and its members) want to maintain the dual-license (BSD/GPL) for the code, as is the case for all the code that was in the OFA git trees on the OFA server that you pulled from. Will you guys be ensuring that this new user-space package follows that licensing model for accepting any new code into that combined repo ?
RE: [PATCH v4 00/16] Add Paravirtual RDMA Driver
Adit wrote, >Thanks. So does this mean that the libraries distributed via OFED >(openfabrics.org) will be now from the rdma-plumbing git tree? >Or is the switch to happen only when distros start shipping with the 4.9 >kernel by default? We are still discussing this within the OFA EWG for the OFED releases. I suspect that if all of the maintainers of the user-space packages agree to merge into and support this new merged repo-model, then OFED would eventually base their OFED user-space packages on that repo, rather than the individual repo's that are used today. The question is, when. Work has now just started on OFED-4.8, that is based on the kernel.org 4.8 kernel. If this new scheme works with kernel.org 4.8, then it is possible that it could go into that OFED-4.8 Release, but again, we are still looking at the new scheme and evaluating how it affects the community OFED. Woody
Re: [PATCH v4 00/16] Add Paravirtual RDMA Driver
On Wed, Sep 14, 2016 at 07:44:45PM +, Adit Ranadive wrote: > On Wed, Sep 14, 2016 at 10:37:00 -0700, Jason Gunthorpe wrote: > > We desire to use this as the vehical for the userspace included with the 4.9 > > kernel. > > > > I anticipate the tree will be running by Oct 1. > > Thanks. So does this mean that the libraries distributed via OFED > (openfabrics.org) > will be now from the rdma-plumbing git tree? I can't speak to the exact transition schedule each downstream will follow. OFED is a downstream distribution focused on backporting to old distros. I would expect that by the time OFED starts to backport 4.9 they will need to use rdma plumbing. At the cutoff point we will immediately tell all downstreams the latest official upstream source is in rdma-plumbing and they should stop monitoring the old repos. > Or is the switch to happen only when distros start shipping with the > 4.9 kernel by default? Now that we are going down this path with the 4.9 set as the time frame, I wouldn't worry about timing, there is no down-side to you for working within rdma-plumbing, and it guarantees that eventually your code will be widely distributed. Jason
Re: [PATCH v4 00/16] Add Paravirtual RDMA Driver
On Wed, Sep 14, 2016 at 10:37:00 -0700, Jason Gunthorpe wrote: > We desire to use this as the vehical for the userspace included with the 4.9 > kernel. > > I anticipate the tree will be running by Oct 1. Thanks. So does this mean that the libraries distributed via OFED (openfabrics.org) will be now from the rdma-plumbing git tree? Or is the switch to happen only when distros start shipping with the 4.9 kernel by default?
Re: [PATCH v4 00/16] Add Paravirtual RDMA Driver
On Mon, Sep 12, 2016 at 10:43:00PM +, Adit Ranadive wrote: > On Mon, Sep 12, 2016 at 11:03:39 -0700, Jason Gunthorpe wrote: > > On Sun, Sep 11, 2016 at 09:49:10PM -0700, Adit Ranadive wrote: > > > [2] Libpvrdma User-level library - > > > http://git.openfabrics.org/?p=~aditr/libpvrdma.git;a=summary > > > > You will probably find that rdma-plumbing will be the best way to get > > your userspace component into the distributors. > > Hi Jason, > > Sorry I haven't paying attention to that discussion. Do you know how soon > distros will pick up the rdma-plumbing stuff? We desire to use this as the vehical for the userspace included with the 4.9 kernel. I anticipate the tree will be running by Oct 1. Jason
Re: [PATCH v4 00/16] Add Paravirtual RDMA Driver
On Mon, Sep 12, 2016 at 10:43:00PM +, Adit Ranadive wrote: > On Mon, Sep 12, 2016 at 11:03:39 -0700, Jason Gunthorpe wrote: > > On Sun, Sep 11, 2016 at 09:49:10PM -0700, Adit Ranadive wrote: > > > [2] Libpvrdma User-level library - > > > http://git.openfabrics.org/?p=~aditr/libpvrdma.git;a=summary > > > > You will probably find that rdma-plumbing will be the best way to get > > your userspace component into the distributors. > Sorry I haven't paying attention to that discussion. Do you know how soon > distros will pick up the rdma-plumbing stuff? Hopefully things will be clearer after the call on Tuesday. I'm hoping we can begin the process sooner than later. There are 4 new drivers being submitted and at least 2-3 more kernel drivers that are not yet in Debian/SuSE/etc. By far the simplest way to push those 6 new user space drivers into the distros will be via rdma-plumbing, I'm optimistic that the vendors will work together on this. > It seems like it should be fairly straightforward to include the libpvrdma git > within the rdma-plumbing git as well. Let me know what the process is since > I may have to discuss it internally. For now, you can prepare your own clone and create a providers/pvrdma directory similar to the others. It should be very straightforward. Let me know if you have any problems. You should ensure clean compilation on FC24 (for instance using the provided docker tooling), and review the commit stream in rdma-plumbing to make sure you pick up all the systemic fixes that have already been done to the other providers. If Doug accepts your provider upstream before we finalize rdma-plumbing then I will add your git to the giant merge, however please make my life easier and ensure the code does not have any of the systemic problems I alreadly fixed in other providers :( After finalization you'll have to post your user space provider to the mailing list and Doug will pick it up. Jason
Re: [PATCH v4 00/16] Add Paravirtual RDMA Driver
On Mon, Sep 12, 2016 at 11:03:39 -0700, Jason Gunthorpe wrote: > On Sun, Sep 11, 2016 at 09:49:10PM -0700, Adit Ranadive wrote: > > [2] Libpvrdma User-level library - > > http://git.openfabrics.org/?p=~aditr/libpvrdma.git;a=summary > > You will probably find that rdma-plumbing will be the best way to get > your userspace component into the distributors. Hi Jason, Sorry I haven't paying attention to that discussion. Do you know how soon distros will pick up the rdma-plumbing stuff? It seems like it should be fairly straightforward to include the libpvrdma git within the rdma-plumbing git as well. Let me know what the process is since I may have to discuss it internally. Thanks! Adit
Re: [PATCH v4 00/16] Add Paravirtual RDMA Driver
On Sun, Sep 11, 2016 at 09:49:10PM -0700, Adit Ranadive wrote: > [2] Libpvrdma User-level library - > http://git.openfabrics.org/?p=~aditr/libpvrdma.git;a=summary You will probably find that rdma-plumbing will be the best way to get your userspace component into the distributors. http://www.spinics.net/lists/linux-rdma/msg39026.html http://www.spinics.net/lists/linux-rdma/msg39328.html http://www.spinics.net/lists/linux-rdma/msg40014.html http://www.spinics.net/lists/linux-rdma/msg39026.html Jason
[PATCH v4 00/16] Add Paravirtual RDMA Driver
Hi Doug, others,
This patch series adds a driver for a paravirtual RDMA device. The device
is developed for VMware's Virtual Machines and allows existing RDMA
applications to continue to use existing Verbs API when deployed in VMs on
ESXi. We recently did a presentation in the OFA Workshop [1] regarding this
device.
Description and RDMA Support
The virtual device is exposed as a dual function PCIe device. One part is
a virtual network device (VMXNet3) which provides networking properties
like MAC, IP addresses to the RDMA part of the device. The networking
properties are used to register GIDs required by RDMA applications to
communicate.
These patches add support and the all required infrastructure for letting
applications use such a device. We support the mandatory Verbs API as well
as the base memory management extensions (Local Inv, Send with Inv and Fast
Register Work Requests). We currently support both Reliable Connected and
Unreliable Datagram QPs but do not support Shared Receive Queues (SRQs).
Also, we support the following types of Work Requests:
o Send/Receive (with or without Immediate Data)
o RDMA Write (with or without Immediate Data)
o RDMA Read
o Local Invalidate
o Send with Invalidate
o Fast Register Work Requests
This version only adds support for version 1 of RoCE. We will add RoCEv2
support in a future patch. We do support registration of both MAC-based and
IP-based GIDs. I have also created a git tree for our user-level driver [2].
Testing
===
We have tested this internally for various types of Guest OS - Red Hat,
Centos, Ubuntu 12.04/14.04/16.04, Oracle Enterprise Linux, SLES 12
using backported versions of this driver. The tests included several runs
of the performance tests (included with OFED), Intel MPI PingPong benchmark
on OpenMPI, krping for FRWRs. Mellanox has been kind enough to test the
backported version of the driver internally on their hardware using a
VMware provided ESX build. I have also applied and tested this with Doug's
k.o/for-4.9 branch (commit 64278fe). Note, that this patch series should be
applied all together. I split out the commits so that it may be easier to
review.
PVRDMA Resources
[1] OFA Workshop Presentation - https://goo.gl/pHOXJ8
[2] Libpvrdma User-level library -
http://git.openfabrics.org/?p=~aditr/libpvrdma.git;a=summary
---
Changes v3->v4:
- Rebased on for-4.9 branch - commit 64278fe89b729
("Merge branch 'hns-roce' into k.o/for-4.9")
- PATCH [01/16]
- New in v4 - Moved vmxnet3 id to pci_ids.h
- PATCH [02,03/16]
- pvrdma_sge was moved into pvrdma_uapi.h
- PATCH [04/16]
- Removed explicit enum values.
- PATCH [05/16]
- Renamed priviledged -> privileged.
- Added error numbers for command errors.
- Removed unnecessary goto in modify_device.
- Moved pd allocation to after command execution.
- Removed an incorrect atomic_dec.
- PATCH [06/16]
- Renamed priviledged -> privileged.
- Renamed pvrdma_flush_cqe to _pvrdma_flush_cqe since we hold a lock
to call it.
- Added wrapper functions for writing to UARs for CQ/QP.
- The conversion functions are updated as func_name(dst, src) format.
- Renamed max_gs to max_sg.
- Added work struct for net device events.
- PATCH [07/16]
- Updated conversion functions to func_name(dst, src) format.
- Removed unneeded local variables.
- PATCH [08/16]
- Removed the min check and added a BUILD_BUG_ON check for size.
- PATCH [09/16]
- Added a pvrdma_destroy_cq in the error path.
- Renamed pvrdma_flush_cqe to _pvrdma_flush_cqe since we need a lock to
be held while calling this.
- Updated to use wrapper for UAR write for CQ.
- Ensure that poll_cq does not return error values.
- PATCH [10/16]
- Removed an unnecessary comment.
- PATCH [11/16]
- Changed access flag check for DMA MR to using bit operation.
- Removed some local variables.
- PATCH [12/16]
- Removed an unnecessary switch case.
- Unified the returns in pvrdma_create_qp to use one exit point.
- Renamed pvrdma_flush_cqe to _pvrdma_flush_cqe since we need a lock to
be held when calling this.
- Updated to use wrapper for UAR write for QP.
- Updated conversion function to func_name(dst, src) format.
- Renamed max_gs to max_sg.
- Renamed cap variable to req_cap in pvrdma_set_sq/rq_size.
- Changed dev_warn to dev_warn_ratelimited in pvrdma_post_send/recv.
- Added nesting locking for flushing CQs when destroying/resetting a QP.
- Added missing ret value.
- PATCH [13/16]
- Fixed some checkpatch warnings.
- Added support for new get_dev_fw_str API.
- Added event workqueue for netdevice events.
- Restructured the pvrdma_pci_remove function a little bit.
- PATCH [14/16]
- Enforced dependency on VMXNet3 module.
Changes v2->v3:
- I reordered the patches so that the definitions of enums, structures
