Re: Enabling peer to peer device transactions for PCIe devices

;, "Paul Blinzer" <paul.blin...@amd.com>, > "Christian Koenig" <christian.koe...@amd.com>, > "Suravee Suthikulpanit" <suravee.suthikulpa...@amd.com>, "Ben Sander" > <ben.san...@amd.com> > Sent: Tuesday, 24 October, 2017 16:58:24 > Subject: RE: Enabling peer to peer device transactions for PCIe devices > Please don't top post, write shorter lines, and add the odd blank line. > Big blocks of text are hard to read quickly. > OK this time I am very short. peer2peer works Ludwig

Re: Enabling peer to peer device transactions for PCIe devices

;Linux-media" , > "dri-devel" , "linux-pci" > , "John Bridgman" > , "Felix Kuehling" , "Serguei > Sagalovitch" > , "Paul Blinzer" , > "Christian Koenig" , > "Suravee Suthikulpanit" , "Ben Sander&qu

RE: Enabling peer to peer device transactions for PCIe devices

Please don't top post, write shorter lines, and add the odd blank line. Big blocks of text are hard to read quickly. > From: Petrosyan, Ludwig [mailto:ludwig.petros...@desy.de] > Yes I agree it has to be started with the write transaction, according of > PCIe standard all write > transaction are

RE: Enabling peer to peer device transactions for PCIe devices

Please don't top post, write shorter lines, and add the odd blank line. Big blocks of text are hard to read quickly. > From: Petrosyan, Ludwig [mailto:ludwig.petros...@desy.de] > Yes I agree it has to be started with the write transaction, according of > PCIe standard all write > transaction are

Re: Enabling peer to peer device transactions for PCIe devices

.blin...@amd.com>, "Christian Koenig" <christian.koe...@amd.com>, "Suravee Suthikulpanit" <suravee.suthikulpa...@amd.com>, "Ben Sander" <ben.san...@amd.com> Sent: Tuesday, 24 October, 2017 00:04:26 Subject: Re: Enabling peer to peer device transact

Re: Enabling peer to peer device transactions for PCIe devices

; , "dri-devel" , "linux-pci" , "John Bridgman" , "Felix Kuehling" , "Serguei Sagalovitch" , "Paul Blinzer" , "Christian Koenig" , "Suravee Suthikulpanit" , "Ben Sander" Sent: Tuesday, 24 October, 2017 00:04:26

Re: Enabling peer to peer device transactions for PCIe devices

On 23/10/17 10:08 AM, David Laight wrote: It is also worth checking that the hardware actually supports p2p transfers. Writes are more likely to be supported then reads. ISTR that some intel cpus support some p2p writes, but there could easily be errata against them. Ludwig mentioned a PCIe

Re: Enabling peer to peer device transactions for PCIe devices

On 23/10/17 10:08 AM, David Laight wrote: It is also worth checking that the hardware actually supports p2p transfers. Writes are more likely to be supported then reads. ISTR that some intel cpus support some p2p writes, but there could easily be errata against them. Ludwig mentioned a PCIe

RE: Enabling peer to peer device transactions for PCIe devices

From: Petrosyan Ludwig > Sent: 22 October 2017 07:14 > Could be I have done is stupid... > But at first sight it has to be simple: > The PCIe Write transactions are address routed, so if in the packet header > the other endpoint address > is written the TLP has to be routed (by PCIe Switch to the

RE: Enabling peer to peer device transactions for PCIe devices

From: Petrosyan Ludwig > Sent: 22 October 2017 07:14 > Could be I have done is stupid... > But at first sight it has to be simple: > The PCIe Write transactions are address routed, so if in the packet header > the other endpoint address > is written the TLP has to be routed (by PCIe Switch to the

Re: Enabling peer to peer device transactions for PCIe devices

On 22/10/17 12:13 AM, Petrosyan, Ludwig wrote: > But at first sight it has to be simple: > The PCIe Write transactions are address routed, so if in the packet header > the other endpoint address is written the TLP has to be routed (by PCIe > Switch to the endpoint), the DMA reading from the end

Re: Enabling peer to peer device transactions for PCIe devices

On 22/10/17 12:13 AM, Petrosyan, Ludwig wrote: > But at first sight it has to be simple: > The PCIe Write transactions are address routed, so if in the packet header > the other endpoint address is written the TLP has to be routed (by PCIe > Switch to the endpoint), the DMA reading from the end

Re: Enabling peer to peer device transactions for PCIe devices

itch, Serguei" <serguei.sagalovi...@amd.com>, "Blinzer, Paul" <paul.blin...@amd.com>, "Koenig, Christian" <christian.koe...@amd.com>, "Suthikulpanit, Suravee" <suravee.suthikulpa...@amd.com>, "Sander, Ben" <ben.san...@amd.

Re: Enabling peer to peer device transactions for PCIe devices

ig, Christian" , "Suthikulpanit, Suravee" , "Sander, Ben" Sent: Friday, 20 October, 2017 17:48:58 Subject: Re: Enabling peer to peer device transactions for PCIe devices Hi Ludwig, P2P transactions are still *very* experimental at the moment and take a lot of exper

Re: Enabling peer to peer device transactions for PCIe devices

Hi Ludwig, P2P transactions are still *very* experimental at the moment and take a lot of expertise to get working in a general setup. It will definitely require changes to the kernel, including the drivers of all the devices you are trying to make talk to eachother. If you're up for it you

Re: Enabling peer to peer device transactions for PCIe devices

Hi Ludwig, P2P transactions are still *very* experimental at the moment and take a lot of expertise to get working in a general setup. It will definitely require changes to the kernel, including the drivers of all the devices you are trying to make talk to eachother. If you're up for it you

Re: Enabling peer to peer device transactions for PCIe devices

Dear Linux kernel group my name is Ludwig Petrosyan I am working in DESY (Germany) we are responsible for the control system of  all accelerators in DESY. For a 7-8 years we have switched to MTCA.4 systems and using PCIe as a central Bus. I am mostly responsible for the Linux drivers of the

Re: Enabling peer to peer device transactions for PCIe devices

Dear Linux kernel group my name is Ludwig Petrosyan I am working in DESY (Germany) we are responsible for the control system of  all accelerators in DESY. For a 7-8 years we have switched to MTCA.4 systems and using PCIe as a central Bus. I am mostly responsible for the Linux drivers of the

Re: Enabling peer to peer device transactions for PCIe devices

Am 12.01.2017 um 16:11 schrieb Jerome Glisse: On Wed, Jan 11, 2017 at 10:54:39PM -0600, Stephen Bates wrote: On Fri, January 6, 2017 4:10 pm, Logan Gunthorpe wrote: On 06/01/17 11:26 AM, Jason Gunthorpe wrote: Make a generic API for all of this and you'd have my vote.. IMHO, you must

Re: Enabling peer to peer device transactions for PCIe devices

Am 12.01.2017 um 16:11 schrieb Jerome Glisse: On Wed, Jan 11, 2017 at 10:54:39PM -0600, Stephen Bates wrote: On Fri, January 6, 2017 4:10 pm, Logan Gunthorpe wrote: On 06/01/17 11:26 AM, Jason Gunthorpe wrote: Make a generic API for all of this and you'd have my vote.. IMHO, you must

Re: Enabling peer to peer device transactions for PCIe devices

On 11/01/17 09:54 PM, Stephen Bates wrote: > The iopmem patchset addressed all the use cases above and while it is not > an in kernel API it could have been modified to be one reasonably easily. > As Logan states the driver can then choose to pass the VMAs to user-space > in a manner that makes

Re: Enabling peer to peer device transactions for PCIe devices

On 11/01/17 09:54 PM, Stephen Bates wrote: > The iopmem patchset addressed all the use cases above and while it is not > an in kernel API it could have been modified to be one reasonably easily. > As Logan states the driver can then choose to pass the VMAs to user-space > in a manner that makes

Re: Enabling peer to peer device transactions for PCIe devices

On Thu, Jan 12, 2017 at 10:11:29AM -0500, Jerome Glisse wrote: > On Wed, Jan 11, 2017 at 10:54:39PM -0600, Stephen Bates wrote: > > > What we want is for RDMA, O_DIRECT, etc to just work with special VMAs > > > (ie. at least those backed with ZONE_DEVICE memory). Then > > > GPU/NVME/DAX/whatever

Re: Enabling peer to peer device transactions for PCIe devices

On Thu, Jan 12, 2017 at 10:11:29AM -0500, Jerome Glisse wrote: > On Wed, Jan 11, 2017 at 10:54:39PM -0600, Stephen Bates wrote: > > > What we want is for RDMA, O_DIRECT, etc to just work with special VMAs > > > (ie. at least those backed with ZONE_DEVICE memory). Then > > > GPU/NVME/DAX/whatever

Re: Enabling peer to peer device transactions for PCIe devices

On Wed, Jan 11, 2017 at 10:54:39PM -0600, Stephen Bates wrote: > On Fri, January 6, 2017 4:10 pm, Logan Gunthorpe wrote: > > > > > > On 06/01/17 11:26 AM, Jason Gunthorpe wrote: > > > > > >> Make a generic API for all of this and you'd have my vote.. > >> > >> > >> IMHO, you must support basic

Re: Enabling peer to peer device transactions for PCIe devices

On Wed, Jan 11, 2017 at 10:54:39PM -0600, Stephen Bates wrote: > On Fri, January 6, 2017 4:10 pm, Logan Gunthorpe wrote: > > > > > > On 06/01/17 11:26 AM, Jason Gunthorpe wrote: > > > > > >> Make a generic API for all of this and you'd have my vote.. > >> > >> > >> IMHO, you must support basic

Re: Enabling peer to peer device transactions for PCIe devices

On Fri, January 6, 2017 4:10 pm, Logan Gunthorpe wrote: > > > On 06/01/17 11:26 AM, Jason Gunthorpe wrote: > > >> Make a generic API for all of this and you'd have my vote.. >> >> >> IMHO, you must support basic pinning semantics - that is necessary to >> support generic short lived DMA (eg

Re: Enabling peer to peer device transactions for PCIe devices

On Fri, January 6, 2017 4:10 pm, Logan Gunthorpe wrote: > > > On 06/01/17 11:26 AM, Jason Gunthorpe wrote: > > >> Make a generic API for all of this and you'd have my vote.. >> >> >> IMHO, you must support basic pinning semantics - that is necessary to >> support generic short lived DMA (eg

Re: Enabling peer to peer device transactions for PCIe devices

On 06/01/17 11:26 AM, Jason Gunthorpe wrote: > Make a generic API for all of this and you'd have my vote.. > > IMHO, you must support basic pinning semantics - that is necessary to > support generic short lived DMA (eg filesystem, etc). That hardware > can clearly do that if it can support

Re: Enabling peer to peer device transactions for PCIe devices

On 06/01/17 11:26 AM, Jason Gunthorpe wrote: > Make a generic API for all of this and you'd have my vote.. > > IMHO, you must support basic pinning semantics - that is necessary to > support generic short lived DMA (eg filesystem, etc). That hardware > can clearly do that if it can support

RE: Enabling peer to peer device transactions for PCIe devices

ubject: Re: Enabling peer to peer device transactions for PCIe devices > > On Fri, Jan 06, 2017 at 12:37:22PM -0500, Jerome Glisse wrote: > > On Fri, Jan 06, 2017 at 11:56:30AM -0500, Serguei Sagalovitch wrote: > > > On 2017-01-05 08:58 PM, Jerome Glisse wrote: > > > &g

RE: Enabling peer to peer device transactions for PCIe devices

ubject: Re: Enabling peer to peer device transactions for PCIe devices > > On Fri, Jan 06, 2017 at 12:37:22PM -0500, Jerome Glisse wrote: > > On Fri, Jan 06, 2017 at 11:56:30AM -0500, Serguei Sagalovitch wrote: > > > On 2017-01-05 08:58 PM, Jerome Glisse wrote: > > > &g

Re: Enabling peer to peer device transactions for PCIe devices

On Fri, Jan 06, 2017 at 12:37:22PM -0500, Jerome Glisse wrote: > On Fri, Jan 06, 2017 at 11:56:30AM -0500, Serguei Sagalovitch wrote: > > On 2017-01-05 08:58 PM, Jerome Glisse wrote: > > > On Thu, Jan 05, 2017 at 05:30:34PM -0700, Jason Gunthorpe wrote: > > > > On Thu, Jan 05, 2017 at 06:23:52PM

Re: Enabling peer to peer device transactions for PCIe devices

On Fri, Jan 06, 2017 at 12:37:22PM -0500, Jerome Glisse wrote: > On Fri, Jan 06, 2017 at 11:56:30AM -0500, Serguei Sagalovitch wrote: > > On 2017-01-05 08:58 PM, Jerome Glisse wrote: > > > On Thu, Jan 05, 2017 at 05:30:34PM -0700, Jason Gunthorpe wrote: > > > > On Thu, Jan 05, 2017 at 06:23:52PM

Re: Enabling peer to peer device transactions for PCIe devices

On Fri, Jan 06, 2017 at 11:56:30AM -0500, Serguei Sagalovitch wrote: > On 2017-01-05 08:58 PM, Jerome Glisse wrote: > > On Thu, Jan 05, 2017 at 05:30:34PM -0700, Jason Gunthorpe wrote: > > > On Thu, Jan 05, 2017 at 06:23:52PM -0500, Jerome Glisse wrote: > > > > > > > > I still don't understand

Re: Enabling peer to peer device transactions for PCIe devices

On Fri, Jan 06, 2017 at 11:56:30AM -0500, Serguei Sagalovitch wrote: > On 2017-01-05 08:58 PM, Jerome Glisse wrote: > > On Thu, Jan 05, 2017 at 05:30:34PM -0700, Jason Gunthorpe wrote: > > > On Thu, Jan 05, 2017 at 06:23:52PM -0500, Jerome Glisse wrote: > > > > > > > > I still don't understand

Re: Enabling peer to peer device transactions for PCIe devices

On 2017-01-05 08:58 PM, Jerome Glisse wrote: On Thu, Jan 05, 2017 at 05:30:34PM -0700, Jason Gunthorpe wrote: On Thu, Jan 05, 2017 at 06:23:52PM -0500, Jerome Glisse wrote: I still don't understand what you driving at - you've said in both cases a user VMA exists. In the former case no,

Re: Enabling peer to peer device transactions for PCIe devices

On 2017-01-05 08:58 PM, Jerome Glisse wrote: On Thu, Jan 05, 2017 at 05:30:34PM -0700, Jason Gunthorpe wrote: On Thu, Jan 05, 2017 at 06:23:52PM -0500, Jerome Glisse wrote: I still don't understand what you driving at - you've said in both cases a user VMA exists. In the former case no,

Re: Enabling peer to peer device transactions for PCIe devices

Hello, I've been watching this thread not as a kernel developer, but as an user interested in doing peer-to-peer access between network card and GPU. I believe that merging raw direct access with vma overcomplicates things for our use case. We'll have a very large camera streaming data at high

Re: Enabling peer to peer device transactions for PCIe devices

Hello, I've been watching this thread not as a kernel developer, but as an user interested in doing peer-to-peer access between network card and GPU. I believe that merging raw direct access with vma overcomplicates things for our use case. We'll have a very large camera streaming data at high

Re: Enabling peer to peer device transactions for PCIe devices

On Thu, Jan 05, 2017 at 05:30:34PM -0700, Jason Gunthorpe wrote: > On Thu, Jan 05, 2017 at 06:23:52PM -0500, Jerome Glisse wrote: > > > > I still don't understand what you driving at - you've said in both > > > cases a user VMA exists. > > > > In the former case no, there is no VMA directly but

Re: Enabling peer to peer device transactions for PCIe devices

On Thu, Jan 05, 2017 at 05:30:34PM -0700, Jason Gunthorpe wrote: > On Thu, Jan 05, 2017 at 06:23:52PM -0500, Jerome Glisse wrote: > > > > I still don't understand what you driving at - you've said in both > > > cases a user VMA exists. > > > > In the former case no, there is no VMA directly but

Re: Enabling peer to peer device transactions for PCIe devices

On 2017-01-05 07:30 PM, Jason Gunthorpe wrote: but I am opposed to the idea we need two API paths that the *driver* has to figure out. That is fundamentally not what I want as a driver developer. Give me a common API to convert '__user *' to a scatter list and pin the pages.

Re: Enabling peer to peer device transactions for PCIe devices

On 2017-01-05 07:30 PM, Jason Gunthorpe wrote: but I am opposed to the idea we need two API paths that the *driver* has to figure out. That is fundamentally not what I want as a driver developer. Give me a common API to convert '__user *' to a scatter list and pin the pages.

Re: Enabling peer to peer device transactions for PCIe devices

On Thu, Jan 05, 2017 at 06:23:52PM -0500, Jerome Glisse wrote: > > I still don't understand what you driving at - you've said in both > > cases a user VMA exists. > > In the former case no, there is no VMA directly but if you want one than > a device can provide one. But such VMA is useless as

Re: Enabling peer to peer device transactions for PCIe devices

On Thu, Jan 05, 2017 at 06:23:52PM -0500, Jerome Glisse wrote: > > I still don't understand what you driving at - you've said in both > > cases a user VMA exists. > > In the former case no, there is no VMA directly but if you want one than > a device can provide one. But such VMA is useless as

Re: Enabling peer to peer device transactions for PCIe devices

On Thu, Jan 05, 2017 at 03:42:15PM -0700, Jason Gunthorpe wrote: > On Thu, Jan 05, 2017 at 03:19:36PM -0500, Jerome Glisse wrote: > > > > Always having a VMA changes the discussion - the question is how to > > > create a VMA that reprensents IO device memory, and how do DMA > > > consumers

Re: Enabling peer to peer device transactions for PCIe devices

On Thu, Jan 05, 2017 at 03:42:15PM -0700, Jason Gunthorpe wrote: > On Thu, Jan 05, 2017 at 03:19:36PM -0500, Jerome Glisse wrote: > > > > Always having a VMA changes the discussion - the question is how to > > > create a VMA that reprensents IO device memory, and how do DMA > > > consumers

Re: Enabling peer to peer device transactions for PCIe devices

On Thu, Jan 05, 2017 at 03:19:36PM -0500, Jerome Glisse wrote: > > Always having a VMA changes the discussion - the question is how to > > create a VMA that reprensents IO device memory, and how do DMA > > consumers extract the correct information from that VMA to pass to the > > kernel DMA API

Re: Enabling peer to peer device transactions for PCIe devices

On Thu, Jan 05, 2017 at 03:19:36PM -0500, Jerome Glisse wrote: > > Always having a VMA changes the discussion - the question is how to > > create a VMA that reprensents IO device memory, and how do DMA > > consumers extract the correct information from that VMA to pass to the > > kernel DMA API

Re: Enabling peer to peer device transactions for PCIe devices

On Thu, Jan 05, 2017 at 01:07:19PM -0700, Jason Gunthorpe wrote: > On Thu, Jan 05, 2017 at 02:54:24PM -0500, Jerome Glisse wrote: > > > Mellanox and NVidia support peer to peer with what they market a > > GPUDirect. It only works without IOMMU. It is probably not upstream : > > > >

Re: Enabling peer to peer device transactions for PCIe devices

On Thu, Jan 05, 2017 at 01:07:19PM -0700, Jason Gunthorpe wrote: > On Thu, Jan 05, 2017 at 02:54:24PM -0500, Jerome Glisse wrote: > > > Mellanox and NVidia support peer to peer with what they market a > > GPUDirect. It only works without IOMMU. It is probably not upstream : > > > >

Re: Enabling peer to peer device transactions for PCIe devices

On Thu, Jan 05, 2017 at 02:54:24PM -0500, Jerome Glisse wrote: > Mellanox and NVidia support peer to peer with what they market a > GPUDirect. It only works without IOMMU. It is probably not upstream : > > https://www.mail-archive.com/linux-rdma@vger.kernel.org/msg21402.html > > I thought it

Re: Enabling peer to peer device transactions for PCIe devices

On Thu, Jan 05, 2017 at 02:54:24PM -0500, Jerome Glisse wrote: > Mellanox and NVidia support peer to peer with what they market a > GPUDirect. It only works without IOMMU. It is probably not upstream : > > https://www.mail-archive.com/linux-rdma@vger.kernel.org/msg21402.html > > I thought it

Re: Enabling peer to peer device transactions for PCIe devices

On Thu, Jan 05, 2017 at 12:01:13PM -0700, Jason Gunthorpe wrote: > On Thu, Jan 05, 2017 at 01:39:29PM -0500, Jerome Glisse wrote: > > > 1) peer-to-peer because of userspace specific API like NVidia GPU > > direct (AMD is pushing its own similar API i just can't remember > > marketing

Re: Enabling peer to peer device transactions for PCIe devices

On Thu, Jan 05, 2017 at 12:01:13PM -0700, Jason Gunthorpe wrote: > On Thu, Jan 05, 2017 at 01:39:29PM -0500, Jerome Glisse wrote: > > > 1) peer-to-peer because of userspace specific API like NVidia GPU > > direct (AMD is pushing its own similar API i just can't remember > > marketing

Re: Enabling peer to peer device transactions for PCIe devices

On Thu, Jan 05, 2017 at 01:39:29PM -0500, Jerome Glisse wrote: > 1) peer-to-peer because of userspace specific API like NVidia GPU > direct (AMD is pushing its own similar API i just can't remember > marketing name). This does not happen through a vma, this happens > through

Re: Enabling peer to peer device transactions for PCIe devices

On Thu, Jan 05, 2017 at 01:39:29PM -0500, Jerome Glisse wrote: > 1) peer-to-peer because of userspace specific API like NVidia GPU > direct (AMD is pushing its own similar API i just can't remember > marketing name). This does not happen through a vma, this happens > through

Re: Enabling peer to peer device transactions for PCIe devices

Sorry to revive this thread but it fells through my filters and i miss it. I have been going through it and i think the discussion has been hinder by the fact that distinct problems were merge while they should be address separately. First for peer-to-peer we need to be clear on how this happens.

Re: Enabling peer to peer device transactions for PCIe devices

Sorry to revive this thread but it fells through my filters and i miss it. I have been going through it and i think the discussion has been hinder by the fact that distinct problems were merge while they should be address separately. First for peer-to-peer we need to be clear on how this happens.

Re: Enabling peer to peer device transactions for PCIe devices

On Tue, Dec 6, 2016 at 1:47 PM, Logan Gunthorpe wrote: > Hey, > >> Okay, so clearly this needs a kernel side NVMe specific allocator >> and locking so users don't step on each other.. > > Yup, ideally. That's why device dax isn't ideal for this application: it > doesn't

Re: Enabling peer to peer device transactions for PCIe devices

On Tue, Dec 6, 2016 at 1:47 PM, Logan Gunthorpe wrote: > Hey, > >> Okay, so clearly this needs a kernel side NVMe specific allocator >> and locking so users don't step on each other.. > > Yup, ideally. That's why device dax isn't ideal for this application: it > doesn't provide any way to prevent

Re: Enabling peer to peer device transactions for PCIe devices

Hey, > Okay, so clearly this needs a kernel side NVMe specific allocator > and locking so users don't step on each other.. Yup, ideally. That's why device dax isn't ideal for this application: it doesn't provide any way to prevent users from stepping on each other. > Or as Christoph says some

Re: Enabling peer to peer device transactions for PCIe devices

Hey, > Okay, so clearly this needs a kernel side NVMe specific allocator > and locking so users don't step on each other.. Yup, ideally. That's why device dax isn't ideal for this application: it doesn't provide any way to prevent users from stepping on each other. > Or as Christoph says some

Re: Enabling peer to peer device transactions for PCIe devices

On Tue, Dec 06, 2016 at 09:51:15AM -0700, Logan Gunthorpe wrote: > Hey, > > On 06/12/16 09:38 AM, Jason Gunthorpe wrote: > >>> I'm not opposed to mapping /dev/nvmeX. However, the lookup is trivial > >>> to accomplish in sysfs through /sys/dev/char to find the sysfs path of the > >>> device-dax

Re: Enabling peer to peer device transactions for PCIe devices

On Tue, Dec 06, 2016 at 09:51:15AM -0700, Logan Gunthorpe wrote: > Hey, > > On 06/12/16 09:38 AM, Jason Gunthorpe wrote: > >>> I'm not opposed to mapping /dev/nvmeX. However, the lookup is trivial > >>> to accomplish in sysfs through /sys/dev/char to find the sysfs path of the > >>> device-dax

Re: Enabling peer to peer device transactions for PCIe devices

On Tue, Dec 06, 2016 at 09:38:50AM -0700, Jason Gunthorpe wrote: > > > I'm not opposed to mapping /dev/nvmeX. However, the lookup is trivial > > > to accomplish in sysfs through /sys/dev/char to find the sysfs path of the > > > device-dax instance under the nvme device, or if you already have the

Re: Enabling peer to peer device transactions for PCIe devices

On Tue, Dec 06, 2016 at 09:38:50AM -0700, Jason Gunthorpe wrote: > > > I'm not opposed to mapping /dev/nvmeX. However, the lookup is trivial > > > to accomplish in sysfs through /sys/dev/char to find the sysfs path of the > > > device-dax instance under the nvme device, or if you already have the

Re: Enabling peer to peer device transactions for PCIe devices

Hey, On 06/12/16 09:38 AM, Jason Gunthorpe wrote: >>> I'm not opposed to mapping /dev/nvmeX. However, the lookup is trivial >>> to accomplish in sysfs through /sys/dev/char to find the sysfs path of the >>> device-dax instance under the nvme device, or if you already have the nvme >>> sysfs path

Re: Enabling peer to peer device transactions for PCIe devices

Hey, On 06/12/16 09:38 AM, Jason Gunthorpe wrote: >>> I'm not opposed to mapping /dev/nvmeX. However, the lookup is trivial >>> to accomplish in sysfs through /sys/dev/char to find the sysfs path of the >>> device-dax instance under the nvme device, or if you already have the nvme >>> sysfs path

Re: Enabling peer to peer device transactions for PCIe devices

> > I'm not opposed to mapping /dev/nvmeX. However, the lookup is trivial > > to accomplish in sysfs through /sys/dev/char to find the sysfs path of the > > device-dax instance under the nvme device, or if you already have the nvme > > sysfs path the dax instance(s) will appear under the "dax"

Re: Enabling peer to peer device transactions for PCIe devices

> > I'm not opposed to mapping /dev/nvmeX. However, the lookup is trivial > > to accomplish in sysfs through /sys/dev/char to find the sysfs path of the > > device-dax instance under the nvme device, or if you already have the nvme > > sysfs path the dax instance(s) will appear under the "dax"

Re: Enabling peer to peer device transactions for PCIe devices

>>> I've already recommended that iopmem not be a block device and >>> instead be a device-dax instance. I also don't think it should claim >>> the PCI ID, rather the driver that wants to map one of its bars this >>> way can register the memory region with the device-dax core. >>> >>> I'm not sure

Re: Enabling peer to peer device transactions for PCIe devices

>>> I've already recommended that iopmem not be a block device and >>> instead be a device-dax instance. I also don't think it should claim >>> the PCI ID, rather the driver that wants to map one of its bars this >>> way can register the memory region with the device-dax core. >>> >>> I'm not sure

Re: Enabling peer to peer device transactions for PCIe devices

On Mon, Dec 05, 2016 at 12:46:14PM -0700, Jason Gunthorpe wrote: > In any event the allocator still needs to track which regions are in > use and be able to hook 'free' from userspace. That does suggest it > should be integrated into the nvme driver and not a bolt on driver.. Two totally

Re: Enabling peer to peer device transactions for PCIe devices

On Mon, Dec 05, 2016 at 12:46:14PM -0700, Jason Gunthorpe wrote: > In any event the allocator still needs to track which regions are in > use and be able to hook 'free' from userspace. That does suggest it > should be integrated into the nvme driver and not a bolt on driver.. Two totally

Re: Enabling peer to peer device transactions for PCIe devices

On 05/12/16 12:46 PM, Jason Gunthorpe wrote: NVMe might have to deal with pci-e hot-unplug, which is a similar problem-class to the GPU case.. Sure, but if the NVMe device gets hot-unplugged it means that all the CMB mappings are useless and need to be torn down. This probably means

Re: Enabling peer to peer device transactions for PCIe devices

On 05/12/16 12:46 PM, Jason Gunthorpe wrote: NVMe might have to deal with pci-e hot-unplug, which is a similar problem-class to the GPU case.. Sure, but if the NVMe device gets hot-unplugged it means that all the CMB mappings are useless and need to be torn down. This probably means

Re: Enabling peer to peer device transactions for PCIe devices

On Mon, Dec 05, 2016 at 12:27:20PM -0700, Logan Gunthorpe wrote: > > > On 05/12/16 12:14 PM, Jason Gunthorpe wrote: > >But CMB sounds much more like the GPU case where there is a > >specialized allocator handing out the BAR to consumers, so I'm not > >sure a general purpose chardev makes a lot

Re: Enabling peer to peer device transactions for PCIe devices

On Mon, Dec 05, 2016 at 12:27:20PM -0700, Logan Gunthorpe wrote: > > > On 05/12/16 12:14 PM, Jason Gunthorpe wrote: > >But CMB sounds much more like the GPU case where there is a > >specialized allocator handing out the BAR to consumers, so I'm not > >sure a general purpose chardev makes a lot

Re: Enabling peer to peer device transactions for PCIe devices

On 05/12/16 12:14 PM, Jason Gunthorpe wrote: But CMB sounds much more like the GPU case where there is a specialized allocator handing out the BAR to consumers, so I'm not sure a general purpose chardev makes a lot of sense? I don't think it will ever need to be as complicated as the GPU

Re: Enabling peer to peer device transactions for PCIe devices

On 05/12/16 12:14 PM, Jason Gunthorpe wrote: But CMB sounds much more like the GPU case where there is a specialized allocator handing out the BAR to consumers, so I'm not sure a general purpose chardev makes a lot of sense? I don't think it will ever need to be as complicated as the GPU

Re: Enabling peer to peer device transactions for PCIe devices

On Mon, Dec 05, 2016 at 10:48:58AM -0800, Dan Williams wrote: > On Mon, Dec 5, 2016 at 10:39 AM, Logan Gunthorpe wrote: > > On 05/12/16 11:08 AM, Dan Williams wrote: > >> > >> I've already recommended that iopmem not be a block device and instead > >> be a device-dax

Re: Enabling peer to peer device transactions for PCIe devices

On Mon, Dec 05, 2016 at 10:48:58AM -0800, Dan Williams wrote: > On Mon, Dec 5, 2016 at 10:39 AM, Logan Gunthorpe wrote: > > On 05/12/16 11:08 AM, Dan Williams wrote: > >> > >> I've already recommended that iopmem not be a block device and instead > >> be a device-dax instance. I also don't think

Re: Enabling peer to peer device transactions for PCIe devices

On Mon, Dec 5, 2016 at 10:39 AM, Logan Gunthorpe wrote: > On 05/12/16 11:08 AM, Dan Williams wrote: >> >> I've already recommended that iopmem not be a block device and instead >> be a device-dax instance. I also don't think it should claim the PCI >> ID, rather the driver

Re: Enabling peer to peer device transactions for PCIe devices

On Mon, Dec 5, 2016 at 10:39 AM, Logan Gunthorpe wrote: > On 05/12/16 11:08 AM, Dan Williams wrote: >> >> I've already recommended that iopmem not be a block device and instead >> be a device-dax instance. I also don't think it should claim the PCI >> ID, rather the driver that wants to map one

Re: Enabling peer to peer device transactions for PCIe devices

On 05/12/16 11:08 AM, Dan Williams wrote: I've already recommended that iopmem not be a block device and instead be a device-dax instance. I also don't think it should claim the PCI ID, rather the driver that wants to map one of its bars this way can register the memory region with the

Re: Enabling peer to peer device transactions for PCIe devices

On 05/12/16 11:08 AM, Dan Williams wrote: I've already recommended that iopmem not be a block device and instead be a device-dax instance. I also don't think it should claim the PCI ID, rather the driver that wants to map one of its bars this way can register the memory region with the

Re: Enabling peer to peer device transactions for PCIe devices

On Mon, Dec 5, 2016 at 10:02 AM, Jason Gunthorpe wrote: > On Mon, Dec 05, 2016 at 09:40:38AM -0800, Dan Williams wrote: > >> > If it is kernel only with physical addresess we don't need a uAPI for >> > it, so I'm not sure #1 is at all related to iopmem. >> > >> >

Re: Enabling peer to peer device transactions for PCIe devices

On Mon, Dec 5, 2016 at 10:02 AM, Jason Gunthorpe wrote: > On Mon, Dec 05, 2016 at 09:40:38AM -0800, Dan Williams wrote: > >> > If it is kernel only with physical addresess we don't need a uAPI for >> > it, so I'm not sure #1 is at all related to iopmem. >> > >> > Most people who want #1 probably

Re: Enabling peer to peer device transactions for PCIe devices

On Mon, Dec 05, 2016 at 09:40:38AM -0800, Dan Williams wrote: > > If it is kernel only with physical addresess we don't need a uAPI for > > it, so I'm not sure #1 is at all related to iopmem. > > > > Most people who want #1 probably can just mmap > > /sys/../pci/../resourceX to get a user handle

Re: Enabling peer to peer device transactions for PCIe devices

On Mon, Dec 05, 2016 at 09:40:38AM -0800, Dan Williams wrote: > > If it is kernel only with physical addresess we don't need a uAPI for > > it, so I'm not sure #1 is at all related to iopmem. > > > > Most people who want #1 probably can just mmap > > /sys/../pci/../resourceX to get a user handle

Re: Enabling peer to peer device transactions for PCIe devices

On Mon, Dec 5, 2016 at 9:18 AM, Jason Gunthorpe wrote: > On Sun, Dec 04, 2016 at 07:23:00AM -0600, Stephen Bates wrote: >> Hi All >> >> This has been a great thread (thanks to Alex for kicking it off) and I >> wanted to jump in and maybe try and put some summary

Re: Enabling peer to peer device transactions for PCIe devices

On Mon, Dec 5, 2016 at 9:18 AM, Jason Gunthorpe wrote: > On Sun, Dec 04, 2016 at 07:23:00AM -0600, Stephen Bates wrote: >> Hi All >> >> This has been a great thread (thanks to Alex for kicking it off) and I >> wanted to jump in and maybe try and put some summary around the >> discussion. I also

Re: Enabling peer to peer device transactions for PCIe devices

On Sun, Dec 04, 2016 at 07:23:00AM -0600, Stephen Bates wrote: > Hi All > > This has been a great thread (thanks to Alex for kicking it off) and I > wanted to jump in and maybe try and put some summary around the > discussion. I also wanted to propose we include this as a topic for LFS/MM >

Re: Enabling peer to peer device transactions for PCIe devices

On Sun, Dec 04, 2016 at 07:23:00AM -0600, Stephen Bates wrote: > Hi All > > This has been a great thread (thanks to Alex for kicking it off) and I > wanted to jump in and maybe try and put some summary around the > discussion. I also wanted to propose we include this as a topic for LFS/MM >

Re: Enabling peer to peer device transactions for PCIe devices

Hi All This has been a great thread (thanks to Alex for kicking it off) and I wanted to jump in and maybe try and put some summary around the discussion. I also wanted to propose we include this as a topic for LFS/MM because I think we need more discussion on the best way to add this

Re: Enabling peer to peer device transactions for PCIe devices

Hi All This has been a great thread (thanks to Alex for kicking it off) and I wanted to jump in and maybe try and put some summary around the discussion. I also wanted to propose we include this as a topic for LFS/MM because I think we need more discussion on the best way to add this

Re: Enabling peer to peer device transactions for PCIe devices

On 11/30/2016 6:23 PM, Jason Gunthorpe wrote: >> and O_DIRECT operations that access GPU memory. > This goes through user space so there is still a VMA.. > >> Also, HMM's migration between two GPUs could use peer to peer in the >> kernel, although that is intended to be handled by the GPU driver

Re: Enabling peer to peer device transactions for PCIe devices

On 11/30/2016 6:23 PM, Jason Gunthorpe wrote: >> and O_DIRECT operations that access GPU memory. > This goes through user space so there is still a VMA.. > >> Also, HMM's migration between two GPUs could use peer to peer in the >> kernel, although that is intended to be handled by the GPU driver

  1   2   3   >