Re: Enabling peer to peer device transactions for PCIe devices

2017-01-06 Thread Serguei Sagalovitch
On 2017-01-05 08:58 PM, Jerome Glisse wrote: On Thu, Jan 05, 2017 at 05:30:34PM -0700, Jason Gunthorpe wrote: On Thu, Jan 05, 2017 at 06:23:52PM -0500, Jerome Glisse wrote: I still don't understand what you driving at - you've said in both cases a user VMA exists. In the former case no,

Re: Enabling peer to peer device transactions for PCIe devices

2017-01-05 Thread Serguei Sagalovitch
. Completely agreed. IMHO there is no sense to duplicate the same logic everywhere as well as trying to find places where it is missing. Sincerely yours, Serguei Sagalovitch -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majord...@vger.kerne

Re: Enabling peer to peer device transactions for PCIe devices

2016-11-30 Thread Serguei Sagalovitch
On 2016-11-30 11:23 AM, Jason Gunthorpe wrote: Yes, that sounds fine. Can we simply kill the process from the GPU driver? Or do we need to extend the OOM killer to manage GPU pages? I don't know.. We could use send_sig_info to send signal from kernel to user space. So theoretically GPU

Re: Enabling peer to peer device transactions for PCIe devices

2016-11-28 Thread Serguei Sagalovitch
On 2016-11-28 04:36 PM, Logan Gunthorpe wrote: On 28/11/16 12:35 PM, Serguei Sagalovitch wrote: As soon as PeerDirect mapping is called then GPU must not "move" the such memory. It is by PeerDirect design. It is similar how it is works with system memory and RDMA MR: when "

Re: Enabling peer to peer device transactions for PCIe devices

2016-11-28 Thread Serguei Sagalovitch
On 2016-11-28 01:20 PM, Logan Gunthorpe wrote: On 28/11/16 09:57 AM, Jason Gunthorpe wrote: On PeerDirect, we have some kind of a middle-ground solution for pinning GPU memory. We create a non-ODP MR pointing to VRAM but rely on user-space and the GPU not to migrate it. If they do, the MR gets

Re: Enabling peer to peer device transactions for PCIe devices

2016-11-28 Thread Serguei Sagalovitch
On 2016-11-27 09:02 AM, Haggai Eran wrote On PeerDirect, we have some kind of a middle-ground solution for pinning GPU memory. We create a non-ODP MR pointing to VRAM but rely on user-space and the GPU not to migrate it. If they do, the MR gets destroyed immediately. This should work on legacy

Re: Enabling peer to peer device transactions for PCIe devices

2016-11-25 Thread Serguei Sagalovitch
On 2016-11-25 03:26 PM, Felix Kuehling wrote: On 16-11-25 12:20 PM, Serguei Sagalovitch wrote: A white list may end up being rather complicated if it has to cover different CPU generations and system architectures. I feel this is a decision user space could easily make. Logan I agreed

Re: Enabling peer to peer device transactions for PCIe devices

2016-11-25 Thread Serguei Sagalovitch
On 2016-11-25 02:34 PM, Jason Gunthorpe wrote: On Fri, Nov 25, 2016 at 12:16:30PM -0500, Serguei Sagalovitch wrote: b) Allocation may not have CPU address at all - only GPU one. But you don't expect RDMA to work in the case, right? GPU people need to stop doing this windowed memory stuff

Re: Enabling peer to peer device transactions for PCIe devices

2016-11-25 Thread Serguei Sagalovitch
On 2016-11-25 08:22 AM, Christian König wrote: Serguei, what is your plan in GPU land for migration? Ie if I have a CPU mapped page and the GPU moves it to VRAM, it becomes non-cachable - do you still allow the CPU to access it? Or do you swap it back to cachable memory if the CPU touches it?

Re: Enabling peer to peer device transactions for PCIe devices

2016-11-25 Thread Serguei Sagalovitch
Well, I guess there's some consensus building to do. The existing options are: * Device DAX: which could work but the problem I see with it is that it only allows one application to do these transfers. Or there would have to be some user-space coordination to figure which application gets what

Re: Enabling peer to peer device transactions for PCIe devices

2016-11-25 Thread Serguei Sagalovitch
A white list may end up being rather complicated if it has to cover different CPU generations and system architectures. I feel this is a decision user space could easily make. Logan I agreed that it is better to leave up to user space to check what is working and what is not. I found that

Re: Enabling peer to peer device transactions for PCIe devices

2016-11-24 Thread Serguei Sagalovitch
On 2016-11-24 11:26 AM, Jason Gunthorpe wrote: On Thu, Nov 24, 2016 at 10:45:18AM +0100, Christian König wrote: Am 24.11.2016 um 00:25 schrieb Jason Gunthorpe: There is certainly nothing about the hardware that cares about ZONE_DEVICE vs System memory. Well that is clearly not so simple.

Re: Enabling peer to peer device transactions for PCIe devices

2016-11-23 Thread Serguei Sagalovitch
On 2016-11-23 02:12 PM, Jason Gunthorpe wrote: On Wed, Nov 23, 2016 at 10:40:47AM -0800, Dan Williams wrote: I don't think that was designed for the case where the backing memory is a special/static physical address range rather than anonymous "System RAM", right? The hardware doesn't care

Re: Enabling peer to peer device transactions for PCIe devices

2016-11-23 Thread Serguei Sagalovitch
On 2016-11-23 03:51 AM, Christian König wrote: Am 23.11.2016 um 08:49 schrieb Daniel Vetter: On Tue, Nov 22, 2016 at 01:21:03PM -0800, Dan Williams wrote: On Tue, Nov 22, 2016 at 1:03 PM, Daniel Vetter <dan...@ffwll.ch> wrote: On Tue, Nov 22, 2016 at 9:35 PM, Serguei Sagal

Re: Enabling peer to peer device transactions for PCIe devices

2016-11-23 Thread Serguei Sagalovitch
On 2016-11-23 12:27 PM, Bart Van Assche wrote: On 11/23/2016 09:13 AM, Logan Gunthorpe wrote: IMO any memory that has been registered for a P2P transaction should be locked from being evicted. So if there's a get_user_pages call it needs to be pinned until the put_page. The main issue being

Re: Enabling peer to peer device transactions for PCIe devices

2016-11-23 Thread Serguei Sagalovitch
On 2016-11-23 02:05 PM, Jason Gunthorpe wrote: On Wed, Nov 23, 2016 at 10:13:03AM -0700, Logan Gunthorpe wrote: an MR would be very tricky. The MR may be relied upon by another host and the kernel would have to inform user-space the MR was invalid then user-space would have to tell the remote

Re: Enabling peer to peer device transactions for PCIe devices

2016-11-22 Thread Serguei Sagalovitch
On 2016-11-22 03:10 PM, Daniel Vetter wrote: On Tue, Nov 22, 2016 at 9:01 PM, Dan Williams <dan.j.willi...@intel.com> wrote: On Tue, Nov 22, 2016 at 10:59 AM, Serguei Sagalovitch <serguei.sagalovi...@amd.com> wrote: I personally like "device-DAX" idea but my conc