Re: Enabling peer to peer device transactions for PCIe devices

2016-12-04 Thread Haggai Eran
On 11/30/2016 6:23 PM, Jason Gunthorpe wrote: >> and O_DIRECT operations that access GPU memory. > This goes through user space so there is still a VMA.. > >> Also, HMM's migration between two GPUs could use peer to peer in the >> kernel, although that is intended to be handled by the GPU driver

Re: Enabling peer to peer device transactions for PCIe devices

2016-12-04 Thread Haggai Eran
On 11/30/2016 8:01 PM, Logan Gunthorpe wrote: > > > On 30/11/16 09:23 AM, Jason Gunthorpe wrote: >>> Two cases I can think of are RDMA access to an NVMe device's controller >>> memory buffer, >> >> I'm not sure on the use model there.. > > The NVMe fabrics stuff could probably make use of this.

Re: Enabling peer to peer device transactions for PCIe devices

2016-12-04 Thread Haggai Eran
ould use send_sig_info to send signal from kernel to user space. So > theoretically GPU driver > could issue KILL signal to some process. > >> On Wed, Nov 30, 2016 at 12:45:58PM +0200, Haggai Eran wrote: >>> I think we can achieve the kernel's needs with ZONE_DEVICE and D

Re: Enabling peer to peer device transactions for PCIe devices

2016-11-30 Thread Haggai Eran
On 11/28/2016 9:02 PM, Jason Gunthorpe wrote: > On Mon, Nov 28, 2016 at 06:19:40PM +0000, Haggai Eran wrote: >>>> GPU memory. We create a non-ODP MR pointing to VRAM but rely on >>>> user-space and the GPU not to migrate it. If they do, the MR gets >>>>

Re: Enabling peer to peer device transactions for PCIe devices

2016-11-28 Thread Haggai Eran
On Mon, 2016-11-28 at 09:57 -0700, Jason Gunthorpe wrote: > On Sun, Nov 27, 2016 at 04:02:16PM +0200, Haggai Eran wrote: > > I think blocking mmu notifiers against something that is basically > > controlled by user-space can be problematic. This can block things > > like >

Re: Enabling peer to peer device transactions for PCIe devices

2016-11-28 Thread Haggai Eran
On Mon, 2016-11-28 at 09:48 -0500, Serguei Sagalovitch wrote: > On 2016-11-27 09:02 AM, Haggai Eran wrote > > > > On PeerDirect, we have some kind of a middle-ground solution for > > pinning > > GPU memory. We create a non-ODP MR pointing to VRAM but rely on

Re: Enabling peer to peer device transactions for PCIe devices

2016-11-27 Thread Haggai Eran
On 11/25/2016 9:32 PM, Jason Gunthorpe wrote: > On Fri, Nov 25, 2016 at 02:22:17PM +0100, Christian König wrote: > >>> Like you say below we have to handle short lived in the usual way, and >>> that covers basically every device except IB MRs, including the >>> command queue on a NVMe drive. >>

Re: Enabling peer to peer device transactions for PCIe devices

2016-11-27 Thread Haggai Eran
On 11/25/2016 9:32 PM, Jason Gunthorpe wrote: > On Fri, Nov 25, 2016 at 02:22:17PM +0100, Christian König wrote: > >>> Like you say below we have to handle short lived in the usual way, and >>> that covers basically every device except IB MRs, including the >>> command queue on a NVMe drive. >> >>