Enabling peer to peer device transactions for PCIe devices

2017-01-06 Thread Serguei Sagalovitch
On 2017-01-05 08:58 PM, Jerome Glisse wrote: > On Thu, Jan 05, 2017 at 05:30:34PM -0700, Jason Gunthorpe wrote: >> On Thu, Jan 05, 2017 at 06:23:52PM -0500, Jerome Glisse wrote: >> I still don't understand what you driving at - you've said in both cases a user VMA exists. >>> In the forme

Enabling peer to peer device transactions for PCIe devices

2017-01-05 Thread Serguei Sagalovitch
list and pin > the pages. Completely agreed. IMHO there is no sense to duplicate the same logic everywhere as well as trying to find places where it is missing. Sincerely yours, Serguei Sagalovitch

Enabling peer to peer device transactions for PCIe devices

2016-11-30 Thread Serguei Sagalovitch
On 2016-11-30 11:23 AM, Jason Gunthorpe wrote: >> Yes, that sounds fine. Can we simply kill the process from the GPU driver? >> Or do we need to extend the OOM killer to manage GPU pages? > I don't know.. We could use send_sig_info to send signal from kernel to user space. So theoretically GPU d

Enabling peer to peer device transactions for PCIe devices

2016-11-28 Thread Serguei Sagalovitch
On 2016-11-28 04:36 PM, Logan Gunthorpe wrote: > On 28/11/16 12:35 PM, Serguei Sagalovitch wrote: >> As soon as PeerDirect mapping is called then GPU must not "move" the >> such memory. It is by PeerDirect design. It is similar how it is works >> with

Enabling peer to peer device transactions for PCIe devices

2016-11-28 Thread Serguei Sagalovitch
On 2016-11-28 01:20 PM, Logan Gunthorpe wrote: > > On 28/11/16 09:57 AM, Jason Gunthorpe wrote: >>> On PeerDirect, we have some kind of a middle-ground solution for pinning >>> GPU memory. We create a non-ODP MR pointing to VRAM but rely on >>> user-space and the GPU not to migrate it. If they do,

Enabling peer to peer device transactions for PCIe devices

2016-11-28 Thread Serguei Sagalovitch
On 2016-11-27 09:02 AM, Haggai Eran wrote > On PeerDirect, we have some kind of a middle-ground solution for pinning > GPU memory. We create a non-ODP MR pointing to VRAM but rely on > user-space and the GPU not to migrate it. If they do, the MR gets > destroyed immediately. This should work on leg

Enabling peer to peer device transactions for PCIe devices

2016-11-25 Thread Serguei Sagalovitch
On 2016-11-25 03:26 PM, Felix Kuehling wrote: > On 16-11-25 12:20 PM, Serguei Sagalovitch wrote: >>> A white list may end up being rather complicated if it has to cover >>> different CPU generations and system architectures. I feel this is a >>> decisio

Enabling peer to peer device transactions for PCIe devices

2016-11-25 Thread Serguei Sagalovitch
On 2016-11-25 02:34 PM, Jason Gunthorpe wrote: > On Fri, Nov 25, 2016 at 12:16:30PM -0500, Serguei Sagalovitch wrote: > >> b) Allocation may not have CPU address at all - only GPU one. > But you don't expect RDMA to work in the case, right? > > GPU people need to stop

Enabling peer to peer device transactions for PCIe devices

2016-11-25 Thread Serguei Sagalovitch
> Well, I guess there's some consensus building to do. The existing > options are: > > * Device DAX: which could work but the problem I see with it is that it > only allows one application to do these transfers. Or there would have > to be some user-space coordination to figure which application g

Enabling peer to peer device transactions for PCIe devices

2016-11-25 Thread Serguei Sagalovitch
> A white list may end up being rather complicated if it has to cover > different CPU generations and system architectures. I feel this is a > decision user space could easily make. > > Logan I agreed that it is better to leave up to user space to check what is working and what is not. I found th

Enabling peer to peer device transactions for PCIe devices

2016-11-25 Thread Serguei Sagalovitch
On 2016-11-25 08:22 AM, Christian König wrote: > >> Serguei, what is your plan in GPU land for migration? Ie if I have a >> CPU mapped page and the GPU moves it to VRAM, it becomes non-cachable >> - do you still allow the CPU to access it? Or do you swap it back to >> cachable memory if the CPU to

Enabling peer to peer device transactions for PCIe devices

2016-11-24 Thread Serguei Sagalovitch
On 2016-11-24 11:26 AM, Jason Gunthorpe wrote: > On Thu, Nov 24, 2016 at 10:45:18AM +0100, Christian König wrote: >> Am 24.11.2016 um 00:25 schrieb Jason Gunthorpe: >>> There is certainly nothing about the hardware that cares >>> about ZONE_DEVICE vs System memory. >> Well that is clearly not so

Enabling peer to peer device transactions for PCIe devices

2016-11-23 Thread Serguei Sagalovitch
On 2016-11-23 02:32 PM, Jason Gunthorpe wrote: > On Wed, Nov 23, 2016 at 02:14:40PM -0500, Serguei Sagalovitch wrote: >> On 2016-11-23 02:05 PM, Jason Gunthorpe wrote: >>> As Bart says, it would be best to be combined with something like >>> Mellanox's ODP MRs, w

Enabling peer to peer device transactions for PCIe devices

2016-11-23 Thread Serguei Sagalovitch
On 2016-11-23 03:51 AM, Christian König wrote: > Am 23.11.2016 um 08:49 schrieb Daniel Vetter: >> On Tue, Nov 22, 2016 at 01:21:03PM -0800, Dan Williams wrote: >>> On Tue, Nov 22, 2016 at 1:03 PM, Daniel Vetter wrote: >>>> On Tue, Nov 22, 2016 at 9:35 PM, S

Enabling peer to peer device transactions for PCIe devices

2016-11-23 Thread Serguei Sagalovitch
On 2016-11-23 02:12 PM, Jason Gunthorpe wrote: > On Wed, Nov 23, 2016 at 10:40:47AM -0800, Dan Williams wrote: > >> I don't think that was designed for the case where the backing memory >> is a special/static physical address range rather than anonymous >> "System RAM", right? > The hardware does

Enabling peer to peer device transactions for PCIe devices

2016-11-23 Thread Serguei Sagalovitch
On 2016-11-23 02:05 PM, Jason Gunthorpe wrote: > On Wed, Nov 23, 2016 at 10:13:03AM -0700, Logan Gunthorpe wrote: > >> an MR would be very tricky. The MR may be relied upon by another host >> and the kernel would have to inform user-space the MR was invalid then >> user-space would have to tell th

Enabling peer to peer device transactions for PCIe devices

2016-11-23 Thread Serguei Sagalovitch
On 2016-11-23 12:27 PM, Bart Van Assche wrote: > On 11/23/2016 09:13 AM, Logan Gunthorpe wrote: >> IMO any memory that has been registered for a P2P transaction should be >> locked from being evicted. So if there's a get_user_pages call it needs >> to be pinned until the put_page. The main issue be

Enabling peer to peer device transactions for PCIe devices

2016-11-22 Thread Serguei Sagalovitch
On 2016-11-22 04:03 PM, Daniel Vetter wrote: > On Tue, Nov 22, 2016 at 9:35 PM, Serguei Sagalovitch > wrote: >> On 2016-11-22 03:10 PM, Daniel Vetter wrote: >>> On Tue, Nov 22, 2016 at 9:01 PM, Dan Williams >>> wrote: >>>> On Tue, Nov 22, 2016 at

Enabling peer to peer device transactions for PCIe devices

2016-11-22 Thread Serguei Sagalovitch
On 2016-11-22 03:10 PM, Daniel Vetter wrote: > On Tue, Nov 22, 2016 at 9:01 PM, Dan Williams > wrote: >> On Tue, Nov 22, 2016 at 10:59 AM, Serguei Sagalovitch >> wrote: >>> I personally like "device-DAX" idea but my concerns are: >>> &

Enabling peer to peer device transactions for PCIe devices

2016-11-22 Thread Serguei Sagalovitch
ata to the new location so CPU pointer should point to the new physical location/address (and may be not in PCI device memory at all)? Sincerely yours, Serguei Sagalovitch On 2016-11-22 01:11 PM, Dan Williams wrote: > On Mon, Nov 21, 2016 at 12:36 PM, Deucher, Alexander > wrote: >

[RFC] drm/radeon: userfence IOCTL

2015-04-13 Thread Serguei Sagalovitch
running in loop which read/write from this BO. It could also spawn processes which will do the same thing. IMHO the only way to improve situation is to have GPU Scheduler to allow "unloading" such application. Sincerely yours, Serguei Sagalovitch On 15-04-13 11:39 AM, Jerome Glisse wro

[RFC] drm/radeon: userfence IOCTL

2015-04-13 Thread Serguei Sagalovitch
> Another alternative would be to use the userspace mapping to check the BO value This is what I was thinking. Sincerely yours, Serguei Sagalovitch On 15-04-13 11:35 AM, Christian König wrote: > On 13.04.2015 17:25, Serguei Sagalovitch wrote: >> > the BO to be kept in the same

[RFC] drm/radeon: userfence IOCTL

2015-04-13 Thread Serguei Sagalovitch
hen TTM needs to move BO to another location it should also update CPU "mapping" correctly so user will always read / write the correct data. Did I miss anything? Sincerely yours, Serguei Sagalovitch On 15-04-13 10:52 AM, Christian König wrote: > Hello everyone, > > we hav