Re: [RFC] Make use of non-dynamic dmabuf in RDMA

2021-09-01 Thread Christian König
Am 01.09.21 um 13:20 schrieb Gal Pressman: On 24/08/2021 20:32, Jason Gunthorpe wrote: On Tue, Aug 24, 2021 at 10:27:23AM -0700, John Hubbard wrote: On 8/24/21 2:32 AM, Christian König wrote: Am 24.08.21 um 11:06 schrieb Gal Pressman: On 23/08/2021 13:43, Christian König wrote: Am

Re: [RFC] Make use of non-dynamic dmabuf in RDMA

2021-08-25 Thread Oded Gabbay
On Wed, Aug 25, 2021 at 6:14 PM Christian König wrote: > > Am 25.08.21 um 16:47 schrieb Jason Gunthorpe: > > On Wed, Aug 25, 2021 at 03:51:14PM +0200, Christian König wrote: > >> Am 25.08.21 um 14:38 schrieb Jason Gunthorpe: > >>> On Wed, Aug 25, 2021 at 02:27:08PM +0200, Christian König wrote: >

Re: [RFC] Make use of non-dynamic dmabuf in RDMA

2021-08-25 Thread Jason Gunthorpe
On Wed, Aug 25, 2021 at 05:14:06PM +0200, Christian König wrote: > Yeah, that's exactly what I'm talking about by adding cgroup or similar. You > need a knob to control this. We have the pinned memory ulimit today. A pinned memory cgroup might be interesting, but even containrs are covered

Re: [RFC] Make use of non-dynamic dmabuf in RDMA

2021-08-25 Thread Christian König
Am 25.08.21 um 16:47 schrieb Jason Gunthorpe: On Wed, Aug 25, 2021 at 03:51:14PM +0200, Christian König wrote: Am 25.08.21 um 14:38 schrieb Jason Gunthorpe: On Wed, Aug 25, 2021 at 02:27:08PM +0200, Christian König wrote: Am 25.08.21 um 14:18 schrieb Jason Gunthorpe: On Wed, Aug 25, 2021 at

Re: [RFC] Make use of non-dynamic dmabuf in RDMA

2021-08-25 Thread Jason Gunthorpe
On Wed, Aug 25, 2021 at 03:51:14PM +0200, Christian König wrote: > Am 25.08.21 um 14:38 schrieb Jason Gunthorpe: > > On Wed, Aug 25, 2021 at 02:27:08PM +0200, Christian König wrote: > > > Am 25.08.21 um 14:18 schrieb Jason Gunthorpe: > > > > On Wed, Aug 25, 2021 at 08:17:51AM +0200, Christian

Re: [RFC] Make use of non-dynamic dmabuf in RDMA

2021-08-25 Thread Christian König
Am 25.08.21 um 14:38 schrieb Jason Gunthorpe: On Wed, Aug 25, 2021 at 02:27:08PM +0200, Christian König wrote: Am 25.08.21 um 14:18 schrieb Jason Gunthorpe: On Wed, Aug 25, 2021 at 08:17:51AM +0200, Christian König wrote: The only real option where you could do P2P with buffer pinning are

Re: [RFC] Make use of non-dynamic dmabuf in RDMA

2021-08-25 Thread Jason Gunthorpe
On Wed, Aug 25, 2021 at 08:17:51AM +0200, Christian König wrote: > The only real option where you could do P2P with buffer pinning are those > compute boards where we know that everything is always accessible to > everybody and we will never need to migrate anything. But even then you want > some

Re: [RFC] Make use of non-dynamic dmabuf in RDMA

2021-08-25 Thread Jason Gunthorpe
On Wed, Aug 25, 2021 at 02:27:08PM +0200, Christian König wrote: > Am 25.08.21 um 14:18 schrieb Jason Gunthorpe: > > On Wed, Aug 25, 2021 at 08:17:51AM +0200, Christian König wrote: > > > > > The only real option where you could do P2P with buffer pinning are those > > > compute boards where we

Re: [RFC] Make use of non-dynamic dmabuf in RDMA

2021-08-25 Thread Christian König
Am 25.08.21 um 14:18 schrieb Jason Gunthorpe: On Wed, Aug 25, 2021 at 08:17:51AM +0200, Christian König wrote: The only real option where you could do P2P with buffer pinning are those compute boards where we know that everything is always accessible to everybody and we will never need to

Re: [RFC] Make use of non-dynamic dmabuf in RDMA

2021-08-25 Thread John Hubbard
On 8/24/21 11:17 PM, Christian König wrote: ... I think it depends on the user, if the user creates memory which is permanently located on the GPU then it should be pinnable in this way without force migration. But if the memory is inherently migratable then it just cannot be pinned in the GPU

Re: [RFC] Make use of non-dynamic dmabuf in RDMA

2021-08-25 Thread Christian König
Am 24.08.21 um 19:32 schrieb Jason Gunthorpe: On Tue, Aug 24, 2021 at 10:27:23AM -0700, John Hubbard wrote: On 8/24/21 2:32 AM, Christian König wrote: Am 24.08.21 um 11:06 schrieb Gal Pressman: On 23/08/2021 13:43, Christian König wrote: Am 21.08.21 um 11:16 schrieb Gal Pressman: On

RE: [RFC] Make use of non-dynamic dmabuf in RDMA

2021-08-24 Thread Xiong, Jianxin
RAMEWORK > ; dri-devel de...@lists.freedesktop.org>; Linux Kernel Mailing List > ; linux-rdma ; > Gabbay, > Oded (Habana) ; Tayar, Tomer (Habana) ; > Yossi Leybovich ; Alexander > Matushevsky ; Leon Romanovsky ; Xiong, > Jianxin > Subject: Re: [RFC] Make use of non

Re: [RFC] Make use of non-dynamic dmabuf in RDMA

2021-08-24 Thread Alex Deucher
On Tue, Aug 24, 2021 at 3:16 PM Dave Airlie wrote: > > On Wed, 25 Aug 2021 at 03:36, John Hubbard wrote: > > > > On 8/24/21 10:32 AM, Jason Gunthorpe wrote: > > ... > > >>> And yes at least for the amdgpu driver we migrate the memory to host > > >>> memory as soon as it is pinned and I would

Re: [RFC] Make use of non-dynamic dmabuf in RDMA

2021-08-24 Thread Jason Gunthorpe
On Wed, Aug 25, 2021 at 05:15:52AM +1000, Dave Airlie wrote: > On Wed, 25 Aug 2021 at 03:36, John Hubbard wrote: > > > > On 8/24/21 10:32 AM, Jason Gunthorpe wrote: > > ... > > >>> And yes at least for the amdgpu driver we migrate the memory to host > > >>> memory as soon as it is pinned and I

Re: [RFC] Make use of non-dynamic dmabuf in RDMA

2021-08-24 Thread Dave Airlie
On Wed, 25 Aug 2021 at 03:36, John Hubbard wrote: > > On 8/24/21 10:32 AM, Jason Gunthorpe wrote: > ... > >>> And yes at least for the amdgpu driver we migrate the memory to host > >>> memory as soon as it is pinned and I would expect that other GPU drivers > >>> do something similar. > >> > >>

Re: [RFC] Make use of non-dynamic dmabuf in RDMA

2021-08-24 Thread John Hubbard
On 8/24/21 10:32 AM, Jason Gunthorpe wrote: ... And yes at least for the amdgpu driver we migrate the memory to host memory as soon as it is pinned and I would expect that other GPU drivers do something similar. Well...for many topologies, migrating to host memory will result in a dramatically

Re: [RFC] Make use of non-dynamic dmabuf in RDMA

2021-08-24 Thread Jason Gunthorpe
On Tue, Aug 24, 2021 at 10:27:23AM -0700, John Hubbard wrote: > On 8/24/21 2:32 AM, Christian König wrote: > > Am 24.08.21 um 11:06 schrieb Gal Pressman: > > > On 23/08/2021 13:43, Christian König wrote: > > > > Am 21.08.21 um 11:16 schrieb Gal Pressman: > > > > > On 20/08/2021 17:32, Jason

Re: [RFC] Make use of non-dynamic dmabuf in RDMA

2021-08-24 Thread John Hubbard
On 8/24/21 2:32 AM, Christian König wrote: Am 24.08.21 um 11:06 schrieb Gal Pressman: On 23/08/2021 13:43, Christian König wrote: Am 21.08.21 um 11:16 schrieb Gal Pressman: On 20/08/2021 17:32, Jason Gunthorpe wrote: On Fri, Aug 20, 2021 at 03:58:33PM +0300, Gal Pressman wrote: ... IIUC,

Re: [RFC] Make use of non-dynamic dmabuf in RDMA

2021-08-24 Thread Christian König
Am 24.08.21 um 11:06 schrieb Gal Pressman: On 23/08/2021 13:43, Christian König wrote: Am 21.08.21 um 11:16 schrieb Gal Pressman: On 20/08/2021 17:32, Jason Gunthorpe wrote: On Fri, Aug 20, 2021 at 03:58:33PM +0300, Gal Pressman wrote: Though it would've been nicer if we could agree on a

Re: [RFC] Make use of non-dynamic dmabuf in RDMA

2021-08-23 Thread Christian König
Am 21.08.21 um 11:16 schrieb Gal Pressman: On 20/08/2021 17:32, Jason Gunthorpe wrote: On Fri, Aug 20, 2021 at 03:58:33PM +0300, Gal Pressman wrote: Though it would've been nicer if we could agree on a solution that could work for more than 1-2 RDMA devices, using the existing tools the RDMA

Re: [RFC] Make use of non-dynamic dmabuf in RDMA

2021-08-21 Thread Gal Pressman
On 20/08/2021 17:32, Jason Gunthorpe wrote: > On Fri, Aug 20, 2021 at 03:58:33PM +0300, Gal Pressman wrote: > >> Though it would've been nicer if we could agree on a solution that could work >> for more than 1-2 RDMA devices, using the existing tools the RDMA subsystem >> has. > > I don't think

Re: [RFC] Make use of non-dynamic dmabuf in RDMA

2021-08-20 Thread Jason Gunthorpe
On Fri, Aug 20, 2021 at 03:58:33PM +0300, Gal Pressman wrote: > Though it would've been nicer if we could agree on a solution that could work > for more than 1-2 RDMA devices, using the existing tools the RDMA subsystem > has. I don't think it can really be done, revoke is necessary, and isn't

Re: [RFC] Make use of non-dynamic dmabuf in RDMA

2021-08-20 Thread Jason Gunthorpe
On Fri, Aug 20, 2021 at 09:25:30AM +0200, Daniel Vetter wrote: > On Fri, Aug 20, 2021 at 1:06 AM Jason Gunthorpe wrote: > > On Wed, Aug 18, 2021 at 11:34:51AM +0200, Daniel Vetter wrote: > > > On Wed, Aug 18, 2021 at 9:45 AM Gal Pressman wrote: > > > > > > > > Hey all, > > > > > > > > Currently,

Re: [RFC] Make use of non-dynamic dmabuf in RDMA

2021-08-20 Thread Daniel Vetter
On Fri, Aug 20, 2021 at 1:06 AM Jason Gunthorpe wrote: > On Wed, Aug 18, 2021 at 11:34:51AM +0200, Daniel Vetter wrote: > > On Wed, Aug 18, 2021 at 9:45 AM Gal Pressman wrote: > > > > > > Hey all, > > > > > > Currently, the RDMA subsystem can only work with dynamic dmabuf > > > attachments,

Re: [RFC] Make use of non-dynamic dmabuf in RDMA

2021-08-19 Thread Jason Gunthorpe
On Wed, Aug 18, 2021 at 11:34:51AM +0200, Daniel Vetter wrote: > On Wed, Aug 18, 2021 at 9:45 AM Gal Pressman wrote: > > > > Hey all, > > > > Currently, the RDMA subsystem can only work with dynamic dmabuf > > attachments, which requires the RDMA device to support on-demand-paging > > (ODP) which

Re: [RFC] Make use of non-dynamic dmabuf in RDMA

2021-08-18 Thread Daniel Vetter
On Wed, Aug 18, 2021 at 9:45 AM Gal Pressman wrote: > > Hey all, > > Currently, the RDMA subsystem can only work with dynamic dmabuf > attachments, which requires the RDMA device to support on-demand-paging > (ODP) which is not common on most devices (only supported by mlx5). > > While the

Re: [RFC] Make use of non-dynamic dmabuf in RDMA

2021-08-18 Thread Gal Pressman
On 18/08/2021 11:00, Christian König wrote: > Am 18.08.21 um 09:43 schrieb Gal Pressman: >> Hey all, >> >> Currently, the RDMA subsystem can only work with dynamic dmabuf >> attachments, which requires the RDMA device to support on-demand-paging >> (ODP) which is not common on most devices (only

Re: [RFC] Make use of non-dynamic dmabuf in RDMA

2021-08-18 Thread Christian König
Am 18.08.21 um 09:43 schrieb Gal Pressman: Hey all, Currently, the RDMA subsystem can only work with dynamic dmabuf attachments, which requires the RDMA device to support on-demand-paging (ODP) which is not common on most devices (only supported by mlx5). While the dynamic requirement makes