Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-05-07 Thread Jason Gunthorpe
On Mon, May 06, 2024 at 11:50:36PM +, Matthew Brost wrote: > > I think like with the gpu vma stuff we should at least aim for the core > > data structures, and more importantly, the locking design and how it > > interacts with core mm services to be common code. > > I believe this is a

Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-05-06 Thread Matthew Brost
On Mon, May 06, 2024 at 03:04:15PM +0200, Daniel Vetter wrote: > On Sat, May 04, 2024 at 11:03:03AM +1000, Dave Airlie wrote: > > > Let me know if this understanding is correct. > > > > > > Or what would you like to do in such situation? > > > > > > > > > > > Not sure how it is really a good idea.

Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-05-06 Thread Jason Gunthorpe
On Fri, May 03, 2024 at 08:29:39PM +, Zeng, Oak wrote: > > > But we have use case where we want to fault-in pages other than the > > > page which contains the GPU fault address, e.g., user malloc'ed or > > > mmap'ed 8MiB buffer, and no CPU touching of this buffer before GPU > > > access it.

Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-05-06 Thread Daniel Vetter
On Sat, May 04, 2024 at 11:03:03AM +1000, Dave Airlie wrote: > > Let me know if this understanding is correct. > > > > Or what would you like to do in such situation? > > > > > > > > Not sure how it is really a good idea. > > > > > > Adaptive locality of memory is still an unsolved problem in

Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-05-03 Thread Dave Airlie
> Let me know if this understanding is correct. > > Or what would you like to do in such situation? > > > > > Not sure how it is really a good idea. > > > > Adaptive locality of memory is still an unsolved problem in Linux, > > sadly. > > > > > > However, the migration stuff should really not be

RE: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-05-03 Thread Zeng, Oak
, Himal Prasad > ; Bommu, Krishnaiah > ; Vishwanathapura, Niranjana > ; Leon Romanovsky > > Subject: Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table > from hmm range > > On Fri, May 03, 2024 at 02:43:19PM +, Zeng, Oak wrote: > > > > 2. >

Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-05-03 Thread Jason Gunthorpe
On Fri, May 03, 2024 at 02:43:19PM +, Zeng, Oak wrote: > > > 2. > > > Then call hmm_range_fault a second time > > > Setting the hmm_range start/end only to cover valid pfns > > > With all valid pfns, set the REQ_FAULT flag > > > > Why would you do this? The first already did the faults you

RE: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-05-03 Thread Zeng, Oak
, Himal Prasad > ; Bommu, Krishnaiah > ; Vishwanathapura, Niranjana > ; Leon Romanovsky > > Subject: Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table > from hmm range > > On Thu, May 02, 2024 at 07:25:50PM +, Zeng, Oak wrote: > > Hi Jason, > > &g

Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-05-03 Thread Jason Gunthorpe
On Thu, May 02, 2024 at 07:25:50PM +, Zeng, Oak wrote: > Hi Jason, > > I tried to understand how you supposed us to use hmm range fault... it seems > you want us to call hmm range fault two times on each gpu page fault: > 1. > Call Hmm_range_fault first time, pfn of the fault address is

RE: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-05-02 Thread Zeng, Oak
l Vetter ; Zeng, Oak ; dri- > de...@lists.freedesktop.org; intel...@lists.freedesktop.org; Brost, > Matthew ; Welty, Brian > ; Ghimiray, Himal Prasad > ; Bommu, Krishnaiah > ; Vishwanathapura, Niranjana > ; Leon Romanovsky > > Subject: Re: [PATCH 06/23] drm/xe/svm: Introduc

Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-05-02 Thread Thomas Hellström
On Thu, 2024-05-02 at 09:46 -0300, Jason Gunthorpe wrote: > On Thu, May 02, 2024 at 11:11:04AM +0200, Thomas Hellström wrote: > > > It's true the cpu vma lookup is a remnant from amdkfd. The idea > > here is > > to replace that with fixed prefaulting ranges of tunable size. So > > far, > > as you

Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-05-02 Thread Jason Gunthorpe
On Thu, May 02, 2024 at 11:11:04AM +0200, Thomas Hellström wrote: > It's true the cpu vma lookup is a remnant from amdkfd. The idea here is > to replace that with fixed prefaulting ranges of tunable size. So far, > as you mention, the prefaulting range has been determined by the CPU > vma size.

Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-05-02 Thread Thomas Hellström
Hi, Although I haven't had a chance yet to eye through the current code, some comments along the way. On Thu, 2024-05-02 at 10:04 +0200, Daniel Vetter wrote: > On Tue, Apr 30, 2024 at 09:09:15PM -0300, Jason Gunthorpe wrote: > > On Tue, Apr 30, 2024 at 08:57:48PM +0200, Daniel Vetter wrote: > >

Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-05-02 Thread Daniel Vetter
On Tue, Apr 30, 2024 at 09:09:15PM -0300, Jason Gunthorpe wrote: > On Tue, Apr 30, 2024 at 08:57:48PM +0200, Daniel Vetter wrote: > > On Tue, Apr 30, 2024 at 02:30:02PM -0300, Jason Gunthorpe wrote: > > > On Mon, Apr 29, 2024 at 10:25:48AM +0200, Thomas Hellström wrote: > > > > > > > > Yes there

Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-04-30 Thread Jason Gunthorpe
On Tue, Apr 30, 2024 at 08:57:48PM +0200, Daniel Vetter wrote: > On Tue, Apr 30, 2024 at 02:30:02PM -0300, Jason Gunthorpe wrote: > > On Mon, Apr 29, 2024 at 10:25:48AM +0200, Thomas Hellström wrote: > > > > > > Yes there is another common scheme where you bind a window of CPU to > > > > a > > >

Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-04-30 Thread Daniel Vetter
On Tue, Apr 30, 2024 at 02:30:02PM -0300, Jason Gunthorpe wrote: > On Mon, Apr 29, 2024 at 10:25:48AM +0200, Thomas Hellström wrote: > > > > Yes there is another common scheme where you bind a window of CPU to > > > a > > > window on the device and mirror a fixed range, but this is a quite > > >

Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-04-30 Thread Jason Gunthorpe
On Mon, Apr 29, 2024 at 10:25:48AM +0200, Thomas Hellström wrote: > > Yes there is another common scheme where you bind a window of CPU to > > a > > window on the device and mirror a fixed range, but this is a quite > > different thing. It is not SVA, it has a fixed range, and it is > > probably

Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-04-29 Thread Thomas Hellström
On Fri, 2024-04-26 at 13:35 -0300, Jason Gunthorpe wrote: > On Fri, Apr 26, 2024 at 04:49:26PM +0200, Thomas Hellström wrote: > > On Fri, 2024-04-26 at 09:00 -0300, Jason Gunthorpe wrote: > > > On Fri, Apr 26, 2024 at 11:55:05AM +0200, Thomas Hellström wrote: > > > > First, the gpu_vma structure

Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-04-26 Thread Jason Gunthorpe
On Fri, Apr 26, 2024 at 04:49:26PM +0200, Thomas Hellström wrote: > On Fri, 2024-04-26 at 09:00 -0300, Jason Gunthorpe wrote: > > On Fri, Apr 26, 2024 at 11:55:05AM +0200, Thomas Hellström wrote: > > > First, the gpu_vma structure is something that partitions the > > > gpu_vm > > > that holds

Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-04-26 Thread Thomas Hellström
On Fri, 2024-04-26 at 09:00 -0300, Jason Gunthorpe wrote: > On Fri, Apr 26, 2024 at 11:55:05AM +0200, Thomas Hellström wrote: > > First, the gpu_vma structure is something that partitions the > > gpu_vm > > that holds gpu-related range metadata, like what to mirror, desired > > gpu > > caching

Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-04-26 Thread Jason Gunthorpe
On Fri, Apr 26, 2024 at 11:55:05AM +0200, Thomas Hellström wrote: > First, the gpu_vma structure is something that partitions the gpu_vm > that holds gpu-related range metadata, like what to mirror, desired gpu > caching policies etc. These are managed (created, removed and split) > mainly from

Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-04-26 Thread Thomas Hellström
Hi, Jason. I've quickly read through the discussion here and have a couple of questions and clarifications to hopefully help moving forward on aligning on an approach for this. Let's for simplicity initially ignore migration and assume this is on integrated hardware since it seems like it's

Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-04-24 Thread Jason Gunthorpe
On Wed, Apr 24, 2024 at 11:59:18PM +, Zeng, Oak wrote: > Hi Jason, > > I went through the conversation b/t you and Matt. I think we are pretty much > aligned. Here is what I get from this threads: > > 1) hmm range fault size, gpu page table map size : you prefer bigger > gpu vma size and

RE: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-04-24 Thread Zeng, Oak
freedesktop.org; intel...@lists.freedesktop.org; Brost, > Matthew ; thomas.hellst...@linux.intel.com; > Welty, Brian ; Ghimiray, Himal Prasad > ; Bommu, Krishnaiah > ; Vishwanathapura, Niranjana > ; Leon Romanovsky > > Subject: Re: [PATCH 06/23] drm/xe/svm: Introduce a helpe

Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-04-24 Thread Jason Gunthorpe
On Wed, Apr 24, 2024 at 04:56:57PM +, Matthew Brost wrote: > > What "meta data" is there for a SVA mapping? The entire page table is > > an SVA. > > If we have allocated memory for GPU page tables in the range, This is encoded directly in the radix tree. > if range > has been invalidated,

Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-04-24 Thread Matthew Brost
On Wed, Apr 24, 2024 at 01:44:11PM -0300, Jason Gunthorpe wrote: > On Wed, Apr 24, 2024 at 04:35:17PM +, Matthew Brost wrote: > > On Wed, Apr 24, 2024 at 10:57:54AM -0300, Jason Gunthorpe wrote: > > > On Wed, Apr 24, 2024 at 02:31:36AM +, Matthew Brost wrote: > > > > > > > AMD seems to

Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-04-24 Thread Jason Gunthorpe
On Wed, Apr 24, 2024 at 04:35:17PM +, Matthew Brost wrote: > On Wed, Apr 24, 2024 at 10:57:54AM -0300, Jason Gunthorpe wrote: > > On Wed, Apr 24, 2024 at 02:31:36AM +, Matthew Brost wrote: > > > > > AMD seems to register notifiers on demand for parts of the address space > > > [1], I

Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-04-24 Thread Matthew Brost
On Wed, Apr 24, 2024 at 10:57:54AM -0300, Jason Gunthorpe wrote: > On Wed, Apr 24, 2024 at 02:31:36AM +, Matthew Brost wrote: > > > AMD seems to register notifiers on demand for parts of the address space > > [1], I think Nvidia's open source driver does this too (can look this up > > if

Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-04-24 Thread Jason Gunthorpe
On Wed, Apr 24, 2024 at 02:31:36AM +, Matthew Brost wrote: > AMD seems to register notifiers on demand for parts of the address space > [1], I think Nvidia's open source driver does this too (can look this up > if needed). We (Intel) also do this in Xe and the i915 for userptrs > (explictly

Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-04-24 Thread Jason Gunthorpe
On Tue, Apr 23, 2024 at 09:17:03PM +, Zeng, Oak wrote: > > On Tue, Apr 09, 2024 at 04:45:22PM +, Zeng, Oak wrote: > > > > > > I saw, I am saying this should not be done. You cannot unmap bits of > > > > a sgl mapping if an invalidation comes in. > > > > > > You are right, if we register a

Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-04-23 Thread Matthew Brost
ian > > ; Ghimiray, Himal Prasad > > ; Bommu, Krishnaiah > > ; Vishwanathapura, Niranjana > > ; Leon Romanovsky > > Subject: Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table > > from > > hmm range > > > > On Tue, Apr 09, 2024 a

RE: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-04-23 Thread Zeng, Oak
ktop.org; intel...@lists.freedesktop.org; Brost, > Matthew > ; thomas.hellst...@linux.intel.com; Welty, Brian > ; Ghimiray, Himal Prasad > ; Bommu, Krishnaiah > ; Vishwanathapura, Niranjana > ; Leon Romanovsky > Subject: Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table &g

Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-04-09 Thread Matthew Brost
Matthew ; thomas.hellst...@linux.intel.com; > > Welty, Brian ; Ghimiray, Himal Prasad > > ; Bommu, Krishnaiah > > ; Vishwanathapura, Niranjana > > ; Leon Romanovsky > > Subject: Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table > > from > &g

Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-04-09 Thread Jason Gunthorpe
On Tue, Apr 09, 2024 at 04:45:22PM +, Zeng, Oak wrote: > > I saw, I am saying this should not be done. You cannot unmap bits of > > a sgl mapping if an invalidation comes in. > > You are right, if we register a huge mmu interval notifier to cover > the whole address space, then we should use

RE: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-04-09 Thread Zeng, Oak
vsky > Subject: Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table > from > hmm range > > On Fri, Apr 05, 2024 at 04:42:14PM +, Zeng, Oak wrote: > > > > Above codes deal with a case where dma map is not needed. As I > > > > understand

Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-04-05 Thread Jason Gunthorpe
On Fri, Apr 05, 2024 at 04:42:14PM +, Zeng, Oak wrote: > > > Above codes deal with a case where dma map is not needed. As I > > > understand it, whether we need a dma map depends on the devices > > > topology. For example, when device access host memory or another > > > device's memory through

RE: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-04-05 Thread Zeng, Oak
Brian ; Ghimiray, Himal Prasad > ; Bommu, Krishnaiah > ; Vishwanathapura, Niranjana > ; Leon Romanovsky > Subject: Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table > from > hmm range > > On Fri, Apr 05, 2024 at 03:33:10AM +, Zeng, Oak wrote: > > &

Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-04-05 Thread Jason Gunthorpe
On Fri, Apr 05, 2024 at 03:33:10AM +, Zeng, Oak wrote: > > > > I didn't look at this series a lot but I wanted to make a few > > remarks.. This I don't like quite a lot. Yes, the DMA API interaction > > with hmm_range_fault is pretty bad, but it should not be hacked > > around like this. Leon

RE: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-04-04 Thread Zeng, Oak
imal Prasad > ; Bommu, Krishnaiah > ; Vishwanathapura, Niranjana > ; Leon Romanovsky > Subject: Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table > from > hmm range > > On Wed, Jan 17, 2024 at 05:12:06PM -0500, Oak Zeng wrote: > > +/** > > + *

Re: [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range

2024-04-04 Thread Jason Gunthorpe
On Wed, Jan 17, 2024 at 05:12:06PM -0500, Oak Zeng wrote: > +/** > + * xe_svm_build_sg() - build a scatter gather table for all the physical > pages/pfn > + * in a hmm_range. > + * > + * @range: the hmm range that we build the sg table from. range->hmm_pfns[] > + * has the pfn numbers of pages