On 11/20/25 16:53, Matthew Brost wrote: > On Thu, Nov 20, 2025 at 02:58:58PM +1100, Balbir Singh wrote: >> On 11/20/25 14:15, Matthew Brost wrote: >>> On Thu, Nov 20, 2025 at 01:59:09PM +1100, Balbir Singh wrote: >>>> On 11/20/25 13:50, Balbir Singh wrote: >>>>> On 11/20/25 13:40, Matthew Brost wrote: >>>>>> On Wed, Nov 12, 2025 at 10:52:43AM +1100, Balbir Singh wrote: >>>>>>> On 11/12/25 10:43, Andrew Morton wrote: >>>>>>>> On Thu, 9 Oct 2025 03:33:33 -0700 Matthew Brost >>>>>>>> <[email protected]> wrote: >>>>>>>> >>>>>>>>>>>> This patch series introduces support for Transparent Huge Page >>>>>>>>>>>> (THP) migration in zone device-private memory. The implementation >>>>>>>>>>>> enables >>>>>>>>>>>> efficient migration of large folios between system memory and >>>>>>>>>>>> device-private memory >>>>>>>>>>> >>>>>>>>>>> Lots of chatter for the v6 series, but none for v7. I hope that's a >>>>>>>>>>> good sign. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I hope so too, I've tried to address the comments in v6. >>>>>>>>>> >>>>>>>>> >>>>>>>>> Circling back to this series, we will itegrate and test this version. >>>>>>>> >>>>>>>> How'd it go? >>>>>>>> >>>>>> >>>>>> My apologies for the delay—I got distracted by other tasks in Xe (my >>>>>> driver) and was out for a bit. Unfortunately, this series breaks >>>>>> something in the existing core MM code for the Xe SVM implementation. I >>>>>> have an extensive test case that hammers on SVM, which fully passes >>>>>> prior to applying this series, but fails randomly with the series >>>>>> applied (to drm-tip-rc6) due to the below kernel lockup. >>>>>> >>>>>> I've tried to trace where the migration PTE gets installed but not >>>>>> removed or isolate a test case which causes this failure but no luck so >>>>>> far. I'll keep digging as I have time. >>>>>> >>>>>> Beyond that, if I enable Xe SVM + THP, it seems to mostly work (though >>>>>> the same issue as above eventually occurs), but I do need two additional >>>>>> core MM patches—one is new code required for Xe, and the other could be >>>>>> considered a bug fix. Those patches can included when Xe merges SVM THP >>>>>> support but we need at least not break Xe SVM before this series merges. >>>>>> >>>>>> Stack trace: >>>>>> >>>>>> INFO: task kworker/u65:2:1642 blocked for more than 30 >>>>>> seconds. >>>>>> [ 212.624286] Tainted: G S W 6.18.0-rc6-xe+ #1719 >>>>>> [ 212.630561] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>> disables this message. >>>>>> [ 212.638285] task:kworker/u65:2 state:D stack:0 pid:1642 >>>>>> tgid:1642 ppid:2 task_flags:0x4208060 flags:0x00080000 >>>>>> [ 212.638288] Workqueue: xe_page_fault_work_queue >>>>>> xe_pagefault_queue_work [xe] >>>>>> [ 212.638323] Call Trace: >>>>>> [ 212.638324] <TASK> >>>>>> [ 212.638325] __schedule+0x4b0/0x990 >>>>>> [ 212.638330] schedule+0x22/0xd0 >>>>>> [ 212.638331] io_schedule+0x41/0x60 >>>>>> [ 212.638333] migration_entry_wait_on_locked+0x1d8/0x2d0 >>>>>> [ 212.638336] ? __pfx_wake_page_function+0x10/0x10 >>>>>> [ 212.638339] migration_entry_wait+0xd2/0xe0 >>>>>> [ 212.638341] hmm_vma_walk_pmd+0x7c9/0x8d0 >>>>>> [ 212.638343] walk_pgd_range+0x51d/0xa40 >>>>>> [ 212.638345] __walk_page_range+0x75/0x1e0 >>>>>> [ 212.638347] walk_page_range_mm+0x138/0x1f0 >>>>>> [ 212.638349] hmm_range_fault+0x59/0xa0 >>>>>> [ 212.638351] drm_gpusvm_get_pages+0x194/0x7b0 [drm_gpusvm_helper] >>>>>> [ 212.638354] drm_gpusvm_range_get_pages+0x2d/0x40 [drm_gpusvm_helper] >>>>>> [ 212.638355] __xe_svm_handle_pagefault+0x259/0x900 [xe] >>>>>> [ 212.638375] ? update_load_avg+0x7f/0x6c0 >>>>>> [ 212.638377] ? update_curr+0x13d/0x170 >>>>>> [ 212.638379] xe_svm_handle_pagefault+0x37/0x90 [xe] >>>>>> [ 212.638396] xe_pagefault_queue_work+0x2da/0x3c0 [xe] >>>>>> [ 212.638420] process_one_work+0x16e/0x2e0 >>>>>> [ 212.638422] worker_thread+0x284/0x410 >>>>>> [ 212.638423] ? __pfx_worker_thread+0x10/0x10 >>>>>> [ 212.638425] kthread+0xec/0x210 >>>>>> [ 212.638427] ? __pfx_kthread+0x10/0x10 >>>>>> [ 212.638428] ? __pfx_kthread+0x10/0x10 >>>>>> [ 212.638430] ret_from_fork+0xbd/0x100 >>>>>> [ 212.638433] ? __pfx_kthread+0x10/0x10 >>>>>> [ 212.638434] ret_from_fork_asm+0x1a/0x30 >>>>>> [ 212.638436] </TASK> >>>>>> >>>>> >>>>> Hi, Matt >>>>> >>>>> Thanks for the report, two questions >>>>> >>>>> 1. Are you using mm/mm-unstable, we've got some fixes in there (including >>>>> fixes to remove_migration_pmd()) >>> >>> remove_migration_pmd - This is a PTE migration entry. >>> >> >> I don't have your symbols, I thought we were hitting, the following >> condition in the walk >> >> if (thp_migration_supported() && pmd_is_migration_entry(pmd)) { >> >> But sounds like you are not, PMD/THP has not been enabled in this case >> > > No, migration_entry_wait rather than pmd_migration_entry_wait. > >> >>>>> - Generally a left behind migration entry is a symptom of a failed >>>>> migration that did not clean up >>>>> after itself. >>> >>> I'm on drm-tip as I generally need the latest version of my driver >>> because of the speed we move at. >>> >>> Yes, I agree it looks like somehow a migration PTE is not getting >>> properly removed. >>> >>> I'm happy to cherry pick any patches that you think might be helpful >>> into my tree. >>> >> >> Could you try the mm/mm-new tree with the current xe driver? >> > > Unfortunately, this is a tough one. We land a lot of patches in Xe/DRM, > so bringing the driver up to date with an MM branch is difficult, and > I’m not an expert at merging branches. It would be nice if, in the DRM > flow, we could merge patches from outside our subsystem into a > bleeding-edge kernel for the things we typically care about—but we’d > need a maintainer to sign up for that. > >> In general, w.r.t failure, I would check for the following >> >> 1. Are the dst_pfns in migrate_vma_pages() setup correctly by the device >> driver? >> 2. Any failures in folio_migrate_mapping()? >> 3. In migrate_vma_finalize() check to see if remove_migration_ptes() failed >> >> If (3) fails that will explain the left over migration entries >> > > Good tips, but think I got it via biscet. > > Offending patch is: > > 'mm/migrate_device: handle partially mapped folios during collection' > > The failing test case involves some remap-related issue. It’s a > parameterized test, so I honestly couldn’t tell you exactly what it’s > doing beyond the fact that it seems nonsensical but stresses remap. I > thought commit '66d81853fa3d selftests/mm/hmm-tests: partial unmap, > mremap and anon_write tests' would catch this, but it looks like I need > to make the remap HMM test cases a bit more robust—similar to my > driver-side tests. I can take an action item to follow up on this. > > Good news, I can tell you how to fix this... > > In 'mm/migrate_device: handle partially mapped folios during collection': > > 109 +#if 0 > 110 + folio = page ? page_folio(page) : NULL; > 111 + if (folio && folio_test_large(folio)) { > 112 + int ret; > 113 + > 114 + pte_unmap_unlock(ptep, ptl); > 115 + ret = migrate_vma_split_folio(folio, > 116 + > migrate->fault_page); > 117 + > 118 + if (ret) { > 119 + ptep = pte_offset_map_lock(mm, > pmdp, addr, &ptl); > 120 + goto next; > 121 + } > 122 + > 123 + addr = start; > 124 + goto again; > 125 + } > 126 +#endif > > You can probably just delete this and use my patch below, but if you > want to try fixing it with a quick look: if migrate_vma_split_folio > fails, you probably need to collect a hole. On success, you likely want > to continue executing the remainder of the loop. I can try playing with > this tomorrow, but it’s late here. > > I had privately sent you a version of this patch as a fix for Xe, and > this one seems to work: > > [PATCH] mm/migrate: Split THP found in middle of PMD during page collection > > The migrate layer is not coded to handle a THP found in the middle of a > PMD. This can occur if a user manipulates mappings with mremap(). If a > THP is found mid-PMD during page collection, split it. > > Cc: Balbir Singh <[email protected]> > Signed-off-by: Matthew Brost <[email protected]> > --- > mm/migrate_device.c | 37 +++++++++++++++++++++++++++++++++++-- > 1 file changed, 35 insertions(+), 2 deletions(-) > > diff --git a/mm/migrate_device.c b/mm/migrate_device.c > index abd9f6850db6..9ffc025bad50 100644 > --- a/mm/migrate_device.c > +++ b/mm/migrate_device.c > @@ -65,6 +65,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, > struct vm_area_struct *vma = walk->vma; > struct mm_struct *mm = vma->vm_mm; > unsigned long addr = start, unmapped = 0; > + struct folio *split_folio = NULL; > spinlock_t *ptl; > pte_t *ptep; > > @@ -107,10 +108,11 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, > } > } > > - ptep = pte_offset_map_lock(mm, pmdp, addr, &ptl); > + ptep = pte_offset_map_lock(mm, pmdp, start, &ptl); > if (!ptep) > goto again; > arch_enter_lazy_mmu_mode(); > + ptep += (addr - start) / PAGE_SIZE; > > for (; addr < end; addr += PAGE_SIZE, ptep++) { > struct dev_pagemap *pgmap; > @@ -209,6 +211,11 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, > bool anon_exclusive; > pte_t swp_pte; > > + if (folio_order(folio)) { > + split_folio = folio; > + goto split; > + } > + > flush_cache_page(vma, addr, pte_pfn(pte)); > anon_exclusive = folio_test_anon(folio) && > PageAnonExclusive(page); > @@ -287,8 +294,34 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, > if (unmapped) > flush_tlb_range(walk->vma, start, end); > > +split: > arch_leave_lazy_mmu_mode(); > - pte_unmap_unlock(ptep - 1, ptl); > + pte_unmap_unlock(ptep - 1 + !!split_folio, ptl); > + > + /* > + * XXX: No clean way to support higher-order folios that don't match > PMD > + * boundaries for now — split them instead. Once mTHP support lands, > add > + * proper support for this case. > + * > + * The test, which exposed this as problematic, remapped (memremap) a > + * large folio to an unaligned address, resulting in the folio being > + * found in the middle of the PTEs. The requested number of pages was > + * less than the folio size. Likely to be handled gracefully by upper > + * layers eventually, but not yet. > + */ > + if (split_folio) { > + int ret; > + > + ret = split_folio(split_folio); > + if (fault_folio != split_folio) > + folio_unlock(split_folio); > + folio_put(split_folio); > + if (ret) > + return migrate_vma_collect_skip(addr, end, walk); > + > + split_folio = NULL; > + goto again; > + } > > return 0; > } > > If I apply the #if 0 change along with my patch above (plus one core > MM patch needed for Xe that adds a support function), Xe SVM fully > passes our test cases with both THP enabled and disabled. > Excellent work! Since you found this, do you mind sending the fix to Andrew as a fixup to the original patch. Since I don't have the test case, I have no way of validating the change or any change on top of it would continue to work
FYI: The original code does something similar, I might be missing the migrate_vma_collect_skip() bits. Thanks! Balbir
