On Thu, Aug 13, 2026 at 06:23:09PM -0700, Matthew Brost wrote: > On Wed, Aug 12, 2026 at 04:28:47PM +0800, Neil Zhong wrote: > > Hi Matt, > > > > > Can you give me instructions on how to recreate this on our end and your > > > machine, memory details? I have a bunch of various reproducers which I > > > have been using for shrinker work and the more the better. > > > > Yes. The test system and workload are as follows. > > > > - Panther Lake iGPU using shared system memory > > - 8 GiB installed memory; /proc/meminfo reports 7,723,392 KiB > > - 4 KiB base pages > > - four zram swap devices, 3,858,416 KiB in total > > - Linux 6.18.15, x86_64, PREEMPT_DYNAMIC > > - X11 fullscreen output at 3840x2160 and 60 Hz > > - a 3840x2160, 60 fps, HEVC HDR video played continuously in a loop > > > > The player uses hardware decoding. Each decoded frame is imported through > > DMA-BUF and processed by libplacebo/OpenGL for HDR tone mapping before > > presentation. It calls glFlush() for every rendered frame. During steady > > playback, the player accounts for about 1.85 GiB of logical Xe BO > > allocation, including about 672 MiB reported as shared. > > > > The player is not currently public. However, an equivalent pipeline should > > This will become a problem if we introduce a uAPI, as we need an > open-source consumer of that uAPI. Fortunately, it looks like we are > already moving toward pinning uAPIs for several other reasons as well. > > > reproduce the condition if it keeps the decoded surfaces, HDR intermediate > > render targets and presentation surfaces alive, rather than creating a > > small synthetic BO set. > > > > No additional memory-pressure tool was used for the first reproduction. > > I start the video, let its working set reach steady state, and then capture > > ten minutes while playback continues. On the 8 GiB system, the priority-only > > run had MemAvailable between 2.89 and 3.15 GiB, while 97.8% of kswapd > > wakeups > > were for order-10 allocations. Thus this reproduces without forcing an > > We have upstream fixes for the order-10 allocations to avoid triggering > reclaim for anything other than order-0 or order-9 allocations. I shared > that patch in a previous reply, and it is probably worth pulling in as a > mitigation, plus the core MM series. > > > order-0 shortage. > > > > I collected the trace with: > > > > sudo ./capture_xe_memory_churn.sh \ > > -t 600 \ > > -s 0.2 \ > > -o ./xe-memory-churn > > > > `-t 600` records ten minutes. `-t 0` can instead be used to record until > > Ctrl-C. The 0.2 second option is only the /proc and TTM-pool sampling > > interval; ftrace events are recorded continuously. > > > > The player also logs one line after each frame submission in this form: > > > > gl_sw_submit_frame timing: ... Flush=123.456 ms > > > > The log prefix contains the wall-clock timestamp. An equivalent reproducer > > can record the time immediately after glFlush() returns and the measured > > duration. The trace script inserts a wall-clock epoch marker into a > > mono_raw ftrace stream so that the two timelines can be aligned. > > > > The figures in my previous email came from the 6.18.15 kernel with [9]-[13] > > backported, the fragmentation check applied to direct reclaim as well, and > > the high-plus-min watermark experiment described there. The default-device > > watermark capture used vm.min_free_kbytes=131072. The follow-up used 50000; > > the workload and trace procedure were otherwise unchanged. > > > > > Also a quick write up how you extracted these numbers from reproducer so > > > I can recreate on my end. > > > > The capture script creates temporary entry and return kprobes for: > > > > ttm_tt_backup() > > ttm_tt_restore() > > ttm_pool_alloc() > > ttm_pool_free() > > ttm_pool_shrink() > > > > It also traces the Xe shrinker, TTM restore and cache-attribute functions, > > kswapd and direct-reclaim events, compaction, and allocation > > fragmentation. > > > > At ttm_tt_backup() entry, the probe records the ttm_tt pointer and > > num_pages. At return, it records the positive return value, which is the > > number of pages actually backed up. I sum those successful return values, > > not the requested page count, when reporting backup volume. A later > > ttm_tt_restore() is matched to the most recent successful backup using the > > same ttm_tt pointer. That provides per-object backup-to-restore latency and > > repeated-cycle counts. > > > > The byte-volume calculation is: > > > > backup bytes = sum(successful backup return pages) * PAGE_SIZE > > restore bytes = sum(num_pages for matched restores) * PAGE_SIZE > > > > The reported backup-plus-restore volume is the sum of those two values. > > It is cumulative migration/copy traffic, not resident memory and not net > > memory freed. A shmem backup remains resident system memory unless those > > shmem pages are subsequently swapped out. > > > > > This customer, in particular, utilizes priority bands to express this > > > heuristic (e.g., the compositor is the highest priority, any > > > non-privileged UI-related content is normal priority, and everything > > > else is low priority). > > > > I think priority bands are useful for relative reclaim ordering, but they > > do not by themselves express the guarantee needed here. > > > > Consider a system with one large GPU workload. If nearly all reclaimable > > BOs belong to that client and are placed in the high-priority band, the low > > bands will contain few or no candidates. When enough memory is requested, > > the shrinker must eventually enter the high band. In that case, high > > priority delays reclaim but does not prevent it. The priority-only test > > showed exactly this limitation: over ten minutes there were 6,743 > > successful backups and 6,693 restores, with 157 Flush calls over 16.667 ms > > and a maximum Flush of 494.434 ms. > > > > I am not suggesting that every BO of a high-priority client should be > > unreclaimable. Whether reclaim is acceptable depends on the workload, and > > the kernel cannot infer that semantic from WC, VM-bound state, client count > > or BO size alone. For example, the driver-visible behavior of these two > > workloads can look very similar: > > > > 1. Foreground 4K60 HDR playback. Its active decoded surfaces, HDR render > > targets and presentation surfaces have a 16.667 ms deadline. Backing > > them up and restoring them causes an immediate and clearly visible > > product failure. These BOs should avoid eviction and shrinking while > > they are part of the active visual pipeline. > > > > 2. Background image recognition or classification. Its BOs have no > > presentation deadline. Reclaiming them under system memory pressure > > is reasonable, even if the job later has to reconstruct its working > > set. > > In this case, do you have two VMs sharing BOs? If the foreground and > background tasks share a single VM, it does not matter which BOs are > shrunk or restored because, when a VM is validated during an exec IOCTL, > all BOs mapped within that VM are restored. > > I assume that in this case you are using two VMs that share buffers as > needed via dma-buf. Is that correct? If not, pinning is not going to > help unless the pinned set includes everything. > > > > > Priority bands cannot distinguish those cases if both clients assign their > > Assuming there are two VMs here, set your foreground queue's priority to > NORMAL and your background task queue's priority to LOW. (Alternatively, > if the foreground task has CAP_SYS_ADMIN privileges, you could set it to > HIGH, etc.) This should cause the background BOs to be shrunk rather > than the foreground BOs. > > There is another problem in Xe when VMs share BOs. The way a VM is > locked and validated during exec IOCTLs can cause cross-VM lock > contention get stuck behing shrinking. This issue exists regardless of > whether a priority-based solution or pinning is used. > > For example, assume there are two VMs, one for the foreground workload > and one for the background workload. We correctly evict the background > BOs when needed, and some set of dma-buf buffers is shared between the > two VMs. Below a flow that shows a contention problem: > > 1. a private BO from background is shrunk > 2. background exec IOCTL > 2.1. grab all dma-resv locks (including some shared with > foreground) > 2.2. restore shunk BO > 2.3. submit GPU job > 3. foreground exec IOCTL (in parallel with 2) > 3.1 grab all dma-resv locks (including some shared with > background) > 3.2. submit GPU job > > In this example 3.1 can get stuck behind 2.2 (an unrelated restore), > thus 3 can miss a presentation deadline. > > We likely need our Xe IOCTL and GPUVM code to be smart enough to perform > multiple lock-and-validate passes. For example, we could initially lock > only the eviction set, validate everything, and then repeat the process > until the eviction set is empty. Only after that would we lock and > validate everything else. > > This is an existing problem that really needs to be fixed. I'll probably > take a look at addressing it. >
Here is an attempt at fixing the cross-VM issue: https://patchwork.freedesktop.org/series/172205/ Matt > > current working set a high relative priority. This is why I think the > > business semantic has to come from userspace. Priority bands can remain the > > general ordering mechanism, while a separate, explicit and bounded > > NO_SHRINK or latency-critical mark protects only the BOs in an active visual > > pipeline. The mark should be removed as soon as a surface leaves that > > working set. > > > > > To be clear - this would be an addition to fixes discussed above, right? > > > > Yes. I see explicit workload-semantic protection as an addition to the > > core MM fragmentation/shrinker fixes and the Xe/TTM priority bands, not a > > replacement for either. I agree that testing series 168651 together with > > the priority bands is still useful for general working-set preservation. > > It can reduce accidental reclaim, while an accounted NO_SHRINK contract > > handles the smaller set for which a missed presentation deadline is not an > > acceptable reclaim tradeoff. > > I don't think anyone is opposed to pinning if we can get the right > permission control in place (most likely cgroups). > > Matt > > > > > Thanks, > > Neil
