On 25.08.25 21:10, David Hildenbrand wrote: > On 21.08.25 10:10, Christian König wrote: >> On 20.08.25 17:23, David Hildenbrand wrote: >>> CCing Lorenzo >>> >>> On 20.08.25 16:33, Christian König wrote: >>>> Hi everyone, >>>> >>>> sorry for CCing so many people, but that rabbit hole turned out to be >>>> deeper than originally thought. >>>> >>>> TTM always had problems with UC/WC mappings on 32bit systems and drivers >>>> often had to revert to hacks like using GFP_DMA32 to get things working >>>> while having no rational explanation why that helped (see the TTM AGP, >>>> radeon and nouveau driver code for that). >>>> >>>> It turned out that the PAT implementation we use on x86 not only enforces >>>> the same caching attributes for pages in the linear kernel mapping, but >>>> also for highmem pages through a separate R/B tree. >>>> >>>> That was unexpected and TTM never updated that R/B tree for highmem pages, >>>> so the function pgprot_set_cachemode() just overwrote the caching >>>> attributes drivers passed in to vmf_insert_pfn_prot() and that essentially >>>> caused all kind of random trouble. >>>> >>>> An R/B tree is potentially not a good data structure to hold thousands if >>>> not millions of different attributes for each page, so updating that is >>>> probably not the way to solve this issue. >>>> >>>> Thomas pointed out that the i915 driver is using apply_page_range() >>>> instead of vmf_insert_pfn_prot() to circumvent the PAT implementation and >>>> just fill in the page tables with what the driver things is the right >>>> caching attribute. >>> >>> I assume you mean apply_to_page_range() -- same issue in patch subjects. >> >> Oh yes, of course. Sorry. >> >>> Oh this sounds horrible. Why oh why do we have these hacks in core-mm and >>> have drivers abuse them :( >> >> Yeah I was also a bit hesitated to use that, but the performance advantage >> is so high that we probably can't avoid the general approach. >> >>> Honestly, apply_to_pte_range() is just the entry in doing all kinds of >>> weird crap to page tables because "you know better". >> >> Exactly that's the problem I'm pointing out, drivers *do* know it better. >> The core memory management has applied incorrect values which caused all >> kind of the trouble. >> >> The problem is not a bug in PAT nor TTM/drivers but rather how they interact >> with each other. >> >> What I don't understand is why do we have the PAT in the first place? No >> other architecture does it this way. > > Probably because no other architecture has these weird glitches I assume ... > skimming over memtype_reserve() and friends there are quite some corner cases > the code is handling (BIOS, ACPI, low ISA, system RAM, ...) > > > I did a lot of work on the higher PAT level functions, but I am no expert on > the lower level management functions, and in particular all the special cases > with different memory types. > > IIRC, the goal of the PAT subsystem is to make sure that no two page tables > map the same PFN with different caching attributes.
Yeah, that actually makes sense. Thomas from Intel recently explained the technical background to me: Some x86 CPUs write back cache lines even if they aren't dirty and what can happen is that because of the linear mapping the CPU speculatively loads a cache line which is elsewhere mapped uncached. So the end result is that the writeback of not dirty cache lines potentially corrupts the data in the otherwise uncached system memory. But that a) only applies to memory in the linear mapping and b) only to a handful of x86 CPU types (e.g. recently Intels Luna Lake, AMD Athlons produced before 2004, maybe others). > It treats ordinary system RAM (IORESOURCE_SYSTEM_RAM) usually in a special > way: no special caching mode. > > For everything else, it expects that someone first reserves a memory range > for a specific caching mode. > > For example, remap_pfn_range()...->pfnmap_track()->memtype_reserve() will > make sure that there are no conflicts, to the call memtype_kernel_map_sync() > to make sure the identity mapping is updated to the new type. > > In case someone ends up calling pfnmap_setup_cachemode(), the expectation is > that there was a previous call to memtype_reserve_io() or similar, such that > pfnmap_setup_cachemode() will find that caching mode. > > > So my assumption would be that that is missing for the drivers here? Well yes and no. See the PAT is optimized for applying specific caching attributes to ranges [A..B] (e.g. it uses an R/B tree). But what drivers do here is that they have single pages (usually for get_free_page or similar) and want to apply a certain caching attribute to it. So what would happen is that we completely clutter the R/B tree used by the PAT with thousands if not millions of entries. > > Last time I asked where this reservation is done, Peter Xu explained [1] it > at least for VFIO: > > vfio_pci_core_mmap > pci_iomap > pci_iomap_range > ... > __ioremap_caller > memtype_reserve > > > Now, could it be that something like that is missing in these drivers > (ioremap etc)? Well that would solve the issue temporary, but I'm pretty sure that will just go boom at a different place then :( One possibility would be to say that the PAT only overrides the attributes if they aren't normal cached and leaves everything else alone. What do you think? Thanks, Christian. > > > > [1] https://lkml.kernel.org/r/aBDXr-Qp4z0tS50P@x1.local > > >> >> Is that because of the of x86 CPUs which have problems when different page >> tables contain different caching attributes for the same physical memory? > > Yes, but I don't think x86 is special here. >
