On 02.06.2025 16:27, Roger Pau Monné wrote: > On Mon, Jun 02, 2025 at 11:46:52AM +0200, Jan Beulich wrote: >> On 30.05.2025 11:23, Roger Pau Monne wrote: >>> The Xen PCI device (vendor ID 0x5853) exposed to x86 HVM guests doesn't >>> have the functionality of a traditional PCI device. The exposed MIO BAR is >>> used by some guests (including Linux) as a safe place to map foreign >>> memory, including the grant table itself. >>> >>> Traditionally BARs from devices have the uncacheable (UC) cache attribute >>> from the MTRR, to ensure correct functionality of such devices. hvmloader >>> mimics this behaviour and sets the MTRR attributes of both the low and high >>> PCI MMIO windows (where BARs of PCI devices reside) as UC in MTRR. >>> >>> This however causes performance issues for the users of the Xen PCI device >>> BAR, as for the purposes of mapping remote memory there's no need to use >>> the UC attribute. On Intel systems this is worked around by using iPAT, >>> that allows the hypervisor to force the effective cache attribute of a p2m >>> entry regardless of the guest PAT value. AMD however doesn't have an >>> equivalent of iPAT, and guest PAT values are always considered. >>> >>> Linux commit: >>> >>> 41925b105e34 xen: replace xen_remap() with memremap() >>> >>> Attempted to mitigate this by forcing mappings of the grant-table to use >>> the write-back (WB) cache attribute. However Linux memremap() takes MTRRs >>> into account to calculate which PAT type to use, and seeing the MTRR cache >>> attribute for the region being UC the PAT also ends up as UC, regardless of >>> the caller having requested WB. >>> >>> As a workaround to allow current Linux to map the grant-table as WB using >>> memremap() special case the Xen PCI device BAR in hvmloader and don't set >>> its cache attribute as UC. >> >> Can we (fully compatibly) make such a change? IOW do we know all possible >> guests would be at least unaffected (ideally affected positively)? Imo ... > > Is there any other possible usage for the xenpci MMIO BAR?
How do you / we know? > My > understanding is it was introduced for this specific purpose; to > signal a safe place to map the grant-table or foreign mappings, which > in both cases want to have an effective WB cache attribute. It's a hack. It's relatively easy to imagine that someone might have built a 2nd hack on top of this 1st one. >>> Such workaround in hvmloader should also be >>> paired with a fix for Linux so it attempts to change the MTRR of the Xen >>> PCI device BAR to WB by itself. >>> >>> Overall, the long term solution would be to provide the guest with a safe >>> range in the guest physical address space where mappings to foreign pages >>> can be created. >> >> ... this is the only viable path. > > I agree, however this will take more time to materialize IMO. Needs a > patch to Linux, plus possible backports, and then distros picking the > updates. > > While I agree this needs fixing in Linux, I don't see any downsides of > doing the workaround in hvmloader also, as a faster way to get it > deployed with just a Xen update. The (maybe merely theoretical) downside is that we may regress something somewhere. Hence why I think that for starters this new behavior should be optional, default off. Later, once proven to work in practice for a fair while, we could then consider changing the default. (Then again I certainly realize that adding yet another control is quite a bit of extra effort, too.) Jan
