> -----Original Message----- > From: Alex Williamson <[email protected]> > Sent: Tuesday, September 22, 2026 7:43 AM > To: Manish Honap <[email protected]> > Cc: [email protected]; Ankit Agrawal <[email protected]>; [email protected]; > [email protected]; [email protected]; Srirangan > Madhavan <[email protected]>; [email protected]; > [email protected]; [email protected]; [email protected]; > [email protected]; [email protected]; [email protected]; Yishai > Hadas <[email protected]>; Shameer Kolothum Thodi > <[email protected]>; [email protected]; [email protected]; > [email protected]; [email protected]; [email protected]; Neo Jia > <[email protected]>; Krishnakant Jaju <[email protected]>; Vikram Sethi > <[email protected]>; Zhi Wang <[email protected]>; linux- > [email protected]; [email protected]; [email protected]; > [email protected]; [email protected]; linux- > [email protected]; [email protected]; [email protected] > Subject: Re: [PATCH v5 22/27] vfio/cxl: Clear the HDM access gate after a hot > reset > > External email: Use caution opening links or attachments > > > On Thu, 17 Sep 2026 00:05:35 +0530 > <[email protected]> wrote: > > > From: Manish Honap <[email protected]> > > > > The HDM memory fault inserts a PFN only when the decoder is in a > > known-good state (cxl->hdm_valid), so the host CPU never faults the > > coherent range into a disabled decoder. > > > > VFIO_DEVICE_PCI_HOT_RESET drives a plain secondary bus reset rather > > than the CXL reset sequence, so nothing restores the decoder in the > > reset_done handler. A secondary bus reset only reaches a CXL endpoint > > when the upstream port has SBR unmasked (CXL r3.1 sec 8.1.5.2); in > > that case the reset decommits the decoder, so clear cxl->hdm_valid to > > keep a later fault from inserting a PFN into a dead decoder. > > > > A subsequent VFIO_DEVICE_RESET runs the CXL reset sequence and > > restores it, and a masked SBR is a no-op that leaves the decoder intact. > > > > vfio_cxl_sbr_unmasked() mirrors the cxl_sbr_masked() check the PCI > > core uses (drivers/pci/pci.c) for its own CXL bus reset. > > This all sounds very broken. If a hot reset is masked, it should fail, not > silently > do nothing. If a hot reset is unmasked, we can't just invent that the user > needs to follow it with a reset ioctl, and the decoder is dead for the > remainder > of their session otherwise. > > So on one hand, we have a reset we can't use because it doesn't do anything, > and on the other we have a reset that we can't use because in doing > something it makes the device effectively unrecoverable, without some > assumption to perform a reset ioctl. > > Are bus resets just not supported on CXL devices through vfio-pci? > Thanks,
I will try to handle both these cases in v6 correctly: 1. Masked with Unmask SBR clear in the port's CXL DVSEC, the SBR does nothing to the device (CXL r3.1 sec 8.1.5.2). The PCI core already refuses pci_reset_bus_function() in that case. I will add the same check to the bus and slot reset paths and their probes, so VFIO_DEVICE_GET_PCI_HOT_RESET_INFO and VFIO_DEVICE_PCI_HOT_RESET fail for the device instead of doing nothing. A VMM will then use VFIO_DEVICE_RESET, which runs the CXL reset. I will add the PCI core change in its own patch. 2. Unmasked The reset clears the decoder, so vfio-cxl's reset_done restores it with the same cxl-core helper the PCI core's cxl_bus reset method calls after its SBR, and reopens the HDM gate. If the restore fails, the hot reset ioctl returns the error. No follow-up VFIO_DEVICE_RESET is needed. > > Alex > > > > Assisted-by: LLM > > Signed-off-by: Manish Honap <[email protected]> > > --- > > drivers/vfio/pci/cxl/vfio_cxl_core.c | 38 > > ++++++++++++++++++++++++++++ > > 1 file changed, 38 insertions(+) > > > > diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c > > b/drivers/vfio/pci/cxl/vfio_cxl_core.c > > index 37e8a3b54cfb..395b4a5b0956 100644 > > --- a/drivers/vfio/pci/cxl/vfio_cxl_core.c > > +++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c > > @@ -514,8 +514,46 @@ static void vfio_cxl_reset_prepare(struct > > vfio_pci_core_device *vdev) { } > > > > +/* > > + * A secondary bus reset only reaches a CXL endpoint when the > > +upstream port > > + * has SBR unmasked (CXL r3.1 sec 8.1.5.2); otherwise the Bridge > > +Control SBR > > + * bit is ignored and the decoder is left intact. Mirrors the > > +cxl_sbr_masked() > > + * check the PCI core uses for its own CXL bus reset. > > + */ > > +static bool vfio_cxl_sbr_unmasked(struct pci_dev *pdev) { > > + struct pci_dev *bridge = pci_upstream_bridge(pdev); > > + u16 dvsec, ctl; > > + > > + if (!bridge) > > + return false; > > + > > + dvsec = pci_find_dvsec_capability(bridge, PCI_VENDOR_ID_CXL, > > + PCI_DVSEC_CXL_PORT); > > + if (!dvsec) > > + return false; > > + > > + if (pci_read_config_word(bridge, dvsec + PCI_DVSEC_CXL_PORT_CTL, > &ctl)) > > + return false; > > + > > + return ctl & PCI_DVSEC_CXL_PORT_CTL_UNMASK_SBR; > > +} > > + > > static void vfio_cxl_reset_done(struct vfio_pci_core_device *vdev) { > > + struct vfio_cxl_state *cxl = vdev->cxl; > > + > > + /* > > + * VFIO_DEVICE_PCI_HOT_RESET drives a plain secondary bus reset, not > the > > + * CXL-aware cxl_reset_bus_function(), so nothing restores the HDM > > + * decoder here. When the upstream port has SBR unmasked the reset > > + * decommits the decoder; gate host access to the HDM range so a later > > + * fault cannot insert a PFN into a dead decoder. A VFIO_DEVICE_RESET > > + * then runs the CXL reset sequence and restores it. A masked SBR is a > > + * no-op and leaves the decoder intact. > > + */ > > + if (vfio_cxl_sbr_unmasked(vdev->pdev)) > > + cxl->hdm_valid = false; > > } > > > > static const struct vfio_cxl_ops vfio_cxl_ops = {
