Re: [Xen-devel] [PATCH v12 1/6] IOMMU: add a timeout parameter for device IOTLB invalidation
On June 27, 2016 4:29 PM, Jan Beulich wrote: > >>> On 27.06.16 at 10:19, wrote: > > On June 27, 2016 4:03 PM, Jan Beulich wrote: > >> >>> On 24.06.16 at 07:51, wrote: > >> > From: Quan Xu > >> > > >> > The parameter 'iommu_dev_iotlb_timeout' specifies the timeout of > >> > device IOTLB invalidation in milliseconds. By default, the timeout > >> > is > >> > 1000 milliseconds, which can be boot-time changed. > >> > > >> > We also confirmed with VT-d hardware team that 1 milliseconds is > >> > large enough for VT-d IOMMU internal invalidation. > >> > > >> > the existing panic() is eliminated and we bubble up the timeout of > >> > device IOTLB invalidation for further processing, as the PCI-e > >> > Address Translation Services (ATS) mandates a timeout of > >> > 60 seconds for device IOTLB invalidation. Obviously we can't spin > >> > for > >> > 60 seconds or otherwise Xen hypervisor hangs. > >> > > >> > Add a __must_check annotation. The followup patch titled 'VT-d > >> > IOTLB/Context/IEC flush issue' addresses the __mustcheck. > >> > That is the other callers of this routine (two or three levels up) > >> > ignore the return code. This patch does not address this but the > >> > other does. > >> > >> The patch itself looks okay, > > > > Jan, thanks for your review. > > > >> but I'm confused by this paragraph: > >> There's no patch with the named title later in this series. And > >> having gone through this patch I also don't see what remains to be > >> addressed wrt the __must_check-s getting added here. > > > > This paragraph was added from a few rounds ago. I will drop it in next > > version. > > Well, if dropping this paragraph is all that's needed, Yes, > I can do this while > committing: Patches 1-3 appear to be ready to go in. > Ah, that's great. Thanks. Quan ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v12 1/6] IOMMU: add a timeout parameter for device IOTLB invalidation
>>> On 27.06.16 at 10:19, wrote: > On June 27, 2016 4:03 PM, Jan Beulich wrote: >> >>> On 24.06.16 at 07:51, wrote: >> > From: Quan Xu >> > >> > The parameter 'iommu_dev_iotlb_timeout' specifies the timeout of >> > device IOTLB invalidation in milliseconds. By default, the timeout is >> > 1000 milliseconds, which can be boot-time changed. >> > >> > We also confirmed with VT-d hardware team that 1 milliseconds is large >> > enough for VT-d IOMMU internal invalidation. >> > >> > the existing panic() is eliminated and we bubble up the timeout of >> > device IOTLB invalidation for further processing, as the PCI-e Address >> > Translation Services (ATS) mandates a timeout of >> > 60 seconds for device IOTLB invalidation. Obviously we can't spin for >> > 60 seconds or otherwise Xen hypervisor hangs. >> > >> > Add a __must_check annotation. The followup patch titled 'VT-d >> > IOTLB/Context/IEC flush issue' addresses the __mustcheck. >> > That is the other callers of this routine (two or three levels up) >> > ignore the return code. This patch does not address this but the other >> > does. >> >> The patch itself looks okay, > > Jan, thanks for your review. > >> but I'm confused by this paragraph: >> There's no patch with the named title later in this series. And having gone >> through this patch I also don't see what remains to be addressed wrt the >> __must_check-s getting added here. > > This paragraph was added from a few rounds ago. I will drop it in next > version. Well, if dropping this paragraph is all that's needed, I can do this while committing: Patches 1-3 appear to be ready to go in. Jan ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v12 1/6] IOMMU: add a timeout parameter for device IOTLB invalidation
On June 27, 2016 4:03 PM, Jan Beulich wrote: > >>> On 24.06.16 at 07:51, wrote: > > From: Quan Xu > > > > The parameter 'iommu_dev_iotlb_timeout' specifies the timeout of > > device IOTLB invalidation in milliseconds. By default, the timeout is > > 1000 milliseconds, which can be boot-time changed. > > > > We also confirmed with VT-d hardware team that 1 milliseconds is large > > enough for VT-d IOMMU internal invalidation. > > > > the existing panic() is eliminated and we bubble up the timeout of > > device IOTLB invalidation for further processing, as the PCI-e Address > > Translation Services (ATS) mandates a timeout of > > 60 seconds for device IOTLB invalidation. Obviously we can't spin for > > 60 seconds or otherwise Xen hypervisor hangs. > > > > Add a __must_check annotation. The followup patch titled 'VT-d > > IOTLB/Context/IEC flush issue' addresses the __mustcheck. > > That is the other callers of this routine (two or three levels up) > > ignore the return code. This patch does not address this but the other > > does. > > The patch itself looks okay, Jan, thanks for your review. > but I'm confused by this paragraph: > There's no patch with the named title later in this series. And having gone > through this patch I also don't see what remains to be addressed wrt the > __must_check-s getting added here. > This paragraph was added from a few rounds ago. I will drop it in next version. Quan ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v12 1/6] IOMMU: add a timeout parameter for device IOTLB invalidation
>>> On 24.06.16 at 07:51, wrote: > From: Quan Xu > > The parameter 'iommu_dev_iotlb_timeout' specifies the timeout > of device IOTLB invalidation in milliseconds. By default, the > timeout is 1000 milliseconds, which can be boot-time changed. > > We also confirmed with VT-d hardware team that 1 milliseconds > is large enough for VT-d IOMMU internal invalidation. > > the existing panic() is eliminated and we bubble up the timeout > of device IOTLB invalidation for further processing, as the > PCI-e Address Translation Services (ATS) mandates a timeout of > 60 seconds for device IOTLB invalidation. Obviously we can't > spin for 60 seconds or otherwise Xen hypervisor hangs. > > Add a __must_check annotation. The followup patch titled > 'VT-d IOTLB/Context/IEC flush issue' addresses the __mustcheck. > That is the other callers of this routine (two or three levels up) > ignore the return code. This patch does not address this but the > other does. The patch itself looks okay, but I'm confused by this paragraph: There's no patch with the named title later in this series. And having gone through this patch I also don't see what remains to be addressed wrt the __must_check-s getting added here. Jan ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v12 1/6] IOMMU: add a timeout parameter for device IOTLB invalidation
> From: Xu, Quan > Sent: Friday, June 24, 2016 1:52 PM > > From: Quan Xu > > The parameter 'iommu_dev_iotlb_timeout' specifies the timeout > of device IOTLB invalidation in milliseconds. By default, the > timeout is 1000 milliseconds, which can be boot-time changed. > > We also confirmed with VT-d hardware team that 1 milliseconds > is large enough for VT-d IOMMU internal invalidation. > > the existing panic() is eliminated and we bubble up the timeout > of device IOTLB invalidation for further processing, as the > PCI-e Address Translation Services (ATS) mandates a timeout of > 60 seconds for device IOTLB invalidation. Obviously we can't > spin for 60 seconds or otherwise Xen hypervisor hangs. > > Add a __must_check annotation. The followup patch titled > 'VT-d IOTLB/Context/IEC flush issue' addresses the __mustcheck. > That is the other callers of this routine (two or three levels up) > ignore the return code. This patch does not address this but the > other does. > > Signed-off-by: Quan Xu > Acked-by: Kevin Tian ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
[Xen-devel] [PATCH v12 1/6] IOMMU: add a timeout parameter for device IOTLB invalidation
From: Quan Xu The parameter 'iommu_dev_iotlb_timeout' specifies the timeout of device IOTLB invalidation in milliseconds. By default, the timeout is 1000 milliseconds, which can be boot-time changed. We also confirmed with VT-d hardware team that 1 milliseconds is large enough for VT-d IOMMU internal invalidation. the existing panic() is eliminated and we bubble up the timeout of device IOTLB invalidation for further processing, as the PCI-e Address Translation Services (ATS) mandates a timeout of 60 seconds for device IOTLB invalidation. Obviously we can't spin for 60 seconds or otherwise Xen hypervisor hangs. Add a __must_check annotation. The followup patch titled 'VT-d IOTLB/Context/IEC flush issue' addresses the __mustcheck. That is the other callers of this routine (two or three levels up) ignore the return code. This patch does not address this but the other does. Signed-off-by: Quan Xu CC: Julien Grall CC: Jan Beulich CC: Kevin Tian CC: Feng Wu CC: Suravee Suthikulpanit --- v12: 1. enhance commit message. 2. change timeout from 1ms to 1000ms. 3. change IOMMU_QI_TIMEOUT to VTD_QI_TIMEOUT, with VTD_QI_TIMEOUT having its MILLISECS() dropped. 4. enhance the whole expression of 'timeout = ...' 5. drop a blank line that doesn't belong here. --- docs/misc/xen-command-line.markdown | 9 + xen/drivers/passthrough/iommu.c | 3 +++ xen/drivers/passthrough/vtd/qinval.c | 32 +--- xen/include/xen/iommu.h | 2 ++ 4 files changed, 35 insertions(+), 11 deletions(-) diff --git a/docs/misc/xen-command-line.markdown b/docs/misc/xen-command-line.markdown index 7a271c0..0046f0d 100644 --- a/docs/misc/xen-command-line.markdown +++ b/docs/misc/xen-command-line.markdown @@ -1015,6 +1015,15 @@ debug hypervisor only). >> Enable IOMMU debugging code (implies `verbose`). +### iommu\_dev\_iotlb\_timeout +> `= ` + +> Default: `1000` + +Specify the timeout of the device IOTLB invalidation in milliseconds. +By default, the timeout is 1000 ms. When you see error 'Queue invalidate +wait descriptor timed out', try increasing this value. + ### iommu\_inclusive\_mapping (VT-d) > `= ` diff --git a/xen/drivers/passthrough/iommu.c b/xen/drivers/passthrough/iommu.c index ef80b3c..7656aeb 100644 --- a/xen/drivers/passthrough/iommu.c +++ b/xen/drivers/passthrough/iommu.c @@ -24,6 +24,9 @@ static void parse_iommu_param(char *s); static void iommu_dump_p2m_table(unsigned char key); +unsigned int __read_mostly iommu_dev_iotlb_timeout = 1000; +integer_param("iommu_dev_iotlb_timeout", iommu_dev_iotlb_timeout); + /* * The 'iommu' parameter enables the IOMMU. Optional comma separated * value may contain: diff --git a/xen/drivers/passthrough/vtd/qinval.c b/xen/drivers/passthrough/vtd/qinval.c index aa7841a..4788d5f 100644 --- a/xen/drivers/passthrough/vtd/qinval.c +++ b/xen/drivers/passthrough/vtd/qinval.c @@ -28,6 +28,8 @@ #include "vtd.h" #include "extern.h" +#define VTD_QI_TIMEOUT 1 + static void print_qi_regs(struct iommu *iommu) { u64 val; @@ -130,10 +132,10 @@ static void queue_invalidate_iotlb(struct iommu *iommu, spin_unlock_irqrestore(&iommu->register_lock, flags); } -static int queue_invalidate_wait(struct iommu *iommu, -u8 iflag, u8 sw, u8 fn) +static int __must_check queue_invalidate_wait(struct iommu *iommu, + u8 iflag, u8 sw, u8 fn, + bool_t flush_dev_iotlb) { -s_time_t start_time; volatile u32 poll_slot = QINVAL_STAT_INIT; unsigned int index; unsigned long flags; @@ -163,14 +165,20 @@ static int queue_invalidate_wait(struct iommu *iommu, /* Now we don't support interrupt method */ if ( sw ) { +s_time_t timeout; + /* In case all wait descriptor writes to same addr with same data */ -start_time = NOW(); +timeout = NOW() + MILLISECS(flush_dev_iotlb ? +iommu_dev_iotlb_timeout : VTD_QI_TIMEOUT); + while ( poll_slot != QINVAL_STAT_DONE ) { -if ( NOW() > (start_time + DMAR_OPERATION_TIMEOUT) ) +if ( NOW() > timeout ) { print_qi_regs(iommu); -panic("queue invalidate wait descriptor was not executed"); +printk(XENLOG_WARNING VTDPREFIX + " Queue invalidate wait descriptor timed out\n"); +return -ETIMEDOUT; } cpu_relax(); } @@ -180,12 +188,14 @@ static int queue_invalidate_wait(struct iommu *iommu, return -EOPNOTSUPP; } -static int invalidate_sync(struct iommu *iommu) +static int __must_check invalidate_sync(struct iommu *iommu, +bool_t flush_dev_iotlb) { struct qi_ctrl *qi_ctrl = iommu_qi_ctrl(iommu); if ( qi_ctrl->qinval_maddr ) -return queue_invalidate_wait(iommu, 0, 1, 1);