Re: [Xen-devel] [PATCH v15 0/3] VT-d Device-TLB flush issue

[Xu, Quan]

On July 08, 2016 2:52 PM, Jan Beulich <jbeul...@suse.com> wrote:
> >>> On 08.07.16 at 08:39, <jbeul...@suse.com> wrote:
> > From: Quan Xu <quan...@intel.com>
> >
> > these patches fix current timeout concern and also allow limited ATS
> > support.
> >
> > these patches are the rest ones:
> > 1. move the domain crash logic up to the generic IOMMU layer
> >
> > 2. If Device-TLB flush timed out, we hide the target ATS device
> >immediately. By hiding the device, we make sure it can't be
> >assigned to any domain any longer (see device_assigned).
> >
> > Quan Xu (3):
> >   IOMMU/ATS: use a struct pci_dev * instead of SBDF
> >   IOMMU: add domain crash logic
> >   VT-d: fix Device-TLB flush timeout issue
> 
> I should probably have said somewhere that obviously I didn't test this for 
> the
> actual ATS case, as I don't have access to any suitable device. Therefore,
> alongside the necessary ack-s, it would be nice if we could also get a 
> Tested-by
> from someone who does have an ATS device.
> 

(today I just come back to return my laptop and while test it) thanks for your 
work and education :):)...

Tested-by: Quan Xu <quan...@intel.com>

Quan
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [v9 00/19] QEMU:Xen stubdom vTPM for HVM virtual machine(QEMU Part)

[Xu, Quan]

Emil, Thanks for your  effort ( today I just come back to return my laptop).

btw, sstabell...@kernel.org may be the right email.
 Stefan / Stefano,  could you help us review these patches? Thanks in advance!!

Quan 

 
On July 10, 2016 7:48 PM, Emil Condrea  wrote:
> *INTRODUCTION*
> The goal of virtual Trusted Platform Module (vTPM) is to provide a TPM
> functionality to virtual machines (Fedora, Ubuntu, Redhat, Windows .etc).
> This allows programs to interact with a TPM in a virtual machine the same way
> they interact with a TPM on the physical system. Each virtual machine gets its
> own unique, emulated, software TPM. Each major component of vTPM is
> implemented as a stubdom, providing secure separation guaranteed by the
> hypervisor.
> 
> The vTPM stubdom is a Xen mini-OS domain that emulates a TPM for the
> virtual machine to use. It is a small wrapper around the Berlios TPM emulator.
> TPM commands are passed from mini-os TPM backend driver.
> 
> *ARCHITECTURE*
> The architecture of stubdom vTPM for HVM virtual machine:
> 
> ++
> | Windows/Linux DomU | ...
> ||  ^|
> |v  ||
> |  Qemu tpm1.2 Tis   |
> ||  ^|
> |v  ||
> | XenStubdoms backend|
> ++
>  |  ^
>  v  |
> ++
> |  XenDevOps |
> ++
>  |  ^
>  v  |
> ++
> |  mini-os/tpmback   |
> ||  ^|
> |v  ||
> |   vtpm-stubdom | ...
> ||  ^|
> |v  ||
> |  mini-os/tpmfront  |
> ++
>  |  ^
>  v  |
> ++
> |  mini-os/tpmback   |
> ||  ^|
> |v  ||
> |  vtpmmgr-stubdom   |
> ||  ^|
> |v  ||
> |  mini-os/tpm_tis   |
> ++
>  |  ^
>  v  |
> ++
> |Hardware TPM|
> ++
> 
>  * Windows/Linux DomU:
> The HVM based guest that wants to use a vTPM. There may be
> more than one of these.
> 
>  * Qemu tpm1.2 Tis:
> Implementation of the tpm1.2 Tis interface for HVM virtual
> machines. It is Qemu emulation device.
> 
>  * vTPM xenstubdoms driver:
> Qemu vTPM driver. This driver provides vtpm initialization
> and sending data and commends to a para-virtualized vtpm
> stubdom.
> 
>  * XenDevOps:
> Register Xen stubdom vTPM frontend driver, and transfer any
> request/repond between TPM xenstubdoms driver and Xen vTPM
> stubdom. Facilitate communications between Xen vTPM stubdom
> and vTPM xenstubdoms driver.
> 
>  * mini-os/tpmback:
> Mini-os TPM backend driver. The Linux frontend driver connects
> to this backend driver to facilitate communications between the
> Linux DomU and its vTPM. This driver is also used by vtpmmgr
> stubdom to communicate with vtpm-stubdom.
> 
>  * vtpm-stubdom:
> A mini-os stub domain that implements a vTPM. There is a
> one to one mapping between running vtpm-stubdom instances and
> logical vtpms on the system. The vTPM Platform Configuration
> Registers (PCRs) are all initialized to zero.
> 
>  * mini-os/tpmfront:
> Mini-os TPM frontend driver. The vTPM mini-os domain vtpm
> stubdom uses this driver to communicate with vtpmmgr-stubdom.
> This driver could also be used separately to implement a mini-os
> domain that wishes to use a vTPM of its own.
> 
>  * vtpmmgr-stubdom:
> A mini-os domain that implements the vTPM manager. There is only
> one vTPM manager and it should be running during the entire lifetime
> of the machine. vtpmmgr domain securely stores encryption keys for
> each of the vtpms and accesses to the hardware TPM to get the root of
> trust for the entire system.
> 
>  * mini-os/tpm_tis:
> Mini-os TPM version 1.2 TPM Interface Specification (TIS) driver.
> This driver used by vtpmmgr-stubdom to talk directly to the hardware
> TPM. Communication is facilitated by mapping hardware memory pages
> into vtpmmgr stubdom.
> 
>  * Hardware TPM: The physical TPM 1.2 that is soldered onto the
> motherboard.
> 
> ---
> Changes in v9
> High level changes: (each patch has a detailed history versioning)
>  * rebase on upstream qemu
>  * refactor qemu xendevs, xenstore functions in order to be shared with both
> backend and frontends
>  * convert tpm 

Re: [Xen-devel] [PATCH v14 3/3] IOMMU: fix vt-d Device-TLB flush timeout issue

[Xu, Quan]

Jan, 
Just check it, do you agree to pick up this series?

Quan

On July 05, 2016 9:46 PM, Jan Beulich  wrote:
> >>> On 04.07.16 at 11:11,  wrote:
> > --- a/xen/drivers/passthrough/iommu.c
> > +++ b/xen/drivers/passthrough/iommu.c
> > @@ -361,6 +361,30 @@ int iommu_iotlb_flush_all(struct domain *d)
> >  return rc;
> >  }
> >
> > +void iommu_dev_iotlb_flush_timeout(struct domain *d,
> 
> const
> 
> > +   struct pci_dev *pdev) {
> > +pcidevs_lock();
> > +
> > +ASSERT(pdev->domain);
> > +if ( d != pdev->domain )
> > +{
> > +pcidevs_unlock();
> > +return;
> > +}
> > +
> > +list_del(>domain_list);
> > +pdev->domain = NULL;
> > +pci_hide_existing_device(pdev);
> > +if ( !d->is_shutting_down && printk_ratelimit() )
> > +printk(XENLOG_ERR
> > +   "dom%d: ATS device %04x:%02x:%02x.%u flush failed\n",
> > +   d->domain_id, pdev->seg, pdev->bus, PCI_SLOT(pdev->devfn),
> > +   PCI_FUNC(pdev->devfn));
> > +
> > +pcidevs_unlock();
> > +}
> 
> I'm missing the domain_crash() part here (which would be the only reason
> why the parameter above can't be const).
> 
> > +static int __must_check dev_invalidate_sync(struct iommu *iommu,
> > +struct pci_dev *pdev, u16
> > +did)
> 
> const
> 
> Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH RFC v1 09/14] Makefile: delete STUBDOMPATH target

[Xu, Quan]

Wei,
I have spent a lot of days to clean up the code of vtpm / vtpmmgr. but there 
are tricky issue to boot domu on TPM2.0..
So I am afraid I can't finish it as today is my last working day at intel.

Now, these patch are on https://github.com/virt2x/vtpm2vtpmmgr 
Sorry for that.
Quan

On April 01, 2016 6:03 PM, Wei Liu <wei.l...@citrix.com> wrote:
> On Fri, Apr 01, 2016 at 01:41:40AM +0000, Xu, Quan wrote:
> > On March 31, 2016 9:50pm, Wei Liu <wei.l...@citrix.com> wrote:
> > > On Thu, Mar 31, 2016 at 10:21:22AM +, Xu, Quan wrote:
> > > > On March 11, 2016 12:53am, Wei Liu <wei.l...@citrix.com> wrote:
> > > > > -build: $(STUBDOMPATH)
> > > > > +build: $(STUBDOM_BUILD)
> > > >
> > > > Wei,
> > > > in original code, in stubdom/vtpm and stubdom/vtpmmgr, the code
> > > > style is inconsistent and ugly.
> > > I personally prefer small patches in a series, but I don't object to
> > > having large
> > > patch(es) either.
> >
> > ok. I think so too. It may be one file per patch.
> >
> >
> > > At the end of the day, I think Daniel's opinion matters most.
> > > After a plan is agreed upon, you can then provide a branch for us to pull 
> > > in.
> >
> > I think I am not authorized to branch, could you provide a branch and tell 
> > me
> how to commit to that branch?
> > Sorry, I am unfamiliar with the upstream process.
> >
> 
> Oh, you don't need to branch on the main tree (xen.git or future stubdom.git).
> You can just do the following:
> 
>   $ git clone xen.git # or stubdom.git
>   $ cd xen.git # or stubdom.git
>   $ git branch wip.vtpm-coding-style-fix-v1
>   ... do work, commit as you go alone ...
>   $ git push $some_public_remote wip.vtpm-coding-style-fix-v1
>   ... post your patch series on xen-devel, along with the git repository
>   and branch
> 
> When your patches are all acked by Daniel, you can then fold all the tags into
> your own branch (wip.vtpm-coding-style-fix-v$X-acked)
> and prod committers to pull from that branch.
> 
> > > Note
> > > that upstream don't test vtpm in any fashion so you do need to run your
> tests.
> > >
> > I will test it. I think I hope Daniel could help my patches.
> > btw, I have made a quick patch to fix the seal/unseal issue. I will send out
> later.
> >
> > > The only thing that matters to me is that when you will do it -- you
> > > will need to patch a different tree after I split off stubdom. In
> > > order to minimise the fuss one of us will need to wait for the other.
> > >
> > Once you have done, please let me know.
> 
> OK, I will try to sort it out within April.  Feel free to ping me if I drop 
> the ball.
> 
> Wei.
> 
> > Quan
> 
> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v14 3/3] IOMMU: fix vt-d Device-TLB flush timeout issue

[Xu, Quan]

On July 05, 2016 3:00 PM, Jan Beulich <jbeul...@suse.com> wrote:
> >>> On 05.07.16 at 04:04, <quan...@intel.com> wrote:
> > On July 05, 2016 8:47 AM, Tian, Kevin <kevin.t...@intel.com> wrote:
> >> > From: Xu, Quan
> >> > Sent: Monday, July 04, 2016 5:12 PM
> >> >
> >> > From: Quan Xu <quan...@intel.com>
> >> >
> >> > If Device-TLB flush timed out, we hide the target ATS device
> >> > immediately. By hiding the device, we make sure it can't be
> >> > assigned to any domain any longer (see device_assigned).
> >> >
> >> > Signed-off-by: Quan Xu <quan...@intel.com>
> >> >
> >> > CC: Jan Beulich <jbeul...@suse.com>
> >> > CC: Kevin Tian <kevin.t...@intel.com>
> >> > CC: Feng Wu <feng...@intel.com>
> >>
> >> Acked-by: Kevin Tian <kevin.t...@intel.com>
> >
> > Jan,  now there patches  are all acked-by Kevin. Is it ready to go in?
> > If not,  could you send out your comments, I shall be very grateful
> > and fix it today.
> 
> You sent the most recent version yesterday. Please allow more than just one
> day before you ping.
> 

Sorry for that, as today is my last working day at Intel Corporation. 
I really want to close it as a  milestone (but if there is still some comment, 
I will continue to fix it).

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v14 3/3] IOMMU: fix vt-d Device-TLB flush timeout issue

[Xu, Quan]

On July 05, 2016 8:47 AM, Tian, Kevin <kevin.t...@intel.com> wrote:
> > From: Xu, Quan
> > Sent: Monday, July 04, 2016 5:12 PM
> >
> > From: Quan Xu <quan...@intel.com>
> >
> > If Device-TLB flush timed out, we hide the target ATS device
> > immediately. By hiding the device, we make sure it can't be assigned
> > to any domain any longer (see device_assigned).
> >
> > Signed-off-by: Quan Xu <quan...@intel.com>
> >
> > CC: Jan Beulich <jbeul...@suse.com>
> > CC: Kevin Tian <kevin.t...@intel.com>
> > CC: Feng Wu <feng...@intel.com>
> 
> Acked-by: Kevin Tian <kevin.t...@intel.com>

Jan,  now there patches  are all acked-by Kevin. Is it ready to go in?
If not,  could you send out your comments, I shall be very grateful and fix it 
today.

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [PATCH v14 1/3] IOMMU/x86: use a struct pci_dev* instead of SBDF

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

A struct pci_dev* instead of SBDF is stored inside struct
pci_ats_dev and parameter to *_ats_device().

Also use ats_dev for "struct pci_ats_dev" variable, while
pdev (_pdev, if there is already a pdev) for "struct pci_dev"
in the scope of IOMMU.

Signed-off-by: Quan Xu <quan...@intel.com>
Acked-by: Kevin Tian <kevin.t...@intel.com>

CC: Jan Beulich <jbeul...@suse.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>
CC: Suravee Suthikulpanit <suravee.suthikulpa...@amd.com>

---
v14: change 'ats_pdev' to 'ats_dev'.
---
 xen/drivers/passthrough/amd/iommu_cmd.c | 19 
 xen/drivers/passthrough/amd/pci_amd_iommu.c |  4 +-
 xen/drivers/passthrough/ats.h   | 10 ++---
 xen/drivers/passthrough/vtd/intremap.c  |  8 ++--
 xen/drivers/passthrough/vtd/iommu.c | 14 +++---
 xen/drivers/passthrough/vtd/x86/ats.c   | 24 +++
 xen/drivers/passthrough/x86/ats.c   | 67 ++---
 7 files changed, 84 insertions(+), 62 deletions(-)

diff --git a/xen/drivers/passthrough/amd/iommu_cmd.c 
b/xen/drivers/passthrough/amd/iommu_cmd.c
index 7c9d9be..7e010e6 100644
--- a/xen/drivers/passthrough/amd/iommu_cmd.c
+++ b/xen/drivers/passthrough/amd/iommu_cmd.c
@@ -289,35 +289,34 @@ void amd_iommu_flush_iotlb(u8 devfn, const struct pci_dev 
*pdev,
 unsigned long flags;
 struct amd_iommu *iommu;
 unsigned int req_id, queueid, maxpend;
-struct pci_ats_dev *ats_pdev;
+struct pci_ats_dev *ats_dev;
 
 if ( !ats_enabled )
 return;
 
-ats_pdev = get_ats_device(pdev->seg, pdev->bus, pdev->devfn);
-if ( ats_pdev == NULL )
+ats_dev = get_ats_device(pdev);
+if ( ats_dev == NULL )
 return;
 
-if ( !pci_ats_enabled(ats_pdev->seg, ats_pdev->bus, ats_pdev->devfn) )
+if ( !pci_ats_enabled(pdev->seg, pdev->bus, pdev->devfn) )
 return;
 
-iommu = find_iommu_for_device(ats_pdev->seg,
-  PCI_BDF2(ats_pdev->bus, ats_pdev->devfn));
+iommu = find_iommu_for_device(pdev->seg, PCI_BDF2(pdev->bus, pdev->devfn));
 
 if ( !iommu )
 {
 AMD_IOMMU_DEBUG("%s: Can't find iommu for %04x:%02x:%02x.%u\n",
-__func__, ats_pdev->seg, ats_pdev->bus,
-PCI_SLOT(ats_pdev->devfn), PCI_FUNC(ats_pdev->devfn));
+__func__, pdev->seg, pdev->bus,
+PCI_SLOT(pdev->devfn), PCI_FUNC(pdev->devfn));
 return;
 }
 
 if ( !iommu_has_cap(iommu, PCI_CAP_IOTLB_SHIFT) )
 return;
 
-req_id = get_dma_requestor_id(iommu->seg, PCI_BDF2(ats_pdev->bus, devfn));
+req_id = get_dma_requestor_id(iommu->seg, PCI_BDF2(pdev->bus, 
pdev->devfn));
 queueid = req_id;
-maxpend = ats_pdev->ats_queue_depth & 0xff;
+maxpend = ats_dev->ats_queue_depth & 0xff;
 
 /* send INVALIDATE_IOTLB_PAGES command */
 spin_lock_irqsave(>lock, flags);
diff --git a/xen/drivers/passthrough/amd/pci_amd_iommu.c 
b/xen/drivers/passthrough/amd/pci_amd_iommu.c
index 7761241..dad4a71 100644
--- a/xen/drivers/passthrough/amd/pci_amd_iommu.c
+++ b/xen/drivers/passthrough/amd/pci_amd_iommu.c
@@ -162,7 +162,7 @@ static void amd_iommu_setup_domain_device(
  !pci_ats_enabled(iommu->seg, bus, pdev->devfn) )
 {
 if ( devfn == pdev->devfn )
-enable_ats_device(iommu->seg, bus, devfn, iommu);
+enable_ats_device(iommu, pdev);
 
 amd_iommu_flush_iotlb(devfn, pdev, INV_IOMMU_ALL_PAGES_ADDRESS, 0);
 }
@@ -356,7 +356,7 @@ void amd_iommu_disable_domain_device(struct domain *domain,
 if ( devfn == pdev->devfn &&
  pci_ats_device(iommu->seg, bus, devfn) &&
  pci_ats_enabled(iommu->seg, bus, devfn) )
-disable_ats_device(iommu->seg, bus, devfn);
+disable_ats_device(pdev);
 }
 
 static int reassign_device(struct domain *source, struct domain *target,
diff --git a/xen/drivers/passthrough/ats.h b/xen/drivers/passthrough/ats.h
index 5c91572..47ff22d 100644
--- a/xen/drivers/passthrough/ats.h
+++ b/xen/drivers/passthrough/ats.h
@@ -19,9 +19,7 @@
 
 struct pci_ats_dev {
 struct list_head list;
-u16 seg;
-u8 bus;
-u8 devfn;
+struct pci_dev *pdev;
 u16 ats_queue_depth;/* ATS device invalidation queue depth */
 const void *iommu;  /* No common IOMMU struct so use void pointer */
 };
@@ -34,9 +32,9 @@ struct pci_ats_dev {
 extern struct list_head ats_devices;
 extern bool_t ats_enabled;
 
-int enable_ats_device(int seg, int bus, int devfn, const void *iommu);
-void disable_ats_device(int seg, int bus, int devfn);
-struct pci_ats_dev *get_ats_device(int seg, int bus, int devfn);
+int enable_ats_device(const void *iommu, struct pci_dev *

[Xen-devel] [PATCH v14 2/3] IOMMU: add domain crash logic

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Add domain crash logic to the generic IOMMU layer to benefit
all platforms.

No spamming of the log can occur. For DomU, we avoid logging any
message for already dying domains. For Dom0, that'll still be more
verbose than we'd really like, but it at least wouldn't outright
flood the console.

Signed-off-by: Quan Xu <quan...@intel.com>
Acked-by: Kevin Tian <kevin.t...@intel.com>

CC: Julien Grall <julien.gr...@arm.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>
CC: Jan Beulich <jbeul...@suse.com>
CC: Suravee Suthikulpanit <suravee.suthikulpa...@amd.com>
---
 xen/drivers/passthrough/iommu.c | 30 --
 xen/drivers/passthrough/vtd/iommu.c | 11 +++
 2 files changed, 39 insertions(+), 2 deletions(-)

diff --git a/xen/drivers/passthrough/iommu.c b/xen/drivers/passthrough/iommu.c
index 7656aeb..d793f5d 100644
--- a/xen/drivers/passthrough/iommu.c
+++ b/xen/drivers/passthrough/iommu.c
@@ -318,21 +318,47 @@ int iommu_iotlb_flush(struct domain *d, unsigned long gfn,
   unsigned int page_count)
 {
 const struct domain_iommu *hd = dom_iommu(d);
+int rc;
 
 if ( !iommu_enabled || !hd->platform_ops || !hd->platform_ops->iotlb_flush 
)
 return 0;
 
-return hd->platform_ops->iotlb_flush(d, gfn, page_count);
+rc = hd->platform_ops->iotlb_flush(d, gfn, page_count);
+if ( unlikely(rc) )
+{
+if ( !d->is_shutting_down && printk_ratelimit() )
+printk(XENLOG_ERR
+   "d%d: IOMMU IOTLB flush failed: %d, gfn %#lx, page count 
%u\n",
+   d->domain_id, rc, gfn, page_count);
+
+if ( !is_hardware_domain(d) )
+domain_crash(d);
+}
+
+return rc;
 }
 
 int iommu_iotlb_flush_all(struct domain *d)
 {
 const struct domain_iommu *hd = dom_iommu(d);
+int rc;
 
 if ( !iommu_enabled || !hd->platform_ops || 
!hd->platform_ops->iotlb_flush_all )
 return 0;
 
-return hd->platform_ops->iotlb_flush_all(d);
+rc = hd->platform_ops->iotlb_flush_all(d);
+if ( unlikely(rc) )
+{
+if ( !d->is_shutting_down && printk_ratelimit() )
+printk(XENLOG_ERR
+   "d%d: IOMMU IOTLB flush all failed: %d\n",
+   d->domain_id, rc);
+
+if ( !is_hardware_domain(d) )
+domain_crash(d);
+}
+
+return rc;
 }
 
 int __init iommu_setup(void)
diff --git a/xen/drivers/passthrough/vtd/iommu.c 
b/xen/drivers/passthrough/vtd/iommu.c
index cc34497..a02b4c40 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -1847,6 +1847,17 @@ int iommu_pte_flush(struct domain *d, u64 gfn, u64 *pte,
 }
 }
 
+if ( unlikely(rc) )
+{
+if ( !d->is_shutting_down && printk_ratelimit() )
+printk(XENLOG_ERR VTDPREFIX
+   " d%d: IOMMU pages flush failed: %d\n",
+   d->domain_id, rc);
+
+if ( !is_hardware_domain(d) )
+domain_crash(d);
+}
+
 return rc;
 }
 
-- 
1.9.1


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v14 0/3] VT-d Device-TLB flush issue

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

these patches fix current timeout concern and also allow limited ATS support.

these patches are the rest ones:
1. move the domain crash logic up to the generic IOMMU layer

2. If Device-TLB flush timed out, we hide the target ATS device
   immediately. By hiding the device, we make sure it can't be
   assigned to any domain any longer (see device_assigned).

---
Not covered in this series:

a) Eliminate the panic() in IOMMU_WAIT_OP, used only in VT-d register 
read/write.
   Further discussion is required on whether and how to improve it.
b) Handle IOTLB/Context/IEC flush timeout.
---
Quan Xu (3):
  IOMMU/x86: use a struct pci_dev* instead of SBDF
  IOMMU: add domain crash logic
  IOMMU: fix vt-d Device-TLB flush timeout issue

 xen/drivers/passthrough/amd/iommu_cmd.c | 19 
 xen/drivers/passthrough/amd/pci_amd_iommu.c |  4 +-
 xen/drivers/passthrough/ats.h   | 10 ++---
 xen/drivers/passthrough/iommu.c | 54 ++-
 xen/drivers/passthrough/pci.c   |  6 +--
 xen/drivers/passthrough/vtd/extern.h|  5 ++-
 xen/drivers/passthrough/vtd/intremap.c  |  8 ++--
 xen/drivers/passthrough/vtd/iommu.c | 25 ---
 xen/drivers/passthrough/vtd/qinval.c| 56 ++--
 xen/drivers/passthrough/vtd/x86/ats.c   | 21 +
 xen/drivers/passthrough/x86/ats.c   | 67 ++---
 xen/include/xen/iommu.h |  3 ++
 xen/include/xen/pci.h   |  1 +
 13 files changed, 195 insertions(+), 84 deletions(-)

-- 
1.9.1


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v14 3/3] IOMMU: fix vt-d Device-TLB flush timeout issue

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

If Device-TLB flush timed out, we hide the target ATS device
immediately. By hiding the device, we make sure it can't be
assigned to any domain any longer (see device_assigned).

Signed-off-by: Quan Xu <quan...@intel.com>

CC: Jan Beulich <jbeul...@suse.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>

---
v14: release the lock before return.
---
 xen/drivers/passthrough/iommu.c   | 24 +++
 xen/drivers/passthrough/pci.c |  6 ++--
 xen/drivers/passthrough/vtd/extern.h  |  5 ++--
 xen/drivers/passthrough/vtd/qinval.c  | 56 +++
 xen/drivers/passthrough/vtd/x86/ats.c | 11 ++-
 xen/include/xen/iommu.h   |  3 ++
 xen/include/xen/pci.h |  1 +
 7 files changed, 79 insertions(+), 27 deletions(-)

diff --git a/xen/drivers/passthrough/iommu.c b/xen/drivers/passthrough/iommu.c
index d793f5d..2353c7d 100644
--- a/xen/drivers/passthrough/iommu.c
+++ b/xen/drivers/passthrough/iommu.c
@@ -361,6 +361,30 @@ int iommu_iotlb_flush_all(struct domain *d)
 return rc;
 }
 
+void iommu_dev_iotlb_flush_timeout(struct domain *d,
+   struct pci_dev *pdev)
+{
+pcidevs_lock();
+
+ASSERT(pdev->domain);
+if ( d != pdev->domain )
+{
+pcidevs_unlock();
+return;
+}
+
+list_del(>domain_list);
+pdev->domain = NULL;
+pci_hide_existing_device(pdev);
+if ( !d->is_shutting_down && printk_ratelimit() )
+printk(XENLOG_ERR
+   "dom%d: ATS device %04x:%02x:%02x.%u flush failed\n",
+   d->domain_id, pdev->seg, pdev->bus, PCI_SLOT(pdev->devfn),
+   PCI_FUNC(pdev->devfn));
+
+pcidevs_unlock();
+}
+
 int __init iommu_setup(void)
 {
 int rc = -ENODEV;
diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
index bb5f344..58bfb79 100644
--- a/xen/drivers/passthrough/pci.c
+++ b/xen/drivers/passthrough/pci.c
@@ -419,7 +419,7 @@ static void free_pdev(struct pci_seg *pseg, struct pci_dev 
*pdev)
 xfree(pdev);
 }
 
-static void _pci_hide_device(struct pci_dev *pdev)
+void pci_hide_existing_device(struct pci_dev *pdev)
 {
 if ( pdev->domain )
 return;
@@ -436,7 +436,7 @@ int __init pci_hide_device(int bus, int devfn)
 pdev = alloc_pdev(get_pseg(0), bus, devfn);
 if ( pdev )
 {
-_pci_hide_device(pdev);
+pci_hide_existing_device(pdev);
 rc = 0;
 }
 pcidevs_unlock();
@@ -466,7 +466,7 @@ int __init pci_ro_device(int seg, int bus, int devfn)
 }
 
 __set_bit(PCI_BDF2(bus, devfn), pseg->ro_map);
-_pci_hide_device(pdev);
+pci_hide_existing_device(pdev);
 
 return 0;
 }
diff --git a/xen/drivers/passthrough/vtd/extern.h 
b/xen/drivers/passthrough/vtd/extern.h
index 45357f2..efaff28 100644
--- a/xen/drivers/passthrough/vtd/extern.h
+++ b/xen/drivers/passthrough/vtd/extern.h
@@ -25,6 +25,7 @@
 
 #define VTDPREFIX "[VT-D]"
 
+struct pci_ats_dev;
 extern bool_t rwbf_quirk;
 
 void print_iommu_regs(struct acpi_drhd_unit *drhd);
@@ -60,8 +61,8 @@ int dev_invalidate_iotlb(struct iommu *iommu, u16 did,
  u64 addr, unsigned int size_order, u64 type);
 
 int __must_check qinval_device_iotlb_sync(struct iommu *iommu,
-  u32 max_invs_pend,
-  u16 sid, u16 size, u64 addr);
+  struct pci_ats_dev *ats_dev,
+  u16 did, u16 size, u64 addr);
 
 unsigned int get_cache_line_size(void);
 void cacheline_flush(char *);
diff --git a/xen/drivers/passthrough/vtd/qinval.c 
b/xen/drivers/passthrough/vtd/qinval.c
index 4492b29..7a5c433 100644
--- a/xen/drivers/passthrough/vtd/qinval.c
+++ b/xen/drivers/passthrough/vtd/qinval.c
@@ -27,11 +27,11 @@
 #include "dmar.h"
 #include "vtd.h"
 #include "extern.h"
+#include "../ats.h"
 
 #define VTD_QI_TIMEOUT 1
 
-static int __must_check invalidate_sync(struct iommu *iommu,
-bool_t flush_dev_iotlb);
+static int __must_check invalidate_sync(struct iommu *iommu);
 
 static void print_qi_regs(struct iommu *iommu)
 {
@@ -103,7 +103,7 @@ static int __must_check 
queue_invalidate_context_sync(struct iommu *iommu,
 
 unmap_vtd_domain_page(qinval_entries);
 
-return invalidate_sync(iommu, 0);
+return invalidate_sync(iommu);
 }
 
 static int __must_check queue_invalidate_iotlb_sync(struct iommu *iommu,
@@ -140,7 +140,7 @@ static int __must_check queue_invalidate_iotlb_sync(struct 
iommu *iommu,
 qinval_update_qtail(iommu, index);
 spin_unlock_irqrestore(>register_lock, flags);
 
-return invalidate_sync(iommu, 0);
+return invalidate_sync(iommu);
 }
 
 static int __must_check queue_invalidate_wa

Re: [Xen-devel] [PATCH v13 3/3] IOMMU: fix vt-d Device-TLB flush timeout issue

[Xu, Quan]

On July 04, 2016 2:16 PM, Tian, Kevin <kevin.t...@intel.com> wrote:
> > From: Xu, Quan
> > Sent: Wednesday, June 29, 2016 2:00 PM
> >
> > From: Quan Xu <quan...@intel.com>
> >
> > If Device-TLB flush timed out, we hide the target ATS device
> > immediately. By hiding the device, we make sure it can't be assigned
> > to any domain any longer (see device_assigned).
> >
> > Signed-off-by: Quan Xu <quan...@intel.com>
> >
> > CC: Jan Beulich <jbeul...@suse.com>
> > CC: Kevin Tian <kevin.t...@intel.com>
> > CC: Feng Wu <feng...@intel.com>
> >
> > ---
> > v13:
> >1. drop domain crash logic, which is added to the vendor
> >   independent layer in patch #2.
> >2. rename dev_invalidate_iotlb_timeout() to
> iommu_dev_iotlb_flush_timeout()
> >   and move it to the vendor independent layer.
> > ---
> >  xen/drivers/passthrough/iommu.c   | 21 +
> >  xen/drivers/passthrough/pci.c |  6 ++--
> >  xen/drivers/passthrough/vtd/extern.h  |  5 ++--
> > xen/drivers/passthrough/vtd/qinval.c  | 56
> > +++
> >  xen/drivers/passthrough/vtd/x86/ats.c | 11 ++-
> >  xen/include/xen/iommu.h   |  3 ++
> >  xen/include/xen/pci.h |  1 +
> >  7 files changed, 76 insertions(+), 27 deletions(-)
> >
> > diff --git a/xen/drivers/passthrough/iommu.c
> > b/xen/drivers/passthrough/iommu.c index d793f5d..5db8ae6 100644
> > --- a/xen/drivers/passthrough/iommu.c
> > +++ b/xen/drivers/passthrough/iommu.c
> > @@ -361,6 +361,27 @@ int iommu_iotlb_flush_all(struct domain *d)
> >  return rc;
> >  }
> >
> > +void iommu_dev_iotlb_flush_timeout(struct domain *d,
> > +   struct pci_dev *pdev) {
> > +pcidevs_lock();
> > +
> > +ASSERT(pdev->domain);
> > +if ( d != pdev->domain )
> > +return;
> 
> return w/o releasing the lock!
> 
Yes, I really need releasing the lock before return.

> and is above scenario actually possible (a flush timeout is captured when the
> device doesn't belong to previous domain)? If not, better to move the
> condition into ASSERT.

IMO, this is possible.
  -- not all of call trees of device iotlb flush are under pcidevs_lock, (.i.e  
...--iommu_iotlb_flush()-- xenmem_add_to_physmap()... )
  -- In extreme cases , the domain may has been freed or the device may has 
been detached or even attached to another domain.
That's also why to introduce a domain point here.

> 
> > +
> > +list_del(>domain_list);
> > +pdev->domain = NULL;
> > +pci_hide_existing_device(pdev);
> > +if ( !d->is_shutting_down && printk_ratelimit() )
> > +printk(XENLOG_ERR
> > +   "dom%d: ATS device %04x:%02x:%02x.%u flush failed\n",
> > +   d->domain_id, pdev->seg, pdev->bus, PCI_SLOT(pdev->devfn),
> > +   PCI_FUNC(pdev->devfn));
> > +
> > +pcidevs_unlock();
> 
> please move above warning out of the lock.
> 

I think I'm better leave it as is.

as I use 'pdev' to print information, as similar as pci_release_devices().
If I use seg, bus, devfn variables directly, instead of 'pdev', I agree to move 
out of the lock, as similar as:

iommu_do_pci_domctl()
{
 case XEN_DOMCTL_assign_device... 
}

 correct me if I am not right.

Quan




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v13 1/3] IOMMU/x86: use a struct pci_dev* instead of SBDF

[Xu, Quan]

On July 04, 2016 1:35 PM, Tian, Kevin <kevin.t...@intel.com> wrote:
> > From: Xu, Quan
> > Sent: Wednesday, June 29, 2016 2:00 PM
> >
> > From: Quan Xu <quan...@intel.com>
> >
> > A struct pci_dev* instead of SBDF is stored inside struct pci_ats_dev
> > and parameter to *_ats_device().
> >
> > Also use ats_dev for "struct pci_ats_dev" variable, while pdev (_pdev,
> > if there is already a pdev) for "struct pci_dev"
> > in the scope of IOMMU.
> >
> > Signed-off-by: Quan Xu <quan...@intel.com>
> >
> > CC: Jan Beulich <jbeul...@suse.com>
> > CC: Kevin Tian <kevin.t...@intel.com>
> > CC: Feng Wu <feng...@intel.com>
> > CC: Suravee Suthikulpanit <suravee.suthikulpa...@amd.com>
> >
> 
> Acked-by: Kevin Tian <kevin.t...@intel.com>, with one minor comment:
> 
> > diff --git a/xen/drivers/passthrough/amd/iommu_cmd.c
> > b/xen/drivers/passthrough/amd/iommu_cmd.c
> > index 7c9d9be..934977a 100644
> > --- a/xen/drivers/passthrough/amd/iommu_cmd.c
> > +++ b/xen/drivers/passthrough/amd/iommu_cmd.c
> > @@ -294,28 +294,27 @@ void amd_iommu_flush_iotlb(u8 devfn, const
> > struct pci_dev *pdev,
> >  if ( !ats_enabled )
> >  return;
> >
> > -ats_pdev = get_ats_device(pdev->seg, pdev->bus, pdev->devfn);
> > +ats_pdev = get_ats_device(pdev);
> >  if ( ats_pdev == NULL )
> >  return;
> >
> 
> To align with other places, better to change ats_pdev to ats_dev too.


Indeed,  thanks very much!!
I will update it soon.
- Quan 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] FW: vTPM detaching issue

[Xu, Quan]

On June 15, 2016 7:49 PM, < wei.l...@citrix.com > wrote:
> On Tue, Jun 14, 2016 at 12:32:19PM +0200, Andrea Genuise wrote:
> [...]
> > libxl: debug: libxl_aoutils.c:88:xswait_timeout_callback: backend
> > /local/domain/748/backend/vtpm/749/0/state (hoping for state change to
> 6):
> > xswait timeout (path=/local/domain/748/backend/vtpm/749/0/state)
> > libxl: debug: libxl_event.c:677:libxl__ev_xswatch_deregister: watch
> > w=0x1c4c1f0 wpath=/local/domain/748/backend/vtpm/749/0/state
> token=3/0:
> > deregister slotnum=3
> > libxl: debug: libxl_event.c:867:devstate_callback: backend
> > /local/domain/748/backend/vtpm/749/0/state wanted state 6  timed out
> 
> This. The toolstack is waiting for the state to change to 6. But that never
> happened.
> 
> > libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
> > w=0x1c4c1f0: deregister unregistered
> > libxl: debug: libxl_device.c:937:device_backend_callback: calling
> > device_backend_cleanup
> > libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
> > w=0x1c4c1f0: deregister unregistered
> > libxl: debug: libxl_device.c:943:device_backend_callback: Timeout
> > reached, initiating forced remove
> 
> I think this is due to interaction between frontend and backend, but I'm not 
> an
> expert on vtpm so I don't have further comment.
> 

Daniel,  are you following this issue?  --Quan



___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v13 3/3] IOMMU: fix vt-d Device-TLB flush timeout issue

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

If Device-TLB flush timed out, we hide the target ATS device
immediately. By hiding the device, we make sure it can't be
assigned to any domain any longer (see device_assigned).

Signed-off-by: Quan Xu <quan...@intel.com>

CC: Jan Beulich <jbeul...@suse.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>

---
v13:
   1. drop domain crash logic, which is added to the vendor
  independent layer in patch #2.
   2. rename dev_invalidate_iotlb_timeout() to iommu_dev_iotlb_flush_timeout()
  and move it to the vendor independent layer.
---
 xen/drivers/passthrough/iommu.c   | 21 +
 xen/drivers/passthrough/pci.c |  6 ++--
 xen/drivers/passthrough/vtd/extern.h  |  5 ++--
 xen/drivers/passthrough/vtd/qinval.c  | 56 +++
 xen/drivers/passthrough/vtd/x86/ats.c | 11 ++-
 xen/include/xen/iommu.h   |  3 ++
 xen/include/xen/pci.h |  1 +
 7 files changed, 76 insertions(+), 27 deletions(-)

diff --git a/xen/drivers/passthrough/iommu.c b/xen/drivers/passthrough/iommu.c
index d793f5d..5db8ae6 100644
--- a/xen/drivers/passthrough/iommu.c
+++ b/xen/drivers/passthrough/iommu.c
@@ -361,6 +361,27 @@ int iommu_iotlb_flush_all(struct domain *d)
 return rc;
 }
 
+void iommu_dev_iotlb_flush_timeout(struct domain *d,
+   struct pci_dev *pdev)
+{
+pcidevs_lock();
+
+ASSERT(pdev->domain);
+if ( d != pdev->domain )
+return;
+
+list_del(>domain_list);
+pdev->domain = NULL;
+pci_hide_existing_device(pdev);
+if ( !d->is_shutting_down && printk_ratelimit() )
+printk(XENLOG_ERR
+   "dom%d: ATS device %04x:%02x:%02x.%u flush failed\n",
+   d->domain_id, pdev->seg, pdev->bus, PCI_SLOT(pdev->devfn),
+   PCI_FUNC(pdev->devfn));
+
+pcidevs_unlock();
+}
+
 int __init iommu_setup(void)
 {
 int rc = -ENODEV;
diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
index bb5f344..58bfb79 100644
--- a/xen/drivers/passthrough/pci.c
+++ b/xen/drivers/passthrough/pci.c
@@ -419,7 +419,7 @@ static void free_pdev(struct pci_seg *pseg, struct pci_dev 
*pdev)
 xfree(pdev);
 }
 
-static void _pci_hide_device(struct pci_dev *pdev)
+void pci_hide_existing_device(struct pci_dev *pdev)
 {
 if ( pdev->domain )
 return;
@@ -436,7 +436,7 @@ int __init pci_hide_device(int bus, int devfn)
 pdev = alloc_pdev(get_pseg(0), bus, devfn);
 if ( pdev )
 {
-_pci_hide_device(pdev);
+pci_hide_existing_device(pdev);
 rc = 0;
 }
 pcidevs_unlock();
@@ -466,7 +466,7 @@ int __init pci_ro_device(int seg, int bus, int devfn)
 }
 
 __set_bit(PCI_BDF2(bus, devfn), pseg->ro_map);
-_pci_hide_device(pdev);
+pci_hide_existing_device(pdev);
 
 return 0;
 }
diff --git a/xen/drivers/passthrough/vtd/extern.h 
b/xen/drivers/passthrough/vtd/extern.h
index 45357f2..efaff28 100644
--- a/xen/drivers/passthrough/vtd/extern.h
+++ b/xen/drivers/passthrough/vtd/extern.h
@@ -25,6 +25,7 @@
 
 #define VTDPREFIX "[VT-D]"
 
+struct pci_ats_dev;
 extern bool_t rwbf_quirk;
 
 void print_iommu_regs(struct acpi_drhd_unit *drhd);
@@ -60,8 +61,8 @@ int dev_invalidate_iotlb(struct iommu *iommu, u16 did,
  u64 addr, unsigned int size_order, u64 type);
 
 int __must_check qinval_device_iotlb_sync(struct iommu *iommu,
-  u32 max_invs_pend,
-  u16 sid, u16 size, u64 addr);
+  struct pci_ats_dev *ats_dev,
+  u16 did, u16 size, u64 addr);
 
 unsigned int get_cache_line_size(void);
 void cacheline_flush(char *);
diff --git a/xen/drivers/passthrough/vtd/qinval.c 
b/xen/drivers/passthrough/vtd/qinval.c
index 4492b29..7a5c433 100644
--- a/xen/drivers/passthrough/vtd/qinval.c
+++ b/xen/drivers/passthrough/vtd/qinval.c
@@ -27,11 +27,11 @@
 #include "dmar.h"
 #include "vtd.h"
 #include "extern.h"
+#include "../ats.h"
 
 #define VTD_QI_TIMEOUT 1
 
-static int __must_check invalidate_sync(struct iommu *iommu,
-bool_t flush_dev_iotlb);
+static int __must_check invalidate_sync(struct iommu *iommu);
 
 static void print_qi_regs(struct iommu *iommu)
 {
@@ -103,7 +103,7 @@ static int __must_check 
queue_invalidate_context_sync(struct iommu *iommu,
 
 unmap_vtd_domain_page(qinval_entries);
 
-return invalidate_sync(iommu, 0);
+return invalidate_sync(iommu);
 }
 
 static int __must_check queue_invalidate_iotlb_sync(struct iommu *iommu,
@@ -140,7 +140,7 @@ static int __must_check queue_invalidate_iotlb_sync(struct 
iommu *iommu,
 qinval_update_qtail(iommu, index);
 spin_unlock_irqrest

[Xen-devel] [PATCH v13 0/3] VT-d Device-TLB flush issue

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

This patches fix current timeout concern and also allow limited ATS support.

these patches are the rest ones:
1. move the domain crash logic up to the generic IOMMU layer

2. If Device-TLB flush timed out, we hide the target ATS device
   immediately. By hiding the device, we make sure it can't be
   assigned to any domain any longer (see device_assigned).

---
Not covered in this series:

a) Eliminate the panic() in IOMMU_WAIT_OP, used only in VT-d register 
read/write.
   Further discussion is required on whether and how to improve it.
b) Handle IOTLB/Context/IEC flush timeout.
---
Quan Xu (3):
  IOMMU/x86: use a struct pci_dev* instead of SBDF
  IOMMU: add domain crash logic
  IOMMU: fix vt-d Device-TLB flush timeout issue

 xen/drivers/passthrough/amd/iommu_cmd.c | 13 +++---
 xen/drivers/passthrough/amd/pci_amd_iommu.c |  4 +-
 xen/drivers/passthrough/ats.h   | 10 ++---
 xen/drivers/passthrough/iommu.c | 51 +-
 xen/drivers/passthrough/pci.c   |  6 +--
 xen/drivers/passthrough/vtd/extern.h|  5 ++-
 xen/drivers/passthrough/vtd/intremap.c  |  8 ++--
 xen/drivers/passthrough/vtd/iommu.c | 25 ---
 xen/drivers/passthrough/vtd/qinval.c| 56 ++--
 xen/drivers/passthrough/vtd/x86/ats.c   | 21 +
 xen/drivers/passthrough/x86/ats.c   | 67 ++---
 xen/include/xen/iommu.h |  3 ++
 xen/include/xen/pci.h   |  1 +
 13 files changed, 189 insertions(+), 81 deletions(-)

-- 
1.9.1


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v13 2/3] IOMMU: add domain crash logic

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Add domain crash logic to the generic IOMMU layer to benefit
all platforms.

No spamming of the log can occur. For DomU, we avoid logging any
message for already dying domains. For Dom0, that'll still be more
verbose than we'd really like, but it at least wouldn't outright
flood the console.

Signed-off-by: Quan Xu <quan...@intel.com>

CC: Julien Grall <julien.gr...@arm.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>
CC: Jan Beulich <jbeul...@suse.com>
CC: Suravee Suthikulpanit <suravee.suthikulpa...@amd.com>

---
v13: enhance commit message
---
 xen/drivers/passthrough/iommu.c | 30 --
 xen/drivers/passthrough/vtd/iommu.c | 11 +++
 2 files changed, 39 insertions(+), 2 deletions(-)

diff --git a/xen/drivers/passthrough/iommu.c b/xen/drivers/passthrough/iommu.c
index 7656aeb..d793f5d 100644
--- a/xen/drivers/passthrough/iommu.c
+++ b/xen/drivers/passthrough/iommu.c
@@ -318,21 +318,47 @@ int iommu_iotlb_flush(struct domain *d, unsigned long gfn,
   unsigned int page_count)
 {
 const struct domain_iommu *hd = dom_iommu(d);
+int rc;
 
 if ( !iommu_enabled || !hd->platform_ops || !hd->platform_ops->iotlb_flush 
)
 return 0;
 
-return hd->platform_ops->iotlb_flush(d, gfn, page_count);
+rc = hd->platform_ops->iotlb_flush(d, gfn, page_count);
+if ( unlikely(rc) )
+{
+if ( !d->is_shutting_down && printk_ratelimit() )
+printk(XENLOG_ERR
+   "d%d: IOMMU IOTLB flush failed: %d, gfn %#lx, page count 
%u\n",
+   d->domain_id, rc, gfn, page_count);
+
+if ( !is_hardware_domain(d) )
+domain_crash(d);
+}
+
+return rc;
 }
 
 int iommu_iotlb_flush_all(struct domain *d)
 {
 const struct domain_iommu *hd = dom_iommu(d);
+int rc;
 
 if ( !iommu_enabled || !hd->platform_ops || 
!hd->platform_ops->iotlb_flush_all )
 return 0;
 
-return hd->platform_ops->iotlb_flush_all(d);
+rc = hd->platform_ops->iotlb_flush_all(d);
+if ( unlikely(rc) )
+{
+if ( !d->is_shutting_down && printk_ratelimit() )
+printk(XENLOG_ERR
+   "d%d: IOMMU IOTLB flush all failed: %d\n",
+   d->domain_id, rc);
+
+if ( !is_hardware_domain(d) )
+domain_crash(d);
+}
+
+return rc;
 }
 
 int __init iommu_setup(void)
diff --git a/xen/drivers/passthrough/vtd/iommu.c 
b/xen/drivers/passthrough/vtd/iommu.c
index cc34497..a02b4c40 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -1847,6 +1847,17 @@ int iommu_pte_flush(struct domain *d, u64 gfn, u64 *pte,
 }
 }
 
+if ( unlikely(rc) )
+{
+if ( !d->is_shutting_down && printk_ratelimit() )
+printk(XENLOG_ERR VTDPREFIX
+   " d%d: IOMMU pages flush failed: %d\n",
+   d->domain_id, rc);
+
+if ( !is_hardware_domain(d) )
+domain_crash(d);
+}
+
 return rc;
 }
 
-- 
1.9.1


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v13 1/3] IOMMU/x86: use a struct pci_dev* instead of SBDF

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

A struct pci_dev* instead of SBDF is stored inside struct
pci_ats_dev and parameter to *_ats_device().

Also use ats_dev for "struct pci_ats_dev" variable, while
pdev (_pdev, if there is already a pdev) for "struct pci_dev"
in the scope of IOMMU.

Signed-off-by: Quan Xu <quan...@intel.com>

CC: Jan Beulich <jbeul...@suse.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>
CC: Suravee Suthikulpanit <suravee.suthikulpa...@amd.com>

---
v13:
   1. use ats_dev for "struct pci_ats_dev" variable, while
  pdev (_pdev, if there is already a pdev) for "struct pci_dev"
  in the scope of IOMMU.
   2. add local variables seg, bus, and devfn, which will
  greatly reduce the number of changes in *_ats_device().
   3. convert SBDF into struct pci_dev* to disable_ats_device()
  get_ats_device() as well.
  (
   enable_ats_device() and disable_ats_device() can't have
   the const added to 'struct pci_dev *pdev', otherwise then
   compilation fails. also Afaict that would in turn eliminate
   the need for some of the changes further up. but get_ats_device()
   can.
  )
---
 xen/drivers/passthrough/amd/iommu_cmd.c | 13 +++---
 xen/drivers/passthrough/amd/pci_amd_iommu.c |  4 +-
 xen/drivers/passthrough/ats.h   | 10 ++---
 xen/drivers/passthrough/vtd/intremap.c  |  8 ++--
 xen/drivers/passthrough/vtd/iommu.c | 14 +++---
 xen/drivers/passthrough/vtd/x86/ats.c   | 24 +++
 xen/drivers/passthrough/x86/ats.c   | 67 ++---
 7 files changed, 81 insertions(+), 59 deletions(-)

diff --git a/xen/drivers/passthrough/amd/iommu_cmd.c 
b/xen/drivers/passthrough/amd/iommu_cmd.c
index 7c9d9be..934977a 100644
--- a/xen/drivers/passthrough/amd/iommu_cmd.c
+++ b/xen/drivers/passthrough/amd/iommu_cmd.c
@@ -294,28 +294,27 @@ void amd_iommu_flush_iotlb(u8 devfn, const struct pci_dev 
*pdev,
 if ( !ats_enabled )
 return;
 
-ats_pdev = get_ats_device(pdev->seg, pdev->bus, pdev->devfn);
+ats_pdev = get_ats_device(pdev);
 if ( ats_pdev == NULL )
 return;
 
-if ( !pci_ats_enabled(ats_pdev->seg, ats_pdev->bus, ats_pdev->devfn) )
+if ( !pci_ats_enabled(pdev->seg, pdev->bus, pdev->devfn) )
 return;
 
-iommu = find_iommu_for_device(ats_pdev->seg,
-  PCI_BDF2(ats_pdev->bus, ats_pdev->devfn));
+iommu = find_iommu_for_device(pdev->seg, PCI_BDF2(pdev->bus, pdev->devfn));
 
 if ( !iommu )
 {
 AMD_IOMMU_DEBUG("%s: Can't find iommu for %04x:%02x:%02x.%u\n",
-__func__, ats_pdev->seg, ats_pdev->bus,
-PCI_SLOT(ats_pdev->devfn), PCI_FUNC(ats_pdev->devfn));
+__func__, pdev->seg, pdev->bus,
+PCI_SLOT(pdev->devfn), PCI_FUNC(pdev->devfn));
 return;
 }
 
 if ( !iommu_has_cap(iommu, PCI_CAP_IOTLB_SHIFT) )
 return;
 
-req_id = get_dma_requestor_id(iommu->seg, PCI_BDF2(ats_pdev->bus, devfn));
+req_id = get_dma_requestor_id(iommu->seg, PCI_BDF2(pdev->bus, 
pdev->devfn));
 queueid = req_id;
 maxpend = ats_pdev->ats_queue_depth & 0xff;
 
diff --git a/xen/drivers/passthrough/amd/pci_amd_iommu.c 
b/xen/drivers/passthrough/amd/pci_amd_iommu.c
index 7761241..dad4a71 100644
--- a/xen/drivers/passthrough/amd/pci_amd_iommu.c
+++ b/xen/drivers/passthrough/amd/pci_amd_iommu.c
@@ -162,7 +162,7 @@ static void amd_iommu_setup_domain_device(
  !pci_ats_enabled(iommu->seg, bus, pdev->devfn) )
 {
 if ( devfn == pdev->devfn )
-enable_ats_device(iommu->seg, bus, devfn, iommu);
+enable_ats_device(iommu, pdev);
 
 amd_iommu_flush_iotlb(devfn, pdev, INV_IOMMU_ALL_PAGES_ADDRESS, 0);
 }
@@ -356,7 +356,7 @@ void amd_iommu_disable_domain_device(struct domain *domain,
 if ( devfn == pdev->devfn &&
  pci_ats_device(iommu->seg, bus, devfn) &&
  pci_ats_enabled(iommu->seg, bus, devfn) )
-disable_ats_device(iommu->seg, bus, devfn);
+disable_ats_device(pdev);
 }
 
 static int reassign_device(struct domain *source, struct domain *target,
diff --git a/xen/drivers/passthrough/ats.h b/xen/drivers/passthrough/ats.h
index 5c91572..47ff22d 100644
--- a/xen/drivers/passthrough/ats.h
+++ b/xen/drivers/passthrough/ats.h
@@ -19,9 +19,7 @@
 
 struct pci_ats_dev {
 struct list_head list;
-u16 seg;
-u8 bus;
-u8 devfn;
+struct pci_dev *pdev;
 u16 ats_queue_depth;/* ATS device invalidation queue depth */
 const void *iommu;  /* No common IOMMU struct so use void pointer */
 };
@@ -34,9 +32,9 @@ struct pci_ats_dev {
 extern struct list_head ats_devices;
 extern bo

Re: [Xen-devel] [PATCH v12 6/6] vt-d: fix vt-d Device-TLB flush timeout issue

[Xu, Quan]

On June 27, 2016 11:21 PM, Jan Beulich  wrote:
> >>> On 27.06.16 at 14:56,  wrote:
> > On June 27, 2016 4:24 PM, Jan Beulich  wrote:
> >> >>> On 24.06.16 at 07:51,  wrote:
> >> > @@ -199,24 +199,73 @@ static int __must_check
> >> queue_invalidate_wait(struct iommu *iommu,
> >> >  return -EOPNOTSUPP;
> >> >  }
> >> >
> >> > -static int __must_check invalidate_sync(struct iommu *iommu,
> >> > -bool_t flush_dev_iotlb)
> >> > +static int __must_check invalidate_sync(struct iommu *iommu)
> >> >  {
> >> >  struct qi_ctrl *qi_ctrl = iommu_qi_ctrl(iommu);
> >> >
> >> >  ASSERT(qi_ctrl->qinval_maddr);
> >> >
> >> > -return queue_invalidate_wait(iommu, 0, 1, 1, flush_dev_iotlb);
> >> > +return queue_invalidate_wait(iommu, 0, 1, 1, 0); }
> >> > +
> >> > +static void dev_invalidate_iotlb_timeout(struct iommu *iommu, u16 did,
> >> > + struct pci_dev *pdev) {
> >> > +struct domain *d = NULL;
> >> > +
> >> > +if ( test_bit(did, iommu->domid_bitmap) )
> >> > +d = rcu_lock_domain_by_id(iommu->domid_map[did]);
> >> > +
> >> > +/*
> >> > + * In case the domain has been freed or the IOMMU domid bitmap is
> >> > + * not valid, the device no longer belongs to this domain.
> >> > + */
> >> > +if ( d == NULL )
> >> > +return;
> >> > +
> >> > +pcidevs_lock();
> >> > +ASSERT(pdev->domain);
> >> > +list_del(>domain_list);
> >> > +pdev->domain = NULL;
> >> > +pci_hide_existing_device(pdev);
> >> > +pcidevs_unlock();
> >> > +
> >> > +if ( !d->is_shutting_down && printk_ratelimit() )
> >> > +printk(XENLOG_WARNING VTDPREFIX
> >> > +   " dom%d: ATS device %04x:%02x:%02x.%u flush failed\n",
> >> > +   d->domain_id, pdev->seg, pdev->bus,
> >> > +   PCI_SLOT(pdev->devfn), PCI_FUNC(pdev->devfn));
> >> > +
> >> > +if ( !is_hardware_domain(d) )
> >> > +domain_crash(d);
> >> > +
> >> > +rcu_unlock_domain(d);
> >> > +}
> >>
> >> So in an earlier patch in this series you (supposedly) moved similar
> >> logic up to the vendor independent layer. I think this then would
> >> better get moved up too, if at all possible.
> >>
> >
> > To be honest, I have not much reason for leaving domain crash here and
> > I was aware of this problem, but crash_domain() here is not harmful
> > (as the 'd->is_shutting_down' is Set when to crash, and once the 'd-
> >is_shutting_down'
> > is Set then return  in domain_shutdown()  ).
> > In case crash domain directly, it may help us narrow down the 'window'
> > (the domain is still running)..
> >
> > To me, moving the logic up is acceptable.
> >
> > In next version, could I only drop:
> >
> > +if ( !is_hardware_domain(d) )
> > +domain_crash(d);
> >
> > In this patch, and leave the rest as is ?
> 
> Not really - the entire function looks like it could move out of vtd/, as I 
> can't
> see anything VT-d specific in it.
> 

Yes, it could be out of vtd, and then benefit arm/amd  IOMMU to hide ATS device.

But 'did'  and 'iommu->domid_bitmap' are really vtd specific. Both of them are 
to get domain* structure, not a big deal, and then I can use domain_id instead.

IMO, the domain* structure is a must here,
As mentioned, not all of call trees of device iotlb flush are under 
pcidevs_lock, (.i.e  ...--iommu_iotlb_flush()-- xenmem_add_to_physmap()... )
In extreme cases, the domain may has been freed or the device may has been 
detached or even attached to another domain ( I also need to add 'if 
(pdev->domain == d )' before to hide device).
the domain* structure can help us check above cases.

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v12 4/6] IOMMU/x86: using a struct pci_dev* instead of SBDF

[Xu, Quan]

On June 27, 2016 11:19 PM, Jan Beulich  wrote:
> >>> On 27.06.16 at 13:11,  wrote:
> > On June 27, 2016 4:17 PM, Jan Beulich  wrote:
> >> >>> On 24.06.16 at 07:51,  wrote:
> >> > @@ -98,7 +104,13 @@ void disable_ats_device(int seg, int bus, int
> >> > devfn)
> >>
> >> For symmetry reasons this function would also get converted to taking
> >> const struct pci_dev *.
> >>
> >
> > What about ' struct pci_dev *', without const?
> 
> Sure - since the other one apparently can't have the const added, this one
> doesn't need to either (but please nevertheless do add it it that's actually
> possible without having to cast away constness somewhere.
> 

Indeed.. -Quan
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v12 6/6] vt-d: fix vt-d Device-TLB flush timeout issue

[Xu, Quan]

On June 27, 2016 4:24 PM, Jan Beulich  wrote:
> >>> On 24.06.16 at 07:51,  wrote:
> > @@ -199,24 +199,73 @@ static int __must_check
> queue_invalidate_wait(struct iommu *iommu,
> >  return -EOPNOTSUPP;
> >  }
> >
> > -static int __must_check invalidate_sync(struct iommu *iommu,
> > -bool_t flush_dev_iotlb)
> > +static int __must_check invalidate_sync(struct iommu *iommu)
> >  {
> >  struct qi_ctrl *qi_ctrl = iommu_qi_ctrl(iommu);
> >
> >  ASSERT(qi_ctrl->qinval_maddr);
> >
> > -return queue_invalidate_wait(iommu, 0, 1, 1, flush_dev_iotlb);
> > +return queue_invalidate_wait(iommu, 0, 1, 1, 0); }
> > +
> > +static void dev_invalidate_iotlb_timeout(struct iommu *iommu, u16 did,
> > + struct pci_dev *pdev) {
> > +struct domain *d = NULL;
> > +
> > +if ( test_bit(did, iommu->domid_bitmap) )
> > +d = rcu_lock_domain_by_id(iommu->domid_map[did]);
> > +
> > +/*
> > + * In case the domain has been freed or the IOMMU domid bitmap is
> > + * not valid, the device no longer belongs to this domain.
> > + */
> > +if ( d == NULL )
> > +return;
> > +
> > +pcidevs_lock();
> > +ASSERT(pdev->domain);
> > +list_del(>domain_list);
> > +pdev->domain = NULL;
> > +pci_hide_existing_device(pdev);
> > +pcidevs_unlock();
> > +
> > +if ( !d->is_shutting_down && printk_ratelimit() )
> > +printk(XENLOG_WARNING VTDPREFIX
> > +   " dom%d: ATS device %04x:%02x:%02x.%u flush failed\n",
> > +   d->domain_id, pdev->seg, pdev->bus,
> > +   PCI_SLOT(pdev->devfn), PCI_FUNC(pdev->devfn));
> > +
> > +if ( !is_hardware_domain(d) )
> > +domain_crash(d);
> > +
> > +rcu_unlock_domain(d);
> > +}
> 
> So in an earlier patch in this series you (supposedly) moved similar logic up 
> to
> the vendor independent layer. I think this then would better get moved up
> too, if at all possible.
> 

To be honest, I have not much reason for leaving domain crash here and I was 
aware of this problem, but crash_domain() here is not harmful (as the 
'd->is_shutting_down' is Set when to crash, and once the 'd->is_shutting_down' 
is Set then return  in domain_shutdown()  ).
In case crash domain directly, it may help us narrow down the 'window' (the 
domain is still running)..

To me, moving the logic up is acceptable.

In next version, could I only drop:

+if ( !is_hardware_domain(d) )
+domain_crash(d);

In this patch, and leave the rest as is ?

Quan



___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v12 4/6] IOMMU/x86: using a struct pci_dev* instead of SBDF

[Xu, Quan]

On June 27, 2016 4:17 PM, Jan Beulich  wrote:
> >>> On 24.06.16 at 07:51,  wrote:
> > --- a/xen/drivers/passthrough/x86/ats.c
> > +++ b/xen/drivers/passthrough/x86/ats.c
> > @@ -22,26 +22,34 @@ LIST_HEAD(ats_devices);  bool_t __read_mostly
> > ats_enabled = 0;  boolean_param("ats", ats_enabled);
> >
> > -int enable_ats_device(int seg, int bus, int devfn, const void *iommu)
> > +int enable_ats_device(const void *iommu, struct pci_dev *pci_dev)
> 
> Is there anything preventing the second parameter to become a pointer to
> const too? Afaict that would in turn eliminate the need for some of the
> changes further up.
> 

If I make the second parameter to const, then compilation fails:

"""
  ats.c: In function 'enable_ats_device':
ats.c:71:23: error: assignment discards 'const' qualifier from pointer target 
type [-Werror]
 ats_dev->pdev = pdev;
   ^
"""

Also I will hide pci_dev as device IOTLB flush error, with changing of pci_dev.
A const 'struct pci_dev *'  in  'struct pci_ats_dev' is not working..  I have 
not verified this, correct me if  I am wrong.


> >  {
> >  struct pci_ats_dev *pdev = NULL;
> >  u32 value;
> >  int pos;
> >
> > -pos = pci_find_ext_capability(seg, bus, devfn, PCI_EXT_CAP_ID_ATS);
> > +pos = pci_find_ext_capability(pci_dev->seg, pci_dev->bus, pci_dev-
> >devfn,
> > +  PCI_EXT_CAP_ID_ATS);
> 
> Please add local variables seg, bus, and devfn, which will greatly reduce the
> number of changes you need to do to this function (and which likely will also
> produce better code).

Agreed. Good idea.

> 
> > @@ -98,7 +104,13 @@ void disable_ats_device(int seg, int bus, int
> > devfn)
> 
> For symmetry reasons this function would also get converted to taking const
> struct pci_dev *.
> 

What about ' struct pci_dev *', without const?

> > @@ -120,7 +132,13 @@ struct pci_ats_dev *get_ats_device(int seg, int
> > bus, int devfn)
> 
> And this one then probably too.
> 

Ditto.

Quan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] FIXME question

[Xu, Quan]

Hi,

When I read IOMMU code,
In xen/drivers/passthrough/vtd/intremap.c : pi_update_irte()..
There are a FIXME --
''
* FIXME: For performance reasons we should store the 'iommu' pointer in
* 'struct msi_desc' in some other place, so we don't need to waste
* time searching it here.
"

IMO, we are better to store the 'iommu' pointer in pci_dev, then
could I fix it as:

1. save a void *iommu  in pci_dev structure:

--- a/xen/include/xen/pci.h
+++ b/xen/include/xen/pci.h
@@ -83,6 +83,8 @@ struct pci_dev {
 #define PT_FAULT_THRESHOLD 10
 } fault;
 u64 vf_rlen[6];
+
+   void *iommu; /* No common IOMMU struct so use void pointer */
 };



2. Save iommu pointer in 'struct pci_dev' when to add device:

--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -1994,6 +1994,7 @@ static int intel_iommu_enable_device(struct pci_dev *pdev)
 if ( ret <= 0 )
 return ret;

+pdev->iommu = drhd->iommu;
 ret = enable_ats_device(pdev->seg, pdev->bus, pdev->devfn, drhd->iommu);

3. use iommu pointer from pci_dev instead of calling 
acpi_find_matched_drhd_unit each time (also fix the related code).


-Quan
 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v12 1/6] IOMMU: add a timeout parameter for device IOTLB invalidation

[Xu, Quan]

On June 27, 2016 4:29 PM, Jan Beulich <jbeul...@suse.com> wrote:
> >>> On 27.06.16 at 10:19, <quan...@intel.com> wrote:
> > On June 27, 2016 4:03 PM, Jan Beulich <jbeul...@suse.com> wrote:
> >> >>> On 24.06.16 at 07:51, <quan...@intel.com> wrote:
> >> > From: Quan Xu <quan...@intel.com>
> >> >
> >> > The parameter 'iommu_dev_iotlb_timeout' specifies the timeout of
> >> > device IOTLB invalidation in milliseconds. By default, the timeout
> >> > is
> >> > 1000 milliseconds, which can be boot-time changed.
> >> >
> >> > We also confirmed with VT-d hardware team that 1 milliseconds is
> >> > large enough for VT-d IOMMU internal invalidation.
> >> >
> >> > the existing panic() is eliminated and we bubble up the timeout of
> >> > device IOTLB invalidation for further processing, as the PCI-e
> >> > Address Translation Services (ATS) mandates a timeout of
> >> > 60 seconds for device IOTLB invalidation. Obviously we can't spin
> >> > for
> >> > 60 seconds or otherwise Xen hypervisor hangs.
> >> >
> >> > Add a __must_check annotation. The followup patch titled 'VT-d
> >> > IOTLB/Context/IEC flush issue' addresses the __mustcheck.
> >> > That is the other callers of this routine (two or three levels up)
> >> > ignore the return code. This patch does not address this but the
> >> > other does.
> >>
> >> The patch itself looks okay,
> >
> > Jan, thanks for your review.
> >
> >> but I'm confused by this paragraph:
> >> There's no patch with the named title later in this series. And
> >> having gone through this patch I also don't see what remains to be
> >> addressed wrt the __must_check-s getting added here.
> >
> > This paragraph was added from a few rounds ago. I will drop it in next
> > version.
> 
> Well, if dropping this paragraph is all that's needed, 

Yes,

> I can do this while
> committing: Patches 1-3 appear to be ready to go in.
> 

Ah, that's great. Thanks.

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v12 1/6] IOMMU: add a timeout parameter for device IOTLB invalidation

[Xu, Quan]

On June 27, 2016 4:03 PM, Jan Beulich <jbeul...@suse.com> wrote:
> >>> On 24.06.16 at 07:51, <quan...@intel.com> wrote:
> > From: Quan Xu <quan...@intel.com>
> >
> > The parameter 'iommu_dev_iotlb_timeout' specifies the timeout of
> > device IOTLB invalidation in milliseconds. By default, the timeout is
> > 1000 milliseconds, which can be boot-time changed.
> >
> > We also confirmed with VT-d hardware team that 1 milliseconds is large
> > enough for VT-d IOMMU internal invalidation.
> >
> > the existing panic() is eliminated and we bubble up the timeout of
> > device IOTLB invalidation for further processing, as the PCI-e Address
> > Translation Services (ATS) mandates a timeout of
> > 60 seconds for device IOTLB invalidation. Obviously we can't spin for
> > 60 seconds or otherwise Xen hypervisor hangs.
> >
> > Add a __must_check annotation. The followup patch titled 'VT-d
> > IOTLB/Context/IEC flush issue' addresses the __mustcheck.
> > That is the other callers of this routine (two or three levels up)
> > ignore the return code. This patch does not address this but the other
> > does.
> 
> The patch itself looks okay,

Jan, thanks for your review.

> but I'm confused by this paragraph:
> There's no patch with the named title later in this series. And having gone
> through this patch I also don't see what remains to be addressed wrt the
> __must_check-s getting added here.
> 

This paragraph was added from a few rounds ago. I will drop it in next version.

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v12 4/6] IOMMU/x86: using a struct pci_dev* instead of SBDF

[Xu, Quan]

On June 24, 2016 7:46 PM, Tian, Kevin <kevin.t...@intel.com> wrote:
> > From: Xu, Quan
> > Sent: Friday, June 24, 2016 1:52 PM
> >
> > From: Quan Xu <quan...@intel.com>
> >
> > a struct pci_dev* instead of SBDF is stored inside struct pci_ats_dev
> > and parameter to enable_ats_device().
> >
> > Signed-off-by: Quan Xu <quan...@intel.com>
> 
> Can we unify the naming convention throughout the patch, e.g.
> always using ats_pdev for "struct pci_ats_dev" variable,

Kevin, Is it 'ats_dev'? -Quan

> while pdev for "struct
> pci_dev". It's quite confusing when reading the patch which has both named
> as pdev in various places I know the confusion is also in original code, 
> but
> please take this chance to clean them up. :-)

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v12 6/6] vt-d: fix vt-d Device-TLB flush timeout issue

[Xu, Quan]

On June 24, 2016 7:55 PM, Tian, Kevin <kevin.t...@intel.com> wrote:
> > From: Xu, Quan
> > Sent: Friday, June 24, 2016 1:52 PM
> > diff --git a/xen/drivers/passthrough/vtd/extern.h
> > b/xen/drivers/passthrough/vtd/extern.h
> > index 45357f2..efaff28 100644
> > --- a/xen/drivers/passthrough/vtd/extern.h
> > +++ b/xen/drivers/passthrough/vtd/extern.h
> > @@ -25,6 +25,7 @@
> >
> >  #define VTDPREFIX "[VT-D]"
> >
> > +struct pci_ats_dev;
> >  extern bool_t rwbf_quirk;
> >
> >  void print_iommu_regs(struct acpi_drhd_unit *drhd); @@ -60,8 +61,8 @@
> > int dev_invalidate_iotlb(struct iommu *iommu, u16 did,
> >   u64 addr, unsigned int size_order, u64
> > type);
> >
> >  int __must_check qinval_device_iotlb_sync(struct iommu *iommu,
> > -  u32 max_invs_pend,
> > -  u16 sid, u16 size, u64 addr);
> > +  struct pci_ats_dev *ats_dev,
> > +  u16 did, u16 size, u64
> > + addr);
> >
> >  unsigned int get_cache_line_size(void);  void cacheline_flush(char
> > *); diff --git a/xen/drivers/passthrough/vtd/qinval.c
> > b/xen/drivers/passthrough/vtd/qinval.c
> > index 4492b29..e4e2771 100644
> > --- a/xen/drivers/passthrough/vtd/qinval.c
> > +++ b/xen/drivers/passthrough/vtd/qinval.c
> > @@ -27,11 +27,11 @@
> >  #include "dmar.h"
> >  #include "vtd.h"
> >  #include "extern.h"
> > +#include "../ats.h"
> 
> Earlier you said:
> >1. a forward declaration struct pci_ats_dev*, instead of
> >   including ats.h.
>

This context is 'in extern.h', but..

> But above you still have ats.h included.
> 

.. I really need to include 'ats.h' here, as the 'struct pci_ats_dev*' is used 
in this file.

> >
> >  #define VTD_QI_TIMEOUT 1
> >
> > -static int __must_check invalidate_sync(struct iommu *iommu,
> > -bool_t flush_dev_iotlb);
> > +static int __must_check invalidate_sync(struct iommu *iommu);
> 
> I don't understand the rationale behind. In earlier patch you introduce a new
> parameter which is however just removed later here
> 
In earlier patch, refactor invalidate_sync() to indicate whether we need to 
flush device IOTLB or not.
change it back here, as I add a specific function - dev_invalidate_sync() for 
device IOTLB invalidation..

Quan





___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v12 5/6] IOMMU: move the domain crash logic up to the generic IOMMU layer

[Xu, Quan]

On June 24, 2016 7:48 PM, Tian, Kevin <kevin.t...@intel.com> wrote:
> > From: Xu, Quan
> > Sent: Friday, June 24, 2016 1:52 PM
> >
> > From: Quan Xu <quan...@intel.com>
> >
> > Signed-off-by: Quan Xu <quan...@intel.com>
> >
> > CC: Julien Grall <julien.gr...@arm.com>
> > CC: Kevin Tian <kevin.t...@intel.com>
> > CC: Feng Wu <feng...@intel.com>
> > CC: Jan Beulich <jbeul...@suse.com>
> > CC: Suravee Suthikulpanit <suravee.suthikulpa...@amd.com>
> > ---
> >  xen/drivers/passthrough/iommu.c | 30
> > --
> >  xen/drivers/passthrough/vtd/iommu.c | 11 +++
> >  2 files changed, 39 insertions(+), 2 deletions(-)
> 
> when you say "moving the logic up", I don't see any lines being deleted. Looks
> you are just "adding the domain crash logic"?

Yes, it is 'adding'..

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v12 4/6] IOMMU/x86: using a struct pci_dev* instead of SBDF

[Xu, Quan]

On June 24, 2016 7:46 PM, Tian, Kevin <kevin.t...@intel.com> wrote:
> > From: Xu, Quan
> > Sent: Friday, June 24, 2016 1:52 PM
> >
> > From: Quan Xu <quan...@intel.com>
> >
> > a struct pci_dev* instead of SBDF is stored inside struct pci_ats_dev
> > and parameter to enable_ats_device().
> >
> > Signed-off-by: Quan Xu <quan...@intel.com>
> 
> Can we unify the naming convention throughout the patch, e.g.
> always using ats_pdev for "struct pci_ats_dev" variable, while pdev for 
> "struct
> pci_dev". It's quite confusing when reading the patch which has both named
> as pdev in various places I know the confusion is also in original code, 
> but
> please take this chance to clean them up. :-)

Make sense. I'll fix it in next patch soon.

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v12 3/6] vt-d: convert conditionals of qi_ctrl->qinval_maddr into ASSERT()s

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

QI ought to have got disabled if any of the IOMMU table setup
failed. A QI function (other than enable_qinval) is unreachable
when qi_ctrl->qinval_maddr is zero.

Signed-off-by: Quan Xu <quan...@intel.com>

CC: Jan Beulich <jbeul...@suse.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>
---
 xen/drivers/passthrough/vtd/qinval.c | 52 
 1 file changed, 23 insertions(+), 29 deletions(-)

diff --git a/xen/drivers/passthrough/vtd/qinval.c 
b/xen/drivers/passthrough/vtd/qinval.c
index 46c4c8f..4492b29 100644
--- a/xen/drivers/passthrough/vtd/qinval.c
+++ b/xen/drivers/passthrough/vtd/qinval.c
@@ -204,10 +204,9 @@ static int __must_check invalidate_sync(struct iommu 
*iommu,
 {
 struct qi_ctrl *qi_ctrl = iommu_qi_ctrl(iommu);
 
-if ( qi_ctrl->qinval_maddr )
-return queue_invalidate_wait(iommu, 0, 1, 1, flush_dev_iotlb);
+ASSERT(qi_ctrl->qinval_maddr);
 
-return 0;
+return queue_invalidate_wait(iommu, 0, 1, 1, flush_dev_iotlb);
 }
 
 int qinval_device_iotlb_sync(struct iommu *iommu,
@@ -297,10 +296,11 @@ static int __must_check flush_context_qi(void *_iommu, 
u16 did,
  u16 sid, u8 fm, u64 type,
  bool_t flush_non_present_entry)
 {
-int ret = 0;
 struct iommu *iommu = (struct iommu *)_iommu;
 struct qi_ctrl *qi_ctrl = iommu_qi_ctrl(iommu);
 
+ASSERT(qi_ctrl->qinval_maddr);
+
 /*
  * In the non-present entry flush case, if hardware doesn't cache
  * non-present entry we do nothing and if hardware cache non-present
@@ -315,11 +315,8 @@ static int __must_check flush_context_qi(void *_iommu, u16 
did,
 did = 0;
 }
 
-if ( qi_ctrl->qinval_maddr != 0 )
-ret = queue_invalidate_context_sync(iommu, did, sid, fm,
-type >> 
DMA_CCMD_INVL_GRANU_OFFSET);
-
-return ret;
+return queue_invalidate_context_sync(iommu, did, sid, fm,
+ type >> DMA_CCMD_INVL_GRANU_OFFSET);
 }
 
 static int __must_check flush_iotlb_qi(void *_iommu, u16 did, u64 addr,
@@ -328,10 +325,12 @@ static int __must_check flush_iotlb_qi(void *_iommu, u16 
did, u64 addr,
bool_t flush_dev_iotlb)
 {
 u8 dr = 0, dw = 0;
-int ret = 0;
+int ret = 0, rc;
 struct iommu *iommu = (struct iommu *)_iommu;
 struct qi_ctrl *qi_ctrl = iommu_qi_ctrl(iommu);
 
+ASSERT(qi_ctrl->qinval_maddr);
+
 /*
  * In the non-present entry flush case, if hardware doesn't cache
  * non-present entry we do nothing and if hardware cache non-present
@@ -346,28 +345,23 @@ static int __must_check flush_iotlb_qi(void *_iommu, u16 
did, u64 addr,
 did = 0;
 }
 
-if ( qi_ctrl->qinval_maddr != 0 )
+/* use queued invalidation */
+if (cap_write_drain(iommu->cap))
+dw = 1;
+if (cap_read_drain(iommu->cap))
+dr = 1;
+/* Need to conside the ih bit later */
+rc = queue_invalidate_iotlb_sync(iommu,
+ type >> DMA_TLB_FLUSH_GRANU_OFFSET,
+ dr, dw, did, size_order, 0, addr);
+if ( !ret )
+ret = rc;
+
+if ( flush_dev_iotlb )
 {
-int rc;
-
-/* use queued invalidation */
-if (cap_write_drain(iommu->cap))
-dw = 1;
-if (cap_read_drain(iommu->cap))
-dr = 1;
-/* Need to conside the ih bit later */
-rc = queue_invalidate_iotlb_sync(iommu,
- type >> DMA_TLB_FLUSH_GRANU_OFFSET,
- dr, dw, did, size_order, 0, addr);
+rc = dev_invalidate_iotlb(iommu, did, addr, size_order, type);
 if ( !ret )
 ret = rc;
-
-if ( flush_dev_iotlb )
-{
-rc = dev_invalidate_iotlb(iommu, did, addr, size_order, type);
-if ( !ret )
-ret = rc;
-}
 }
 return ret;
 }
-- 
1.9.1


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v12 6/6] vt-d: fix vt-d Device-TLB flush timeout issue

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

If Device-TLB flush timed out, we hide the target ATS device
immediately and crash the domain owning this ATS device. If
impacted domain is hardware domain, just throw out a warning.

By hiding the device, we make sure it can't be assigned to any
domain any longer (see device_assigned).

Signed-off-by: Quan Xu <quan...@intel.com>

CC: Jan Beulich <jbeul...@suse.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>

---
v12:
   1. a forward declaration struct pci_ats_dev*, instead of
  including ats.h.
   2. eliminate the loop.
   3. use the same logic for logging and crashing as I did in
  other series (despite I have moved the domain crash logic
  up to the generic IOMMU layer, I think I am better still
  leave it as is).
   4. enhance dev_invalidate_sync() with ASSERT().
---
 xen/drivers/passthrough/pci.c |  6 +--
 xen/drivers/passthrough/vtd/extern.h  |  5 ++-
 xen/drivers/passthrough/vtd/qinval.c  | 75 +--
 xen/drivers/passthrough/vtd/x86/ats.c |  9 +
 xen/include/xen/pci.h |  1 +
 5 files changed, 71 insertions(+), 25 deletions(-)

diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
index 98936f55c..843dc88 100644
--- a/xen/drivers/passthrough/pci.c
+++ b/xen/drivers/passthrough/pci.c
@@ -419,7 +419,7 @@ static void free_pdev(struct pci_seg *pseg, struct pci_dev 
*pdev)
 xfree(pdev);
 }
 
-static void _pci_hide_device(struct pci_dev *pdev)
+void pci_hide_existing_device(struct pci_dev *pdev)
 {
 if ( pdev->domain )
 return;
@@ -436,7 +436,7 @@ int __init pci_hide_device(int bus, int devfn)
 pdev = alloc_pdev(get_pseg(0), bus, devfn);
 if ( pdev )
 {
-_pci_hide_device(pdev);
+pci_hide_existing_device(pdev);
 rc = 0;
 }
 pcidevs_unlock();
@@ -466,7 +466,7 @@ int __init pci_ro_device(int seg, int bus, int devfn)
 }
 
 __set_bit(PCI_BDF2(bus, devfn), pseg->ro_map);
-_pci_hide_device(pdev);
+pci_hide_existing_device(pdev);
 
 return 0;
 }
diff --git a/xen/drivers/passthrough/vtd/extern.h 
b/xen/drivers/passthrough/vtd/extern.h
index 45357f2..efaff28 100644
--- a/xen/drivers/passthrough/vtd/extern.h
+++ b/xen/drivers/passthrough/vtd/extern.h
@@ -25,6 +25,7 @@
 
 #define VTDPREFIX "[VT-D]"
 
+struct pci_ats_dev;
 extern bool_t rwbf_quirk;
 
 void print_iommu_regs(struct acpi_drhd_unit *drhd);
@@ -60,8 +61,8 @@ int dev_invalidate_iotlb(struct iommu *iommu, u16 did,
  u64 addr, unsigned int size_order, u64 type);
 
 int __must_check qinval_device_iotlb_sync(struct iommu *iommu,
-  u32 max_invs_pend,
-  u16 sid, u16 size, u64 addr);
+  struct pci_ats_dev *ats_dev,
+  u16 did, u16 size, u64 addr);
 
 unsigned int get_cache_line_size(void);
 void cacheline_flush(char *);
diff --git a/xen/drivers/passthrough/vtd/qinval.c 
b/xen/drivers/passthrough/vtd/qinval.c
index 4492b29..e4e2771 100644
--- a/xen/drivers/passthrough/vtd/qinval.c
+++ b/xen/drivers/passthrough/vtd/qinval.c
@@ -27,11 +27,11 @@
 #include "dmar.h"
 #include "vtd.h"
 #include "extern.h"
+#include "../ats.h"
 
 #define VTD_QI_TIMEOUT 1
 
-static int __must_check invalidate_sync(struct iommu *iommu,
-bool_t flush_dev_iotlb);
+static int __must_check invalidate_sync(struct iommu *iommu);
 
 static void print_qi_regs(struct iommu *iommu)
 {
@@ -103,7 +103,7 @@ static int __must_check 
queue_invalidate_context_sync(struct iommu *iommu,
 
 unmap_vtd_domain_page(qinval_entries);
 
-return invalidate_sync(iommu, 0);
+return invalidate_sync(iommu);
 }
 
 static int __must_check queue_invalidate_iotlb_sync(struct iommu *iommu,
@@ -140,7 +140,7 @@ static int __must_check queue_invalidate_iotlb_sync(struct 
iommu *iommu,
 qinval_update_qtail(iommu, index);
 spin_unlock_irqrestore(>register_lock, flags);
 
-return invalidate_sync(iommu, 0);
+return invalidate_sync(iommu);
 }
 
 static int __must_check queue_invalidate_wait(struct iommu *iommu,
@@ -199,24 +199,73 @@ static int __must_check queue_invalidate_wait(struct 
iommu *iommu,
 return -EOPNOTSUPP;
 }
 
-static int __must_check invalidate_sync(struct iommu *iommu,
-bool_t flush_dev_iotlb)
+static int __must_check invalidate_sync(struct iommu *iommu)
 {
 struct qi_ctrl *qi_ctrl = iommu_qi_ctrl(iommu);
 
 ASSERT(qi_ctrl->qinval_maddr);
 
-return queue_invalidate_wait(iommu, 0, 1, 1, flush_dev_iotlb);
+return queue_invalidate_wait(iommu, 0, 1, 1, 0);
+}
+
+static void dev_invalidate_iotlb_timeout(struct iommu *iommu, u16 did,
+ struct p

[Xen-devel] [PATCH v12 2/6] vt-d: synchronize for Device-TLB flush one by one

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Today we do Device-TLB flush synchronization after issuing flush
requests for all ATS devices belonging to a VM. Doing so however
imposes a limitation, i.e. that we can not figure out which flush
request is blocked in the flush queue list, based on VT-d spec.

To prepare correct Device-TLB flush timeout handling in next patch,
we change the behavior to synchronize for every Device-TLB flush
request. So the Device-TLB flush interface is changed a little bit,
by checking timeout within the function instead of outside of function.

Accordingly we also do a similar change for flush interfaces of
IOTLB/IEC/Context, i.e. moving synchronization into the function.
Since there is no user of a non-synced interface, we just rename
existing ones with _sync suffix.

Signed-off-by: Quan Xu <quan...@intel.com>
Reviewed-by: Jan Beulich <jbeul...@suse.com>

CC: Jan Beulich <jbeul...@suse.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>
---
 xen/drivers/passthrough/vtd/extern.h  |  5 +--
 xen/drivers/passthrough/vtd/qinval.c  | 65 ---
 xen/drivers/passthrough/vtd/x86/ats.c |  8 ++---
 3 files changed, 45 insertions(+), 33 deletions(-)

diff --git a/xen/drivers/passthrough/vtd/extern.h 
b/xen/drivers/passthrough/vtd/extern.h
index 6772839..45357f2 100644
--- a/xen/drivers/passthrough/vtd/extern.h
+++ b/xen/drivers/passthrough/vtd/extern.h
@@ -59,8 +59,9 @@ int ats_device(const struct pci_dev *, const struct 
acpi_drhd_unit *);
 int dev_invalidate_iotlb(struct iommu *iommu, u16 did,
  u64 addr, unsigned int size_order, u64 type);
 
-int qinval_device_iotlb(struct iommu *iommu,
-u32 max_invs_pend, u16 sid, u16 size, u64 addr);
+int __must_check qinval_device_iotlb_sync(struct iommu *iommu,
+  u32 max_invs_pend,
+  u16 sid, u16 size, u64 addr);
 
 unsigned int get_cache_line_size(void);
 void cacheline_flush(char *);
diff --git a/xen/drivers/passthrough/vtd/qinval.c 
b/xen/drivers/passthrough/vtd/qinval.c
index 4788d5f..46c4c8f 100644
--- a/xen/drivers/passthrough/vtd/qinval.c
+++ b/xen/drivers/passthrough/vtd/qinval.c
@@ -30,6 +30,9 @@
 
 #define VTD_QI_TIMEOUT 1
 
+static int __must_check invalidate_sync(struct iommu *iommu,
+bool_t flush_dev_iotlb);
+
 static void print_qi_regs(struct iommu *iommu)
 {
 u64 val;
@@ -69,8 +72,10 @@ static void qinval_update_qtail(struct iommu *iommu, 
unsigned int index)
 dmar_writeq(iommu->reg, DMAR_IQT_REG, (val << QINVAL_INDEX_SHIFT));
 }
 
-static void queue_invalidate_context(struct iommu *iommu,
-u16 did, u16 source_id, u8 function_mask, u8 granu)
+static int __must_check queue_invalidate_context_sync(struct iommu *iommu,
+  u16 did, u16 source_id,
+  u8 function_mask,
+  u8 granu)
 {
 unsigned long flags;
 unsigned int index;
@@ -97,10 +102,14 @@ static void queue_invalidate_context(struct iommu *iommu,
 spin_unlock_irqrestore(>register_lock, flags);
 
 unmap_vtd_domain_page(qinval_entries);
+
+return invalidate_sync(iommu, 0);
 }
 
-static void queue_invalidate_iotlb(struct iommu *iommu,
-u8 granu, u8 dr, u8 dw, u16 did, u8 am, u8 ih, u64 addr)
+static int __must_check queue_invalidate_iotlb_sync(struct iommu *iommu,
+u8 granu, u8 dr, u8 dw,
+u16 did, u8 am, u8 ih,
+u64 addr)
 {
 unsigned long flags;
 unsigned int index;
@@ -130,6 +139,8 @@ static void queue_invalidate_iotlb(struct iommu *iommu,
 unmap_vtd_domain_page(qinval_entries);
 qinval_update_qtail(iommu, index);
 spin_unlock_irqrestore(>register_lock, flags);
+
+return invalidate_sync(iommu, 0);
 }
 
 static int __must_check queue_invalidate_wait(struct iommu *iommu,
@@ -199,8 +210,9 @@ static int __must_check invalidate_sync(struct iommu *iommu,
 return 0;
 }
 
-int qinval_device_iotlb(struct iommu *iommu,
-u32 max_invs_pend, u16 sid, u16 size, u64 addr)
+int qinval_device_iotlb_sync(struct iommu *iommu,
+ u32 max_invs_pend,
+ u16 sid, u16 size, u64 addr)
 {
 unsigned long flags;
 unsigned int index;
@@ -229,15 +241,17 @@ int qinval_device_iotlb(struct iommu *iommu,
 qinval_update_qtail(iommu, index);
 spin_unlock_irqrestore(>register_lock, flags);
 
-return 0;
+return invalidate_sync(iommu, 1);
 }
 
-static void queue_invalidate_iec(struct iommu *iommu, u8 granu, u8 im, u16 
iidx)
+static int __must_check queue_invalidate_iec_sync(struct iommu *iommu,
+   

[Xen-devel] [PATCH v12 0/6] VT-d Device-TLB flush issue

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

This patches fix current timeout concern and also allow limited ATS support:

1. add a timeout parameter for device IOTLB invalidation

The parameter 'iommu_dev_iotlb_timeout' specifies the timeout
of device IOTLB invalidation in milliseconds. By default, the
timeout is 1000 milliseconds, which can be boot-time changed.

We also confirmed with VT-d hardware team that 1 milliseconds
is large enough for VT-d IOMMU internal invalidation.

the existing panic() is eliminated and we bubble up the timeout
of device IOTLB invalidation for further processing, as the
PCI-e Address Translation Services (ATS) mandates a timeout of
60 seconds for device IOTLB invalidation. Obviously we can't
spin for 60 seconds or otherwise Xen hypervisor hangs.

2. today we do Device-TLB flush synchronization after issuing flush
requests for all ATS devices belonging to a VM. Doing so however
imposes a limitation, i.e. that we can not figure out which flush
request is blocked in the flush queue list, based on VT-d spec.

To prepare correct Device-TLB flush timeout handling in next patch,
we change the behavior to synchronize for every Device-TLB flush
request. So the Device-TLB flush interface is changed a little bit,
by checking timeout within the function instead of outside of function.

Accordingly we also do a similar change for flush interfaces of
IOTLB/IEC/Context, i.e. moving synchronization into the function.
Since there is no user of a non-synced interface, we just rename
existing ones with _sync suffix.

3. move the domain crash logic up to the generic IOMMU layer

4. If Device-TLB flush timed out, we hide the target ATS device
immediately and crash the domain owning this ATS device. If
impacted domain is hardware domain, just throw out a warning.

By hiding the device, we make sure it can't be assigned to any
domain any longer (see device_assigned).

---
Not covered in this series:

a) Eliminate the panic() in IOMMU_WAIT_OP, used only in VT-d register 
read/write.
   Further discussion is required on whether and how to improve it.
b) Handle IOTLB/Context/IEC flush timeout (after v12 review, I try to send 
out
   brain storming).
---
Quan Xu (6):
  IOMMU: add a timeout parameter for device IOTLB invalidation
  vt-d: synchronize for Device-TLB flush one by one
  vt-d: convert conditionals of qi_ctrl->qinval_maddr into ASSERT()s
  IOMMU/x86: using a struct pci_dev* instead of SBDF
  IOMMU: move the domain crash logic up to the generic IOMMU layer
  vt-d: fix vt-d Device-TLB flush timeout issue

 docs/misc/xen-command-line.markdown |   9 ++
 xen/drivers/passthrough/amd/iommu_cmd.c |  11 +-
 xen/drivers/passthrough/amd/pci_amd_iommu.c |   2 +-
 xen/drivers/passthrough/ats.h   |   6 +-
 xen/drivers/passthrough/iommu.c |  33 +-
 xen/drivers/passthrough/pci.c   |   6 +-
 xen/drivers/passthrough/vtd/extern.h|   6 +-
 xen/drivers/passthrough/vtd/iommu.c |  19 +++-
 xen/drivers/passthrough/vtd/qinval.c| 168 +++-
 xen/drivers/passthrough/vtd/x86/ats.c   |  23 ++--
 xen/drivers/passthrough/x86/ats.c   |  52 ++---
 xen/include/xen/iommu.h |   2 +
 xen/include/xen/pci.h   |   1 +
 13 files changed, 238 insertions(+), 100 deletions(-)

-- 
1.9.1


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v12 1/6] IOMMU: add a timeout parameter for device IOTLB invalidation

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

The parameter 'iommu_dev_iotlb_timeout' specifies the timeout
of device IOTLB invalidation in milliseconds. By default, the
timeout is 1000 milliseconds, which can be boot-time changed.

We also confirmed with VT-d hardware team that 1 milliseconds
is large enough for VT-d IOMMU internal invalidation.

the existing panic() is eliminated and we bubble up the timeout
of device IOTLB invalidation for further processing, as the
PCI-e Address Translation Services (ATS) mandates a timeout of
60 seconds for device IOTLB invalidation. Obviously we can't
spin for 60 seconds or otherwise Xen hypervisor hangs.

Add a __must_check annotation. The followup patch titled
'VT-d IOTLB/Context/IEC flush issue' addresses the __mustcheck.
That is the other callers of this routine (two or three levels up)
ignore the return code. This patch does not address this but the
other does.

Signed-off-by: Quan Xu <quan...@intel.com>

CC: Julien Grall <julien.gr...@arm.com>
CC: Jan Beulich <jbeul...@suse.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>
CC: Suravee Suthikulpanit <suravee.suthikulpa...@amd.com>
---
v12:
   1. enhance commit message.
   2. change timeout from 1ms to 1000ms.
   3. change IOMMU_QI_TIMEOUT to VTD_QI_TIMEOUT, with VTD_QI_TIMEOUT
  having its MILLISECS() dropped.
   4. enhance the whole expression of 'timeout = ...'
   5. drop a blank line that doesn't belong here.
---
 docs/misc/xen-command-line.markdown  |  9 +
 xen/drivers/passthrough/iommu.c  |  3 +++
 xen/drivers/passthrough/vtd/qinval.c | 32 +---
 xen/include/xen/iommu.h  |  2 ++
 4 files changed, 35 insertions(+), 11 deletions(-)

diff --git a/docs/misc/xen-command-line.markdown 
b/docs/misc/xen-command-line.markdown
index 7a271c0..0046f0d 100644
--- a/docs/misc/xen-command-line.markdown
+++ b/docs/misc/xen-command-line.markdown
@@ -1015,6 +1015,15 @@ debug hypervisor only).
 
 >> Enable IOMMU debugging code (implies `verbose`).
 
+### iommu\_dev\_iotlb\_timeout
+> `= `
+
+> Default: `1000`
+
+Specify the timeout of the device IOTLB invalidation in milliseconds.
+By default, the timeout is 1000 ms. When you see error 'Queue invalidate
+wait descriptor timed out', try increasing this value.
+
 ### iommu\_inclusive\_mapping (VT-d)
 > `= `
 
diff --git a/xen/drivers/passthrough/iommu.c b/xen/drivers/passthrough/iommu.c
index ef80b3c..7656aeb 100644
--- a/xen/drivers/passthrough/iommu.c
+++ b/xen/drivers/passthrough/iommu.c
@@ -24,6 +24,9 @@
 static void parse_iommu_param(char *s);
 static void iommu_dump_p2m_table(unsigned char key);
 
+unsigned int __read_mostly iommu_dev_iotlb_timeout = 1000;
+integer_param("iommu_dev_iotlb_timeout", iommu_dev_iotlb_timeout);
+
 /*
  * The 'iommu' parameter enables the IOMMU.  Optional comma separated
  * value may contain:
diff --git a/xen/drivers/passthrough/vtd/qinval.c 
b/xen/drivers/passthrough/vtd/qinval.c
index aa7841a..4788d5f 100644
--- a/xen/drivers/passthrough/vtd/qinval.c
+++ b/xen/drivers/passthrough/vtd/qinval.c
@@ -28,6 +28,8 @@
 #include "vtd.h"
 #include "extern.h"
 
+#define VTD_QI_TIMEOUT 1
+
 static void print_qi_regs(struct iommu *iommu)
 {
 u64 val;
@@ -130,10 +132,10 @@ static void queue_invalidate_iotlb(struct iommu *iommu,
 spin_unlock_irqrestore(>register_lock, flags);
 }
 
-static int queue_invalidate_wait(struct iommu *iommu,
-u8 iflag, u8 sw, u8 fn)
+static int __must_check queue_invalidate_wait(struct iommu *iommu,
+  u8 iflag, u8 sw, u8 fn,
+  bool_t flush_dev_iotlb)
 {
-s_time_t start_time;
 volatile u32 poll_slot = QINVAL_STAT_INIT;
 unsigned int index;
 unsigned long flags;
@@ -163,14 +165,20 @@ static int queue_invalidate_wait(struct iommu *iommu,
 /* Now we don't support interrupt method */
 if ( sw )
 {
+s_time_t timeout;
+
 /* In case all wait descriptor writes to same addr with same data */
-start_time = NOW();
+timeout = NOW() + MILLISECS(flush_dev_iotlb ?
+iommu_dev_iotlb_timeout : VTD_QI_TIMEOUT);
+
 while ( poll_slot != QINVAL_STAT_DONE )
 {
-if ( NOW() > (start_time + DMAR_OPERATION_TIMEOUT) )
+if ( NOW() > timeout )
 {
 print_qi_regs(iommu);
-panic("queue invalidate wait descriptor was not executed");
+printk(XENLOG_WARNING VTDPREFIX
+   " Queue invalidate wait descriptor timed out\n");
+return -ETIMEDOUT;
 }
 cpu_relax();
 }
@@ -180,12 +188,14 @@ static int queue_invalidate_wait(struct iommu *iommu,
 return -EOPNOTSUPP;
 }
 
-static int invalidate_sync(struct iommu *iommu)
+static i

[Xen-devel] [PATCH v12 4/6] IOMMU/x86: using a struct pci_dev* instead of SBDF

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

a struct pci_dev* instead of SBDF is stored inside struct
pci_ats_dev and parameter to enable_ats_device().

Signed-off-by: Quan Xu <quan...@intel.com>

CC: Jan Beulich <jbeul...@suse.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>
CC: Suravee Suthikulpanit <suravee.suthikulpa...@amd.com>
---
 xen/drivers/passthrough/amd/iommu_cmd.c | 11 +++---
 xen/drivers/passthrough/amd/pci_amd_iommu.c |  2 +-
 xen/drivers/passthrough/ats.h   |  6 ++--
 xen/drivers/passthrough/vtd/iommu.c |  8 ++---
 xen/drivers/passthrough/vtd/x86/ats.c   | 20 ---
 xen/drivers/passthrough/x86/ats.c   | 52 +++--
 6 files changed, 62 insertions(+), 37 deletions(-)

diff --git a/xen/drivers/passthrough/amd/iommu_cmd.c 
b/xen/drivers/passthrough/amd/iommu_cmd.c
index 7c9d9be..b3094f3 100644
--- a/xen/drivers/passthrough/amd/iommu_cmd.c
+++ b/xen/drivers/passthrough/amd/iommu_cmd.c
@@ -298,24 +298,23 @@ void amd_iommu_flush_iotlb(u8 devfn, const struct pci_dev 
*pdev,
 if ( ats_pdev == NULL )
 return;
 
-if ( !pci_ats_enabled(ats_pdev->seg, ats_pdev->bus, ats_pdev->devfn) )
+if ( !pci_ats_enabled(pdev->seg, pdev->bus, pdev->devfn) )
 return;
 
-iommu = find_iommu_for_device(ats_pdev->seg,
-  PCI_BDF2(ats_pdev->bus, ats_pdev->devfn));
+iommu = find_iommu_for_device(pdev->seg, PCI_BDF2(pdev->bus, pdev->devfn));
 
 if ( !iommu )
 {
 AMD_IOMMU_DEBUG("%s: Can't find iommu for %04x:%02x:%02x.%u\n",
-__func__, ats_pdev->seg, ats_pdev->bus,
-PCI_SLOT(ats_pdev->devfn), PCI_FUNC(ats_pdev->devfn));
+__func__, pdev->seg, pdev->bus,
+PCI_SLOT(pdev->devfn), PCI_FUNC(pdev->devfn));
 return;
 }
 
 if ( !iommu_has_cap(iommu, PCI_CAP_IOTLB_SHIFT) )
 return;
 
-req_id = get_dma_requestor_id(iommu->seg, PCI_BDF2(ats_pdev->bus, devfn));
+req_id = get_dma_requestor_id(iommu->seg, PCI_BDF2(pdev->bus, devfn));
 queueid = req_id;
 maxpend = ats_pdev->ats_queue_depth & 0xff;
 
diff --git a/xen/drivers/passthrough/amd/pci_amd_iommu.c 
b/xen/drivers/passthrough/amd/pci_amd_iommu.c
index 7761241..64ca78e 100644
--- a/xen/drivers/passthrough/amd/pci_amd_iommu.c
+++ b/xen/drivers/passthrough/amd/pci_amd_iommu.c
@@ -162,7 +162,7 @@ static void amd_iommu_setup_domain_device(
  !pci_ats_enabled(iommu->seg, bus, pdev->devfn) )
 {
 if ( devfn == pdev->devfn )
-enable_ats_device(iommu->seg, bus, devfn, iommu);
+enable_ats_device(iommu, pdev);
 
 amd_iommu_flush_iotlb(devfn, pdev, INV_IOMMU_ALL_PAGES_ADDRESS, 0);
 }
diff --git a/xen/drivers/passthrough/ats.h b/xen/drivers/passthrough/ats.h
index 5c91572..1359841 100644
--- a/xen/drivers/passthrough/ats.h
+++ b/xen/drivers/passthrough/ats.h
@@ -19,9 +19,7 @@
 
 struct pci_ats_dev {
 struct list_head list;
-u16 seg;
-u8 bus;
-u8 devfn;
+struct pci_dev *pci_dev;
 u16 ats_queue_depth;/* ATS device invalidation queue depth */
 const void *iommu;  /* No common IOMMU struct so use void pointer */
 };
@@ -34,7 +32,7 @@ struct pci_ats_dev {
 extern struct list_head ats_devices;
 extern bool_t ats_enabled;
 
-int enable_ats_device(int seg, int bus, int devfn, const void *iommu);
+int enable_ats_device(const void *iommu, struct pci_dev *pci_dev);
 void disable_ats_device(int seg, int bus, int devfn);
 struct pci_ats_dev *get_ats_device(int seg, int bus, int devfn);
 
diff --git a/xen/drivers/passthrough/vtd/iommu.c 
b/xen/drivers/passthrough/vtd/iommu.c
index f010612..1b0a0f0 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -1461,8 +1461,8 @@ int domain_context_mapping_one(
 return rc;
 }
 
-static int domain_context_mapping(
-struct domain *domain, u8 devfn, const struct pci_dev *pdev)
+static int domain_context_mapping(struct domain *domain, u8 devfn,
+  struct pci_dev *pdev)
 {
 struct acpi_drhd_unit *drhd;
 int ret = 0;
@@ -1498,7 +1498,7 @@ static int domain_context_mapping(
 ret = domain_context_mapping_one(domain, drhd->iommu, bus, devfn,
  pdev);
 if ( !ret && devfn == pdev->devfn && ats_device(pdev, drhd) > 0 )
-enable_ats_device(seg, bus, devfn, drhd->iommu);
+enable_ats_device(drhd->iommu, pdev);
 
 break;
 
@@ -1994,7 +1994,7 @@ static int intel_iommu_enable_device(struct pci_dev *pdev)
 if ( ret <= 0 )
 return ret;
 
-ret = enable_ats_device(pdev->seg, pdev->bus, pdev->devfn, drhd->iommu);
+ret = enable_ats_de

[Xen-devel] [PATCH v12 5/6] IOMMU: move the domain crash logic up to the generic IOMMU layer

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Signed-off-by: Quan Xu <quan...@intel.com>

CC: Julien Grall <julien.gr...@arm.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>
CC: Jan Beulich <jbeul...@suse.com>
CC: Suravee Suthikulpanit <suravee.suthikulpa...@amd.com>
---
 xen/drivers/passthrough/iommu.c | 30 --
 xen/drivers/passthrough/vtd/iommu.c | 11 +++
 2 files changed, 39 insertions(+), 2 deletions(-)

diff --git a/xen/drivers/passthrough/iommu.c b/xen/drivers/passthrough/iommu.c
index 7656aeb..d793f5d 100644
--- a/xen/drivers/passthrough/iommu.c
+++ b/xen/drivers/passthrough/iommu.c
@@ -318,21 +318,47 @@ int iommu_iotlb_flush(struct domain *d, unsigned long gfn,
   unsigned int page_count)
 {
 const struct domain_iommu *hd = dom_iommu(d);
+int rc;
 
 if ( !iommu_enabled || !hd->platform_ops || !hd->platform_ops->iotlb_flush 
)
 return 0;
 
-return hd->platform_ops->iotlb_flush(d, gfn, page_count);
+rc = hd->platform_ops->iotlb_flush(d, gfn, page_count);
+if ( unlikely(rc) )
+{
+if ( !d->is_shutting_down && printk_ratelimit() )
+printk(XENLOG_ERR
+   "d%d: IOMMU IOTLB flush failed: %d, gfn %#lx, page count 
%u\n",
+   d->domain_id, rc, gfn, page_count);
+
+if ( !is_hardware_domain(d) )
+domain_crash(d);
+}
+
+return rc;
 }
 
 int iommu_iotlb_flush_all(struct domain *d)
 {
 const struct domain_iommu *hd = dom_iommu(d);
+int rc;
 
 if ( !iommu_enabled || !hd->platform_ops || 
!hd->platform_ops->iotlb_flush_all )
 return 0;
 
-return hd->platform_ops->iotlb_flush_all(d);
+rc = hd->platform_ops->iotlb_flush_all(d);
+if ( unlikely(rc) )
+{
+if ( !d->is_shutting_down && printk_ratelimit() )
+printk(XENLOG_ERR
+   "d%d: IOMMU IOTLB flush all failed: %d\n",
+   d->domain_id, rc);
+
+if ( !is_hardware_domain(d) )
+domain_crash(d);
+}
+
+return rc;
 }
 
 int __init iommu_setup(void)
diff --git a/xen/drivers/passthrough/vtd/iommu.c 
b/xen/drivers/passthrough/vtd/iommu.c
index 1b0a0f0..82332c8 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -1847,6 +1847,17 @@ int iommu_pte_flush(struct domain *d, u64 gfn, u64 *pte,
 }
 }
 
+if ( unlikely(rc) )
+{
+if ( !d->is_shutting_down && printk_ratelimit() )
+printk(XENLOG_ERR VTDPREFIX
+   " d%d: IOMMU pages flush failed: %d\n",
+   d->domain_id, rc);
+
+if ( !is_hardware_domain(d) )
+domain_crash(d);
+}
+
 return rc;
 }
 
-- 
1.9.1


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [Patch v11 3/3] vt-d: fix vt-d Device-TLB flush timeout issue

[Xu, Quan]

On June 23, 2016 12:18 AM, Jan Beulich  wrote:
> >>> On 22.06.16 at 17:54,  wrote:
> > On June 17, 2016 3:01 PM, Jan Beulich  wrote:
> >> And again I don't understand: ASSERT()s are to verify assumed state.
> >> If
> > static
> >> code analysis resulted in understanding a function is unreachable
> >> when qi_ctrl->qinval_maddr is zero (because qinval ought to have got
> >> disabled if
> > any
> >> of the table setup failed), then adding ASSERT() would (a) document
> >> that and
> >> (b) allow to know quickly if something broke that assumption.
> >
> > other than enable_qinval() -- yes, I need to convert conditionals of
> > qi_ctrl->qinval_maddr into  ASSERT()s..
> > But in enable_qinval(), I am still not quite sure whether I need to
> > convert these conditionals of  qi_ctrl->qinval_maddr into ASSERT()s or
> > not.
> 
> No, I don't think you want to so there - you'd bring the system down in case
> of an actual initialization error. ASSERT()s should only be used on conditions
> controlled entirely by the hypervisor.
> 

Jan, thank you. Now I am clear.

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [Patch v11 3/3] vt-d: fix vt-d Device-TLB flush timeout issue

[Xu, Quan]

On June 17, 2016 3:01 PM, Jan Beulich  wrote:
> And again I don't understand: ASSERT()s are to verify assumed state. If static
> code analysis resulted in understanding a function is unreachable when
> qi_ctrl->qinval_maddr is zero (because qinval ought to have got disabled if 
> any
> of the table setup failed), then adding ASSERT() would (a) document that and
> (b) allow to know quickly if something broke that assumption.

other than enable_qinval() -- yes, I need to convert conditionals of 
qi_ctrl->qinval_maddr into
 ASSERT()s..
But in enable_qinval(), I am still not quite sure whether I need to convert 
these conditionals of
 qi_ctrl->qinval_maddr into ASSERT()s or not.

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v10 2/3] vt-d: propagate the IOMMU Device-TLB flush error up to ME phantom functions

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Signed-off-by: Quan Xu <quan...@intel.com>
Acked-by: Kevin Tian <kevin.t...@intel.com>
Reviewed-by: Jan Beulich <jbeul...@suse.com>

CC: Jan Beulich <jbeul...@suse.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>
---
 xen/drivers/passthrough/vtd/extern.h |  3 ++-
 xen/drivers/passthrough/vtd/iommu.c  |  8 
 xen/drivers/passthrough/vtd/quirks.c | 27 +--
 3 files changed, 23 insertions(+), 15 deletions(-)

diff --git a/xen/drivers/passthrough/vtd/extern.h 
b/xen/drivers/passthrough/vtd/extern.h
index cbe0286..6772839 100644
--- a/xen/drivers/passthrough/vtd/extern.h
+++ b/xen/drivers/passthrough/vtd/extern.h
@@ -91,7 +91,8 @@ int is_igd_vt_enabled_quirk(void);
 void platform_quirks_init(void);
 void vtd_ops_preamble_quirk(struct iommu* iommu);
 void vtd_ops_postamble_quirk(struct iommu* iommu);
-void me_wifi_quirk(struct domain *domain, u8 bus, u8 devfn, int map);
+int __must_check me_wifi_quirk(struct domain *domain,
+   u8 bus, u8 devfn, int map);
 void pci_vtd_quirk(const struct pci_dev *);
 bool_t platform_supports_intremap(void);
 bool_t platform_supports_x2apic(void);
diff --git a/xen/drivers/passthrough/vtd/iommu.c 
b/xen/drivers/passthrough/vtd/iommu.c
index 999cdf6..3692113 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -1455,8 +1455,8 @@ int domain_context_mapping_one(
 
 unmap_vtd_domain_page(context_entries);
 
-if ( !seg )
-me_wifi_quirk(domain, bus, devfn, MAP_ME_PHANTOM_FUNC);
+if ( !seg && !rc )
+rc = me_wifi_quirk(domain, bus, devfn, MAP_ME_PHANTOM_FUNC);
 
 return rc;
 }
@@ -1605,8 +1605,8 @@ int domain_context_unmap_one(
 spin_unlock(>lock);
 unmap_vtd_domain_page(context_entries);
 
-if ( !iommu->intel->drhd->segment )
-me_wifi_quirk(domain, bus, devfn, UNMAP_ME_PHANTOM_FUNC);
+if ( !iommu->intel->drhd->segment && !rc )
+rc = me_wifi_quirk(domain, bus, devfn, UNMAP_ME_PHANTOM_FUNC);
 
 return rc;
 }
diff --git a/xen/drivers/passthrough/vtd/quirks.c 
b/xen/drivers/passthrough/vtd/quirks.c
index 473d1fc..91f96ac 100644
--- a/xen/drivers/passthrough/vtd/quirks.c
+++ b/xen/drivers/passthrough/vtd/quirks.c
@@ -331,10 +331,12 @@ void __init platform_quirks_init(void)
  * assigning Intel integrated wifi device to a guest.
  */
 
-static void map_me_phantom_function(struct domain *domain, u32 dev, int map)
+static int __must_check map_me_phantom_function(struct domain *domain,
+u32 dev, int map)
 {
 struct acpi_drhd_unit *drhd;
 struct pci_dev *pdev;
+int rc;
 
 /* find ME VT-d engine base on a real ME device */
 pdev = pci_get_pdev(0, 0, PCI_DEVFN(dev, 0));
@@ -342,23 +344,26 @@ static void map_me_phantom_function(struct domain 
*domain, u32 dev, int map)
 
 /* map or unmap ME phantom function */
 if ( map )
-domain_context_mapping_one(domain, drhd->iommu, 0,
-   PCI_DEVFN(dev, 7), NULL);
+rc = domain_context_mapping_one(domain, drhd->iommu, 0,
+PCI_DEVFN(dev, 7), NULL);
 else
-domain_context_unmap_one(domain, drhd->iommu, 0,
- PCI_DEVFN(dev, 7));
+rc = domain_context_unmap_one(domain, drhd->iommu, 0,
+  PCI_DEVFN(dev, 7));
+
+return rc;
 }
 
-void me_wifi_quirk(struct domain *domain, u8 bus, u8 devfn, int map)
+int me_wifi_quirk(struct domain *domain, u8 bus, u8 devfn, int map)
 {
 u32 id;
+int rc = 0;
 
 id = pci_conf_read32(0, 0, 0, 0, 0);
 if ( IS_CTG(id) )
 {
 /* quit if ME does not exist */
 if ( pci_conf_read32(0, 0, 3, 0, 0) == 0x )
-return;
+return 0;
 
 /* if device is WLAN device, map ME phantom device 0:3.7 */
 id = pci_conf_read32(0, bus, PCI_SLOT(devfn), PCI_FUNC(devfn), 0);
@@ -372,7 +377,7 @@ void me_wifi_quirk(struct domain *domain, u8 bus, u8 devfn, 
int map)
 case 0x423b8086:
 case 0x423c8086:
 case 0x423d8086:
-map_me_phantom_function(domain, 3, map);
+rc = map_me_phantom_function(domain, 3, map);
 break;
 default:
 break;
@@ -382,7 +387,7 @@ void me_wifi_quirk(struct domain *domain, u8 bus, u8 devfn, 
int map)
 {
 /* quit if ME does not exist */
 if ( pci_conf_read32(0, 0, 22, 0, 0) == 0x )
-return;
+return 0;
 
 /* if device is WLAN device, map ME phantom device 0:22.7 */
 id = pci_conf_read32(0, bus, PCI_SLOT(devfn), PCI_FUNC(devfn), 0);
@@ -398,12 +403,14 @@ void me_wifi_quirk(struct domain *domain, u8 bus, u8 
devfn, int map)
 

[Xen-devel] [PATCH v10 3/3] vt-d: add __must_check annotation to IOMMU flush pointers and handlers

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Signed-off-by: Quan Xu <quan...@intel.com>
Acked-by: Kevin Tian <kevin.t...@intel.com>
Reviewed-by: Jan Beulich <jbeul...@suse.com>

CC: Jan Beulich <jbeul...@suse.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>
---
 xen/drivers/passthrough/vtd/iommu.c  | 50 ++--
 xen/drivers/passthrough/vtd/iommu.h  | 11 +---
 xen/drivers/passthrough/vtd/qinval.c | 14 +-
 3 files changed, 39 insertions(+), 36 deletions(-)

diff --git a/xen/drivers/passthrough/vtd/iommu.c 
b/xen/drivers/passthrough/vtd/iommu.c
index 3692113..f010612 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -335,10 +335,9 @@ static void iommu_flush_write_buffer(struct iommu *iommu)
 }
 
 /* return value determine if we need a write buffer flush */
-static int flush_context_reg(
-void *_iommu,
-u16 did, u16 source_id, u8 function_mask, u64 type,
-int flush_non_present_entry)
+static int __must_check flush_context_reg(void *_iommu, u16 did, u16 source_id,
+  u8 function_mask, u64 type,
+  bool_t flush_non_present_entry)
 {
 struct iommu *iommu = (struct iommu *) _iommu;
 u64 val = 0;
@@ -389,7 +388,7 @@ static int flush_context_reg(
 }
 
 static int __must_check iommu_flush_context_global(struct iommu *iommu,
-   int flush_non_present_entry)
+   bool_t 
flush_non_present_entry)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 return flush->context(iommu, 0, 0, 0, DMA_CCMD_GLOBAL_INVL,
@@ -399,7 +398,7 @@ static int __must_check iommu_flush_context_global(struct 
iommu *iommu,
 static int __must_check iommu_flush_context_device(struct iommu *iommu,
u16 did, u16 source_id,
u8 function_mask,
-   int flush_non_present_entry)
+   bool_t 
flush_non_present_entry)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 return flush->context(iommu, did, source_id, function_mask,
@@ -408,9 +407,10 @@ static int __must_check iommu_flush_context_device(struct 
iommu *iommu,
 }
 
 /* return value determine if we need a write buffer flush */
-static int flush_iotlb_reg(void *_iommu, u16 did,
-   u64 addr, unsigned int size_order, u64 type,
-   int flush_non_present_entry, int flush_dev_iotlb)
+static int __must_check flush_iotlb_reg(void *_iommu, u16 did, u64 addr,
+unsigned int size_order, u64 type,
+bool_t flush_non_present_entry,
+bool_t flush_dev_iotlb)
 {
 struct iommu *iommu = (struct iommu *) _iommu;
 int tlb_offset = ecap_iotlb_offset(iommu->ecap);
@@ -475,8 +475,8 @@ static int flush_iotlb_reg(void *_iommu, u16 did,
 }
 
 static int __must_check iommu_flush_iotlb_global(struct iommu *iommu,
- int flush_non_present_entry,
- int flush_dev_iotlb)
+ bool_t 
flush_non_present_entry,
+ bool_t flush_dev_iotlb)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 int status;
@@ -494,8 +494,8 @@ static int __must_check iommu_flush_iotlb_global(struct 
iommu *iommu,
 }
 
 static int __must_check iommu_flush_iotlb_dsi(struct iommu *iommu, u16 did,
-  int flush_non_present_entry,
-  int flush_dev_iotlb)
+  bool_t flush_non_present_entry,
+  bool_t flush_dev_iotlb)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 int status;
@@ -514,8 +514,8 @@ static int __must_check iommu_flush_iotlb_dsi(struct iommu 
*iommu, u16 did,
 
 static int __must_check iommu_flush_iotlb_psi(struct iommu *iommu, u16 did,
   u64 addr, unsigned int order,
-  int flush_non_present_entry,
-  int flush_dev_iotlb)
+  bool_t flush_non_present_entry,
+  bool_t flush_dev_iotlb)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 int status;
@@ -549,7 +549,7 @@ static int __must_check iommu_flush_all(void)
 {
 struct acpi_drhd_unit *drhd;
 struct iommu *iommu;
-int flush_dev_

[Xen-devel] [PATCH v10 1/3] vt-d: fix the IOMMU flush issue

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

The propagation value from IOMMU flush interfaces may be positive, which
indicates callers need to flush cache, not one of faliures.

when the propagation value is positive, this patch fixes this flush issue
as follows:
  - call iommu_flush_write_buffer() to flush cache.
  - return zero.

Signed-off-by: Quan Xu <quan...@intel.com>
Acked-by: Kevin Tian <kevin.t...@intel.com>

CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>
CC: Keir Fraser <k...@xen.org>
CC: Jan Beulich <jbeul...@suse.com>
CC: Andrew Cooper <andrew.coop...@citrix.com>

---
v10: change 'if ( context_rc >= 0 )' to 'if ( rc >= 0 )'
---
 xen/drivers/passthrough/vtd/iommu.c | 150 +---
 1 file changed, 103 insertions(+), 47 deletions(-)

diff --git a/xen/drivers/passthrough/vtd/iommu.c 
b/xen/drivers/passthrough/vtd/iommu.c
index 48edb67..999cdf6 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -388,17 +388,18 @@ static int flush_context_reg(
 return 0;
 }
 
-static int iommu_flush_context_global(
-struct iommu *iommu, int flush_non_present_entry)
+static int __must_check iommu_flush_context_global(struct iommu *iommu,
+   int flush_non_present_entry)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 return flush->context(iommu, 0, 0, 0, DMA_CCMD_GLOBAL_INVL,
  flush_non_present_entry);
 }
 
-static int iommu_flush_context_device(
-struct iommu *iommu, u16 did, u16 source_id,
-u8 function_mask, int flush_non_present_entry)
+static int __must_check iommu_flush_context_device(struct iommu *iommu,
+   u16 did, u16 source_id,
+   u8 function_mask,
+   int flush_non_present_entry)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 return flush->context(iommu, did, source_id, function_mask,
@@ -473,8 +474,9 @@ static int flush_iotlb_reg(void *_iommu, u16 did,
 return 0;
 }
 
-static int iommu_flush_iotlb_global(struct iommu *iommu,
-int flush_non_present_entry, int flush_dev_iotlb)
+static int __must_check iommu_flush_iotlb_global(struct iommu *iommu,
+ int flush_non_present_entry,
+ int flush_dev_iotlb)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 int status;
@@ -491,8 +493,9 @@ static int iommu_flush_iotlb_global(struct iommu *iommu,
 return status;
 }
 
-static int iommu_flush_iotlb_dsi(struct iommu *iommu, u16 did,
-int flush_non_present_entry, int flush_dev_iotlb)
+static int __must_check iommu_flush_iotlb_dsi(struct iommu *iommu, u16 did,
+  int flush_non_present_entry,
+  int flush_dev_iotlb)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 int status;
@@ -509,9 +512,10 @@ static int iommu_flush_iotlb_dsi(struct iommu *iommu, u16 
did,
 return status;
 }
 
-static int iommu_flush_iotlb_psi(
-struct iommu *iommu, u16 did, u64 addr, unsigned int order,
-int flush_non_present_entry, int flush_dev_iotlb)
+static int __must_check iommu_flush_iotlb_psi(struct iommu *iommu, u16 did,
+  u64 addr, unsigned int order,
+  int flush_non_present_entry,
+  int flush_dev_iotlb)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 int status;
@@ -546,17 +550,37 @@ static int __must_check iommu_flush_all(void)
 struct acpi_drhd_unit *drhd;
 struct iommu *iommu;
 int flush_dev_iotlb;
+int rc = 0;
 
 flush_all_cache();
 for_each_drhd_unit ( drhd )
 {
+int context_rc, iotlb_rc;
+
 iommu = drhd->iommu;
-iommu_flush_context_global(iommu, 0);
+context_rc = iommu_flush_context_global(iommu, 0);
 flush_dev_iotlb = find_ats_dev_drhd(iommu) ? 1 : 0;
-iommu_flush_iotlb_global(iommu, 0, flush_dev_iotlb);
+iotlb_rc = iommu_flush_iotlb_global(iommu, 0, flush_dev_iotlb);
+
+/*
+ * The current logic for returns:
+ *   - positive  invoke iommu_flush_write_buffer to flush cache.
+ *   - zero  on success.
+ *   - negative  on failure. Continue to flush IOMMU IOTLB on a
+ *   best effort basis.
+ */
+if ( context_rc > 0 || iotlb_rc > 0 )
+iommu_flush_write_buffer(iommu);
+if ( rc >= 0 )
+rc = context_rc;
+if ( rc >= 0 )
+rc = iotlb_rc;
 }
 
-return 0;
+if ( rc > 0 )
+rc = 0;
+
+return r

[Xen-devel] [PATCH v10 0/3] Check VT-d Device-TLB flush error

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

This patch set is a prereq patch set for Patch:'VT-d Device-TLB flush issue'.

While IOMMU Device-TLB flush timed out, xen calls panic() at present. However 
the existing panic()
is going to be eliminated, so we must propagate the IOMMU Device-TLB flush 
error up to the call trees.

--
Quan Xu (3):
  vt-d: fix the IOMMU flush issue
  vt-d: propagate the IOMMU Device-TLB flush error up to ME phantom
functions
  vt-d: add __must_check annotation to IOMMU flush pointers and handlers

 xen/drivers/passthrough/vtd/extern.h |   3 +-
 xen/drivers/passthrough/vtd/iommu.c  | 184 +++
 xen/drivers/passthrough/vtd/iommu.h  |  11 ++-
 xen/drivers/passthrough/vtd/qinval.c |  14 +--
 xen/drivers/passthrough/vtd/quirks.c |  27 +++--
 5 files changed, 153 insertions(+), 86 deletions(-)

-- 
1.9.1


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v9 1/3] vt-d: fix the IOMMU flush issue

[Xu, Quan]

On June 21, 2016 9:25 PM, Jan Beulich  wrote:
> >>> On 17.06.16 at 05:37,  wrote:
> > @@ -546,17 +550,37 @@ static int __must_check iommu_flush_all(void)
> >  struct acpi_drhd_unit *drhd;
> >  struct iommu *iommu;
> >  int flush_dev_iotlb;
> > +int rc = 0;
> >
> >  flush_all_cache();
> >  for_each_drhd_unit ( drhd )
> >  {
> > +int context_rc, iotlb_rc;
> > +
> >  iommu = drhd->iommu;
> > -iommu_flush_context_global(iommu, 0);
> > +context_rc = iommu_flush_context_global(iommu, 0);
> >  flush_dev_iotlb = find_ats_dev_drhd(iommu) ? 1 : 0;
> > -iommu_flush_iotlb_global(iommu, 0, flush_dev_iotlb);
> > +iotlb_rc = iommu_flush_iotlb_global(iommu, 0,
> > + flush_dev_iotlb);
> > +
> > +/*
> > + * The current logic for returns:
> > + *   - positive  invoke iommu_flush_write_buffer to flush cache.
> > + *   - zero  on success.
> > + *   - negative  on failure. Continue to flush IOMMU IOTLB on a
> > + *   best effort basis.
> > + */
> > +if ( context_rc > 0 || iotlb_rc > 0 )
> > +iommu_flush_write_buffer(iommu);
> > +if ( context_rc >= 0 )
> 
> Wasn't this meant to be just "rc"? (I can't, btw, imagine Kevin's ack to be
> rightfully retained with a change like this.)
>
SORRY, it is 'rc'. It is really my mistake here, but Kevin's ack is right as 
the previous v8 was:

+if ( rc >= 0 )
+rc = iommu_rc;
+if ( rc >= 0 )
+rc = iommu_ret;

,, I will send it out again with this fix.

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [Patch v11 3/3] vt-d: fix vt-d Device-TLB flush timeout issue (+ crash logic )

[Xu, Quan]

On June 18, 2016 5:54 PM, Xu, Quan <quan...@intel.com> wrote:
> On June 17, 2016 9:35 PM, Julien Grall <julien.gr...@arm.com> wrote:
> > On 17/06/16 09:51, Xu, Quan wrote:
> > > + arm/amd maintainers..
> > >
> > > On June 01, 2016 5:05 PM, Xu, Quan <quan...@intel.com> wrote:
> > >> If Device-TLB flush timed out, we hide the target ATS device
> > >> immediately and crash the domain owning this ATS device. If
> > >> impacted domain is hardware domain, just throw out a warning.
> > >>
> > >> By hiding the device, we make sure it can't be assigned to any
> > >> domain any longer (see device_assigned).
> > >
> > > DomU (other than Dom0) gets crashed when a device IOTLB flush times
> out.
> > I suppose that's what you will want on ARM/AMD then too.
> >
> > Correct it is what we want for ARM :).
> >
> 
> 
> That's good :):)..
> Julien, thanks.
> 
> > > We need to move up the crash logic , as similar as iommu_map_page()
> > > /
> > iommu_unmap_page().
> > >
> > >  - add the crash logic to iommu_iotlb_flush() / 
> > > iommu_iotlb_flush_all().
> > >
> > >  - when IOMMU/MMU share page tables, we need to fix it one by one.
> > >  -- on amd, we need to add the crash logic to
> > amd_iommu_flush_pages().
> > >  -- on intel, we need to add the crash logic to iommu_pte_flush().
> > >  -- on arm, we benefit that we add the crash logic to
> > iommu_iotlb_flush().
> >
> > Right.
> >
> 
> :):)
> 
> > >
> > >
> > > Taken together, we need to add crash logic to
> > >   iommu_iotlb_flush() /
> > > iommu_iotlb_flush_all() /
> > iommu_pte_flush() / amd_iommu_flush_pages().
> >




Think twice,  now I am inclined to leave amd_iommu_flush_pages() as is,
As the IOTLB flush error has not been yet bubbled up on AMD platform, it is 
pointless to  add
domain crash logic there.

Quan




> > For iommu_iotlb_* yes as it is common code. I don't know for the others.
> >
> IMO, that's is enough. I hope the other maintainers can give some comments.
> Thanks!!
> 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [Patch v11 3/3] vt-d: fix vt-d Device-TLB flush timeout issue (+ crash logic )

[Xu, Quan]

On June 17, 2016 9:35 PM, Julien Grall <julien.gr...@arm.com> wrote:
> On 17/06/16 09:51, Xu, Quan wrote:
> > + arm/amd maintainers..
> >
> > On June 01, 2016 5:05 PM, Xu, Quan <quan...@intel.com> wrote:
> >> If Device-TLB flush timed out, we hide the target ATS device
> >> immediately and crash the domain owning this ATS device. If impacted
> >> domain is hardware domain, just throw out a warning.
> >>
> >> By hiding the device, we make sure it can't be assigned to any domain
> >> any longer (see device_assigned).
> >
> > DomU (other than Dom0) gets crashed when a device IOTLB flush times out.
> I suppose that's what you will want on ARM/AMD then too.
> 
> Correct it is what we want for ARM :).
> 


That's good :):)..
Julien, thanks.

> > We need to move up the crash logic , as similar as iommu_map_page() /
> iommu_unmap_page().
> >
> >  - add the crash logic to iommu_iotlb_flush() / iommu_iotlb_flush_all().
> >
> >  - when IOMMU/MMU share page tables, we need to fix it one by one.
> >  -- on amd, we need to add the crash logic to
> amd_iommu_flush_pages().
> >  -- on intel, we need to add the crash logic to iommu_pte_flush().
> >  -- on arm, we benefit that we add the crash logic to
> iommu_iotlb_flush().
> 
> Right.
> 

:):)

> >
> >
> > Taken together, we need to add crash logic to
> >   iommu_iotlb_flush() / iommu_iotlb_flush_all() /
> iommu_pte_flush() / amd_iommu_flush_pages().
> 
> For iommu_iotlb_* yes as it is common code. I don't know for the others.
> 
IMO, that's is enough. I hope the other maintainers can give some comments. 
Thanks!!

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [Patch v11 3/3] vt-d: fix vt-d Device-TLB flush timeout issue (+ crash logic )

[Xu, Quan]

+ arm/amd maintainers..

On June 01, 2016 5:05 PM, Xu, Quan <quan...@intel.com> wrote:
> If Device-TLB flush timed out, we hide the target ATS device immediately and
> crash the domain owning this ATS device. If impacted domain is hardware
> domain, just throw out a warning.
> 
> By hiding the device, we make sure it can't be assigned to any domain any
> longer (see device_assigned).

DomU (other than Dom0) gets crashed when a device IOTLB flush times out. I 
suppose that's what you will want on ARM/AMD then too.
We need to move up the crash logic , as similar as iommu_map_page() / 
iommu_unmap_page().

- add the crash logic to iommu_iotlb_flush() / iommu_iotlb_flush_all().

- when IOMMU/MMU share page tables, we need to fix it one by one.
-- on amd, we need to add the crash logic to amd_iommu_flush_pages().
-- on intel, we need to add the crash logic to iommu_pte_flush().
-- on arm, we benefit that we add the crash logic to 
iommu_iotlb_flush().


Taken together, we need to add crash logic to
 iommu_iotlb_flush() / iommu_iotlb_flush_all() / 
iommu_pte_flush() / amd_iommu_flush_pages().

Thoughts?

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [Patch v11 3/3] vt-d: fix vt-d Device-TLB flush timeout issue

[Xu, Quan]

On June 17, 2016 3:01 PM, Jan Beulich  wrote:
> >>> On 17.06.16 at 08:08,  wrote:
> 
> > On June 16, 2016 5:05 PM, Jan Beulich  wrote:
> >> >>> On 16.06.16 at 10:42,  wrote:
> >> > On June 02, 2016 7:07 PM, Jan Beulich  wrote:
> >> >> >>> On 01.06.16 at 11:05,  wrote:
> >> >> > +static void dev_invalidate_iotlb_timeout(struct iommu *iommu, u16
> did,
> >> >> > + struct pci_ats_dev
> >> >> > +*ats_dev) {
> >> >> > +struct domain *d = NULL;
> >> >> > +struct pci_dev *pdev;
> >> >> > +
> >> >> > +if ( test_bit(did, iommu->domid_bitmap) )
> >> >> > +d = rcu_lock_domain_by_id(iommu->domid_map[did]);
> >> >> > +
> >> >> > +/*
> >> >> > + * In case the domain has been freed or the IOMMU domid bitmap
> is
> >> >> > + * not valid, the device no longer belongs to this domain.
> >> >> > + */
> >> >> > +if ( d == NULL )
> >> >> > +return;
> >> >> > +
> >> >> > +pcidevs_lock();
> >> >> > +
> >> >> > +for_each_pdev(d, pdev)
> >> >> > +{
> >> >> > +if ( (pdev->seg == ats_dev->seg) &&
> >> >> > + (pdev->bus == ats_dev->bus) &&
> >> >> > + (pdev->devfn == ats_dev->devfn) )
> >> >> > +{
> >> >> > +ASSERT(pdev->domain);
> >> >> > +list_del(>domain_list);
> >> >> > +pdev->domain = NULL;
> >> >> > +pci_hide_existing_device(pdev);
> >> >> > +break;
> >> >> > +}
> >> >> > +}
> >> >> > +
> >> >> > +pcidevs_unlock();
> >> >>
> >> >> ... this loop (and locking). (Of course such a change may better
> >> >> be done in another preparatory patch.)
> >> >>
> >> >
> >> > To eliminate the locking?  I am afraid the locking is still a must
> >> > here even without the loop, also referring  to device_assigned()..
> >>
> >> If the entire loop gets eliminated, what would be left is
> >>
> >> pcidevs_lock();
> >> pcidevs_unlock();
> >>
> >> which I don't think does any good.
> >
> > Why? I can't follow it..
> 
> I don't understand your question. Can you explain what use above code
> sequence is, in your opinion? Or else - what does the "why"
> refer to?
> 

Ah, there may be a gap between us. without this loop,  these pdev operation 
should be still there, such as,


+ASSERT(pdev->domain);
+list_del(>domain_list);
+pdev->domain = NULL;
+pci_hide_existing_device(pdev);

So, the left is not only:
   pcidevs_lock();
   pcidevs_unlock();


> >> >> > +static int __must_check dev_invalidate_sync(struct iommu
> >> >> > +*iommu,
> >> >> > +u16
> >> >> did,
> >> >> > +struct pci_ats_dev
> >> >> > +*ats_dev) {
> >> >> > +struct qi_ctrl *qi_ctrl = iommu_qi_ctrl(iommu);
> >> >> > +int rc = 0;
> >> >> > +
> >> >> > +if ( qi_ctrl->qinval_maddr )
> >> >> > +{
> >> >> > +rc = queue_invalidate_wait(iommu, 0, 1, 1, 1);
> >> >> > +
> >> >> > +if ( rc == -ETIMEDOUT )
> >> >> > +dev_invalidate_iotlb_timeout(iommu, did, ats_dev);
> >> >> > +}
> >> >> > +
> >> >> > +return rc;
> >> >> > +}
> >> >>
> >> >> I've never really understood why invalidate_sync() returns success
> >> >> when it didn't do anything. Now that you copy this same behavior
> >> >> here, I really need to ask you to explain that.
> >> >>
> >> >
> >> > It is acceptable to me, returning success when it didn't do
> >> > anything
> >> > -- this is worth reflection and criticism:(..
> >> > It is better:
> >> > +if ( qi_ctrl->qinval_maddr )
> >> > +...
> >> > +else
> >> > +rc = -ENOENT;
> >>
> >> Right. And perhaps a separate patch to make invalidate_sync() do the
> same.
> >
> > Agreed.
> >
> >> Question is whether this really ought to be a conditional, or whether
> > instead
> >> this code is unreachable when qinval is not in use, in which case
> >> these conditionals would imo better be converted to ASSERT()s.
> >>
> >
> > IMO, this should be a conditional.
> > As mentioned below, strictly speaking, this is a bug. We can't
> > ASSERT() based on a bug..
> > A coming patch may fix it..
> 
> And again I don't understand: ASSERT()s are to verify assumed state. If static
> code analysis resulted in understanding a function is unreachable when
> qi_ctrl->qinval_maddr is zero (because qinval ought to have got disabled if 
> any
> of the table setup failed), then adding ASSERT() would (a) document that and
> (b) allow to know quickly if something broke that assumption.
> 

You are right.  A separate patch does this.

> But then again I may simply misunderstand your wording: "We can't ASSERT()
> based on a bug" is really pretty unclear to me.
> 

I supposed the variable in ASSERT() is always true, but disable_qinval() needs 
to make
qi_ctrl->qinval_maddr  zero, but today it doesn't do this -- a bug.
With your explanation,  I got it now. Thanks.

Re: [Xen-devel] [Patch v11 3/3] vt-d: fix vt-d Device-TLB flush timeout issue

[Xu, Quan]

On June 16, 2016 5:05 PM, Jan Beulich  wrote:
> >>> On 16.06.16 at 10:42,  wrote:
> > On June 02, 2016 7:07 PM, Jan Beulich  wrote:
> >> >>> On 01.06.16 at 11:05,  wrote:
> >> > +static void dev_invalidate_iotlb_timeout(struct iommu *iommu, u16 did,
> >> > + struct pci_ats_dev
> >> > +*ats_dev) {
> >> > +struct domain *d = NULL;
> >> > +struct pci_dev *pdev;
> >> > +
> >> > +if ( test_bit(did, iommu->domid_bitmap) )
> >> > +d = rcu_lock_domain_by_id(iommu->domid_map[did]);
> >> > +
> >> > +/*
> >> > + * In case the domain has been freed or the IOMMU domid bitmap is
> >> > + * not valid, the device no longer belongs to this domain.
> >> > + */
> >> > +if ( d == NULL )
> >> > +return;
> >> > +
> >> > +pcidevs_lock();
> >> > +
> >> > +for_each_pdev(d, pdev)
> >> > +{
> >> > +if ( (pdev->seg == ats_dev->seg) &&
> >> > + (pdev->bus == ats_dev->bus) &&
> >> > + (pdev->devfn == ats_dev->devfn) )
> >> > +{
> >> > +ASSERT(pdev->domain);
> >> > +list_del(>domain_list);
> >> > +pdev->domain = NULL;
> >> > +pci_hide_existing_device(pdev);
> >> > +break;
> >> > +}
> >> > +}
> >> > +
> >> > +pcidevs_unlock();
> >>
> >> ... this loop (and locking). (Of course such a change may better be
> >> done in another preparatory patch.)
> >>
> >
> > To eliminate the locking?  I am afraid the locking is still a must
> > here even without the loop, also referring  to device_assigned()..
> 
> If the entire loop gets eliminated, what would be left is
> 
> pcidevs_lock();
> pcidevs_unlock();
> 
> which I don't think does any good.
> 

Why? I can't follow it..


> >> > +static int __must_check dev_invalidate_sync(struct iommu *iommu,
> >> > +u16
> >> did,
> >> > +struct pci_ats_dev
> >> > +*ats_dev) {
> >> > +struct qi_ctrl *qi_ctrl = iommu_qi_ctrl(iommu);
> >> > +int rc = 0;
> >> > +
> >> > +if ( qi_ctrl->qinval_maddr )
> >> > +{
> >> > +rc = queue_invalidate_wait(iommu, 0, 1, 1, 1);
> >> > +
> >> > +if ( rc == -ETIMEDOUT )
> >> > +dev_invalidate_iotlb_timeout(iommu, did, ats_dev);
> >> > +}
> >> > +
> >> > +return rc;
> >> > +}
> >>
> >> I've never really understood why invalidate_sync() returns success
> >> when it didn't do anything. Now that you copy this same behavior
> >> here, I really need to ask you to explain that.
> >>
> >
> > It is acceptable to me, returning success when it didn't do anything
> > -- this is worth reflection and criticism:(..
> > It is better:
> > +if ( qi_ctrl->qinval_maddr )
> > +...
> > +else
> > +rc = -ENOENT;
> 
> Right. And perhaps a separate patch to make invalidate_sync() do the same.

Agreed.

> Question is whether this really ought to be a conditional, or whether instead
> this code is unreachable when qinval is not in use, in which case these
> conditionals would imo better be converted to ASSERT()s.
> 

IMO, this should be a conditional.
As mentioned below, strictly speaking, this is a bug. We can't  ASSERT() based 
on a bug..
A coming patch may fix it..


> > A question:
> > I find the page related to qi_ctrl->qinval_maddr is not freed at all.
> > IMO, In disable_qinval (), we need to do:
> >  - free the page related to qi_ctrl->qinval_maddr.
> >  - qi_ctrl->qinval_maddr = 0;
> 
> Well, that's a correct observation, but not a big problem imo: If this was a 
> per-
> domain resource, it surely would need fixing. But if freeing a couple of these
> pages (one per IOMMU) causes synchronization headaches (e.g. because
> there may still be dangling references to it), then I think freeing them is 
> not a
> must. But if freeing them is safe (like you seem to imply), then I'm certainly
> fine with you fixing this (not that my opinion would matter much here, as I'm
> not the maintainer of this code).
> 

Agreed, thanks for your explanation. At least I will leave it as is in this 
patch set.

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v9 1/3] vt-d: fix the IOMMU flush issue

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

The propagation value from IOMMU flush interfaces may be positive, which
indicates callers need to flush cache, not one of faliures.

when the propagation value is positive, this patch fixes this flush issue
as follows:
  - call iommu_flush_write_buffer() to flush cache.
  - return zero.

Signed-off-by: Quan Xu <quan...@intel.com>
Acked-by: Kevin Tian <kevin.t...@intel.com>

CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>
CC: Keir Fraser <k...@xen.org>
CC: Jan Beulich <jbeul...@suse.com>
CC: Andrew Cooper <andrew.coop...@citrix.com>

---
v9: fix naming issue, changing 'iommu_rc / iommu_ret' to 'context_rc / iotlb_rc'
---
 xen/drivers/passthrough/vtd/iommu.c | 150 +---
 1 file changed, 103 insertions(+), 47 deletions(-)

diff --git a/xen/drivers/passthrough/vtd/iommu.c 
b/xen/drivers/passthrough/vtd/iommu.c
index 48edb67..2679ef6 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -388,17 +388,18 @@ static int flush_context_reg(
 return 0;
 }
 
-static int iommu_flush_context_global(
-struct iommu *iommu, int flush_non_present_entry)
+static int __must_check iommu_flush_context_global(struct iommu *iommu,
+   int flush_non_present_entry)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 return flush->context(iommu, 0, 0, 0, DMA_CCMD_GLOBAL_INVL,
  flush_non_present_entry);
 }
 
-static int iommu_flush_context_device(
-struct iommu *iommu, u16 did, u16 source_id,
-u8 function_mask, int flush_non_present_entry)
+static int __must_check iommu_flush_context_device(struct iommu *iommu,
+   u16 did, u16 source_id,
+   u8 function_mask,
+   int flush_non_present_entry)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 return flush->context(iommu, did, source_id, function_mask,
@@ -473,8 +474,9 @@ static int flush_iotlb_reg(void *_iommu, u16 did,
 return 0;
 }
 
-static int iommu_flush_iotlb_global(struct iommu *iommu,
-int flush_non_present_entry, int flush_dev_iotlb)
+static int __must_check iommu_flush_iotlb_global(struct iommu *iommu,
+ int flush_non_present_entry,
+ int flush_dev_iotlb)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 int status;
@@ -491,8 +493,9 @@ static int iommu_flush_iotlb_global(struct iommu *iommu,
 return status;
 }
 
-static int iommu_flush_iotlb_dsi(struct iommu *iommu, u16 did,
-int flush_non_present_entry, int flush_dev_iotlb)
+static int __must_check iommu_flush_iotlb_dsi(struct iommu *iommu, u16 did,
+  int flush_non_present_entry,
+  int flush_dev_iotlb)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 int status;
@@ -509,9 +512,10 @@ static int iommu_flush_iotlb_dsi(struct iommu *iommu, u16 
did,
 return status;
 }
 
-static int iommu_flush_iotlb_psi(
-struct iommu *iommu, u16 did, u64 addr, unsigned int order,
-int flush_non_present_entry, int flush_dev_iotlb)
+static int __must_check iommu_flush_iotlb_psi(struct iommu *iommu, u16 did,
+  u64 addr, unsigned int order,
+  int flush_non_present_entry,
+  int flush_dev_iotlb)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 int status;
@@ -546,17 +550,37 @@ static int __must_check iommu_flush_all(void)
 struct acpi_drhd_unit *drhd;
 struct iommu *iommu;
 int flush_dev_iotlb;
+int rc = 0;
 
 flush_all_cache();
 for_each_drhd_unit ( drhd )
 {
+int context_rc, iotlb_rc;
+
 iommu = drhd->iommu;
-iommu_flush_context_global(iommu, 0);
+context_rc = iommu_flush_context_global(iommu, 0);
 flush_dev_iotlb = find_ats_dev_drhd(iommu) ? 1 : 0;
-iommu_flush_iotlb_global(iommu, 0, flush_dev_iotlb);
+iotlb_rc = iommu_flush_iotlb_global(iommu, 0, flush_dev_iotlb);
+
+/*
+ * The current logic for returns:
+ *   - positive  invoke iommu_flush_write_buffer to flush cache.
+ *   - zero  on success.
+ *   - negative  on failure. Continue to flush IOMMU IOTLB on a
+ *   best effort basis.
+ */
+if ( context_rc > 0 || iotlb_rc > 0 )
+iommu_flush_write_buffer(iommu);
+if ( context_rc >= 0 )
+rc = context_rc;
+if ( rc >= 0 )
+rc = iotlb_rc;
 }
 
-return 0;
+if ( rc > 0 )
+r

[Xen-devel] [PATCH v9 3/3] vt-d: add __must_check annotation to IOMMU flush pointers and handlers

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Signed-off-by: Quan Xu <quan...@intel.com>
Acked-by: Kevin Tian <kevin.t...@intel.com>
Reviewed-by: Jan Beulich <jbeul...@suse.com>

CC: Jan Beulich <jbeul...@suse.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>
---
 xen/drivers/passthrough/vtd/iommu.c  | 50 ++--
 xen/drivers/passthrough/vtd/iommu.h  | 11 +---
 xen/drivers/passthrough/vtd/qinval.c | 14 +-
 3 files changed, 39 insertions(+), 36 deletions(-)

diff --git a/xen/drivers/passthrough/vtd/iommu.c 
b/xen/drivers/passthrough/vtd/iommu.c
index cc47c30..7d413f1 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -335,10 +335,9 @@ static void iommu_flush_write_buffer(struct iommu *iommu)
 }
 
 /* return value determine if we need a write buffer flush */
-static int flush_context_reg(
-void *_iommu,
-u16 did, u16 source_id, u8 function_mask, u64 type,
-int flush_non_present_entry)
+static int __must_check flush_context_reg(void *_iommu, u16 did, u16 source_id,
+  u8 function_mask, u64 type,
+  bool_t flush_non_present_entry)
 {
 struct iommu *iommu = (struct iommu *) _iommu;
 u64 val = 0;
@@ -389,7 +388,7 @@ static int flush_context_reg(
 }
 
 static int __must_check iommu_flush_context_global(struct iommu *iommu,
-   int flush_non_present_entry)
+   bool_t 
flush_non_present_entry)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 return flush->context(iommu, 0, 0, 0, DMA_CCMD_GLOBAL_INVL,
@@ -399,7 +398,7 @@ static int __must_check iommu_flush_context_global(struct 
iommu *iommu,
 static int __must_check iommu_flush_context_device(struct iommu *iommu,
u16 did, u16 source_id,
u8 function_mask,
-   int flush_non_present_entry)
+   bool_t 
flush_non_present_entry)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 return flush->context(iommu, did, source_id, function_mask,
@@ -408,9 +407,10 @@ static int __must_check iommu_flush_context_device(struct 
iommu *iommu,
 }
 
 /* return value determine if we need a write buffer flush */
-static int flush_iotlb_reg(void *_iommu, u16 did,
-   u64 addr, unsigned int size_order, u64 type,
-   int flush_non_present_entry, int flush_dev_iotlb)
+static int __must_check flush_iotlb_reg(void *_iommu, u16 did, u64 addr,
+unsigned int size_order, u64 type,
+bool_t flush_non_present_entry,
+bool_t flush_dev_iotlb)
 {
 struct iommu *iommu = (struct iommu *) _iommu;
 int tlb_offset = ecap_iotlb_offset(iommu->ecap);
@@ -475,8 +475,8 @@ static int flush_iotlb_reg(void *_iommu, u16 did,
 }
 
 static int __must_check iommu_flush_iotlb_global(struct iommu *iommu,
- int flush_non_present_entry,
- int flush_dev_iotlb)
+ bool_t 
flush_non_present_entry,
+ bool_t flush_dev_iotlb)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 int status;
@@ -494,8 +494,8 @@ static int __must_check iommu_flush_iotlb_global(struct 
iommu *iommu,
 }
 
 static int __must_check iommu_flush_iotlb_dsi(struct iommu *iommu, u16 did,
-  int flush_non_present_entry,
-  int flush_dev_iotlb)
+  bool_t flush_non_present_entry,
+  bool_t flush_dev_iotlb)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 int status;
@@ -514,8 +514,8 @@ static int __must_check iommu_flush_iotlb_dsi(struct iommu 
*iommu, u16 did,
 
 static int __must_check iommu_flush_iotlb_psi(struct iommu *iommu, u16 did,
   u64 addr, unsigned int order,
-  int flush_non_present_entry,
-  int flush_dev_iotlb)
+  bool_t flush_non_present_entry,
+  bool_t flush_dev_iotlb)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 int status;
@@ -549,7 +549,7 @@ static int __must_check iommu_flush_all(void)
 {
 struct acpi_drhd_unit *drhd;
 struct iommu *iommu;
-int flush_dev_

[Xen-devel] [PATCH v9 0/3] Check VT-d Device-TLB flush error

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

This patch set is a prereq patch set for Patch:'VT-d Device-TLB flush issue'.

While IOMMU Device-TLB flush timed out, xen calls panic() at present. However 
the existing panic()
is going to be eliminated, so we must propagate the IOMMU Device-TLB flush 
error up to the call trees.

This patch set is also based on the discussion of 'abstract model of IOMMU 
unmaping/mapping failures'.

---
Quan Xu (3):
  vt-d: fix the IOMMU flush issue
  vt-d: propagate the IOMMU Device-TLB flush error up to ME phantom
functions
  vt-d: add __must_check annotation to IOMMU flush pointers and handlers

 xen/drivers/passthrough/vtd/extern.h |   3 +-
 xen/drivers/passthrough/vtd/iommu.c  | 184 +++
 xen/drivers/passthrough/vtd/iommu.h  |  11 ++-
 xen/drivers/passthrough/vtd/qinval.c |  14 +--
 xen/drivers/passthrough/vtd/quirks.c |  27 +++--
 5 files changed, 153 insertions(+), 86 deletions(-)

-- 
1.9.1


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH] AMD IOMMU: correctly propagate errors from amd_iommu_init()

[Xu, Quan]

On June 16, 2016 4:29 PM, Jan Beulich  wrote:
> >>> On 16.06.16 at 04:03,  wrote:
> > On June 14, 2016 5:03 PM, Jan Beulich  wrote:
> >> -if ( amd_iommu_update_ivrs_mapping_acpi() != 0 )
> >> +rc = amd_iommu_update_ivrs_mapping_acpi();
> >> +if ( rc )
> >>  goto error_out;
> >>
> >>  /* initialize io-apic interrupt remapping entries */
> >> -if ( iommu_intremap && amd_iommu_setup_ioapic_remapping() != 0 )
> >> +if ( iommu_intremap )
> >> +rc = amd_iommu_setup_ioapic_remapping();
> >> +if ( rc )
> >>  goto error_out;
> >
> >
> > Is it better to indent this if() here? Then,
> >
> > +if ( iommu_intremap )
> > +{
> > +rc = amd_iommu_setup_ioapic_remapping();
> > +if ( rc )
> > +goto error_out;
> > +}
> 
> What would this help (apart from increasing LOC and patch size)?
> 

Ah, first of all, it is not a logic issue, but just make the code clear:
  - this if( rc ) is called only if ( iommu_intremap ) is true.
  - this error is from amd_iommu_setup_ioapic_remapping().

Also from '-if ( iommu_intremap && amd_iommu_setup_ioapic_remapping() != 0 
)', I'd like to fix it as my suggestion.

Quan
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [Patch v11 3/3] vt-d: fix vt-d Device-TLB flush timeout issue

[Xu, Quan]

(* I will CC arm/amd maintainer after this vt-d specific discussion, and then 
send out my proposal...)

On June 02, 2016 7:07 PM, Jan Beulich  wrote:
> >>> On 01.06.16 at 11:05,  wrote:
> > --- a/xen/drivers/passthrough/vtd/extern.h
> > +++ b/xen/drivers/passthrough/vtd/extern.h
> > @@ -21,6 +21,7 @@
> >  #define _VTD_EXTERN_H_
> >
> >  #include "dmar.h"
> > +#include "../ats.h"
> 
> Why? You don't de-reference struct pci_ats_dev * in this file, so all you'd 
> need
> is a forward declaration. But then this is not in line with your v11 change
> description above, so I wonder whether you actually sent a stale patch.

Sorry, this patch is really strange to me.

> After all I thought the v10 discussion (see
> http://lists.xenproject.org/archives/html/xen-devel/2016-
> 05/msg02208.html
> ) had made clear that this passing down,

Sure, what you said is very clear. But I read these code again, I found a  
pci_get_pdev_by_domain()
Can also get *pdev without below loop.. (also hardware domain calls 
pci_get_pdev_by_domain() to get pdev.)

To be honest,  now I don't like to add a struct pci_dev * inside struct 
pci_ats_dev, as I need to change
' struct pci_ats_dev *pdev' to ' struct pci_ats_dev * pci_ats_dev ' in some 
functions as well.


> besides reducing the number of
> arguments of some function, would also be meant to eliminate ...
> 
> > +static void dev_invalidate_iotlb_timeout(struct iommu *iommu, u16 did,
> > + struct pci_ats_dev *ats_dev)
> > +{
> > +struct domain *d = NULL;
> > +struct pci_dev *pdev;
> > +
> > +if ( test_bit(did, iommu->domid_bitmap) )
> > +d = rcu_lock_domain_by_id(iommu->domid_map[did]);
> > +
> > +/*
> > + * In case the domain has been freed or the IOMMU domid bitmap is
> > + * not valid, the device no longer belongs to this domain.
> > + */
> > +if ( d == NULL )
> > +return;
> > +
> > +pcidevs_lock();
> > +
> > +for_each_pdev(d, pdev)
> > +{
> > +if ( (pdev->seg == ats_dev->seg) &&
> > + (pdev->bus == ats_dev->bus) &&
> > + (pdev->devfn == ats_dev->devfn) )
> > +{
> > +ASSERT(pdev->domain);
> > +list_del(>domain_list);
> > +pdev->domain = NULL;
> > +pci_hide_existing_device(pdev);
> > +break;
> > +}
> > +}
> > +
> > +pcidevs_unlock();
> 
> ... this loop (and locking). (Of course such a change may better be done in
> another preparatory patch.)
> 

To eliminate the locking?  I am afraid the locking is still a must here even 
without the loop, also referring  to device_assigned()..


> 
> > +static int __must_check dev_invalidate_sync(struct iommu *iommu, u16
> did,
> > +struct pci_ats_dev
> > +*ats_dev) {
> > +struct qi_ctrl *qi_ctrl = iommu_qi_ctrl(iommu);
> > +int rc = 0;
> > +
> > +if ( qi_ctrl->qinval_maddr )
> > +{
> > +rc = queue_invalidate_wait(iommu, 0, 1, 1, 1);
> > +
> > +if ( rc == -ETIMEDOUT )
> > +dev_invalidate_iotlb_timeout(iommu, did, ats_dev);
> > +}
> > +
> > +return rc;
> > +}
> 
> I've never really understood why invalidate_sync() returns success when it
> didn't do anything. Now that you copy this same behavior here, I really need
> to ask you to explain that.
> 

It is acceptable to me, returning success when it didn't do anything -- this is 
worth reflection and criticism:(..
It is better:
+if ( qi_ctrl->qinval_maddr )
+...
+else
+rc = -ENOENT;

A question:
I find the page related to qi_ctrl->qinval_maddr is not freed at all. IMO,
In disable_qinval (), we need to do:
 - free the page related to qi_ctrl->qinval_maddr.
 - qi_ctrl->qinval_maddr = 0;

Right?

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH] AMD IOMMU: correctly propagate errors from amd_iommu_init()

[Xu, Quan]

On June 14, 2016 5:03 PM, Jan Beulich  wrote:
> -if ( amd_iommu_update_ivrs_mapping_acpi() != 0 )
> +rc = amd_iommu_update_ivrs_mapping_acpi();
> +if ( rc )
>  goto error_out;
> 
>  /* initialize io-apic interrupt remapping entries */
> -if ( iommu_intremap && amd_iommu_setup_ioapic_remapping() != 0 )
> +if ( iommu_intremap )
> +rc = amd_iommu_setup_ioapic_remapping();
> +if ( rc )
>  goto error_out;


Is it better to indent this if() here? Then,

+if ( iommu_intremap )
+{
+rc = amd_iommu_setup_ioapic_remapping();
+if ( rc )
+goto error_out;
+}

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v8 02/11] IOMMU/MMU: enhance the call trees of IOMMU unmapping and mapping

[Xu, Quan]

On June 15, 2016 4:42 PM, Jan Beulich <jbeul...@suse.com> wrote:
> >>> On 15.06.16 at 10:35, <quan...@intel.com> wrote:
> > On June 15, 2016 4:22 PM, Tian, Kevin <kevin.t...@intel.com> wrote:
> >> > From: Xu, Quan
> >> > Sent: Wednesday, June 15, 2016 4:16 PM
> >> >
> >> > On June 15, 2016 3:45 PM, Jan Beulich <jbeul...@suse.com> wrote:
> >> > > >>> On 15.06.16 at 03:54, <quan...@intel.com> wrote:
> >> > > > Jan,
> >> > > > Could you help me review patch 7?   Thanks.
> >> > > > Then, I can send out next v9 soon and get started to focus on
> >> > > > next patch set.
> >> > >
> >> > > That patch is fine now from my pov, feel free to stick my R-b on it.
> >> > > I actually have it queued for committing already, pending an ARM
> >> > > ack for patch 4 (which is why yesterday I committed only the
> >> > > first three patches of that series); I didn't check yet which
> >> > > other acks may still be missing on later patches, everything up
> >> > > to patch 8 is ready to go in afaic, pending all necessary acks are in 
> >> > > place.
> >> > >
> >> >
> >> > Jan,
> >> > thanks very much!!
> >> > For patch 4, I will ping arm maintainer in that email.
> >> >
> >> >
> >> >
> >> > Kevin,
> >> > For patch 9, could you help me review it?  With your reply of
> >> > http://lists.xenproject.org/archives/html/xen-devel/2016-
> >> 06/msg01842.html  ,
> >> >there is no issue doing so based on spec. I think what I do is
> >> > changing ' iommu_rc / iommu_ret '  to ' context_rc / iotlb_rc '.
> >> >in this case, I wonder whether this can be fixed upon commit,
> >> > then I am no need to send out v9.
> >> >
> >>
> >> Yes, no more comments except the naming:
> >>
> >> Acked-by: Kevin Tian <kevin.t...@intel.com>
> >
> > Jan,
> > If there are no other problem, could you help me fix these naming
> > issue upon commit?
> 
> I wouldn't want to do that. As said in an earlier reply, I have things up to 
> patch
> 8 queued for commit (which by implication means I'd expect at least 9...11 to
> see another revision).
> 

Got it. I will send out v9 when patch 4 is acked.

> > Also for the comments,  you can change them as you like.
> 
> I'm not clear which comments you refer to here.
> 

For all of comments, especially  the repetitive ones of patch 9.

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v8 02/11] IOMMU/MMU: enhance the call trees of IOMMU unmapping and mapping

[Xu, Quan]

On June 15, 2016 4:22 PM, Tian, Kevin <kevin.t...@intel.com> wrote:
> > From: Xu, Quan
> > Sent: Wednesday, June 15, 2016 4:16 PM
> >
> > On June 15, 2016 3:45 PM, Jan Beulich <jbeul...@suse.com> wrote:
> > > >>> On 15.06.16 at 03:54, <quan...@intel.com> wrote:
> > > > Jan,
> > > > Could you help me review patch 7?   Thanks.
> > > > Then, I can send out next v9 soon and get started to focus on next
> > > > patch set.
> > >
> > > That patch is fine now from my pov, feel free to stick my R-b on it.
> > > I actually have it queued for committing already, pending an ARM ack
> > > for patch 4 (which is why yesterday I committed only the first three
> > > patches of that series); I didn't check yet which other acks may
> > > still be missing on later patches, everything up to patch 8 is ready
> > > to go in afaic, pending all necessary acks are in place.
> > >
> >
> > Jan,
> > thanks very much!!
> > For patch 4, I will ping arm maintainer in that email.
> >
> >
> >
> > Kevin,
> > For patch 9, could you help me review it?  With your reply of
> > http://lists.xenproject.org/archives/html/xen-devel/2016-
> 06/msg01842.html  ,
> >there is no issue doing so based on spec. I think what I do is
> > changing ' iommu_rc / iommu_ret '  to ' context_rc / iotlb_rc '.
> >in this case, I wonder whether this can be fixed upon commit, then
> > I am no need to send out v9.
> >
> 
> Yes, no more comments except the naming:
> 
> Acked-by: Kevin Tian <kevin.t...@intel.com>
> 

Jan,
If there are no other problem, could you help me fix these naming issue upon 
commit?  
Also for the comments,  you can change them as you like.

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v8 04/11] IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU mapping (top level ones)

[Xu, Quan]

On June 13, 2016 11:17 PM, Xu, Quan <quan...@intel.com> wrote:
> From: Quan Xu <quan...@intel.com>
> 
> Signed-off-by: Quan Xu <quan...@intel.com>
> Acked-by: Kevin Tian <kevin.t...@intel.com>
> Acked-by: Suravee Suthikulpanit <suravee.suthikulpa...@amd.com>
> Reviewed-by: Jan Beulich <jbeul...@suse.com>
> 
> CC: Suravee Suthikulpanit <suravee.suthikulpa...@amd.com>
> CC: Stefano Stabellini <sstabell...@kernel.org>
> CC: Julien Grall <julien.gr...@arm.com>
> CC: Kevin Tian <kevin.t...@intel.com>
> CC: Feng Wu <feng...@intel.com>
> CC: Jan Beulich <jbeul...@suse.com>
> CC: Andrew Cooper <andrew.coop...@citrix.com>
> 
> v8: use the Linux coding style for arm code.
> ---
>  xen/drivers/passthrough/arm/smmu.c| 4 ++--
>  xen/drivers/passthrough/vtd/iommu.c   | 7 ---
>  xen/include/asm-x86/hvm/svm/amd-iommu-proto.h | 4 ++--
>  xen/include/xen/iommu.h   | 4 ++--
>  4 files changed, 10 insertions(+), 9 deletions(-)
> 
> diff --git a/xen/drivers/passthrough/arm/smmu.c
> b/xen/drivers/passthrough/arm/smmu.c
> index 1ce4ddf..58fde32 100644
> --- a/xen/drivers/passthrough/arm/smmu.c
> +++ b/xen/drivers/passthrough/arm/smmu.c
> @@ -2745,8 +2745,8 @@ static void
> arm_smmu_iommu_domain_teardown(struct domain *d)
>   xfree(xen_domain);
>  }
> 
> -static int arm_smmu_map_page(struct domain *d, unsigned long gfn,
> -  unsigned long mfn, unsigned int flags)
> +static int __must_check arm_smmu_map_page(struct domain *d, unsigned
> long gfn,
> + unsigned long mfn, unsigned int flags)
>  {
>   p2m_type_t t;
> 

Julien,

Could you help me review this patch? This patch still needs your ack if I have 
fixed coding style.
Thanks in advance.

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v8 02/11] IOMMU/MMU: enhance the call trees of IOMMU unmapping and mapping

[Xu, Quan]

On June 15, 2016 3:45 PM, Jan Beulich  wrote:
> >>> On 15.06.16 at 03:54,  wrote:
> > Jan,
> > Could you help me review patch 7?   Thanks.
> > Then, I can send out next v9 soon and get started to focus on next
> > patch set.
> 
> That patch is fine now from my pov, feel free to stick my R-b on it.
> I actually have it queued for committing already, pending an ARM ack for
> patch 4 (which is why yesterday I committed only the first three patches of
> that series); I didn't check yet which other acks may still be missing on 
> later
> patches, everything up to patch 8 is ready to go in afaic, pending all 
> necessary
> acks are in place.
> 

Jan,
thanks very much!!
For patch 4, I will ping arm maintainer in that email.



Kevin,
For patch 9, could you help me review it?  With your reply of 
http://lists.xenproject.org/archives/html/xen-devel/2016-06/msg01842.html  ,
   there is no issue doing so based on spec. I think what I do is changing ' 
iommu_rc / iommu_ret '  to ' context_rc / iotlb_rc '.
   in this case, I wonder whether this can be fixed upon commit, then I am no 
need to send out v9.



Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [Patch v11 1/3] IOMMU: add a timeout parameter for device IOTLB invalidation

[Xu, Quan]

On June 02, 2016 6:25 PM, Jan Beulich <jbeul...@suse.com> wrote:
> >>> On 01.06.16 at 11:05, <quan...@intel.com> wrote:
> > From: Quan Xu <quan...@intel.com>
> > v11: Change the timeout parameter from 'vtd_qi_timeout' to
> > 'iommu_dev_iotlb_timeout', which is not only for VT-d device
> > IOTLB invalidation, but also for other IOMMU implementations.
> 
> This goes after the first --- separator.
> 

Got it. It should be:

---
v11:
   - ...
   - ...
---


> > --- a/docs/misc/xen-command-line.markdown
> > +++ b/docs/misc/xen-command-line.markdown
> > @@ -996,6 +996,15 @@ debug hypervisor only).
> >
> >  >> Enable IOMMU debugging code (implies `verbose`).
> >
> > +### iommu\_dev\_iotlb\_timeout
> > +> `= `
> > +
> > +> Default: `1`
> 
> So on v10 I had made clear that any timeout reduction from its current value
> is, for the ATS case, not acceptable, unless you have proof that this lower
> value will fit all past, present, and future devices. Otherwise we're risking 
> a
> regression here.
> 

I really misunderstood the 'current value', which should be about 
'DMAR_OPERATION_TIMEOUT MILLISECS(1000) ', instead of ' IOMMU_QI_TIMEOUT 
MILLISECS(1)' in my patch.
So the default is 1000.
 
> > --- a/xen/drivers/passthrough/vtd/qinval.c
> > +++ b/xen/drivers/passthrough/vtd/qinval.c
> > @@ -28,6 +28,8 @@
> >  #include "vtd.h"
> >  #include "extern.h"
> >
> > +#define IOMMU_QI_TIMEOUT MILLISECS(1)
> 
> May I suggest VTD_QI_TIMEOUT (but see also below)?
> 

Agreed. VTD_QI_TIMEOUT is a better one.

> > @@ -163,14 +165,21 @@ static int queue_invalidate_wait(struct iommu
> *iommu,
> >  /* Now we don't support interrupt method */
> >  if ( sw )
> >  {
> > +s_time_t timeout;
> > +
> >  /* In case all wait descriptor writes to same addr with same data 
> > */
> > -start_time = NOW();
> > +timeout = flush_dev_iotlb ?
> > +  (NOW() + iommu_dev_iotlb_timeout * MILLISECS(1)) :
> 
> MILLISECS(iommu_dev_iotlb_timeout)
> 
> > +  (NOW() + IOMMU_QI_TIMEOUT);
> 
> Or really the whole expression should probably simply become
> 
> timeout = NOW() + MILLISECS(flush_dev_iotlb ?
> iommu_dev_iotlb_timeout : VTD_QI_TIMEOUT);
> 
> (of course with VTD_QI_TIMEOUT having its MILLISECS() dropped, and
> suitably line wrapped).
> 


I prefer this later one.

Quan
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v8 02/11] IOMMU/MMU: enhance the call trees of IOMMU unmapping and mapping

[Xu, Quan]

On June 14, 2016 12:37 AM,  George Dunlap <george.dun...@eu.citrix.com> wrote:
> On Mon, Jun 13, 2016 at 4:17 PM, Xu, Quan <quan...@intel.com> wrote:
> > From: Quan Xu <quan...@intel.com>
> >
> > When IOMMU mapping is failed, we issue a best effort rollback,
> > stopping IOMMU mapping, unmapping the previous IOMMU maps and
> then
> > reporting the error up to the call trees. When rollback is not
> > feasible (in early initialization phase or trade-off of complexity)
> > for the hardware domain, we do things on a best effort basis, only throwing
> out an error message.
> >
> > IOMMU unmapping should perhaps continue despite an error, in an
> > attempt to do best effort cleanup.
> >
> > Signed-off-by: Quan Xu <quan...@intel.com>
> > Reviewed-by: Jan Beulich <jbeul...@suse.com>
> > Reviewed-by: Suravee Suthikulpanit <suravee.suthikulpa...@amd.com>
> > Acked-by: Kevin Tian <kevin.t...@intel.com>
> 
> Acked-by: George Dunlap <george.dun...@citrix.com>
> 
> Phew!
> 
> One comment...
> 
> > +while ( i-- )
> > +/*
> > + * IOMMU unmapping should perhaps continue 
> > despite an
> > + * error in an attempt to do best effort 
> > cleanup, and
> > + * consume the error as __must_check 
> > annotation.
> > + */
> > +if ( iommu_unmap_page(p2m->domain, gfn + i) )
> > +continue;
> 
> I'd take out the "perhaps", (since there's no 'perhaps' about it) but other 
> than
> that I think this comment is fine.
> 
> It sounds like Jan had something more along the following in mind:
> 
> /* If statement to satisfy __must_check */
> 
> Either one works.  The shorter one is sufficient, but the longer one isn't too
> much I don't think.
> 
George,
Thanks for your comment.. I think your shorter one is better.

Jan, 
Could you help me review patch 7?   Thanks. 
Then, I can send out next v9 soon and get started to focus on next patch set.

Quan
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v8 09/11] vt-d: fix the IOMMU flush issue

[Xu, Quan]

On June 14, 2016 4:27 PM, Jan Beulich <jbeul...@suse.com> wrote: 
> >>> On 14.06.16 at 10:10, <quan...@intel.com> wrote:
> > On June 13, 2016 11:52 PM, Jan Beulich <jbeul...@suse.com> wrote:
> >> >>> "Xu, Quan" <quan...@intel.com> 06/13/16 5:22 PM >>>
> >> >From: Quan Xu <quan...@intel.com>
> >> >@@ -546,17 +550,37 @@ static int __must_check iommu_flush_all(void)
> >> >struct acpi_drhd_unit *drhd; struct iommu *iommu; int
> >> >flush_dev_iotlb;
> >> >+int rc = 0;
> >>  >
> >> >flush_all_cache();
> >> >for_each_drhd_unit ( drhd )
> >> >{
> >> >+int iommu_rc, iommu_ret;
> >> >+
> >> >iommu = drhd->iommu;
> >> >-iommu_flush_context_global(iommu, 0);
> >> >+iommu_rc = iommu_flush_context_global(iommu, 0);
> >> >flush_dev_iotlb = find_ats_dev_drhd(iommu) ? 1 : 0;
> >> >-iommu_flush_iotlb_global(iommu, 0, flush_dev_iotlb);
> >> >+iommu_ret = iommu_flush_iotlb_global(iommu, 0,
> >> >+ flush_dev_iotlb);
> >> >+
> >> >+/*
> >> >+ * The current logic for returns:
> >> >+ *   - positive  invoke iommu_flush_write_buffer to flush cache.
> >> >+ *   - zero  on success.
> >> >+ *   - negative  on failure. Continue to flush IOMMU IOTLB on a
> >> >+ *   best effort basis.
> >> >+ */
> >> >+if ( iommu_rc > 0 || iommu_ret > 0 )
> >> >+iommu_flush_write_buffer(iommu);
> >> >+if ( rc >= 0 )
> >> >+rc = iommu_rc;
> >> >+if ( rc >= 0 )
> >> >+rc = iommu_ret;
> >>
> >> First of all - is it correct to fold the two
> >> iommu_flush_write_buffer() invocations?
> >>
> >
> > Sure, it is correct..
> >
> > as:
> > - For updates to remapping hardware structures that require
> > context-cache, PASID-cache, IOTLB or IEC invalidation Operations to
> > flush stale entries from the hardware caches, no additional action is
> > required to make the modification Visible to hardware. This is
> > because, hardware performs an implicit write-buffer-flushing as a
> > pre-condition to context-cache, PASID-cache, IOTLB and IEC
> > invalidation operations.
> >
> > - For updates to remapping hardware structures (such as modifying a
> > currently not-present entry) that do not require Context-cache, IOTLB,
> > or IEC invalidations, software must explicitly perform
> > write-buffer-flushing to ensure the updated structures Are visible to
> > hardware.
> 
> But that's not the point. Instead my question was related to Kevin's concern
> towards you making assumptions on the behavior of
> iommu_flush_context_global() vs iommu_flush_iotlb_global(): What if the
> first returned 1 but the second didn't?
> It would seem to me that in such a
> (theoretical) case iommu_flush_write_buffer() might need to be invoked prior
> to the second flush function.

In this case, it is correct too.
As , hardware performs an __implicit__ write-buffer-flushing as a 
__pre-condition__ to IOTLB invalidation operation.
So software is no need to call iommu_flush_write_buffer() to explicitly perform 
write-buffer-flushing for that the first returned 1.

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] vTPM detaching issue

[Xu, Quan]

On June 13, 2016 11:11 PM, Andrea Genuise  wrote:
> I'm not sure if this is a bug or my fault, but when I create a domain
> with a vTPM attached, detaching it sometimes causes the following error
> to be thrown (I post the command sequence):
>

I am afraid this is not a bug. As 'xl vtpm-detach' is to - destroy a domain's 
virtual TPM device.
Based on your record, ...

> [root@localhost ~]# xl create /etc/xen/vtpmmgr-stubdom
> Parsing config from /etc/xen/vtpmmgr-stubdom
> [root@localhost ~]# xl create /etc/xen/vtpm1
> Parsing config from /etc/xen/vtpm1
> [root@localhost ~]# xl create /etc/xen/dom1_ima
> Parsing config from /etc/xen/dom1_ima
> [root@localhost ~]# xl vtpm-detach dom1 vtpm1

... here,  you have detached vtpm on success.

> [root@localhost ~]# xl destroy dom1
> [root@localhost ~]# xl vtpm-detach vtpm1 vtpmmgr

IMO, vtpm-detach doesn't support between vtpm stubdom and vtpmmgr stubdom.

Thanks
Quan Xu

> libxl: error: libxl_device.c:952:device_backend_callback: unable to
> remove device with path /local/domain/18/backend/vtpm/19/0
> libxl: error: libxl.c:1995:device_addrm_aocomplete: unable to remove
> vtpm with id 0
> libxl_device_vtpm_remove failed.
>
> Sometimes the error is raised while detaching vtpmmgr from vtpm (as
> reported), other times while detaching vtpm from domain. I think
> this could be a synchronization problem.
>
> I report some info:
>
> [root@localhost ~]# xl info
> host   : localhost.localdomain
> release: 3.18.25-19.without_tpm.el7.centos.x86_64
> version: #1 SMP Sun Apr 10 18:10:14 CEST 2016
> machine: x86_64
> nr_cpus: 2
> max_cpu_id : 3
> nr_nodes   : 1
> cores_per_socket   : 2
> threads_per_core   : 1
> cpu_mhz: 2394
> hw_caps: 
> bfebfbff:20100800::0900:0408e3fd::0001:
> virt_caps  : hvm hvm_directio
> total_memory   : 3996
> free_memory: 2888
> sharing_freed_memory   : 0
> sharing_used_memory: 0
> outstanding_claims : 0
> free_cpus  : 0
> xen_major  : 4
> xen_minor  : 6
> xen_extra  : .1-5.el7
> xen_version: 4.6.1-5.el7
> xen_caps   : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 
> hvm-3.0-x86_32p hvm-3.0-x86_64
> xen_scheduler  : credit
> xen_pagesize   : 4096
> platform_params: virt_start=0x8000
> xen_changeset  : Tue Mar 29 11:02:43 2016 +0100 git:8210a62-dirty
> xen_commandline: placeholder dom0_mem=1024M,max:1024M cpuinfo 
> com1=115200,8n1 console=com1,tty loglvl=all guest_loglvl=all
> cc_compiler: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-4)
> cc_compile_by  : mockbuild
> cc_compile_domain  : centos.org
> cc_compile_date: Tue Mar 29 12:15:50 UTC 2016
> xend_config_format : 4
>
> [root@localhost ~]# cat /etc/xen/vtpmmgr-stubdom
> name = "vtpmmgr"
> kernel="/usr/lib/xen/boot/vtpmmgr-stubdom.gz"
> memory=16
> disk=["file:/srv/xen/vtpmmgr-stubdom.img,hda,w"]
> iomem=["fed40,5"]
>
> [root@localhost ~]# cat /etc/xen/vtpm1
> name="vtpm1"
> kernel="/usr/lib/xen/boot/vtpm-stubdom.gz"
> memory=8
> disk=["file:/srv/xen/vtpm1.img,hda,w"]
> vtpm=["backend=vtpmmgr,uuid=8aca22b3-768a-41e7-b2cb-123d23901996"]
>
> [root@localhost ~]# cat /etc/xen/dom1_ima
> kernel = "/srv/xen/vmlinuz-xen"
> ramdisk = "/srv/xen/initrd-xen"
> name = "dom1"
> memory = "512"
> disk = [ 'tap:aio:/srv/xen/dom1.img,xvda1,w' ]
> vcpus=1
> root = '/dev/xvda1 ro'
> extra = 'ima_tcb'
> vtpm=['backend=vtpm1']
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v8 09/11] vt-d: fix the IOMMU flush issue

[Xu, Quan]

On June 13, 2016 11:52 PM, Jan Beulich <jbeul...@suse.com> wrote:
> >>> "Xu, Quan" <quan...@intel.com> 06/13/16 5:22 PM >>>
> >From: Quan Xu <quan...@intel.com>
> >@@ -546,17 +550,37 @@ static int __must_check iommu_flush_all(void)
> >struct acpi_drhd_unit *drhd; struct iommu *iommu; int flush_dev_iotlb;
> >+int rc = 0;
>  >
> >flush_all_cache();
> >for_each_drhd_unit ( drhd )
> >{
> >+int iommu_rc, iommu_ret;
> >+
> >iommu = drhd->iommu;
> >-iommu_flush_context_global(iommu, 0);
> >+iommu_rc = iommu_flush_context_global(iommu, 0);
> >flush_dev_iotlb = find_ats_dev_drhd(iommu) ? 1 : 0;
> >-iommu_flush_iotlb_global(iommu, 0, flush_dev_iotlb);
> >+iommu_ret = iommu_flush_iotlb_global(iommu, 0,
> >+ flush_dev_iotlb);
> >+
> >+/*
> >+ * The current logic for returns:
> >+ *   - positive  invoke iommu_flush_write_buffer to flush cache.
> >+ *   - zero  on success.
> >+ *   - negative  on failure. Continue to flush IOMMU IOTLB on a
> >+ *   best effort basis.
> >+ */
> >+if ( iommu_rc > 0 || iommu_ret > 0 )
> >+iommu_flush_write_buffer(iommu);
> >+if ( rc >= 0 )
> >+rc = iommu_rc;
> >+if ( rc >= 0 )
> >+rc = iommu_ret;
> 
> First of all - is it correct to fold the two iommu_flush_write_buffer()
> invocations?
> 

Sure, it is correct.. 

as:
- For updates to remapping hardware structures that require context-cache, 
PASID-cache, IOTLB or IEC invalidation
Operations to flush stale entries from the hardware caches, no additional 
action is required to make the modification
Visible to hardware. This is because, hardware performs an implicit 
write-buffer-flushing as a pre-condition to context-cache,
PASID-cache, IOTLB and IEC invalidation operations.

- For updates to remapping hardware structures (such as modifying a currently 
not-present entry) that do not require
Context-cache, IOTLB, or IEC invalidations, software must explicitly perform 
write-buffer-flushing to ensure the updated structures
Are visible to hardware.



> And then the variable naming is strange - both operations are IOMMU ones,
> so prefixing the variables with iommu_ doesn't help much here. How about
> ctxt_rc and iotlb_rc or some such?
> 

To align to that file, I'm better using context_rc  and iotlb_rc..

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v8 08/11] IOMMU: propagate IOMMU Device-TLB flush error (leaf ones).

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Signed-off-by: Quan Xu <quan...@intel.com>
Acked-by: Kevin Tian <kevin.t...@intel.com>
Reviewed-by: Jan Beulich <jbeul...@suse.com>

CC: Stefano Stabellini <sstabell...@kernel.org>
CC: Julien Grall <julien.gr...@arm.com>
CC: Jan Beulich <jbeul...@suse.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>
---
 xen/drivers/passthrough/arm/smmu.c  | 13 -
 xen/drivers/passthrough/iommu.c |  8 ++--
 xen/drivers/passthrough/vtd/iommu.c | 32 
 xen/include/xen/iommu.h |  5 +++--
 4 files changed, 33 insertions(+), 25 deletions(-)

diff --git a/xen/drivers/passthrough/arm/smmu.c 
b/xen/drivers/passthrough/arm/smmu.c
index 58fde32..8a4b123 100644
--- a/xen/drivers/passthrough/arm/smmu.c
+++ b/xen/drivers/passthrough/arm/smmu.c
@@ -2540,7 +2540,7 @@ static int force_stage = 2;
  */
 static u32 platform_features = ARM_SMMU_FEAT_COHERENT_WALK;
 
-static void arm_smmu_iotlb_flush_all(struct domain *d)
+static int __must_check arm_smmu_iotlb_flush_all(struct domain *d)
 {
struct arm_smmu_xen_domain *smmu_domain = dom_iommu(d)->arch.priv;
struct iommu_domain *cfg;
@@ -2557,13 +2557,16 @@ static void arm_smmu_iotlb_flush_all(struct domain *d)
arm_smmu_tlb_inv_context(cfg->priv);
}
spin_unlock(_domain->lock);
+
+   return 0;
 }
 
-static void arm_smmu_iotlb_flush(struct domain *d, unsigned long gfn,
- unsigned int page_count)
+static int __must_check arm_smmu_iotlb_flush(struct domain *d,
+ unsigned long gfn,
+ unsigned int page_count)
 {
-/* ARM SMMU v1 doesn't have flush by VMA and VMID */
-arm_smmu_iotlb_flush_all(d);
+   /* ARM SMMU v1 doesn't have flush by VMA and VMID */
+   return arm_smmu_iotlb_flush_all(d);
 }
 
 static struct iommu_domain *arm_smmu_get_domain(struct domain *d,
diff --git a/xen/drivers/passthrough/iommu.c b/xen/drivers/passthrough/iommu.c
index a9898fc..ef80b3c 100644
--- a/xen/drivers/passthrough/iommu.c
+++ b/xen/drivers/passthrough/iommu.c
@@ -319,9 +319,7 @@ int iommu_iotlb_flush(struct domain *d, unsigned long gfn,
 if ( !iommu_enabled || !hd->platform_ops || !hd->platform_ops->iotlb_flush 
)
 return 0;
 
-hd->platform_ops->iotlb_flush(d, gfn, page_count);
-
-return 0;
+return hd->platform_ops->iotlb_flush(d, gfn, page_count);
 }
 
 int iommu_iotlb_flush_all(struct domain *d)
@@ -331,9 +329,7 @@ int iommu_iotlb_flush_all(struct domain *d)
 if ( !iommu_enabled || !hd->platform_ops || 
!hd->platform_ops->iotlb_flush_all )
 return 0;
 
-hd->platform_ops->iotlb_flush_all(d);
-
-return 0;
+return hd->platform_ops->iotlb_flush_all(d);
 }
 
 int __init iommu_setup(void)
diff --git a/xen/drivers/passthrough/vtd/iommu.c 
b/xen/drivers/passthrough/vtd/iommu.c
index 0f17afb..48edb67 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -559,8 +559,10 @@ static int __must_check iommu_flush_all(void)
 return 0;
 }
 
-static void __intel_iommu_iotlb_flush(struct domain *d, unsigned long gfn,
-int dma_old_pte_present, unsigned int page_count)
+static int __must_check iommu_flush_iotlb(struct domain *d,
+  unsigned long gfn,
+  bool_t dma_old_pte_present,
+  unsigned int page_count)
 {
 struct domain_iommu *hd = dom_iommu(d);
 struct acpi_drhd_unit *drhd;
@@ -598,16 +600,20 @@ static void __intel_iommu_iotlb_flush(struct domain *d, 
unsigned long gfn,
 iommu_flush_write_buffer(iommu);
 }
 }
+
+return 0;
 }
 
-static void intel_iommu_iotlb_flush(struct domain *d, unsigned long gfn, 
unsigned int page_count)
+static int __must_check iommu_flush_iotlb_pages(struct domain *d,
+unsigned long gfn,
+unsigned int page_count)
 {
-__intel_iommu_iotlb_flush(d, gfn, 1, page_count);
+return iommu_flush_iotlb(d, gfn, 1, page_count);
 }
 
-static void intel_iommu_iotlb_flush_all(struct domain *d)
+static int __must_check iommu_flush_iotlb_all(struct domain *d)
 {
-__intel_iommu_iotlb_flush(d, INVALID_GFN, 0, 0);
+return iommu_flush_iotlb(d, INVALID_GFN, 0, 0);
 }
 
 /* clear one page's page table */
@@ -616,6 +622,7 @@ static int __must_check dma_pte_clear_one(struct domain 
*domain, u64 addr)
 struct domain_iommu *hd = dom_iommu(domain);
 struct dma_pte *page = NULL, *pte = NULL;
 u64 pg_maddr;
+int rc = 0;
 
 spin_lock(>arch.mapping_lock);
 /* get last level pte */
@@ -641,11 +648,11 @@ static int __must_check

[Xen-devel] [PATCH v8 06/11] propagate IOMMU Device-TLB flush error up to EPT update (top level ones)

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Propagate the IOMMU Device-TLB flush error up to the ept_set_entry(),
when VT-d shares EPT page table.

Signed-off-by: Quan Xu <quan...@intel.com>
Acked-by: Kevin Tian <kevin.t...@intel.com>
Reviewed-by: Jan Beulich <jbeul...@suse.com>

CC: Jun Nakajima <jun.nakaj...@intel.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: George Dunlap <george.dun...@eu.citrix.com>
CC: Jan Beulich <jbeul...@suse.com>
CC: Andrew Cooper <andrew.coop...@citrix.com>
CC: Feng Wu <feng...@intel.com>
---
 xen/arch/x86/mm/p2m-ept.c   | 2 +-
 xen/drivers/passthrough/vtd/iommu.c | 6 --
 xen/include/asm-x86/iommu.h | 3 ++-
 3 files changed, 7 insertions(+), 4 deletions(-)

diff --git a/xen/arch/x86/mm/p2m-ept.c b/xen/arch/x86/mm/p2m-ept.c
index a233194..3098c33 100644
--- a/xen/arch/x86/mm/p2m-ept.c
+++ b/xen/arch/x86/mm/p2m-ept.c
@@ -832,7 +832,7 @@ out:
  need_modify_vtd_table )
 {
 if ( iommu_hap_pt_share )
-iommu_pte_flush(d, gfn, _entry->epte, order, vtd_pte_present);
+rc = iommu_pte_flush(d, gfn, _entry->epte, order, 
vtd_pte_present);
 else
 {
 if ( iommu_flags )
diff --git a/xen/drivers/passthrough/vtd/iommu.c 
b/xen/drivers/passthrough/vtd/iommu.c
index e900019..5366267 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -1752,8 +1752,8 @@ static int __must_check intel_iommu_unmap_page(struct 
domain *d,
 return dma_pte_clear_one(d, (paddr_t)gfn << PAGE_SHIFT_4K);
 }
 
-void iommu_pte_flush(struct domain *d, u64 gfn, u64 *pte,
- int order, int present)
+int iommu_pte_flush(struct domain *d, u64 gfn, u64 *pte,
+int order, int present)
 {
 struct acpi_drhd_unit *drhd;
 struct iommu *iommu = NULL;
@@ -1778,6 +1778,8 @@ void iommu_pte_flush(struct domain *d, u64 gfn, u64 *pte,
order, !present, flush_dev_iotlb) )
 iommu_flush_write_buffer(iommu);
 }
+
+return 0;
 }
 
 static int __init vtd_ept_page_compatible(struct iommu *iommu)
diff --git a/xen/include/asm-x86/iommu.h b/xen/include/asm-x86/iommu.h
index e82a2f0..815d77e 100644
--- a/xen/include/asm-x86/iommu.h
+++ b/xen/include/asm-x86/iommu.h
@@ -27,7 +27,8 @@ int iommu_setup_hpet_msi(struct msi_desc *);
 
 /* While VT-d specific, this must get declared in a generic header. */
 int adjust_vtd_irq_affinities(void);
-void iommu_pte_flush(struct domain *d, u64 gfn, u64 *pte, int order, int 
present);
+int __must_check iommu_pte_flush(struct domain *d, u64 gfn, u64 *pte,
+ int order, int present);
 bool_t iommu_supports_eim(void);
 int iommu_enable_x2apic_IR(void);
 void iommu_disable_x2apic_IR(void);
-- 
1.9.1


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v8 07/11] IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU suspending (top level ones)

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Signed-off-by: Quan Xu <quan...@intel.com>
Acked-by: Kevin Tian <kevin.t...@intel.com>
Acked-by: Suravee Suthikulpanit <suravee.suthikulpa...@amd.com>

CC: Jan Beulich <jbeul...@suse.com>
CC: Liu Jinsong <jinsong@alibaba-inc.com>
CC: Keir Fraser <k...@xen.org>
CC: Andrew Cooper <andrew.coop...@citrix.com>
CC: Suravee Suthikulpanit <suravee.suthikulpa...@amd.com>
CC: Stefano Stabellini <sstabell...@kernel.org>
CC: Julien Grall <julien.gr...@arm.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>

v8: enhance the error check of device_power_down():
if call device_power_down() failed,
return a negative error code,
else
return 0.
---
 xen/arch/x86/acpi/power.c | 78 ---
 xen/drivers/passthrough/amd/iommu_init.c  |  9 +++-
 xen/drivers/passthrough/amd/pci_amd_iommu.c   |  2 +-
 xen/drivers/passthrough/iommu.c   |  6 ++-
 xen/drivers/passthrough/vtd/iommu.c   | 35 
 xen/include/asm-x86/hvm/svm/amd-iommu-proto.h |  2 +-
 xen/include/xen/iommu.h   |  4 +-
 7 files changed, 100 insertions(+), 36 deletions(-)

diff --git a/xen/arch/x86/acpi/power.c b/xen/arch/x86/acpi/power.c
index 2885e31..2cb3d13 100644
--- a/xen/arch/x86/acpi/power.c
+++ b/xen/arch/x86/acpi/power.c
@@ -43,36 +43,70 @@ struct acpi_sleep_info acpi_sinfo;
 
 void do_suspend_lowlevel(void);
 
+enum dev_power_saved
+{
+SAVED_NONE,
+SAVED_CONSOLE,
+SAVED_TIME,
+SAVED_I8259A,
+SAVED_IOAPIC,
+SAVED_IOMMU,
+SAVED_LAPIC,
+SAVED_ALL,
+};
+
 static int device_power_down(void)
 {
-console_suspend();
+if ( console_suspend() )
+return SAVED_NONE;
 
-time_suspend();
+if ( time_suspend() )
+return SAVED_CONSOLE;
 
-i8259A_suspend();
+if ( i8259A_suspend() )
+return SAVED_TIME;
 
+/* ioapic_suspend cannot fail */
 ioapic_suspend();
 
-iommu_suspend();
+if ( iommu_suspend() )
+return SAVED_IOAPIC;
 
-lapic_suspend();
+if ( lapic_suspend() )
+return SAVED_IOMMU;
 
-return 0;
+return SAVED_ALL;
 }
 
-static void device_power_up(void)
+static void device_power_up(enum dev_power_saved saved)
 {
-lapic_resume();
-
-iommu_resume();
-
-ioapic_resume();
-
-i8259A_resume();
-
-time_resume();
-
-console_resume();
+switch ( saved )
+{
+case SAVED_ALL:
+case SAVED_LAPIC:
+lapic_resume();
+/* fall through */
+case SAVED_IOMMU:
+iommu_resume();
+/* fall through */
+case SAVED_IOAPIC:
+ioapic_resume();
+/* fall through */
+case SAVED_I8259A:
+i8259A_resume();
+/* fall through */
+case SAVED_TIME:
+time_resume();
+/* fall through */
+case SAVED_CONSOLE:
+console_resume();
+/* fall through */
+case SAVED_NONE:
+break;
+default:
+BUG();
+break;
+}
 }
 
 static void freeze_domains(void)
@@ -165,12 +199,16 @@ static int enter_state(u32 state)
 local_irq_save(flags);
 spin_debug_disable();
 
-if ( (error = device_power_down()) )
+if ( (error = device_power_down()) != SAVED_ALL )
 {
 printk(XENLOG_ERR "Some devices failed to power down.");
 system_state = SYS_STATE_resume;
+device_power_up(error);
+error = -EIO;
 goto done;
 }
+else
+error = 0;
 
 ACPI_FLUSH_CPU_CACHE();
 
@@ -196,7 +234,7 @@ static int enter_state(u32 state)
 write_cr4(cr4 & ~X86_CR4_MCE);
 write_efer(read_efer());
 
-device_power_up();
+device_power_up(SAVED_ALL);
 
 mcheck_init(_cpu_data, 0);
 write_cr4(cr4);
diff --git a/xen/drivers/passthrough/amd/iommu_init.c 
b/xen/drivers/passthrough/amd/iommu_init.c
index 4536106..0b68596 100644
--- a/xen/drivers/passthrough/amd/iommu_init.c
+++ b/xen/drivers/passthrough/amd/iommu_init.c
@@ -1339,7 +1339,14 @@ static void invalidate_all_devices(void)
 iterate_ivrs_mappings(_invalidate_all_devices);
 }
 
-void amd_iommu_suspend(void)
+int amd_iommu_suspend(void)
+{
+amd_iommu_crash_shutdown();
+
+return 0;
+}
+
+void amd_iommu_crash_shutdown(void)
 {
 struct amd_iommu *iommu;
 
diff --git a/xen/drivers/passthrough/amd/pci_amd_iommu.c 
b/xen/drivers/passthrough/amd/pci_amd_iommu.c
index 4a860af..7761241 100644
--- a/xen/drivers/passthrough/amd/pci_amd_iommu.c
+++ b/xen/drivers/passthrough/amd/pci_amd_iommu.c
@@ -633,6 +633,6 @@ const struct iommu_ops amd_iommu_ops = {
 .suspend = amd_iommu_suspend,
 .resume = amd_iommu_resume,
 .share_p2m = amd_iommu_share_p2m,
-.crash_shutdown = amd_iommu_suspend,
+.crash_shutdown = amd_iommu_crash_shutdown,
 .dump_p2m_table = amd_dump_p2m_table,
 };
diff --git a/xen/drivers/passthrough/iommu.c b/xen/drivers/passt

[Xen-devel] [PATCH v8 10/11] vt-d: propagate the IOMMU Device-TLB flush error up to ME phantom functions

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Signed-off-by: Quan Xu <quan...@intel.com>
Acked-by: Kevin Tian <kevin.t...@intel.com>
Reviewed-by: Jan Beulich <jbeul...@suse.com>

CC: Jan Beulich <jbeul...@suse.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>
---
 xen/drivers/passthrough/vtd/extern.h |  3 ++-
 xen/drivers/passthrough/vtd/iommu.c  |  8 
 xen/drivers/passthrough/vtd/quirks.c | 27 +--
 3 files changed, 23 insertions(+), 15 deletions(-)

diff --git a/xen/drivers/passthrough/vtd/extern.h 
b/xen/drivers/passthrough/vtd/extern.h
index cbe0286..6772839 100644
--- a/xen/drivers/passthrough/vtd/extern.h
+++ b/xen/drivers/passthrough/vtd/extern.h
@@ -91,7 +91,8 @@ int is_igd_vt_enabled_quirk(void);
 void platform_quirks_init(void);
 void vtd_ops_preamble_quirk(struct iommu* iommu);
 void vtd_ops_postamble_quirk(struct iommu* iommu);
-void me_wifi_quirk(struct domain *domain, u8 bus, u8 devfn, int map);
+int __must_check me_wifi_quirk(struct domain *domain,
+   u8 bus, u8 devfn, int map);
 void pci_vtd_quirk(const struct pci_dev *);
 bool_t platform_supports_intremap(void);
 bool_t platform_supports_x2apic(void);
diff --git a/xen/drivers/passthrough/vtd/iommu.c 
b/xen/drivers/passthrough/vtd/iommu.c
index 2f046cb..9b882ea 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -1455,8 +1455,8 @@ int domain_context_mapping_one(
 
 unmap_vtd_domain_page(context_entries);
 
-if ( !seg )
-me_wifi_quirk(domain, bus, devfn, MAP_ME_PHANTOM_FUNC);
+if ( !seg && !rc )
+rc = me_wifi_quirk(domain, bus, devfn, MAP_ME_PHANTOM_FUNC);
 
 return rc;
 }
@@ -1605,8 +1605,8 @@ int domain_context_unmap_one(
 spin_unlock(>lock);
 unmap_vtd_domain_page(context_entries);
 
-if ( !iommu->intel->drhd->segment )
-me_wifi_quirk(domain, bus, devfn, UNMAP_ME_PHANTOM_FUNC);
+if ( !iommu->intel->drhd->segment && !rc )
+rc = me_wifi_quirk(domain, bus, devfn, UNMAP_ME_PHANTOM_FUNC);
 
 return rc;
 }
diff --git a/xen/drivers/passthrough/vtd/quirks.c 
b/xen/drivers/passthrough/vtd/quirks.c
index 473d1fc..91f96ac 100644
--- a/xen/drivers/passthrough/vtd/quirks.c
+++ b/xen/drivers/passthrough/vtd/quirks.c
@@ -331,10 +331,12 @@ void __init platform_quirks_init(void)
  * assigning Intel integrated wifi device to a guest.
  */
 
-static void map_me_phantom_function(struct domain *domain, u32 dev, int map)
+static int __must_check map_me_phantom_function(struct domain *domain,
+u32 dev, int map)
 {
 struct acpi_drhd_unit *drhd;
 struct pci_dev *pdev;
+int rc;
 
 /* find ME VT-d engine base on a real ME device */
 pdev = pci_get_pdev(0, 0, PCI_DEVFN(dev, 0));
@@ -342,23 +344,26 @@ static void map_me_phantom_function(struct domain 
*domain, u32 dev, int map)
 
 /* map or unmap ME phantom function */
 if ( map )
-domain_context_mapping_one(domain, drhd->iommu, 0,
-   PCI_DEVFN(dev, 7), NULL);
+rc = domain_context_mapping_one(domain, drhd->iommu, 0,
+PCI_DEVFN(dev, 7), NULL);
 else
-domain_context_unmap_one(domain, drhd->iommu, 0,
- PCI_DEVFN(dev, 7));
+rc = domain_context_unmap_one(domain, drhd->iommu, 0,
+  PCI_DEVFN(dev, 7));
+
+return rc;
 }
 
-void me_wifi_quirk(struct domain *domain, u8 bus, u8 devfn, int map)
+int me_wifi_quirk(struct domain *domain, u8 bus, u8 devfn, int map)
 {
 u32 id;
+int rc = 0;
 
 id = pci_conf_read32(0, 0, 0, 0, 0);
 if ( IS_CTG(id) )
 {
 /* quit if ME does not exist */
 if ( pci_conf_read32(0, 0, 3, 0, 0) == 0x )
-return;
+return 0;
 
 /* if device is WLAN device, map ME phantom device 0:3.7 */
 id = pci_conf_read32(0, bus, PCI_SLOT(devfn), PCI_FUNC(devfn), 0);
@@ -372,7 +377,7 @@ void me_wifi_quirk(struct domain *domain, u8 bus, u8 devfn, 
int map)
 case 0x423b8086:
 case 0x423c8086:
 case 0x423d8086:
-map_me_phantom_function(domain, 3, map);
+rc = map_me_phantom_function(domain, 3, map);
 break;
 default:
 break;
@@ -382,7 +387,7 @@ void me_wifi_quirk(struct domain *domain, u8 bus, u8 devfn, 
int map)
 {
 /* quit if ME does not exist */
 if ( pci_conf_read32(0, 0, 22, 0, 0) == 0x )
-return;
+return 0;
 
 /* if device is WLAN device, map ME phantom device 0:22.7 */
 id = pci_conf_read32(0, bus, PCI_SLOT(devfn), PCI_FUNC(devfn), 0);
@@ -398,12 +403,14 @@ void me_wifi_quirk(struct domain *domain, u8 bus, u8 
devfn, int map)
 

[Xen-devel] [PATCH v8 02/11] IOMMU/MMU: enhance the call trees of IOMMU unmapping and mapping

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

When IOMMU mapping is failed, we issue a best effort rollback, stopping
IOMMU mapping, unmapping the previous IOMMU maps and then reporting the
error up to the call trees. When rollback is not feasible (in early
initialization phase or trade-off of complexity) for the hardware domain,
we do things on a best effort basis, only throwing out an error message.

IOMMU unmapping should perhaps continue despite an error, in an attempt
to do best effort cleanup.

Signed-off-by: Quan Xu <quan...@intel.com>
Reviewed-by: Jan Beulich <jbeul...@suse.com>
Reviewed-by: Suravee Suthikulpanit <suravee.suthikulpa...@amd.com>
Acked-by: Kevin Tian <kevin.t...@intel.com>

CC: Jan Beulich <jbeul...@suse.com>
CC: Andrew Cooper <andrew.coop...@citrix.com>
CC: Jun Nakajima <jun.nakaj...@intel.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: George Dunlap <george.dun...@eu.citrix.com>
CC: Suravee Suthikulpanit <suravee.suthikulpa...@amd.com>
CC: Feng Wu <feng...@intel.com>

v8:
  1. add missing blank
  2. add a brief comment (Jan, if you have a better one, could you help me
 enhance it upon commit?)
---
 xen/arch/x86/mm.c   | 13 ++
 xen/arch/x86/mm/p2m-ept.c   | 39 +++--
 xen/arch/x86/mm/p2m-pt.c| 28 ++---
 xen/arch/x86/mm/p2m.c   | 23 ++---
 xen/arch/x86/x86_64/mm.c|  9 ++-
 xen/drivers/passthrough/amd/pci_amd_iommu.c | 15 +--
 xen/drivers/passthrough/iommu.c | 13 +-
 xen/drivers/passthrough/vtd/x86/vtd.c   | 15 +--
 xen/include/xen/iommu.h |  6 ++---
 9 files changed, 134 insertions(+), 27 deletions(-)

diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index 8d10a3e..ae7c8ab 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -2467,7 +2467,7 @@ static int __get_page_type(struct page_info *page, 
unsigned long type,
int preemptible)
 {
 unsigned long nx, x, y = page->u.inuse.type_info;
-int rc = 0;
+int rc = 0, iommu_ret = 0;
 
 ASSERT(!(type & ~(PGT_type_mask | PGT_pae_xen_l2)));
 
@@ -2578,11 +2578,11 @@ static int __get_page_type(struct page_info *page, 
unsigned long type,
 if ( d && is_pv_domain(d) && unlikely(need_iommu(d)) )
 {
 if ( (x & PGT_type_mask) == PGT_writable_page )
-iommu_unmap_page(d, mfn_to_gmfn(d, page_to_mfn(page)));
+iommu_ret = iommu_unmap_page(d, mfn_to_gmfn(d, 
page_to_mfn(page)));
 else if ( type == PGT_writable_page )
-iommu_map_page(d, mfn_to_gmfn(d, page_to_mfn(page)),
-   page_to_mfn(page),
-   IOMMUF_readable|IOMMUF_writable);
+iommu_ret = iommu_map_page(d, mfn_to_gmfn(d, 
page_to_mfn(page)),
+   page_to_mfn(page),
+   IOMMUF_readable|IOMMUF_writable);
 }
 }
 
@@ -2599,6 +2599,9 @@ static int __get_page_type(struct page_info *page, 
unsigned long type,
 if ( (x & PGT_partial) && !(nx & PGT_partial) )
 put_page(page);
 
+if ( !rc )
+rc = iommu_ret;
+
 return rc;
 }
 
diff --git a/xen/arch/x86/mm/p2m-ept.c b/xen/arch/x86/mm/p2m-ept.c
index 1ed5b47..a233194 100644
--- a/xen/arch/x86/mm/p2m-ept.c
+++ b/xen/arch/x86/mm/p2m-ept.c
@@ -667,6 +667,7 @@ ept_set_entry(struct p2m_domain *p2m, unsigned long gfn, 
mfn_t mfn,
 unsigned long gfn_remainder = gfn;
 unsigned int i, target = order / EPT_TABLE_ORDER;
 int ret, rc = 0;
+bool_t entry_written = 0;
 bool_t direct_mmio = (p2mt == p2m_mmio_direct);
 uint8_t ipat = 0;
 bool_t need_modify_vtd_table = 1;
@@ -812,10 +813,15 @@ ept_set_entry(struct p2m_domain *p2m, unsigned long gfn, 
mfn_t mfn,
 rc = atomic_write_ept_entry(ept_entry, new_entry, target);
 if ( unlikely(rc) )
 old_entry.epte = 0;
-else if ( p2mt != p2m_invalid &&
-  (gfn + (1UL << order) - 1 > p2m->max_mapped_pfn) )
-/* Track the highest gfn for which we have ever had a valid mapping */
-p2m->max_mapped_pfn = gfn + (1UL << order) - 1;
+else
+{
+entry_written = 1;
+
+if ( p2mt != p2m_invalid &&
+ (gfn + (1UL << order) - 1 > p2m->max_mapped_pfn) )
+/* Track the highest gfn for which we have ever had a valid 
mapping */
+p2m->max_mapped_pfn = gfn + (1UL << order) - 1;
+}
 
 out:
 if ( needs_sync )
@@ -831,10 +837,29 @@ out:
 {
 if ( iommu_flags )
 for ( i = 0; i < (1 << order); i++ )
-iommu_map_page(

[Xen-devel] [PATCH v8 11/11] vt-d: add __must_check annotation to IOMMU flush pointers and handlers

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Signed-off-by: Quan Xu <quan...@intel.com>
Acked-by: Kevin Tian <kevin.t...@intel.com>
Reviewed-by: Jan Beulich <jbeul...@suse.com>

CC: Jan Beulich <jbeul...@suse.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>
---
 xen/drivers/passthrough/vtd/iommu.c  | 50 ++--
 xen/drivers/passthrough/vtd/iommu.h  | 11 +---
 xen/drivers/passthrough/vtd/qinval.c | 14 +-
 3 files changed, 39 insertions(+), 36 deletions(-)

diff --git a/xen/drivers/passthrough/vtd/iommu.c 
b/xen/drivers/passthrough/vtd/iommu.c
index 9b882ea..5f48e89 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -335,10 +335,9 @@ static void iommu_flush_write_buffer(struct iommu *iommu)
 }
 
 /* return value determine if we need a write buffer flush */
-static int flush_context_reg(
-void *_iommu,
-u16 did, u16 source_id, u8 function_mask, u64 type,
-int flush_non_present_entry)
+static int __must_check flush_context_reg(void *_iommu, u16 did, u16 source_id,
+  u8 function_mask, u64 type,
+  bool_t flush_non_present_entry)
 {
 struct iommu *iommu = (struct iommu *) _iommu;
 u64 val = 0;
@@ -389,7 +388,7 @@ static int flush_context_reg(
 }
 
 static int __must_check iommu_flush_context_global(struct iommu *iommu,
-   int flush_non_present_entry)
+   bool_t 
flush_non_present_entry)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 return flush->context(iommu, 0, 0, 0, DMA_CCMD_GLOBAL_INVL,
@@ -399,7 +398,7 @@ static int __must_check iommu_flush_context_global(struct 
iommu *iommu,
 static int __must_check iommu_flush_context_device(struct iommu *iommu,
u16 did, u16 source_id,
u8 function_mask,
-   int flush_non_present_entry)
+   bool_t 
flush_non_present_entry)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 return flush->context(iommu, did, source_id, function_mask,
@@ -408,9 +407,10 @@ static int __must_check iommu_flush_context_device(struct 
iommu *iommu,
 }
 
 /* return value determine if we need a write buffer flush */
-static int flush_iotlb_reg(void *_iommu, u16 did,
-   u64 addr, unsigned int size_order, u64 type,
-   int flush_non_present_entry, int flush_dev_iotlb)
+static int __must_check flush_iotlb_reg(void *_iommu, u16 did, u64 addr,
+unsigned int size_order, u64 type,
+bool_t flush_non_present_entry,
+bool_t flush_dev_iotlb)
 {
 struct iommu *iommu = (struct iommu *) _iommu;
 int tlb_offset = ecap_iotlb_offset(iommu->ecap);
@@ -475,8 +475,8 @@ static int flush_iotlb_reg(void *_iommu, u16 did,
 }
 
 static int __must_check iommu_flush_iotlb_global(struct iommu *iommu,
- int flush_non_present_entry,
- int flush_dev_iotlb)
+ bool_t 
flush_non_present_entry,
+ bool_t flush_dev_iotlb)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 int status;
@@ -494,8 +494,8 @@ static int __must_check iommu_flush_iotlb_global(struct 
iommu *iommu,
 }
 
 static int __must_check iommu_flush_iotlb_dsi(struct iommu *iommu, u16 did,
-  int flush_non_present_entry,
-  int flush_dev_iotlb)
+  bool_t flush_non_present_entry,
+  bool_t flush_dev_iotlb)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 int status;
@@ -514,8 +514,8 @@ static int __must_check iommu_flush_iotlb_dsi(struct iommu 
*iommu, u16 did,
 
 static int __must_check iommu_flush_iotlb_psi(struct iommu *iommu, u16 did,
   u64 addr, unsigned int order,
-  int flush_non_present_entry,
-  int flush_dev_iotlb)
+  bool_t flush_non_present_entry,
+  bool_t flush_dev_iotlb)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 int status;
@@ -549,7 +549,7 @@ static int __must_check iommu_flush_all(void)
 {
 struct acpi_drhd_unit *drhd;
 struct iommu *iommu;
-int flush_dev_

[Xen-devel] [PATCH v8 04/11] IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU mapping (top level ones)

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Signed-off-by: Quan Xu <quan...@intel.com>
Acked-by: Kevin Tian <kevin.t...@intel.com>
Acked-by: Suravee Suthikulpanit <suravee.suthikulpa...@amd.com>
Reviewed-by: Jan Beulich <jbeul...@suse.com>

CC: Suravee Suthikulpanit <suravee.suthikulpa...@amd.com>
CC: Stefano Stabellini <sstabell...@kernel.org>
CC: Julien Grall <julien.gr...@arm.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>
CC: Jan Beulich <jbeul...@suse.com>
CC: Andrew Cooper <andrew.coop...@citrix.com>

v8: use the Linux coding style for arm code.
---
 xen/drivers/passthrough/arm/smmu.c| 4 ++--
 xen/drivers/passthrough/vtd/iommu.c   | 7 ---
 xen/include/asm-x86/hvm/svm/amd-iommu-proto.h | 4 ++--
 xen/include/xen/iommu.h   | 4 ++--
 4 files changed, 10 insertions(+), 9 deletions(-)

diff --git a/xen/drivers/passthrough/arm/smmu.c 
b/xen/drivers/passthrough/arm/smmu.c
index 1ce4ddf..58fde32 100644
--- a/xen/drivers/passthrough/arm/smmu.c
+++ b/xen/drivers/passthrough/arm/smmu.c
@@ -2745,8 +2745,8 @@ static void arm_smmu_iommu_domain_teardown(struct domain 
*d)
xfree(xen_domain);
 }
 
-static int arm_smmu_map_page(struct domain *d, unsigned long gfn,
-unsigned long mfn, unsigned int flags)
+static int __must_check arm_smmu_map_page(struct domain *d, unsigned long gfn,
+   unsigned long mfn, unsigned int flags)
 {
p2m_type_t t;
 
diff --git a/xen/drivers/passthrough/vtd/iommu.c 
b/xen/drivers/passthrough/vtd/iommu.c
index 4844193..e900019 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -1687,9 +1687,10 @@ static void iommu_domain_teardown(struct domain *d)
 spin_unlock(>arch.mapping_lock);
 }
 
-static int intel_iommu_map_page(
-struct domain *d, unsigned long gfn, unsigned long mfn,
-unsigned int flags)
+static int __must_check intel_iommu_map_page(struct domain *d,
+ unsigned long gfn,
+ unsigned long mfn,
+ unsigned int flags)
 {
 struct domain_iommu *hd = dom_iommu(d);
 struct dma_pte *page = NULL, *pte = NULL, old, new = { 0 };
diff --git a/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h 
b/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h
index 57b6cc1..ac9f036 100644
--- a/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h
+++ b/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h
@@ -51,8 +51,8 @@ int amd_iommu_init(void);
 int amd_iommu_update_ivrs_mapping_acpi(void);
 
 /* mapping functions */
-int amd_iommu_map_page(struct domain *d, unsigned long gfn, unsigned long mfn,
-   unsigned int flags);
+int __must_check amd_iommu_map_page(struct domain *d, unsigned long gfn,
+unsigned long mfn, unsigned int flags);
 int __must_check amd_iommu_unmap_page(struct domain *d, unsigned long gfn);
 u64 amd_iommu_get_next_table_from_pte(u32 *entry);
 int amd_iommu_reserve_domain_unity_map(struct domain *domain,
diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h
index f45fa5a..2b86710 100644
--- a/xen/include/xen/iommu.h
+++ b/xen/include/xen/iommu.h
@@ -166,8 +166,8 @@ struct iommu_ops {
 #endif /* HAS_PCI */
 
 void (*teardown)(struct domain *d);
-int (*map_page)(struct domain *d, unsigned long gfn, unsigned long mfn,
-unsigned int flags);
+int __must_check (*map_page)(struct domain *d, unsigned long gfn,
+ unsigned long mfn, unsigned int flags);
 int __must_check (*unmap_page)(struct domain *d, unsigned long gfn);
 void (*free_page_table)(struct page_info *);
 #ifdef CONFIG_X86
-- 
1.9.1


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v8 00/11] Check VT-d Device-TLB flush error

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

This patch set is a prereq patch set for Patch:'VT-d Device-TLB flush issue'.

While IOMMU Device-TLB flush timed out, xen calls panic() at present. However 
the existing panic()
is going to be eliminated, so we must propagate the IOMMU Device-TLB flush 
error up to the call trees.

This patch set is also based on the discussion of 'abstract model of IOMMU 
unmaping/mapping failures'


Quan Xu (11):
  IOMMU: handle IOMMU mapping and unmapping failures
  IOMMU/MMU: enhance the call trees of IOMMU unmapping and mapping
  IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU unmapping
(top level ones)
  IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU mapping (top
level ones)
  IOMMU/MMU: propagate IOMMU Device-TLB flush error up to
iommu_iotlb_flush{,_all} (top level ones)
  propagate IOMMU Device-TLB flush error up to EPT update (top level
ones)
  IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU suspending
(top level ones)
  IOMMU: propagate IOMMU Device-TLB flush error (leaf ones).
  vt-d: fix the IOMMU flush issue
  vt-d: propagate the IOMMU Device-TLB flush error up to ME phantom
functions
  vt-d: add __must_check annotation to IOMMU flush pointers and handlers

 xen/arch/arm/p2m.c|   4 +-
 xen/arch/x86/acpi/power.c |  78 ++--
 xen/arch/x86/mm.c |  13 +-
 xen/arch/x86/mm/p2m-ept.c |  41 +++-
 xen/arch/x86/mm/p2m-pt.c  |  28 ++-
 xen/arch/x86/mm/p2m.c |  23 ++-
 xen/arch/x86/x86_64/mm.c  |   9 +-
 xen/common/memory.c   |  12 +-
 xen/drivers/passthrough/amd/iommu_init.c  |   9 +-
 xen/drivers/passthrough/amd/pci_amd_iommu.c   |  17 +-
 xen/drivers/passthrough/arm/smmu.c|  19 +-
 xen/drivers/passthrough/iommu.c   |  62 --
 xen/drivers/passthrough/vtd/extern.h  |   3 +-
 xen/drivers/passthrough/vtd/iommu.c   | 271 +-
 xen/drivers/passthrough/vtd/iommu.h   |  11 +-
 xen/drivers/passthrough/vtd/qinval.c  |  14 +-
 xen/drivers/passthrough/vtd/quirks.c  |  27 ++-
 xen/drivers/passthrough/vtd/x86/vtd.c |  15 +-
 xen/drivers/passthrough/x86/iommu.c   |   5 +-
 xen/include/asm-x86/hvm/svm/amd-iommu-proto.h |   8 +-
 xen/include/asm-x86/iommu.h   |   3 +-
 xen/include/xen/iommu.h   |  26 +--
 22 files changed, 496 insertions(+), 202 deletions(-)

-- 
1.9.1


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v8 03/11] IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU unmapping (top level ones)

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Signed-off-by: Quan Xu <quan...@intel.com>
Acked-by: Kevin Tian <kevin.t...@intel.com>
Acked-by: Suravee Suthikulpanit <suravee.suthikulpa...@amd.com>
Acked-by: Julien Grall <julien.gr...@arm.com>
Reviewed-by: Jan Beulich <jbeul...@suse.com>

CC: Stefano Stabellini <sstabell...@kernel.org>
CC: Julien Grall <julien.gr...@arm.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>
CC: Jan Beulich <jbeul...@suse.com>
CC: Andrew Cooper <andrew.coop...@citrix.com>
CC: Suravee Suthikulpanit <suravee.suthikulpa...@amd.com>
---
 xen/drivers/passthrough/arm/smmu.c|  2 +-
 xen/drivers/passthrough/vtd/iommu.c   | 15 ---
 xen/include/asm-x86/hvm/svm/amd-iommu-proto.h |  2 +-
 xen/include/xen/iommu.h   |  2 +-
 4 files changed, 11 insertions(+), 10 deletions(-)

diff --git a/xen/drivers/passthrough/arm/smmu.c 
b/xen/drivers/passthrough/arm/smmu.c
index 54a03a6..1ce4ddf 100644
--- a/xen/drivers/passthrough/arm/smmu.c
+++ b/xen/drivers/passthrough/arm/smmu.c
@@ -2774,7 +2774,7 @@ static int arm_smmu_map_page(struct domain *d, unsigned 
long gfn,
return guest_physmap_add_entry(d, gfn, mfn, 0, t);
 }
 
-static int arm_smmu_unmap_page(struct domain *d, unsigned long gfn)
+static int __must_check arm_smmu_unmap_page(struct domain *d, unsigned long 
gfn)
 {
/*
 * This function should only be used by gnttab code when the domain
diff --git a/xen/drivers/passthrough/vtd/iommu.c 
b/xen/drivers/passthrough/vtd/iommu.c
index db83949..4844193 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -609,7 +609,7 @@ static void intel_iommu_iotlb_flush_all(struct domain *d)
 }
 
 /* clear one page's page table */
-static void dma_pte_clear_one(struct domain *domain, u64 addr)
+static int __must_check dma_pte_clear_one(struct domain *domain, u64 addr)
 {
 struct domain_iommu *hd = dom_iommu(domain);
 struct dma_pte *page = NULL, *pte = NULL;
@@ -621,7 +621,7 @@ static void dma_pte_clear_one(struct domain *domain, u64 
addr)
 if ( pg_maddr == 0 )
 {
 spin_unlock(>arch.mapping_lock);
-return;
+return 0;
 }
 
 page = (struct dma_pte *)map_vtd_domain_page(pg_maddr);
@@ -631,7 +631,7 @@ static void dma_pte_clear_one(struct domain *domain, u64 
addr)
 {
 spin_unlock(>arch.mapping_lock);
 unmap_vtd_domain_page(page);
-return;
+return 0;
 }
 
 dma_clear_pte(*pte);
@@ -642,6 +642,8 @@ static void dma_pte_clear_one(struct domain *domain, u64 
addr)
 __intel_iommu_iotlb_flush(domain, addr >> PAGE_SHIFT_4K, 1, 1);
 
 unmap_vtd_domain_page(page);
+
+return 0;
 }
 
 static void iommu_free_pagetable(u64 pt_maddr, int level)
@@ -1739,15 +1741,14 @@ static int intel_iommu_map_page(
 return 0;
 }
 
-static int intel_iommu_unmap_page(struct domain *d, unsigned long gfn)
+static int __must_check intel_iommu_unmap_page(struct domain *d,
+   unsigned long gfn)
 {
 /* Do nothing if hardware domain and iommu supports pass thru. */
 if ( iommu_passthrough && is_hardware_domain(d) )
 return 0;
 
-dma_pte_clear_one(d, (paddr_t)gfn << PAGE_SHIFT_4K);
-
-return 0;
+return dma_pte_clear_one(d, (paddr_t)gfn << PAGE_SHIFT_4K);
 }
 
 void iommu_pte_flush(struct domain *d, u64 gfn, u64 *pte,
diff --git a/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h 
b/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h
index 9c51172..57b6cc1 100644
--- a/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h
+++ b/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h
@@ -53,7 +53,7 @@ int amd_iommu_update_ivrs_mapping_acpi(void);
 /* mapping functions */
 int amd_iommu_map_page(struct domain *d, unsigned long gfn, unsigned long mfn,
unsigned int flags);
-int amd_iommu_unmap_page(struct domain *d, unsigned long gfn);
+int __must_check amd_iommu_unmap_page(struct domain *d, unsigned long gfn);
 u64 amd_iommu_get_next_table_from_pte(u32 *entry);
 int amd_iommu_reserve_domain_unity_map(struct domain *domain,
u64 phys_addr, unsigned long size,
diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h
index eaa2c77..f45fa5a 100644
--- a/xen/include/xen/iommu.h
+++ b/xen/include/xen/iommu.h
@@ -168,7 +168,7 @@ struct iommu_ops {
 void (*teardown)(struct domain *d);
 int (*map_page)(struct domain *d, unsigned long gfn, unsigned long mfn,
 unsigned int flags);
-int (*unmap_page)(struct domain *d, unsigned long gfn);
+int __must_check (*unmap_page)(struct domain *d, unsigned long gfn);
 void (*free_page_table)(struct page_info *);
 #ifdef CONFIG_X86
 void (*update_ire_from_apic)(unsigned int apic, unsigned int reg, unsigned 
int value);
-

[Xen-devel] [PATCH v8 09/11] vt-d: fix the IOMMU flush issue

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

The propagation value from IOMMU flush interfaces may be positive, which
indicates callers need to flush cache, not one of faliures.

when the propagation value is positive, this patch fixes this flush issue
as follows:
  - call iommu_flush_write_buffer() to flush cache.
  - return zero.

Signed-off-by: Quan Xu <quan...@intel.com>
Reviewed-by: Jan Beulich <jbeul...@suse.com>

CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>
CC: Keir Fraser <k...@xen.org>
CC: Jan Beulich <jbeul...@suse.com>
CC: Andrew Cooper <andrew.coop...@citrix.com>

v8: drop assertion and check both flush_context and flush_iotlb
whether the return values are positive.
(note: this change is not under Jan's R-b).
---
 xen/drivers/passthrough/vtd/iommu.c | 150 +---
 1 file changed, 103 insertions(+), 47 deletions(-)

diff --git a/xen/drivers/passthrough/vtd/iommu.c 
b/xen/drivers/passthrough/vtd/iommu.c
index 48edb67..2f046cb 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -388,17 +388,18 @@ static int flush_context_reg(
 return 0;
 }
 
-static int iommu_flush_context_global(
-struct iommu *iommu, int flush_non_present_entry)
+static int __must_check iommu_flush_context_global(struct iommu *iommu,
+   int flush_non_present_entry)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 return flush->context(iommu, 0, 0, 0, DMA_CCMD_GLOBAL_INVL,
  flush_non_present_entry);
 }
 
-static int iommu_flush_context_device(
-struct iommu *iommu, u16 did, u16 source_id,
-u8 function_mask, int flush_non_present_entry)
+static int __must_check iommu_flush_context_device(struct iommu *iommu,
+   u16 did, u16 source_id,
+   u8 function_mask,
+   int flush_non_present_entry)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 return flush->context(iommu, did, source_id, function_mask,
@@ -473,8 +474,9 @@ static int flush_iotlb_reg(void *_iommu, u16 did,
 return 0;
 }
 
-static int iommu_flush_iotlb_global(struct iommu *iommu,
-int flush_non_present_entry, int flush_dev_iotlb)
+static int __must_check iommu_flush_iotlb_global(struct iommu *iommu,
+ int flush_non_present_entry,
+ int flush_dev_iotlb)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 int status;
@@ -491,8 +493,9 @@ static int iommu_flush_iotlb_global(struct iommu *iommu,
 return status;
 }
 
-static int iommu_flush_iotlb_dsi(struct iommu *iommu, u16 did,
-int flush_non_present_entry, int flush_dev_iotlb)
+static int __must_check iommu_flush_iotlb_dsi(struct iommu *iommu, u16 did,
+  int flush_non_present_entry,
+  int flush_dev_iotlb)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 int status;
@@ -509,9 +512,10 @@ static int iommu_flush_iotlb_dsi(struct iommu *iommu, u16 
did,
 return status;
 }
 
-static int iommu_flush_iotlb_psi(
-struct iommu *iommu, u16 did, u64 addr, unsigned int order,
-int flush_non_present_entry, int flush_dev_iotlb)
+static int __must_check iommu_flush_iotlb_psi(struct iommu *iommu, u16 did,
+  u64 addr, unsigned int order,
+  int flush_non_present_entry,
+  int flush_dev_iotlb)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 int status;
@@ -546,17 +550,37 @@ static int __must_check iommu_flush_all(void)
 struct acpi_drhd_unit *drhd;
 struct iommu *iommu;
 int flush_dev_iotlb;
+int rc = 0;
 
 flush_all_cache();
 for_each_drhd_unit ( drhd )
 {
+int iommu_rc, iommu_ret;
+
 iommu = drhd->iommu;
-iommu_flush_context_global(iommu, 0);
+iommu_rc = iommu_flush_context_global(iommu, 0);
 flush_dev_iotlb = find_ats_dev_drhd(iommu) ? 1 : 0;
-iommu_flush_iotlb_global(iommu, 0, flush_dev_iotlb);
+iommu_ret = iommu_flush_iotlb_global(iommu, 0, flush_dev_iotlb);
+
+/*
+ * The current logic for returns:
+ *   - positive  invoke iommu_flush_write_buffer to flush cache.
+ *   - zero  on success.
+ *   - negative  on failure. Continue to flush IOMMU IOTLB on a
+ *   best effort basis.
+ */
+if ( iommu_rc > 0 || iommu_ret > 0 )
+iommu_flush_write_buffer(iommu);
+if ( rc >= 0 )
+rc = iommu_rc;
+if ( rc >= 0 )
+rc = 

[Xen-devel] [PATCH v8 05/11] IOMMU/MMU: propagate IOMMU Device-TLB flush error up to iommu_iotlb_flush{, _all} (top level ones)

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Signed-off-by: Quan Xu <quan...@intel.com>
Acked-by: Julien Grall <julien.gr...@arm.com>
Reviewed-by: Kevin Tian <kevin.t...@intel.com>
Reviewed-by: Jan Beulich <jbeul...@suse.com>

CC: Stefano Stabellini <sstabell...@kernel.org>
CC: Julien Grall <julien.gr...@arm.com>
CC: Jan Beulich <jbeul...@suse.com>
CC: Kevin Tian <kevin.t...@intel.com>
---
 xen/arch/arm/p2m.c  |  4 +++-
 xen/common/memory.c | 12 ++--
 xen/drivers/passthrough/iommu.c | 13 +
 xen/drivers/passthrough/x86/iommu.c |  5 +++--
 xen/include/xen/iommu.h |  5 +++--
 5 files changed, 28 insertions(+), 11 deletions(-)

diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
index 6a19c57..65d8f1a 100644
--- a/xen/arch/arm/p2m.c
+++ b/xen/arch/arm/p2m.c
@@ -1178,7 +1178,9 @@ out:
 if ( flush )
 {
 flush_tlb_domain(d);
-iommu_iotlb_flush(d, sgfn, egfn - sgfn);
+ret = iommu_iotlb_flush(d, sgfn, egfn - sgfn);
+if ( !rc )
+rc = ret;
 }
 
 while ( (pg = page_list_remove_head(_pages)) )
diff --git a/xen/common/memory.c b/xen/common/memory.c
index ccc6436..46b1d9f 100644
--- a/xen/common/memory.c
+++ b/xen/common/memory.c
@@ -683,9 +683,17 @@ static int xenmem_add_to_physmap(struct domain *d,
 #ifdef CONFIG_HAS_PASSTHROUGH
 if ( need_iommu(d) )
 {
+int ret;
+
 this_cpu(iommu_dont_flush_iotlb) = 0;
-iommu_iotlb_flush(d, xatp->idx - done, done);
-iommu_iotlb_flush(d, xatp->gpfn - done, done);
+
+ret = iommu_iotlb_flush(d, xatp->idx - done, done);
+if ( unlikely(ret) && rc >= 0 )
+rc = ret;
+
+ret = iommu_iotlb_flush(d, xatp->gpfn - done, done);
+if ( unlikely(ret) && rc >= 0 )
+rc = ret;
 }
 #endif
 
diff --git a/xen/drivers/passthrough/iommu.c b/xen/drivers/passthrough/iommu.c
index ec85352..3a73fab 100644
--- a/xen/drivers/passthrough/iommu.c
+++ b/xen/drivers/passthrough/iommu.c
@@ -311,24 +311,29 @@ static void iommu_free_pagetables(unsigned long unused)
 cpumask_cycle(smp_processor_id(), 
_online_map));
 }
 
-void iommu_iotlb_flush(struct domain *d, unsigned long gfn, unsigned int 
page_count)
+int iommu_iotlb_flush(struct domain *d, unsigned long gfn,
+  unsigned int page_count)
 {
 const struct domain_iommu *hd = dom_iommu(d);
 
 if ( !iommu_enabled || !hd->platform_ops || !hd->platform_ops->iotlb_flush 
)
-return;
+return 0;
 
 hd->platform_ops->iotlb_flush(d, gfn, page_count);
+
+return 0;
 }
 
-void iommu_iotlb_flush_all(struct domain *d)
+int iommu_iotlb_flush_all(struct domain *d)
 {
 const struct domain_iommu *hd = dom_iommu(d);
 
 if ( !iommu_enabled || !hd->platform_ops || 
!hd->platform_ops->iotlb_flush_all )
-return;
+return 0;
 
 hd->platform_ops->iotlb_flush_all(d);
+
+return 0;
 }
 
 int __init iommu_setup(void)
diff --git a/xen/drivers/passthrough/x86/iommu.c 
b/xen/drivers/passthrough/x86/iommu.c
index b64b98f..a18a608 100644
--- a/xen/drivers/passthrough/x86/iommu.c
+++ b/xen/drivers/passthrough/x86/iommu.c
@@ -104,8 +104,9 @@ int arch_iommu_populate_page_table(struct domain *d)
 this_cpu(iommu_dont_flush_iotlb) = 0;
 
 if ( !rc )
-iommu_iotlb_flush_all(d);
-else if ( rc != -ERESTART )
+rc = iommu_iotlb_flush_all(d);
+
+if ( rc && rc != -ERESTART )
 iommu_teardown(d);
 
 return rc;
diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h
index 2b86710..57c9fbc 100644
--- a/xen/include/xen/iommu.h
+++ b/xen/include/xen/iommu.h
@@ -200,8 +200,9 @@ int iommu_do_pci_domctl(struct xen_domctl *, struct domain 
*d,
 int iommu_do_domctl(struct xen_domctl *, struct domain *d,
 XEN_GUEST_HANDLE_PARAM(xen_domctl_t));
 
-void iommu_iotlb_flush(struct domain *d, unsigned long gfn, unsigned int 
page_count);
-void iommu_iotlb_flush_all(struct domain *d);
+int __must_check iommu_iotlb_flush(struct domain *d, unsigned long gfn,
+   unsigned int page_count);
+int __must_check iommu_iotlb_flush_all(struct domain *d);
 
 /*
  * The purpose of the iommu_dont_flush_iotlb optional cpu flag is to
-- 
1.9.1


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v8 01/11] IOMMU: handle IOMMU mapping and unmapping failures

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Treat IOMMU mapping and unmapping failures as a fatal to the DomU
If IOMMU mapping and unmapping failed, crash the DomU and propagate
the error up to the call trees.

No spamming of the log can occur. For DomU, we avoid logging any
message for already dying domains. For Dom0, that'll still be more
verbose than we'd really like, but it at least wouldn't outright
flood the console.

Signed-off-by: Quan Xu <quan...@intel.com>
Reviewed-by: Kevin Tian <kevin.t...@intel.com>
Reviewed-by: Jan Beulich <jbeul...@suse.com>

CC: Jan Beulich <jbeul...@suse.com>
CC: Kevin Tian <kevin.t...@intel.com>
---
 xen/drivers/passthrough/iommu.c | 30 --
 1 file changed, 28 insertions(+), 2 deletions(-)

diff --git a/xen/drivers/passthrough/iommu.c b/xen/drivers/passthrough/iommu.c
index 9d104d2..673e126 100644
--- a/xen/drivers/passthrough/iommu.c
+++ b/xen/drivers/passthrough/iommu.c
@@ -240,21 +240,47 @@ int iommu_map_page(struct domain *d, unsigned long gfn, 
unsigned long mfn,
unsigned int flags)
 {
 const struct domain_iommu *hd = dom_iommu(d);
+int rc;
 
 if ( !iommu_enabled || !hd->platform_ops )
 return 0;
 
-return hd->platform_ops->map_page(d, gfn, mfn, flags);
+rc = hd->platform_ops->map_page(d, gfn, mfn, flags);
+if ( unlikely(rc) )
+{
+if ( !d->is_shutting_down && printk_ratelimit() )
+printk(XENLOG_ERR
+   "d%d: IOMMU mapping gfn %#lx to mfn %#lx failed: %d\n",
+   d->domain_id, gfn, mfn, rc);
+
+if ( !is_hardware_domain(d) )
+domain_crash(d);
+}
+
+return rc;
 }
 
 int iommu_unmap_page(struct domain *d, unsigned long gfn)
 {
 const struct domain_iommu *hd = dom_iommu(d);
+int rc;
 
 if ( !iommu_enabled || !hd->platform_ops )
 return 0;
 
-return hd->platform_ops->unmap_page(d, gfn);
+rc = hd->platform_ops->unmap_page(d, gfn);
+if ( unlikely(rc) )
+{
+if ( !d->is_shutting_down && printk_ratelimit() )
+printk(XENLOG_ERR
+   "d%d: IOMMU unmapping gfn %#lx failed: %d\n",
+   d->domain_id, gfn, rc);
+
+if ( !is_hardware_domain(d) )
+domain_crash(d);
+}
+
+return rc;
 }
 
 static void iommu_free_pagetables(unsigned long unused)
-- 
1.9.1


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v7 09/11] vt-d: fix the IOMMU flush issue

[Xu, Quan]

On June 12, 2016 5:27 PM, Xu, Quan <quan...@intel.com> wrote:
> On June 12, 2016 3:33 PM, Tian, Kevin <kevin.t...@intel.com> wrote:
> > > From: Xu, Quan
> > > Sent: Wednesday, June 08, 2016 4:59 PM @@ -545,18 +549,42 @@
> static
> > > int __must_check iommu_flush_all(void)  {
> > >  struct acpi_drhd_unit *drhd;
> > >  struct iommu *iommu;
> > > -int flush_dev_iotlb;
> > > +int rc = 0;
> > >
> > >  flush_all_cache();
> > >  for_each_drhd_unit ( drhd )
> > >  {
> > >  iommu = drhd->iommu;
> > > -iommu_flush_context_global(iommu, 0);
> > > -flush_dev_iotlb = find_ats_dev_drhd(iommu) ? 1 : 0;
> > > -iommu_flush_iotlb_global(iommu, 0, flush_dev_iotlb);
> > > +/*
> > > + * The current logic for rc returns:
> > > + *   - positive  invoke iommu_flush_write_buffer to flush cache.
> > > + *   - zero  on success.
> > > + *   - negative  on failure. Continue to flush IOMMU IOTLB on a
> > > + *   best effort basis.
> > > + *
> > > + * Moreover, IOMMU flush handlers flush_context_qi and
> > flush_iotlb_qi
> > > + * (or flush_context_reg and flush_iotlb_reg, deep functions in 
> > > the
> > > + * call trees of iommu_flush_context_global and
> > iommu_flush_iotlb_global)
> > > + * are with the same logic to bubble up positive return value.
> > > + */
> > > +rc = iommu_flush_context_global(iommu, 0);
> > > +if ( rc <= 0 )
> > > +{
> > > +int flush_dev_iotlb = find_ats_dev_drhd(iommu) ? 1 : 0;
> > > +int ret = iommu_flush_iotlb_global(iommu, 0,
> > > + flush_dev_iotlb);
> > > +
> > > +ASSERT(ret <= 0);
> > > +if ( !rc )
> > > +rc = ret;
> >
> > I'm dubious about the assertion here. Why can't above call return 1
> > upon error on earlier flush? I digged back your earlier reply like:
> >
> > > Yes, the iommu_flush_iotlb_dsi() can also return 1.
> > > Look at the call tree, at the beginning of
> > > flush_context_qi()/flush_iotlb_qi(), or
> > > flush_context_reg()/flush_iotlb_reg()..
> > >
> > > If rc was negative when we call iommu_flush_context_device(), it is
> > > impossible to return 1 for iommu_flush_iotlb_dsi().
> >
> > But I don't think it a good idea of making so much assumptions about
> > internal implementations of those low level interfaces.
> > Also flush_context may fail for one specific reason which doesn't
> > block flush_iotlb which could get 1 returned when caching mode is
> > disabled. We'd better have return-1 case correctly handled here.
> >
> 
> Your comment looks reasonable here. Could I change it as below:
> 
> -static int iommu_flush_iotlb_psi(
> -struct iommu *iommu, u16 did, u64 addr, unsigned int order,
> -int flush_non_present_entry, int flush_dev_iotlb)
> +static int __must_check iommu_flush_iotlb_psi(struct iommu *iommu, u16
> did,
> +  u64 addr, unsigned int order,
> +  int flush_non_present_entry,
> +  int flush_dev_iotlb)
>  {
>  struct iommu_flush *flush = iommu_get_flush(iommu);
>  int status;
> @@ -546,17 +550,35 @@ static int __must_check iommu_flush_all(void)
>  struct acpi_drhd_unit *drhd;
>  struct iommu *iommu;
>  int flush_dev_iotlb;
> +int rc = 0;
> 
>  flush_all_cache();
>  for_each_drhd_unit ( drhd )
>  {
> +int ret;
> +
>  iommu = drhd->iommu;
> -iommu_flush_context_global(iommu, 0);
> +/*
> + * The current logic for rc returns:
> + *   - positive  invoke iommu_flush_write_buffer to flush cache.
> + *   - zero  on success.
> + *   - negative  on failure. Continue to flush IOMMU IOTLB on a
> + *   best effort basis.
> + */
> +rc = iommu_flush_context_global(iommu, 0);
>  flush_dev_iotlb = find_ats_dev_drhd(iommu) ? 1 : 0;
> -iommu_flush_iotlb_global(iommu, 0, flush_dev_iotlb);
> +ret = iommu_flush_iotlb_global(iommu, 0, flush_dev_iotlb);
> +if ( !rc )
> +rc = ret;
> +
> +if ( rc > 0 || ret > 0 )
> +iommu_flush_write_buffer(iommu);
>  }
> 
>

Re: [Xen-devel] [PATCH v7 09/11] vt-d: fix the IOMMU flush issue

[Xu, Quan]

On June 12, 2016 3:33 PM, Tian, Kevin <kevin.t...@intel.com> wrote:
> > From: Xu, Quan
> > Sent: Wednesday, June 08, 2016 4:59 PM @@ -545,18 +549,42 @@ static
> > int __must_check iommu_flush_all(void)  {
> >  struct acpi_drhd_unit *drhd;
> >  struct iommu *iommu;
> > -int flush_dev_iotlb;
> > +int rc = 0;
> >
> >  flush_all_cache();
> >  for_each_drhd_unit ( drhd )
> >  {
> >  iommu = drhd->iommu;
> > -iommu_flush_context_global(iommu, 0);
> > -flush_dev_iotlb = find_ats_dev_drhd(iommu) ? 1 : 0;
> > -iommu_flush_iotlb_global(iommu, 0, flush_dev_iotlb);
> > +/*
> > + * The current logic for rc returns:
> > + *   - positive  invoke iommu_flush_write_buffer to flush cache.
> > + *   - zero  on success.
> > + *   - negative  on failure. Continue to flush IOMMU IOTLB on a
> > + *   best effort basis.
> > + *
> > + * Moreover, IOMMU flush handlers flush_context_qi and
> flush_iotlb_qi
> > + * (or flush_context_reg and flush_iotlb_reg, deep functions in the
> > + * call trees of iommu_flush_context_global and
> iommu_flush_iotlb_global)
> > + * are with the same logic to bubble up positive return value.
> > + */
> > +rc = iommu_flush_context_global(iommu, 0);
> > +if ( rc <= 0 )
> > +{
> > +int flush_dev_iotlb = find_ats_dev_drhd(iommu) ? 1 : 0;
> > +int ret = iommu_flush_iotlb_global(iommu, 0,
> > + flush_dev_iotlb);
> > +
> > +ASSERT(ret <= 0);
> > +if ( !rc )
> > +rc = ret;
> 
> I'm dubious about the assertion here. Why can't above call return 1 upon error
> on earlier flush? I digged back your earlier reply like:
> 
> > Yes, the iommu_flush_iotlb_dsi() can also return 1.
> > Look at the call tree, at the beginning of
> > flush_context_qi()/flush_iotlb_qi(), or
> > flush_context_reg()/flush_iotlb_reg()..
> >
> > If rc was negative when we call iommu_flush_context_device(), it is
> > impossible to return 1 for iommu_flush_iotlb_dsi().
> 
> But I don't think it a good idea of making so much assumptions about internal
> implementations of those low level interfaces.
> Also flush_context may fail for one specific reason which doesn't block
> flush_iotlb which could get 1 returned when caching mode is disabled. We'd
> better have return-1 case correctly handled here.
> 

Your comment looks reasonable here. Could I change it as below:

-static int iommu_flush_iotlb_psi(
-struct iommu *iommu, u16 did, u64 addr, unsigned int order,
-int flush_non_present_entry, int flush_dev_iotlb)
+static int __must_check iommu_flush_iotlb_psi(struct iommu *iommu, u16 did,
+  u64 addr, unsigned int order,
+  int flush_non_present_entry,
+  int flush_dev_iotlb)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 int status;
@@ -546,17 +550,35 @@ static int __must_check iommu_flush_all(void)
 struct acpi_drhd_unit *drhd;
 struct iommu *iommu;
 int flush_dev_iotlb;
+int rc = 0;

 flush_all_cache();
 for_each_drhd_unit ( drhd )
 {
+int ret;
+
 iommu = drhd->iommu;
-iommu_flush_context_global(iommu, 0);
+/*
+ * The current logic for rc returns:
+ *   - positive  invoke iommu_flush_write_buffer to flush cache.
+ *   - zero  on success.
+ *   - negative  on failure. Continue to flush IOMMU IOTLB on a
+ *   best effort basis.
+ */
+rc = iommu_flush_context_global(iommu, 0);
 flush_dev_iotlb = find_ats_dev_drhd(iommu) ? 1 : 0;
-iommu_flush_iotlb_global(iommu, 0, flush_dev_iotlb);
+ret = iommu_flush_iotlb_global(iommu, 0, flush_dev_iotlb);
+if ( !rc )
+rc = ret;
+
+if ( rc > 0 || ret > 0 )
+iommu_flush_write_buffer(iommu);
 }

-return 0;
+if ( rc > 0 )
+rc = 0;
+
+return rc;
 }








Also, Jan, what's your opinion?

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v7 07/11] IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU suspending (top level ones)

[Xu, Quan]

> -Original Message-
> From: Xen-devel [mailto:xen-devel-boun...@lists.xen.org] On Behalf Of Jan
> Beulich
> Sent: Wednesday, June 08, 2016 10:52 PM
> To: Xu, Quan <quan...@intel.com>
> Cc: Tian, Kevin <kevin.t...@intel.com>; Stefano Stabellini
> <sstabell...@kernel.org>; Wu, Feng <feng...@intel.com>; Liu Jinsong
> <jinsong@alibaba-inc.com>; dario.faggi...@citrix.com; xen-
> de...@lists.xen.org; Julien Grall <julien.gr...@arm.com>; Suravee
> Suthikulpanit <suravee.suthikulpa...@amd.com>; Andrew Cooper
> <andrew.coop...@citrix.com>; Keir Fraser <k...@xen.org>
> Subject: Re: [Xen-devel] [PATCH v7 07/11] IOMMU: propagate IOMMU
> Device-TLB flush error up to IOMMU suspending (top level ones)
> 

On 
> >>> On 08.06.16 at 10:59, <quan...@intel.com> wrote:
> > @@ -169,6 +203,7 @@ static int enter_state(u32 state)
> 
> Right above here we have
> 
> if ( (error = device_power_down()) )
> 
> which is now wrong as long as SAVED_ALL is not zero.
> 
> >  {
> >  printk(XENLOG_ERR "Some devices failed to power down.");
> >  system_state = SYS_STATE_resume;
> > +device_power_up(error);
> >  goto done;
> 
> For the goto you need to adjust "error", or else you return something
> meaningless (a sort of random positive number) to your caller.
> 

Yes, it is still not correct. Could I change it as following: 


-if ( (error = device_power_down()) )
+if ( (error = device_power_down()) != SAVED_ALL )
 {
 printk(XENLOG_ERR "Some devices failed to power down.");
 system_state = SYS_STATE_resume;
+device_power_up(error);
+error = -EIO;
 goto done;
 }

Quan
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v7 05/11] IOMMU/MMU: propagate IOMMU Device-TLB flush error up to iommu_iotlb_flush{, _all} (top level ones)

[Xu, Quan]

On June 12, 2016 2:56 PM, Tian, Kevin  wrote:
> > From: Julien Grall [mailto:julien.gr...@arm.com]
> > Sent: Thursday, June 09, 2016 8:40 PM
> >
> > Hi Jan,
> >
> > On 09/06/16 13:32, Jan Beulich wrote:
> >  On 09.06.16 at 14:24,  wrote:
> > >> So the behavior of iommu_iotlb_flush is up to the IOMMU driver.
> > >> Whilst the behavior of iommu_{,un}map_page are defined by the
> common code.
> > >
> > > I'm certainly up for moving the logic up to the generic IOMMU layer,
> > > if that's feasible.
> >
> > That would be my preference.
> >
> 
> I suppose above comment is for "[Patch v11 0/3] VT-d Device-TLB flush
> issue", where the crash logic better be moved up. 

I think so too. however, I am still reading these AMD/ARM related code.

> Do you have further against
> on this patch then?

Quan
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [BUG] win2008 guest cannot get ip through sriov

[Xu, Quan]

On June 07, 2016 3:50 PM, Jan Beulich  wrote:
> >>> On 07.06.16 at 07:52,  wrote:
> > -vf PT is not working for win2008: the logs:
> >qemu-dm-win2k8.log -- qemu log, vf PT for win2008.
> >xen-win2k8.log -- xen log, vf PT for win2008.
> 
> Hmm, that's very little output. In particular neither qemu nor Xen see _any_
> writes to the MSI-X table (without which interrupts obviously can't get
> enabled for that device).
> 
> Albeit - even in the SLES case only qemu sees such writes, so I'll have to 
> check
> if I made a mistake with the debugging patch.
> 

Jan,  do you have any other suggestions on how could I dig into this issue?

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v7 02/11] IOMMU/MMU: enhance the call trees of IOMMU unmapping and mapping

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

When IOMMU mapping is failed, we issue a best effort rollback, stopping
IOMMU mapping, unmapping the previous IOMMU maps and then reporting the
error up to the call trees. When rollback is not feasible (in early
initialization phase or trade-off of complexity) for the hardware domain,
we do things on a best effort basis, only throwing out an error message.

IOMMU unmapping should perhaps continue despite an error, in an attempt
to do best effort cleanup.

Signed-off-by: Quan Xu <quan...@intel.com>

CC: Keir Fraser <k...@xen.org>
CC: Jan Beulich <jbeul...@suse.com>
CC: Andrew Cooper <andrew.coop...@citrix.com>
CC: Jun Nakajima <jun.nakaj...@intel.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: George Dunlap <george.dun...@eu.citrix.com>

v7:
  1. add __must_check annotation to iommu_map_page() / iommu_unmap_page().
  2. use the same code structure for p2m_pt_set_entry() as I do on the
 EPT side.
  3. fix amd_iommu_hwdom_init() / iommu_hwdom_init() here.
  4. enhance print infomation. It is no need to mention "hardware domain"
 in printk of iommu_hwdom_init().
  5. make the assignment be replaced by its right side becoming
 the variable's initializer.
---
 xen/arch/x86/mm.c   | 13 ++-
 xen/arch/x86/mm/p2m-ept.c   | 34 +++--
 xen/arch/x86/mm/p2m-pt.c| 23 ---
 xen/arch/x86/mm/p2m.c   | 18 ---
 xen/arch/x86/x86_64/mm.c|  4 +++-
 xen/drivers/passthrough/amd/pci_amd_iommu.c | 15 +++--
 xen/drivers/passthrough/iommu.c | 13 ++-
 xen/drivers/passthrough/vtd/x86/vtd.c   | 15 +++--
 xen/include/xen/iommu.h |  6 ++---
 9 files changed, 114 insertions(+), 27 deletions(-)

diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index 8d10a3e..ae7c8ab 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -2467,7 +2467,7 @@ static int __get_page_type(struct page_info *page, 
unsigned long type,
int preemptible)
 {
 unsigned long nx, x, y = page->u.inuse.type_info;
-int rc = 0;
+int rc = 0, iommu_ret = 0;
 
 ASSERT(!(type & ~(PGT_type_mask | PGT_pae_xen_l2)));
 
@@ -2578,11 +2578,11 @@ static int __get_page_type(struct page_info *page, 
unsigned long type,
 if ( d && is_pv_domain(d) && unlikely(need_iommu(d)) )
 {
 if ( (x & PGT_type_mask) == PGT_writable_page )
-iommu_unmap_page(d, mfn_to_gmfn(d, page_to_mfn(page)));
+iommu_ret = iommu_unmap_page(d, mfn_to_gmfn(d, 
page_to_mfn(page)));
 else if ( type == PGT_writable_page )
-iommu_map_page(d, mfn_to_gmfn(d, page_to_mfn(page)),
-   page_to_mfn(page),
-   IOMMUF_readable|IOMMUF_writable);
+iommu_ret = iommu_map_page(d, mfn_to_gmfn(d, 
page_to_mfn(page)),
+   page_to_mfn(page),
+   IOMMUF_readable|IOMMUF_writable);
 }
 }
 
@@ -2599,6 +2599,9 @@ static int __get_page_type(struct page_info *page, 
unsigned long type,
 if ( (x & PGT_partial) && !(nx & PGT_partial) )
 put_page(page);
 
+if ( !rc )
+rc = iommu_ret;
+
 return rc;
 }
 
diff --git a/xen/arch/x86/mm/p2m-ept.c b/xen/arch/x86/mm/p2m-ept.c
index 1ed5b47..5aebc24 100644
--- a/xen/arch/x86/mm/p2m-ept.c
+++ b/xen/arch/x86/mm/p2m-ept.c
@@ -667,6 +667,7 @@ ept_set_entry(struct p2m_domain *p2m, unsigned long gfn, 
mfn_t mfn,
 unsigned long gfn_remainder = gfn;
 unsigned int i, target = order / EPT_TABLE_ORDER;
 int ret, rc = 0;
+bool_t entry_written = 0;
 bool_t direct_mmio = (p2mt == p2m_mmio_direct);
 uint8_t ipat = 0;
 bool_t need_modify_vtd_table = 1;
@@ -812,10 +813,15 @@ ept_set_entry(struct p2m_domain *p2m, unsigned long gfn, 
mfn_t mfn,
 rc = atomic_write_ept_entry(ept_entry, new_entry, target);
 if ( unlikely(rc) )
 old_entry.epte = 0;
-else if ( p2mt != p2m_invalid &&
-  (gfn + (1UL << order) - 1 > p2m->max_mapped_pfn) )
-/* Track the highest gfn for which we have ever had a valid mapping */
-p2m->max_mapped_pfn = gfn + (1UL << order) - 1;
+else
+{
+entry_written = 1;
+
+if ( p2mt != p2m_invalid &&
+ (gfn + (1UL << order) - 1 > p2m->max_mapped_pfn) )
+/* Track the highest gfn for which we have ever had a valid 
mapping */
+p2m->max_mapped_pfn = gfn + (1UL << order) - 1;
+}
 
 out:
 if ( needs_sync )
@@ -831,10 +837,24 @@ out:
 {
 if ( iommu_flags )
 for ( i = 0; i < (1 << order); i++ )
- 

[Xen-devel] [PATCH v7 07/11] IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU suspending (top level ones)

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Signed-off-by: Quan Xu <quan...@intel.com>

CC: Jan Beulich <jbeul...@suse.com>
CC: Liu Jinsong <jinsong@alibaba-inc.com>
CC: Keir Fraser <k...@xen.org>
CC: Andrew Cooper <andrew.coop...@citrix.com>
CC: Suravee Suthikulpanit <suravee.suthikulpa...@amd.com>
CC: Stefano Stabellini <sstabell...@kernel.org>
CC: Julien Grall <julien.gr...@arm.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>

v7:
  1. return SAVED_ALL at the bottom of device_power_down(), instead
 of SAVED_NONE.
  2. drop the 'if ( error > 0 )', calling device_power_up(error)
 without any if().
  3. for vtd_suspend():
   - drop pointless initializer.
   - return 0 at the bottom to make obvious that no error path
 comes there.
---
 xen/arch/x86/acpi/power.c | 73 ---
 xen/drivers/passthrough/amd/iommu_init.c  |  9 +++-
 xen/drivers/passthrough/amd/pci_amd_iommu.c   |  2 +-
 xen/drivers/passthrough/iommu.c   |  6 ++-
 xen/drivers/passthrough/vtd/iommu.c   | 35 +
 xen/include/asm-x86/hvm/svm/amd-iommu-proto.h |  2 +-
 xen/include/xen/iommu.h   |  4 +-
 7 files changed, 96 insertions(+), 35 deletions(-)

diff --git a/xen/arch/x86/acpi/power.c b/xen/arch/x86/acpi/power.c
index 2885e31..717a809 100644
--- a/xen/arch/x86/acpi/power.c
+++ b/xen/arch/x86/acpi/power.c
@@ -43,36 +43,70 @@ struct acpi_sleep_info acpi_sinfo;
 
 void do_suspend_lowlevel(void);
 
+enum dev_power_saved
+{
+SAVED_NONE,
+SAVED_CONSOLE,
+SAVED_TIME,
+SAVED_I8259A,
+SAVED_IOAPIC,
+SAVED_IOMMU,
+SAVED_LAPIC,
+SAVED_ALL,
+};
+
 static int device_power_down(void)
 {
-console_suspend();
+if ( console_suspend() )
+return SAVED_NONE;
 
-time_suspend();
+if ( time_suspend() )
+return SAVED_CONSOLE;
 
-i8259A_suspend();
+if ( i8259A_suspend() )
+return SAVED_TIME;
 
+/* ioapic_suspend cannot fail */
 ioapic_suspend();
 
-iommu_suspend();
+if ( iommu_suspend() )
+return SAVED_IOAPIC;
 
-lapic_suspend();
+if ( lapic_suspend() )
+return SAVED_IOMMU;
 
-return 0;
+return SAVED_ALL;
 }
 
-static void device_power_up(void)
+static void device_power_up(enum dev_power_saved saved)
 {
-lapic_resume();
-
-iommu_resume();
-
-ioapic_resume();
-
-i8259A_resume();
-
-time_resume();
-
-console_resume();
+switch ( saved )
+{
+case SAVED_ALL:
+case SAVED_LAPIC:
+lapic_resume();
+/* fall through */
+case SAVED_IOMMU:
+iommu_resume();
+/* fall through */
+case SAVED_IOAPIC:
+ioapic_resume();
+/* fall through */
+case SAVED_I8259A:
+i8259A_resume();
+/* fall through */
+case SAVED_TIME:
+time_resume();
+/* fall through */
+case SAVED_CONSOLE:
+console_resume();
+/* fall through */
+case SAVED_NONE:
+break;
+default:
+BUG();
+break;
+}
 }
 
 static void freeze_domains(void)
@@ -169,6 +203,7 @@ static int enter_state(u32 state)
 {
 printk(XENLOG_ERR "Some devices failed to power down.");
 system_state = SYS_STATE_resume;
+device_power_up(error);
 goto done;
 }
 
@@ -196,7 +231,7 @@ static int enter_state(u32 state)
 write_cr4(cr4 & ~X86_CR4_MCE);
 write_efer(read_efer());
 
-device_power_up();
+device_power_up(SAVED_ALL);
 
 mcheck_init(_cpu_data, 0);
 write_cr4(cr4);
diff --git a/xen/drivers/passthrough/amd/iommu_init.c 
b/xen/drivers/passthrough/amd/iommu_init.c
index 4536106..0b68596 100644
--- a/xen/drivers/passthrough/amd/iommu_init.c
+++ b/xen/drivers/passthrough/amd/iommu_init.c
@@ -1339,7 +1339,14 @@ static void invalidate_all_devices(void)
 iterate_ivrs_mappings(_invalidate_all_devices);
 }
 
-void amd_iommu_suspend(void)
+int amd_iommu_suspend(void)
+{
+amd_iommu_crash_shutdown();
+
+return 0;
+}
+
+void amd_iommu_crash_shutdown(void)
 {
 struct amd_iommu *iommu;
 
diff --git a/xen/drivers/passthrough/amd/pci_amd_iommu.c 
b/xen/drivers/passthrough/amd/pci_amd_iommu.c
index 4a860af..7761241 100644
--- a/xen/drivers/passthrough/amd/pci_amd_iommu.c
+++ b/xen/drivers/passthrough/amd/pci_amd_iommu.c
@@ -633,6 +633,6 @@ const struct iommu_ops amd_iommu_ops = {
 .suspend = amd_iommu_suspend,
 .resume = amd_iommu_resume,
 .share_p2m = amd_iommu_share_p2m,
-.crash_shutdown = amd_iommu_suspend,
+.crash_shutdown = amd_iommu_crash_shutdown,
 .dump_p2m_table = amd_dump_p2m_table,
 };
diff --git a/xen/drivers/passthrough/iommu.c b/xen/drivers/passthrough/iommu.c
index 3a73fab..a9898fc 100644
--- a/xen/drivers/passthrough/iommu.c
+++ b/xen/drivers/passthrough/iommu.c
@@ -379,10 +379,12 @@ int __init iommu_setup(void)
 re

[Xen-devel] [PATCH v7 01/11] IOMMU: handle IOMMU mapping and unmapping failures

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Treat IOMMU mapping and unmapping failures as a fatal to the DomU
If IOMMU mapping and unmapping failed, crash the DomU and propagate
the error up to the call trees.

No spamming of the log can occur. For DomU, we avoid logging any
message for already dying domains. For Dom0, that'll still be more
verbose than we'd really like, but it at least wouldn't outright
flood the console.

Signed-off-by: Quan Xu <quan...@intel.com>
Reviewed-by: Kevin Tian <kevin.t...@intel.com>
Reviewed-by: Jan Beulich <jbeul...@suse.com>

CC: Jan Beulich <jbeul...@suse.com>
CC: Kevin Tian <kevin.t...@intel.com>
---
 xen/drivers/passthrough/iommu.c | 30 --
 1 file changed, 28 insertions(+), 2 deletions(-)

diff --git a/xen/drivers/passthrough/iommu.c b/xen/drivers/passthrough/iommu.c
index 9d104d2..673e126 100644
--- a/xen/drivers/passthrough/iommu.c
+++ b/xen/drivers/passthrough/iommu.c
@@ -240,21 +240,47 @@ int iommu_map_page(struct domain *d, unsigned long gfn, 
unsigned long mfn,
unsigned int flags)
 {
 const struct domain_iommu *hd = dom_iommu(d);
+int rc;
 
 if ( !iommu_enabled || !hd->platform_ops )
 return 0;
 
-return hd->platform_ops->map_page(d, gfn, mfn, flags);
+rc = hd->platform_ops->map_page(d, gfn, mfn, flags);
+if ( unlikely(rc) )
+{
+if ( !d->is_shutting_down && printk_ratelimit() )
+printk(XENLOG_ERR
+   "d%d: IOMMU mapping gfn %#lx to mfn %#lx failed: %d\n",
+   d->domain_id, gfn, mfn, rc);
+
+if ( !is_hardware_domain(d) )
+domain_crash(d);
+}
+
+return rc;
 }
 
 int iommu_unmap_page(struct domain *d, unsigned long gfn)
 {
 const struct domain_iommu *hd = dom_iommu(d);
+int rc;
 
 if ( !iommu_enabled || !hd->platform_ops )
 return 0;
 
-return hd->platform_ops->unmap_page(d, gfn);
+rc = hd->platform_ops->unmap_page(d, gfn);
+if ( unlikely(rc) )
+{
+if ( !d->is_shutting_down && printk_ratelimit() )
+printk(XENLOG_ERR
+   "d%d: IOMMU unmapping gfn %#lx failed: %d\n",
+   d->domain_id, gfn, rc);
+
+if ( !is_hardware_domain(d) )
+domain_crash(d);
+}
+
+return rc;
 }
 
 static void iommu_free_pagetables(unsigned long unused)
-- 
1.9.1


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v7 08/11] IOMMU: propagate IOMMU Device-TLB flush error (leaf ones).

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Signed-off-by: Quan Xu <quan...@intel.com>
Acked-by: Kevin Tian <kevin.t...@intel.com>
Reviewed-by: Jan Beulich <jbeul...@suse.com>

CC: Stefano Stabellini <sstabell...@kernel.org>
CC: Julien Grall <julien.gr...@arm.com>
CC: Jan Beulich <jbeul...@suse.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>
---
 xen/drivers/passthrough/arm/smmu.c  | 13 -
 xen/drivers/passthrough/iommu.c |  8 ++--
 xen/drivers/passthrough/vtd/iommu.c | 32 
 xen/include/xen/iommu.h |  5 +++--
 4 files changed, 33 insertions(+), 25 deletions(-)

diff --git a/xen/drivers/passthrough/arm/smmu.c 
b/xen/drivers/passthrough/arm/smmu.c
index ee5c89d..1d21568 100644
--- a/xen/drivers/passthrough/arm/smmu.c
+++ b/xen/drivers/passthrough/arm/smmu.c
@@ -2540,7 +2540,7 @@ static int force_stage = 2;
  */
 static u32 platform_features = ARM_SMMU_FEAT_COHERENT_WALK;
 
-static void arm_smmu_iotlb_flush_all(struct domain *d)
+static int __must_check arm_smmu_iotlb_flush_all(struct domain *d)
 {
struct arm_smmu_xen_domain *smmu_domain = dom_iommu(d)->arch.priv;
struct iommu_domain *cfg;
@@ -2557,13 +2557,16 @@ static void arm_smmu_iotlb_flush_all(struct domain *d)
arm_smmu_tlb_inv_context(cfg->priv);
}
spin_unlock(_domain->lock);
+
+   return 0;
 }
 
-static void arm_smmu_iotlb_flush(struct domain *d, unsigned long gfn,
- unsigned int page_count)
+static int __must_check arm_smmu_iotlb_flush(struct domain *d,
+ unsigned long gfn,
+ unsigned int page_count)
 {
-/* ARM SMMU v1 doesn't have flush by VMA and VMID */
-arm_smmu_iotlb_flush_all(d);
+   /* ARM SMMU v1 doesn't have flush by VMA and VMID */
+   return arm_smmu_iotlb_flush_all(d);
 }
 
 static struct iommu_domain *arm_smmu_get_domain(struct domain *d,
diff --git a/xen/drivers/passthrough/iommu.c b/xen/drivers/passthrough/iommu.c
index a9898fc..ef80b3c 100644
--- a/xen/drivers/passthrough/iommu.c
+++ b/xen/drivers/passthrough/iommu.c
@@ -319,9 +319,7 @@ int iommu_iotlb_flush(struct domain *d, unsigned long gfn,
 if ( !iommu_enabled || !hd->platform_ops || !hd->platform_ops->iotlb_flush 
)
 return 0;
 
-hd->platform_ops->iotlb_flush(d, gfn, page_count);
-
-return 0;
+return hd->platform_ops->iotlb_flush(d, gfn, page_count);
 }
 
 int iommu_iotlb_flush_all(struct domain *d)
@@ -331,9 +329,7 @@ int iommu_iotlb_flush_all(struct domain *d)
 if ( !iommu_enabled || !hd->platform_ops || 
!hd->platform_ops->iotlb_flush_all )
 return 0;
 
-hd->platform_ops->iotlb_flush_all(d);
-
-return 0;
+return hd->platform_ops->iotlb_flush_all(d);
 }
 
 int __init iommu_setup(void)
diff --git a/xen/drivers/passthrough/vtd/iommu.c 
b/xen/drivers/passthrough/vtd/iommu.c
index 0f17afb..48edb67 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -559,8 +559,10 @@ static int __must_check iommu_flush_all(void)
 return 0;
 }
 
-static void __intel_iommu_iotlb_flush(struct domain *d, unsigned long gfn,
-int dma_old_pte_present, unsigned int page_count)
+static int __must_check iommu_flush_iotlb(struct domain *d,
+  unsigned long gfn,
+  bool_t dma_old_pte_present,
+  unsigned int page_count)
 {
 struct domain_iommu *hd = dom_iommu(d);
 struct acpi_drhd_unit *drhd;
@@ -598,16 +600,20 @@ static void __intel_iommu_iotlb_flush(struct domain *d, 
unsigned long gfn,
 iommu_flush_write_buffer(iommu);
 }
 }
+
+return 0;
 }
 
-static void intel_iommu_iotlb_flush(struct domain *d, unsigned long gfn, 
unsigned int page_count)
+static int __must_check iommu_flush_iotlb_pages(struct domain *d,
+unsigned long gfn,
+unsigned int page_count)
 {
-__intel_iommu_iotlb_flush(d, gfn, 1, page_count);
+return iommu_flush_iotlb(d, gfn, 1, page_count);
 }
 
-static void intel_iommu_iotlb_flush_all(struct domain *d)
+static int __must_check iommu_flush_iotlb_all(struct domain *d)
 {
-__intel_iommu_iotlb_flush(d, INVALID_GFN, 0, 0);
+return iommu_flush_iotlb(d, INVALID_GFN, 0, 0);
 }
 
 /* clear one page's page table */
@@ -616,6 +622,7 @@ static int __must_check dma_pte_clear_one(struct domain 
*domain, u64 addr)
 struct domain_iommu *hd = dom_iommu(domain);
 struct dma_pte *page = NULL, *pte = NULL;
 u64 pg_maddr;
+int rc = 0;
 
 spin_lock(>arch.mapping_lock);
 /* get last level pte */
@@ -641,11 +648,11 @@ static int __must_check

[Xen-devel] [PATCH v7 05/11] IOMMU/MMU: propagate IOMMU Device-TLB flush error up to iommu_iotlb_flush{, _all} (top level ones)

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Signed-off-by: Quan Xu <quan...@intel.com>
Reviewed-by: Jan Beulich <jbeul...@suse.com>

CC: Stefano Stabellini <sstabell...@kernel.org>
CC: Julien Grall <julien.gr...@arm.com>
CC: Jan Beulich <jbeul...@suse.com>
CC: Kevin Tian <kevin.t...@intel.com>
---
 xen/arch/arm/p2m.c  |  4 +++-
 xen/common/memory.c | 12 ++--
 xen/drivers/passthrough/iommu.c | 13 +
 xen/drivers/passthrough/x86/iommu.c |  5 +++--
 xen/include/xen/iommu.h |  5 +++--
 5 files changed, 28 insertions(+), 11 deletions(-)

diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
index 6a19c57..65d8f1a 100644
--- a/xen/arch/arm/p2m.c
+++ b/xen/arch/arm/p2m.c
@@ -1178,7 +1178,9 @@ out:
 if ( flush )
 {
 flush_tlb_domain(d);
-iommu_iotlb_flush(d, sgfn, egfn - sgfn);
+ret = iommu_iotlb_flush(d, sgfn, egfn - sgfn);
+if ( !rc )
+rc = ret;
 }
 
 while ( (pg = page_list_remove_head(_pages)) )
diff --git a/xen/common/memory.c b/xen/common/memory.c
index ccc6436..46b1d9f 100644
--- a/xen/common/memory.c
+++ b/xen/common/memory.c
@@ -683,9 +683,17 @@ static int xenmem_add_to_physmap(struct domain *d,
 #ifdef CONFIG_HAS_PASSTHROUGH
 if ( need_iommu(d) )
 {
+int ret;
+
 this_cpu(iommu_dont_flush_iotlb) = 0;
-iommu_iotlb_flush(d, xatp->idx - done, done);
-iommu_iotlb_flush(d, xatp->gpfn - done, done);
+
+ret = iommu_iotlb_flush(d, xatp->idx - done, done);
+if ( unlikely(ret) && rc >= 0 )
+rc = ret;
+
+ret = iommu_iotlb_flush(d, xatp->gpfn - done, done);
+if ( unlikely(ret) && rc >= 0 )
+rc = ret;
 }
 #endif
 
diff --git a/xen/drivers/passthrough/iommu.c b/xen/drivers/passthrough/iommu.c
index ec85352..3a73fab 100644
--- a/xen/drivers/passthrough/iommu.c
+++ b/xen/drivers/passthrough/iommu.c
@@ -311,24 +311,29 @@ static void iommu_free_pagetables(unsigned long unused)
 cpumask_cycle(smp_processor_id(), 
_online_map));
 }
 
-void iommu_iotlb_flush(struct domain *d, unsigned long gfn, unsigned int 
page_count)
+int iommu_iotlb_flush(struct domain *d, unsigned long gfn,
+  unsigned int page_count)
 {
 const struct domain_iommu *hd = dom_iommu(d);
 
 if ( !iommu_enabled || !hd->platform_ops || !hd->platform_ops->iotlb_flush 
)
-return;
+return 0;
 
 hd->platform_ops->iotlb_flush(d, gfn, page_count);
+
+return 0;
 }
 
-void iommu_iotlb_flush_all(struct domain *d)
+int iommu_iotlb_flush_all(struct domain *d)
 {
 const struct domain_iommu *hd = dom_iommu(d);
 
 if ( !iommu_enabled || !hd->platform_ops || 
!hd->platform_ops->iotlb_flush_all )
-return;
+return 0;
 
 hd->platform_ops->iotlb_flush_all(d);
+
+return 0;
 }
 
 int __init iommu_setup(void)
diff --git a/xen/drivers/passthrough/x86/iommu.c 
b/xen/drivers/passthrough/x86/iommu.c
index b64b98f..a18a608 100644
--- a/xen/drivers/passthrough/x86/iommu.c
+++ b/xen/drivers/passthrough/x86/iommu.c
@@ -104,8 +104,9 @@ int arch_iommu_populate_page_table(struct domain *d)
 this_cpu(iommu_dont_flush_iotlb) = 0;
 
 if ( !rc )
-iommu_iotlb_flush_all(d);
-else if ( rc != -ERESTART )
+rc = iommu_iotlb_flush_all(d);
+
+if ( rc && rc != -ERESTART )
 iommu_teardown(d);
 
 return rc;
diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h
index 2b86710..57c9fbc 100644
--- a/xen/include/xen/iommu.h
+++ b/xen/include/xen/iommu.h
@@ -200,8 +200,9 @@ int iommu_do_pci_domctl(struct xen_domctl *, struct domain 
*d,
 int iommu_do_domctl(struct xen_domctl *, struct domain *d,
 XEN_GUEST_HANDLE_PARAM(xen_domctl_t));
 
-void iommu_iotlb_flush(struct domain *d, unsigned long gfn, unsigned int 
page_count);
-void iommu_iotlb_flush_all(struct domain *d);
+int __must_check iommu_iotlb_flush(struct domain *d, unsigned long gfn,
+   unsigned int page_count);
+int __must_check iommu_iotlb_flush_all(struct domain *d);
 
 /*
  * The purpose of the iommu_dont_flush_iotlb optional cpu flag is to
-- 
1.9.1


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v7 10/11] vt-d: propagate the IOMMU Device-TLB flush error up to ME phantom functions

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Signed-off-by: Quan Xu <quan...@intel.com>
Reviewed-by: Jan Beulich <jbeul...@suse.com>

CC: Jan Beulich <jbeul...@suse.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>
---
 xen/drivers/passthrough/vtd/extern.h |  3 ++-
 xen/drivers/passthrough/vtd/iommu.c  |  8 
 xen/drivers/passthrough/vtd/quirks.c | 27 +--
 3 files changed, 23 insertions(+), 15 deletions(-)

diff --git a/xen/drivers/passthrough/vtd/extern.h 
b/xen/drivers/passthrough/vtd/extern.h
index cbe0286..6772839 100644
--- a/xen/drivers/passthrough/vtd/extern.h
+++ b/xen/drivers/passthrough/vtd/extern.h
@@ -91,7 +91,8 @@ int is_igd_vt_enabled_quirk(void);
 void platform_quirks_init(void);
 void vtd_ops_preamble_quirk(struct iommu* iommu);
 void vtd_ops_postamble_quirk(struct iommu* iommu);
-void me_wifi_quirk(struct domain *domain, u8 bus, u8 devfn, int map);
+int __must_check me_wifi_quirk(struct domain *domain,
+   u8 bus, u8 devfn, int map);
 void pci_vtd_quirk(const struct pci_dev *);
 bool_t platform_supports_intremap(void);
 bool_t platform_supports_x2apic(void);
diff --git a/xen/drivers/passthrough/vtd/iommu.c 
b/xen/drivers/passthrough/vtd/iommu.c
index 2a55985..8ad862e 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -1470,8 +1470,8 @@ int domain_context_mapping_one(
 
 unmap_vtd_domain_page(context_entries);
 
-if ( !seg )
-me_wifi_quirk(domain, bus, devfn, MAP_ME_PHANTOM_FUNC);
+if ( !seg && !rc )
+rc = me_wifi_quirk(domain, bus, devfn, MAP_ME_PHANTOM_FUNC);
 
 return rc;
 }
@@ -1630,8 +1630,8 @@ int domain_context_unmap_one(
 spin_unlock(>lock);
 unmap_vtd_domain_page(context_entries);
 
-if ( !iommu->intel->drhd->segment )
-me_wifi_quirk(domain, bus, devfn, UNMAP_ME_PHANTOM_FUNC);
+if ( !iommu->intel->drhd->segment && !rc )
+rc = me_wifi_quirk(domain, bus, devfn, UNMAP_ME_PHANTOM_FUNC);
 
 return rc;
 }
diff --git a/xen/drivers/passthrough/vtd/quirks.c 
b/xen/drivers/passthrough/vtd/quirks.c
index 473d1fc..91f96ac 100644
--- a/xen/drivers/passthrough/vtd/quirks.c
+++ b/xen/drivers/passthrough/vtd/quirks.c
@@ -331,10 +331,12 @@ void __init platform_quirks_init(void)
  * assigning Intel integrated wifi device to a guest.
  */
 
-static void map_me_phantom_function(struct domain *domain, u32 dev, int map)
+static int __must_check map_me_phantom_function(struct domain *domain,
+u32 dev, int map)
 {
 struct acpi_drhd_unit *drhd;
 struct pci_dev *pdev;
+int rc;
 
 /* find ME VT-d engine base on a real ME device */
 pdev = pci_get_pdev(0, 0, PCI_DEVFN(dev, 0));
@@ -342,23 +344,26 @@ static void map_me_phantom_function(struct domain 
*domain, u32 dev, int map)
 
 /* map or unmap ME phantom function */
 if ( map )
-domain_context_mapping_one(domain, drhd->iommu, 0,
-   PCI_DEVFN(dev, 7), NULL);
+rc = domain_context_mapping_one(domain, drhd->iommu, 0,
+PCI_DEVFN(dev, 7), NULL);
 else
-domain_context_unmap_one(domain, drhd->iommu, 0,
- PCI_DEVFN(dev, 7));
+rc = domain_context_unmap_one(domain, drhd->iommu, 0,
+  PCI_DEVFN(dev, 7));
+
+return rc;
 }
 
-void me_wifi_quirk(struct domain *domain, u8 bus, u8 devfn, int map)
+int me_wifi_quirk(struct domain *domain, u8 bus, u8 devfn, int map)
 {
 u32 id;
+int rc = 0;
 
 id = pci_conf_read32(0, 0, 0, 0, 0);
 if ( IS_CTG(id) )
 {
 /* quit if ME does not exist */
 if ( pci_conf_read32(0, 0, 3, 0, 0) == 0x )
-return;
+return 0;
 
 /* if device is WLAN device, map ME phantom device 0:3.7 */
 id = pci_conf_read32(0, bus, PCI_SLOT(devfn), PCI_FUNC(devfn), 0);
@@ -372,7 +377,7 @@ void me_wifi_quirk(struct domain *domain, u8 bus, u8 devfn, 
int map)
 case 0x423b8086:
 case 0x423c8086:
 case 0x423d8086:
-map_me_phantom_function(domain, 3, map);
+rc = map_me_phantom_function(domain, 3, map);
 break;
 default:
 break;
@@ -382,7 +387,7 @@ void me_wifi_quirk(struct domain *domain, u8 bus, u8 devfn, 
int map)
 {
 /* quit if ME does not exist */
 if ( pci_conf_read32(0, 0, 22, 0, 0) == 0x )
-return;
+return 0;
 
 /* if device is WLAN device, map ME phantom device 0:22.7 */
 id = pci_conf_read32(0, bus, PCI_SLOT(devfn), PCI_FUNC(devfn), 0);
@@ -398,12 +403,14 @@ void me_wifi_quirk(struct domain *domain, u8 bus, u8 
devfn, int map)
 case 0x423880

[Xen-devel] [PATCH v7 03/11] IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU unmapping (top level ones)

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Signed-off-by: Quan Xu <quan...@intel.com>
Acked-by: Kevin Tian <kevin.t...@intel.com>

CC: Stefano Stabellini <sstabell...@kernel.org>
CC: Julien Grall <julien.gr...@arm.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>
CC: Jan Beulich <jbeul...@suse.com>
CC: Andrew Cooper <andrew.coop...@citrix.com>

v7: just drop 'Reviewed-by: Jan Beulich <jbeul...@suse.com>',
as I haven't added __must_check annotation to iommu_unmap_page()
in previous v6.
---
 xen/drivers/passthrough/arm/smmu.c|  2 +-
 xen/drivers/passthrough/vtd/iommu.c   | 15 ---
 xen/include/asm-x86/hvm/svm/amd-iommu-proto.h |  2 +-
 xen/include/xen/iommu.h   |  2 +-
 4 files changed, 11 insertions(+), 10 deletions(-)

diff --git a/xen/drivers/passthrough/arm/smmu.c 
b/xen/drivers/passthrough/arm/smmu.c
index 54a03a6..1ce4ddf 100644
--- a/xen/drivers/passthrough/arm/smmu.c
+++ b/xen/drivers/passthrough/arm/smmu.c
@@ -2774,7 +2774,7 @@ static int arm_smmu_map_page(struct domain *d, unsigned 
long gfn,
return guest_physmap_add_entry(d, gfn, mfn, 0, t);
 }
 
-static int arm_smmu_unmap_page(struct domain *d, unsigned long gfn)
+static int __must_check arm_smmu_unmap_page(struct domain *d, unsigned long 
gfn)
 {
/*
 * This function should only be used by gnttab code when the domain
diff --git a/xen/drivers/passthrough/vtd/iommu.c 
b/xen/drivers/passthrough/vtd/iommu.c
index db83949..4844193 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -609,7 +609,7 @@ static void intel_iommu_iotlb_flush_all(struct domain *d)
 }
 
 /* clear one page's page table */
-static void dma_pte_clear_one(struct domain *domain, u64 addr)
+static int __must_check dma_pte_clear_one(struct domain *domain, u64 addr)
 {
 struct domain_iommu *hd = dom_iommu(domain);
 struct dma_pte *page = NULL, *pte = NULL;
@@ -621,7 +621,7 @@ static void dma_pte_clear_one(struct domain *domain, u64 
addr)
 if ( pg_maddr == 0 )
 {
 spin_unlock(>arch.mapping_lock);
-return;
+return 0;
 }
 
 page = (struct dma_pte *)map_vtd_domain_page(pg_maddr);
@@ -631,7 +631,7 @@ static void dma_pte_clear_one(struct domain *domain, u64 
addr)
 {
 spin_unlock(>arch.mapping_lock);
 unmap_vtd_domain_page(page);
-return;
+return 0;
 }
 
 dma_clear_pte(*pte);
@@ -642,6 +642,8 @@ static void dma_pte_clear_one(struct domain *domain, u64 
addr)
 __intel_iommu_iotlb_flush(domain, addr >> PAGE_SHIFT_4K, 1, 1);
 
 unmap_vtd_domain_page(page);
+
+return 0;
 }
 
 static void iommu_free_pagetable(u64 pt_maddr, int level)
@@ -1739,15 +1741,14 @@ static int intel_iommu_map_page(
 return 0;
 }
 
-static int intel_iommu_unmap_page(struct domain *d, unsigned long gfn)
+static int __must_check intel_iommu_unmap_page(struct domain *d,
+   unsigned long gfn)
 {
 /* Do nothing if hardware domain and iommu supports pass thru. */
 if ( iommu_passthrough && is_hardware_domain(d) )
 return 0;
 
-dma_pte_clear_one(d, (paddr_t)gfn << PAGE_SHIFT_4K);
-
-return 0;
+return dma_pte_clear_one(d, (paddr_t)gfn << PAGE_SHIFT_4K);
 }
 
 void iommu_pte_flush(struct domain *d, u64 gfn, u64 *pte,
diff --git a/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h 
b/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h
index 9c51172..57b6cc1 100644
--- a/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h
+++ b/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h
@@ -53,7 +53,7 @@ int amd_iommu_update_ivrs_mapping_acpi(void);
 /* mapping functions */
 int amd_iommu_map_page(struct domain *d, unsigned long gfn, unsigned long mfn,
unsigned int flags);
-int amd_iommu_unmap_page(struct domain *d, unsigned long gfn);
+int __must_check amd_iommu_unmap_page(struct domain *d, unsigned long gfn);
 u64 amd_iommu_get_next_table_from_pte(u32 *entry);
 int amd_iommu_reserve_domain_unity_map(struct domain *domain,
u64 phys_addr, unsigned long size,
diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h
index eaa2c77..f45fa5a 100644
--- a/xen/include/xen/iommu.h
+++ b/xen/include/xen/iommu.h
@@ -168,7 +168,7 @@ struct iommu_ops {
 void (*teardown)(struct domain *d);
 int (*map_page)(struct domain *d, unsigned long gfn, unsigned long mfn,
 unsigned int flags);
-int (*unmap_page)(struct domain *d, unsigned long gfn);
+int __must_check (*unmap_page)(struct domain *d, unsigned long gfn);
 void (*free_page_table)(struct page_info *);
 #ifdef CONFIG_X86
 void (*update_ire_from_apic)(unsigned int apic, unsigned int reg, unsigned 
int value);
-- 
1.9.1


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v7 11/11] vt-d: add __must_check annotation to IOMMU flush pointers and handlers

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Signed-off-by: Quan Xu <quan...@intel.com>
Reviewed-by: Jan Beulich <jbeul...@suse.com>

CC: Jan Beulich <jbeul...@suse.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>

v7: using !! instead of ?: .
---
 xen/drivers/passthrough/vtd/iommu.c  | 44 ++--
 xen/drivers/passthrough/vtd/iommu.h  | 11 +
 xen/drivers/passthrough/vtd/qinval.c | 14 ++--
 3 files changed, 36 insertions(+), 33 deletions(-)

diff --git a/xen/drivers/passthrough/vtd/iommu.c 
b/xen/drivers/passthrough/vtd/iommu.c
index 8ad862e..fc5c76c 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -335,10 +335,9 @@ static void iommu_flush_write_buffer(struct iommu *iommu)
 }
 
 /* return value determine if we need a write buffer flush */
-static int flush_context_reg(
-void *_iommu,
-u16 did, u16 source_id, u8 function_mask, u64 type,
-int flush_non_present_entry)
+static int __must_check flush_context_reg(void *_iommu, u16 did, u16 source_id,
+  u8 function_mask, u64 type,
+  bool_t flush_non_present_entry)
 {
 struct iommu *iommu = (struct iommu *) _iommu;
 u64 val = 0;
@@ -389,7 +388,7 @@ static int flush_context_reg(
 }
 
 static int __must_check iommu_flush_context_global(struct iommu *iommu,
-   int flush_non_present_entry)
+   bool_t 
flush_non_present_entry)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 return flush->context(iommu, 0, 0, 0, DMA_CCMD_GLOBAL_INVL,
@@ -399,7 +398,7 @@ static int __must_check iommu_flush_context_global(struct 
iommu *iommu,
 static int __must_check iommu_flush_context_device(struct iommu *iommu,
u16 did, u16 source_id,
u8 function_mask,
-   int flush_non_present_entry)
+   bool_t 
flush_non_present_entry)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 return flush->context(iommu, did, source_id, function_mask,
@@ -408,9 +407,10 @@ static int __must_check iommu_flush_context_device(struct 
iommu *iommu,
 }
 
 /* return value determine if we need a write buffer flush */
-static int flush_iotlb_reg(void *_iommu, u16 did,
-   u64 addr, unsigned int size_order, u64 type,
-   int flush_non_present_entry, int flush_dev_iotlb)
+static int __must_check flush_iotlb_reg(void *_iommu, u16 did, u64 addr,
+unsigned int size_order, u64 type,
+bool_t flush_non_present_entry,
+bool_t flush_dev_iotlb)
 {
 struct iommu *iommu = (struct iommu *) _iommu;
 int tlb_offset = ecap_iotlb_offset(iommu->ecap);
@@ -475,8 +475,8 @@ static int flush_iotlb_reg(void *_iommu, u16 did,
 }
 
 static int __must_check iommu_flush_iotlb_global(struct iommu *iommu,
- int flush_non_present_entry,
- int flush_dev_iotlb)
+ bool_t 
flush_non_present_entry,
+ bool_t flush_dev_iotlb)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 int status;
@@ -494,8 +494,8 @@ static int __must_check iommu_flush_iotlb_global(struct 
iommu *iommu,
 }
 
 static int __must_check iommu_flush_iotlb_dsi(struct iommu *iommu, u16 did,
-  int flush_non_present_entry,
-  int flush_dev_iotlb)
+  bool_t flush_non_present_entry,
+  bool_t flush_dev_iotlb)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 int status;
@@ -514,8 +514,8 @@ static int __must_check iommu_flush_iotlb_dsi(struct iommu 
*iommu, u16 did,
 
 static int __must_check iommu_flush_iotlb_psi(struct iommu *iommu, u16 did,
   u64 addr, unsigned int order,
-  int flush_non_present_entry,
-  int flush_dev_iotlb)
+  bool_t flush_non_present_entry,
+  bool_t flush_dev_iotlb)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 int status;
@@ -570,7 +570,7 @@ static int __must_check iommu_flush_all(void)
 rc = iommu_flush_context_global(iommu, 0);
 if ( rc <= 0 )
 {
-int fl

[Xen-devel] [PATCH v7 09/11] vt-d: fix the IOMMU flush issue

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

The propagation value from IOMMU flush interfaces may be positive, which
indicates callers need to flush cache, not one of faliures.

when the propagation value is positive, this patch fixes this flush issue
as follows:
  - call iommu_flush_write_buffer() to flush cache.
  - return zero.

Signed-off-by: Quan Xu <quan...@intel.com>
Reviewed-by: Jan Beulich <jbeul...@suse.com>

CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>
CC: Keir Fraser <k...@xen.org>
CC: Jan Beulich <jbeul...@suse.com>
CC: Andrew Cooper <andrew.coop...@citrix.com>

v7:
  1. Drop blank lines.
  2. make the assignment be replaced by its right side becoming
 the variable's initializer.
  3. leave the comments as are, no reply from VT-d maintainers yet.
---
 xen/drivers/passthrough/vtd/iommu.c | 167 ++--
 1 file changed, 124 insertions(+), 43 deletions(-)

diff --git a/xen/drivers/passthrough/vtd/iommu.c 
b/xen/drivers/passthrough/vtd/iommu.c
index 48edb67..2a55985 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -388,17 +388,18 @@ static int flush_context_reg(
 return 0;
 }
 
-static int iommu_flush_context_global(
-struct iommu *iommu, int flush_non_present_entry)
+static int __must_check iommu_flush_context_global(struct iommu *iommu,
+   int flush_non_present_entry)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 return flush->context(iommu, 0, 0, 0, DMA_CCMD_GLOBAL_INVL,
  flush_non_present_entry);
 }
 
-static int iommu_flush_context_device(
-struct iommu *iommu, u16 did, u16 source_id,
-u8 function_mask, int flush_non_present_entry)
+static int __must_check iommu_flush_context_device(struct iommu *iommu,
+   u16 did, u16 source_id,
+   u8 function_mask,
+   int flush_non_present_entry)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 return flush->context(iommu, did, source_id, function_mask,
@@ -473,8 +474,9 @@ static int flush_iotlb_reg(void *_iommu, u16 did,
 return 0;
 }
 
-static int iommu_flush_iotlb_global(struct iommu *iommu,
-int flush_non_present_entry, int flush_dev_iotlb)
+static int __must_check iommu_flush_iotlb_global(struct iommu *iommu,
+ int flush_non_present_entry,
+ int flush_dev_iotlb)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 int status;
@@ -491,8 +493,9 @@ static int iommu_flush_iotlb_global(struct iommu *iommu,
 return status;
 }
 
-static int iommu_flush_iotlb_dsi(struct iommu *iommu, u16 did,
-int flush_non_present_entry, int flush_dev_iotlb)
+static int __must_check iommu_flush_iotlb_dsi(struct iommu *iommu, u16 did,
+  int flush_non_present_entry,
+  int flush_dev_iotlb)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 int status;
@@ -509,9 +512,10 @@ static int iommu_flush_iotlb_dsi(struct iommu *iommu, u16 
did,
 return status;
 }
 
-static int iommu_flush_iotlb_psi(
-struct iommu *iommu, u16 did, u64 addr, unsigned int order,
-int flush_non_present_entry, int flush_dev_iotlb)
+static int __must_check iommu_flush_iotlb_psi(struct iommu *iommu, u16 did,
+  u64 addr, unsigned int order,
+  int flush_non_present_entry,
+  int flush_dev_iotlb)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 int status;
@@ -545,18 +549,42 @@ static int __must_check iommu_flush_all(void)
 {
 struct acpi_drhd_unit *drhd;
 struct iommu *iommu;
-int flush_dev_iotlb;
+int rc = 0;
 
 flush_all_cache();
 for_each_drhd_unit ( drhd )
 {
 iommu = drhd->iommu;
-iommu_flush_context_global(iommu, 0);
-flush_dev_iotlb = find_ats_dev_drhd(iommu) ? 1 : 0;
-iommu_flush_iotlb_global(iommu, 0, flush_dev_iotlb);
+/*
+ * The current logic for rc returns:
+ *   - positive  invoke iommu_flush_write_buffer to flush cache.
+ *   - zero  on success.
+ *   - negative  on failure. Continue to flush IOMMU IOTLB on a
+ *   best effort basis.
+ *
+ * Moreover, IOMMU flush handlers flush_context_qi and flush_iotlb_qi
+ * (or flush_context_reg and flush_iotlb_reg, deep functions in the
+ * call trees of iommu_flush_context_global and 
iommu_flush_iotlb_global)
+ * are with the same logic to bubble up positive return value.
+

[Xen-devel] [PATCH v7 00/11] Check VT-d Device-TLB flush error

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

This patch set is a prereq patch set for Patch:'VT-d Device-TLB flush issue'.

While IOMMU Device-TLB flush timed out, xen calls panic() at present. However 
the existing panic()
is going to be eliminated, so we must propagate the IOMMU Device-TLB flush 
error up to the call trees.

This patch set is also based on the discussion of 'abstract model of IOMMU 
unmaping/mapping failures'


Quan Xu (11):
  IOMMU: handle IOMMU mapping and unmapping failures
  IOMMU/MMU: enhance the call trees of IOMMU unmapping and mapping
  IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU unmapping
(top level ones)
  IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU mapping (top
level ones)
  IOMMU/MMU: propagate IOMMU Device-TLB flush error up to
iommu_iotlb_flush{,_all} (top level ones)
  propagate IOMMU Device-TLB flush error up to EPT update (top level
ones)
  IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU suspending
(top level ones)
  IOMMU: propagate IOMMU Device-TLB flush error (leaf ones).
  vt-d: fix the IOMMU flush issue
  vt-d: propagate the IOMMU Device-TLB flush error up to ME phantom
functions
  vt-d: add __must_check annotation to IOMMU flush pointers and handlers

 xen/arch/arm/p2m.c|   4 +-
 xen/arch/x86/acpi/power.c |  73 +--
 xen/arch/x86/mm.c |  13 +-
 xen/arch/x86/mm/p2m-ept.c |  36 +++-
 xen/arch/x86/mm/p2m-pt.c  |  23 ++-
 xen/arch/x86/mm/p2m.c |  18 +-
 xen/arch/x86/x86_64/mm.c  |   4 +-
 xen/common/memory.c   |  12 +-
 xen/drivers/passthrough/amd/iommu_init.c  |   9 +-
 xen/drivers/passthrough/amd/pci_amd_iommu.c   |  17 +-
 xen/drivers/passthrough/arm/smmu.c|  19 +-
 xen/drivers/passthrough/iommu.c   |  62 +-
 xen/drivers/passthrough/vtd/extern.h  |   3 +-
 xen/drivers/passthrough/vtd/iommu.c   | 284 ++
 xen/drivers/passthrough/vtd/iommu.h   |  11 +-
 xen/drivers/passthrough/vtd/qinval.c  |  14 +-
 xen/drivers/passthrough/vtd/quirks.c  |  27 ++-
 xen/drivers/passthrough/vtd/x86/vtd.c |  15 +-
 xen/drivers/passthrough/x86/iommu.c   |   5 +-
 xen/include/asm-x86/hvm/svm/amd-iommu-proto.h |   8 +-
 xen/include/asm-x86/iommu.h   |   3 +-
 xen/include/xen/iommu.h   |  26 +--
 22 files changed, 491 insertions(+), 195 deletions(-)

-- 
1.9.1


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v7 04/11] IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU mapping (top level ones)

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Signed-off-by: Quan Xu <quan...@intel.com>
Acked-by: Kevin Tian <kevin.t...@intel.com>

CC: Suravee Suthikulpanit <suravee.suthikulpa...@amd.com>
CC: Stefano Stabellini <sstabell...@kernel.org>
CC: Julien Grall <julien.gr...@arm.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>
CC: Jan Beulich <jbeul...@suse.com>

v7: drop the amd_iommu_hwdom_init() fix, which has been added in
patch 2 in this v7.
---
 xen/drivers/passthrough/arm/smmu.c| 4 ++--
 xen/drivers/passthrough/vtd/iommu.c   | 7 ---
 xen/include/asm-x86/hvm/svm/amd-iommu-proto.h | 4 ++--
 xen/include/xen/iommu.h   | 4 ++--
 4 files changed, 10 insertions(+), 9 deletions(-)

diff --git a/xen/drivers/passthrough/arm/smmu.c 
b/xen/drivers/passthrough/arm/smmu.c
index 1ce4ddf..ee5c89d 100644
--- a/xen/drivers/passthrough/arm/smmu.c
+++ b/xen/drivers/passthrough/arm/smmu.c
@@ -2745,8 +2745,8 @@ static void arm_smmu_iommu_domain_teardown(struct domain 
*d)
xfree(xen_domain);
 }
 
-static int arm_smmu_map_page(struct domain *d, unsigned long gfn,
-unsigned long mfn, unsigned int flags)
+static int __must_check arm_smmu_map_page(struct domain *d, unsigned long gfn,
+  unsigned long mfn, unsigned int 
flags)
 {
p2m_type_t t;
 
diff --git a/xen/drivers/passthrough/vtd/iommu.c 
b/xen/drivers/passthrough/vtd/iommu.c
index 4844193..e900019 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -1687,9 +1687,10 @@ static void iommu_domain_teardown(struct domain *d)
 spin_unlock(>arch.mapping_lock);
 }
 
-static int intel_iommu_map_page(
-struct domain *d, unsigned long gfn, unsigned long mfn,
-unsigned int flags)
+static int __must_check intel_iommu_map_page(struct domain *d,
+ unsigned long gfn,
+ unsigned long mfn,
+ unsigned int flags)
 {
 struct domain_iommu *hd = dom_iommu(d);
 struct dma_pte *page = NULL, *pte = NULL, old, new = { 0 };
diff --git a/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h 
b/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h
index 57b6cc1..ac9f036 100644
--- a/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h
+++ b/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h
@@ -51,8 +51,8 @@ int amd_iommu_init(void);
 int amd_iommu_update_ivrs_mapping_acpi(void);
 
 /* mapping functions */
-int amd_iommu_map_page(struct domain *d, unsigned long gfn, unsigned long mfn,
-   unsigned int flags);
+int __must_check amd_iommu_map_page(struct domain *d, unsigned long gfn,
+unsigned long mfn, unsigned int flags);
 int __must_check amd_iommu_unmap_page(struct domain *d, unsigned long gfn);
 u64 amd_iommu_get_next_table_from_pte(u32 *entry);
 int amd_iommu_reserve_domain_unity_map(struct domain *domain,
diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h
index f45fa5a..2b86710 100644
--- a/xen/include/xen/iommu.h
+++ b/xen/include/xen/iommu.h
@@ -166,8 +166,8 @@ struct iommu_ops {
 #endif /* HAS_PCI */
 
 void (*teardown)(struct domain *d);
-int (*map_page)(struct domain *d, unsigned long gfn, unsigned long mfn,
-unsigned int flags);
+int __must_check (*map_page)(struct domain *d, unsigned long gfn,
+ unsigned long mfn, unsigned int flags);
 int __must_check (*unmap_page)(struct domain *d, unsigned long gfn);
 void (*free_page_table)(struct page_info *);
 #ifdef CONFIG_X86
-- 
1.9.1


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v7 06/11] propagate IOMMU Device-TLB flush error up to EPT update (top level ones)

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Propagate the IOMMU Device-TLB flush error up to the ept_set_entry(),
when VT-d shares EPT page table.

Signed-off-by: Quan Xu <quan...@intel.com>
Acked-by: Kevin Tian <kevin.t...@intel.com>
Reviewed-by: Jan Beulich <jbeul...@suse.com>

CC: Jun Nakajima <jun.nakaj...@intel.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: George Dunlap <george.dun...@eu.citrix.com>
CC: Jan Beulich <jbeul...@suse.com>
CC: Andrew Cooper <andrew.coop...@citrix.com>
CC: Feng Wu <feng...@intel.com>
---
 xen/arch/x86/mm/p2m-ept.c   | 2 +-
 xen/drivers/passthrough/vtd/iommu.c | 6 --
 xen/include/asm-x86/iommu.h | 3 ++-
 3 files changed, 7 insertions(+), 4 deletions(-)

diff --git a/xen/arch/x86/mm/p2m-ept.c b/xen/arch/x86/mm/p2m-ept.c
index 5aebc24..234d76a 100644
--- a/xen/arch/x86/mm/p2m-ept.c
+++ b/xen/arch/x86/mm/p2m-ept.c
@@ -832,7 +832,7 @@ out:
  need_modify_vtd_table )
 {
 if ( iommu_hap_pt_share )
-iommu_pte_flush(d, gfn, _entry->epte, order, vtd_pte_present);
+rc = iommu_pte_flush(d, gfn, _entry->epte, order, 
vtd_pte_present);
 else
 {
 if ( iommu_flags )
diff --git a/xen/drivers/passthrough/vtd/iommu.c 
b/xen/drivers/passthrough/vtd/iommu.c
index e900019..5366267 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -1752,8 +1752,8 @@ static int __must_check intel_iommu_unmap_page(struct 
domain *d,
 return dma_pte_clear_one(d, (paddr_t)gfn << PAGE_SHIFT_4K);
 }
 
-void iommu_pte_flush(struct domain *d, u64 gfn, u64 *pte,
- int order, int present)
+int iommu_pte_flush(struct domain *d, u64 gfn, u64 *pte,
+int order, int present)
 {
 struct acpi_drhd_unit *drhd;
 struct iommu *iommu = NULL;
@@ -1778,6 +1778,8 @@ void iommu_pte_flush(struct domain *d, u64 gfn, u64 *pte,
order, !present, flush_dev_iotlb) )
 iommu_flush_write_buffer(iommu);
 }
+
+return 0;
 }
 
 static int __init vtd_ept_page_compatible(struct iommu *iommu)
diff --git a/xen/include/asm-x86/iommu.h b/xen/include/asm-x86/iommu.h
index e82a2f0..815d77e 100644
--- a/xen/include/asm-x86/iommu.h
+++ b/xen/include/asm-x86/iommu.h
@@ -27,7 +27,8 @@ int iommu_setup_hpet_msi(struct msi_desc *);
 
 /* While VT-d specific, this must get declared in a generic header. */
 int adjust_vtd_irq_affinities(void);
-void iommu_pte_flush(struct domain *d, u64 gfn, u64 *pte, int order, int 
present);
+int __must_check iommu_pte_flush(struct domain *d, u64 gfn, u64 *pte,
+ int order, int present);
 bool_t iommu_supports_eim(void);
 int iommu_enable_x2apic_IR(void);
 void iommu_disable_x2apic_IR(void);
-- 
1.9.1


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [Patch v6 04/11] IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU mapping (top level ones)

[Xu, Quan]

On June 07, 2016 4:19 PM, Jan Beulich  wrote:
> >>> On 07.06.16 at 09:51,  wrote:
> > I still have one question. If I add __must_check annotation to
> > iommu_unmap_page().
> > How to fix this -- unmapping the previous IOMMU maps when IOMMU
> > mapping is failed..
> > e.g.,
> >
> > ept_set_entry()
> > {
> > ...
> > for ( i = 0; i < (1 << order); i++ )
> > {
> > rc = iommu_map_page(d, gfn + i, mfn_x(mfn) + i, 
> > iommu_flags);
> > if ( unlikely(rc) )
> > {
> > while ( i-- )
> > iommu_unmap_page(p2m->domain, gfn + i);
> >
> > break;
> > }
> > }
> > ...
> > }
> >
> >
> > If we leave it as is, it leads to compilation errors as __must_check
> > annotation. Also we return the first error, so we are no need to
> > cumulate the error of iommu_unmap_page().
> > That's also why I hesitated to add __must_check annotation to
> > iommu_unmap_page().
> 
> Well, with there already being a message logged down the call tree in case of
> error, I think the return value should simply be latched into a local 
> variable,
> and nothing really be done with it (which should satisfy the compiler afaict,
> but it may be that Coverity would grumble about such). We really don't want
> to omit the __must_check just because there are a few cases where the error
> is of no further relevance; the annotation gets put there to make sure we
> catch all (current and future) cases where errors must not be ignored.
> 

Agreed. We really need to add __must_check annotation to iommu_map_page() / 
iommu_unmap_page().


Along with discussion of  a local variable for _for()_ loop,  I need to define 
this local variable into _while()_ loop:
e.g., as above case:
 
+while ( i-- )
+{
+int iommu_ret = iommu_unmap_page(p2m->domain, gfn + i);
+
+break;
+}

,  right?


But I really like this way:

+ int iommu_ret;
+
+while ( i-- )
+{
+iommu_ret = iommu_unmap_page(p2m->domain, gfn + i);
+
+break;
+}

Quan
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [BUG] win2008 guest cannot get ip through sriov

[Xu, Quan]

On June 07, 2016 4:09 PM, Jan Beulich  wrote:
> >>> On 07.06.16 at 09:49,  wrote:
>  On 07.06.16 at 07:52,  wrote:
> >> -vf PT is not working for win2008: the logs:
> >>qemu-dm-win2k8.log -- qemu log, vf PT for win2008.
> >>xen-win2k8.log -- xen log, vf PT for win2008.
> >
> > Hmm, that's very little output. In particular neither qemu nor Xen see
> > _any_ writes to the MSI-X table (without which interrupts obviously
> > can't get enabled for that device).
> >
> > Albeit - even in the SLES case only qemu sees such writes, so I'll
> > have to check if I made a mistake with the debugging patch.
> 
> With the exact same patches I do see MSI-X table writes logged with both SLES
> 11 and SLES 12  guests. What's going on here?
> 

The internet cable is not plugged in for that NIC.

Quan 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [BUG] win2008 guest cannot get ip through sriov

[Xu, Quan]

On June 07, 2016 3:50 PM, Jan Beulich  wrote:
> >>> On 07.06.16 at 07:52,  wrote:
> > -vf PT is not working for win2008: the logs:
> >qemu-dm-win2k8.log -- qemu log, vf PT for win2008.
> >xen-win2k8.log -- xen log, vf PT for win2008.
> 
> Hmm, that's very little output. In particular neither qemu nor Xen see _any_
> writes to the MSI-X table (without which interrupts obviously can't get
> enabled for that device).
> 
> Albeit - even in the SLES case only qemu sees such writes, so I'll have to 
> check
> if I made a mistake with the debugging patch.
> 

I added more print-out when I debugged it.. I'll call Jianzhong, Chang for more 
information..
I __guess__ he makes vf work for win2008.

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [Patch v6 04/11] IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU mapping (top level ones)

[Xu, Quan]

On June 02, 2016 5:21 PM, Jan Beulich  wrote:
> >>> On 02.06.16 at 09:25,  wrote:
> > On June 01, 2016 6:24 PM, Jan Beulich  wrote:
> >> >>> On 31.05.16 at 15:57,  wrote:
> >  As would a respective change to vtd_set_hwdom_mapping(), which
> >> I'm not sure which patch you've put that in.
> >>
> >
> > Sorry,  I missed it. I indeed it need to fix it as similar as above.
> > I wonder whether I could add a __must_check annotation to
> > iommu_map_page() or not, as which may be inconsistent with
> iommu_unmap_page().
> 
> Urgh - are you saying that by the end of the series they aren't _both_
> __must_check? Then I must have overlooked something while reviewing: They
> definitely both ought to be. Or wait - I've pointed this out in the context 
> of this
> patch, still seen below.
> 
> >> > --- a/xen/include/xen/iommu.h
> >> > +++ b/xen/include/xen/iommu.h
> >> > @@ -166,8 +166,8 @@ struct iommu_ops {  #endif /* HAS_PCI */
> >> >
> >> >  void (*teardown)(struct domain *d);
> >> > -int (*map_page)(struct domain *d, unsigned long gfn, unsigned long
> mfn,
> >> > -unsigned int flags);
> >> > +int __must_check (*map_page)(struct domain *d, unsigned long gfn,
> >> > + unsigned long mfn, unsigned int
> >> > + flags);
> >>
> >> With this and with the rule set forth in the context of the
> >> discussion of
> > v5,
> >> iommu_map_page() (as well as any other caller of this hook that do
> >> not themselves _consume_ the error [e.g. hwdom init ones]) should
> >> become or already be __must_check, which afaict isn't the case.
> >
> > But does this rule also apply to these 'void' annotation functions?
> > .e.g, in the call tree of hwdom init ones / domain crash ones, we are
> > no need to bubble up error code, leaving these void annotation as is.
> 
> Note that my previous reply already answered that question (as I expected
> you would otherwise ask): I specifically excluded those functions that
> _consume_ the error, and I did give an example. I really don't know what else 
> I
> could have done to make clear what exceptions are to be expected.
> 
> >> The same then, btw.,
> >> applies to patch 3, and hence I have to withdraw the R-b that you've
> >> got there.
> >
> > I find these callers are grant_table/mm, and we limit __must_check
> > annotation to IOMMU functions for this patch set..
> 
> Talk isn't of those ones. The subject of patch 3 is unmapping, and hence the
> parallel to the one here is that iommu_unmap_page() needs to become
> __must_check there, along with the iommu_ops
> unmap_page() hook.
> 


Jan,
I still have one question. If I add __must_check annotation to 
iommu_unmap_page().
How to fix this -- unmapping the previous IOMMU maps when IOMMU mapping is 
failed..
e.g.,

ept_set_entry()
{
...
for ( i = 0; i < (1 << order); i++ )
{
rc = iommu_map_page(d, gfn + i, mfn_x(mfn) + i, 
iommu_flags);
if ( unlikely(rc) )
{
while ( i-- )
iommu_unmap_page(p2m->domain, gfn + i);

break;
}
}
...
}


If we leave it as is, it leads to compilation errors as __must_check 
annotation. Also we return the first error, so we are no need to cumulate the 
error of iommu_unmap_page().
That's also why I hesitated to add __must_check annotation to 
iommu_unmap_page().

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [BUG] win2008 guest cannot get ip through sriov

[Xu, Quan]

On June 07, 2016 1:52 PM, Xu, Quan <quan...@intel.com> wrote:
> On June 02, 2016 8:09 PM, Jan Beulich <jbeul...@suse.com> wrote:
>

Sorry, this below part is about NIC PF pass-through for win2008..
 
> btw, when I updated the NIC driver for win2008,  actually, it is working.
> Xudong, could you help me verify it again?
> 
> ((
> xen commit: bbfd2d6ccb31a3aeea49c8f9c7884792ddc26e3b
> NIC: Intel Corporation I350 Gigabit Network Connection
>  new win2k8 NIC
> driver ,https://downloadcenter.intel.com/download/18720/Network-
> Adapter-Driver-for-Windows-Server-2008-Final-Release?product=59679
> ))
> 
> qemu-dm-TestDom.log -- qemu log, pf PT for win2008.
> 
> 
... :(:(

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [BUG] win2008 guest cannot get ip through sriov

[Xu, Quan]

On June 02, 2016 8:09 PM, Jan Beulich  wrote:
> Attached both a hypervisor and a qemu patch. Their plus
> debug key M and i output is what I'd like to start with.

Applied these two patches, the attach files are logs.

-vf PT is working for SLES 12. The logs:
   qemu-dm-sles.log -- qemu log, vf PT for sles 12.
   xen-sles.log -- xen log, vf PT for sles 12.

-vf PT is not working for win2008: the logs:
   qemu-dm-win2k8.log -- qemu log, vf PT for win2008.
   xen-win2k8.log -- xen log, vf PT for win2008.


btw, when I updated the NIC driver for win2008,  actually, it is working.  
Xudong, could you help me verify it again?

((
xen commit: bbfd2d6ccb31a3aeea49c8f9c7884792ddc26e3b
NIC: Intel Corporation I350 Gigabit Network Connection
 new win2k8 NIC driver 
,https://downloadcenter.intel.com/download/18720/Network-Adapter-Driver-for-Windows-Server-2008-Final-Release?product=59679
))

qemu-dm-TestDom.log -- qemu log, pf PT for win2008.


... taken together, IMO, from qemu-dm-win2k8.log(also I have read xen/QEMU 
code), the win2k8 vf nic driver is not loading. (any idea, feel free to let me 
know. I will help you try as much as I can..).

Thoughts?

Quan


qemu-dm-sles.log
Description: qemu-dm-sles.log


qemu-dm-TestDom.log
Description: qemu-dm-TestDom.log


qemu-dm-win2k8.log
Description: qemu-dm-win2k8.log


xen-sles.log
Description: xen-sles.log


xen-win2k8.log
Description: xen-win2k8.log
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [BUG] win2008 guest cannot get ip through sriov

[Xu, Quan]

On June 02, 2016 9:30 PM, Jan Beulich <jbeul...@suse.com> wrote:
> >>> On 02.06.16 at 15:05, <quan...@intel.com> wrote:
> > On June 02, 2016 8:09 PM, Jan Beulich <jbeul...@suse.com> wrote:
> >> >>> On 02.06.16 at 13:03, <wei.l...@citrix.com> wrote:
> >> > On Thu, Jun 02, 2016 at 04:38:47AM -0600, Jan Beulich wrote:
> >> >> >>> On 02.06.16 at 12:22, <wei.l...@citrix.com> wrote:
> >> >> > On Thu, Jun 02, 2016 at 07:31:06AM +, Xu, Quan wrote:
> >> >> >> On May 27, 2016 10:06 PM, Jan Beulich <jbeul...@suse.com>
> wrote:
> >> >> >> > >>> On 27.05.16 at 15:34, <wei.l...@citrix.com> wrote:
> >> >> >> > > On Fri, May 27, 2016 at 06:16:30AM -0600, Jan Beulich wrote:
> >> >> >> > >> >>> On 27.05.16 at 12:39, <wei.l...@citrix.com> wrote:
> >> >> >> > >> > Is this a regression? Does it work on previous versions of 
> >> >> >> > >> > Xen?
> >> >> >> > >>
> >> >> >> > >> I think this is what was already reported by other Intel
> >> >> >> > >> people, see e.g. Quan's most recent reply:
> >> >> >> > >> http://lists.xenproject.org/archives/html/xen-devel/2016-
> >> 05/msg01896.
> >> >> >> > >> html It is not clear where the problem is, and not seeing
> >> >> >> > >> the issue myself makes it hard to analyze. In any event
> >> >> >> > >> this quite likely is a regression.
> >> >> >> > >>
> >> >> >> > >
> >> >> >> > > My reading of that email thread and all relevant links
> >> >> >> > > (including the KVM bug report) is that there is a
> >> >> >> > > regression vf driver,
> >> but not in Xen.
> >> >> >> >
> >> >> >> > Just from reading that I would tend to agree. But the report
> >> >> >> > here is about Win2K8.
> >> >> >>
> >> >> >> Do you know which commit is a regression one? I try to find out
> >> >> >> the
> >> >> > regression commit.  That may be helpful to find out the root cause.
> >> >> >>
> >> >> >> Btw, some feedback from QA team, rhel 6.4 VM  doesn't work, but
> >> >> >> rhel 7.2 VM
> >> > does.
> >> >> >
> >> >> > Isn't this at least an indication that the guest could be buggy here?
> >> >> > It could also be both the hypervsior and guest have bugs. But
> >> >> > we're just not sure at this point.
> >> >>
> >> >> Indeed, and (with the many fixes that went in already) I really
> >> >> suspect a combination of both, or some of the involved hypervisor
> >> >> changes having unmasked some guest issue. Regardless, I'm afraid
> >> >> this ought to be treated as a blocker for the release at least
> >> >> until we understand what the issue is. But otoh making it a
> >> >> blocker probably makes sense only if we can expect progress (which
> >> >> we haven't really made for quite long a time).
> >> >>
> >> >
> >> > This issue is on my list, but the information gathered so far isn't
> >> > convincing enough to make it a blocker.
> >> >
> >> > And yes, we need meaningful progress to make it a blocker. To make
> >> > it so, commitment from various parties is needed. Let's start with
> >> > setting out things to look at, who is going to investigate what,
> >> > and a possible timeline for each item.
> >> >
> >> > Jan, can you come up with a list of what sort of information you need?
> >>
> >> Well, I had hoped to avoid that. But now that you ask for it,
> >> providing an
> > initial
> >> debugging patch seems better than a description which may get
> >> misunderstood. Attached both a hypervisor and a qemu patch. Their
> >> plus debug key M and i output is what I'd like to start with.
> >>
> >
> >
> >  I will try these 2 patches.
> >
> > btw. I read the internal Bugzilla carefully. I found that vf is
> > working for
> > win2k8  at  '2014-12-01 14:32:09 EST', but the bug still exi

Re: [Xen-devel] [BUG] win2008 guest cannot get ip through sriov

[Xu, Quan]

On June 02, 2016 8:09 PM, Jan Beulich <jbeul...@suse.com> wrote:
> >>> On 02.06.16 at 13:03, <wei.l...@citrix.com> wrote:
> > On Thu, Jun 02, 2016 at 04:38:47AM -0600, Jan Beulich wrote:
> >> >>> On 02.06.16 at 12:22, <wei.l...@citrix.com> wrote:
> >> > On Thu, Jun 02, 2016 at 07:31:06AM +, Xu, Quan wrote:
> >> >> On May 27, 2016 10:06 PM, Jan Beulich <jbeul...@suse.com> wrote:
> >> >> > >>> On 27.05.16 at 15:34, <wei.l...@citrix.com> wrote:
> >> >> > > On Fri, May 27, 2016 at 06:16:30AM -0600, Jan Beulich wrote:
> >> >> > >> >>> On 27.05.16 at 12:39, <wei.l...@citrix.com> wrote:
> >> >> > >> > Is this a regression? Does it work on previous versions of Xen?
> >> >> > >>
> >> >> > >> I think this is what was already reported by other Intel
> >> >> > >> people, see e.g. Quan's most recent reply:
> >> >> > >> http://lists.xenproject.org/archives/html/xen-devel/2016-
> 05/msg01896.
> >> >> > >> html It is not clear where the problem is, and not seeing the
> >> >> > >> issue myself makes it hard to analyze. In any event this
> >> >> > >> quite likely is a regression.
> >> >> > >>
> >> >> > >
> >> >> > > My reading of that email thread and all relevant links
> >> >> > > (including the KVM bug report) is that there is a regression vf 
> >> >> > > driver,
> but not in Xen.
> >> >> >
> >> >> > Just from reading that I would tend to agree. But the report
> >> >> > here is about Win2K8.
> >> >>
> >> >> Do you know which commit is a regression one? I try to find out
> >> >> the
> >> > regression commit.  That may be helpful to find out the root cause.
> >> >>
> >> >> Btw, some feedback from QA team, rhel 6.4 VM  doesn't work, but
> >> >> rhel 7.2 VM
> > does.
> >> >
> >> > Isn't this at least an indication that the guest could be buggy here?
> >> > It could also be both the hypervsior and guest have bugs. But we're
> >> > just not sure at this point.
> >>
> >> Indeed, and (with the many fixes that went in already) I really
> >> suspect a combination of both, or some of the involved hypervisor
> >> changes having unmasked some guest issue. Regardless, I'm afraid this
> >> ought to be treated as a blocker for the release at least until we
> >> understand what the issue is. But otoh making it a blocker probably
> >> makes sense only if we can expect progress (which we haven't really
> >> made for quite long a time).
> >>
> >
> > This issue is on my list, but the information gathered so far isn't
> > convincing enough to make it a blocker.
> >
> > And yes, we need meaningful progress to make it a blocker. To make it
> > so, commitment from various parties is needed. Let's start with
> > setting out things to look at, who is going to investigate what, and a
> > possible timeline for each item.
> >
> > Jan, can you come up with a list of what sort of information you need?
> 
> Well, I had hoped to avoid that. But now that you ask for it, providing an 
> initial
> debugging patch seems better than a description which may get
> misunderstood. Attached both a hypervisor and a qemu patch. Their plus
> debug key M and i output is what I'd like to start with.
> 


 I will try these 2 patches.

btw. I read the internal Bugzilla carefully. I found that vf is working for 
win2k8  at  '2014-12-01 14:32:09 EST', but the bug still exist on ' 2015-02-11 
15:54:05 EST '.
then, I grepped the commit logs, the below 4 MSI-X related commits are may the 
root cause.


From 6fb3a07bc0ad656b5f76eb9fc961bcd1d3cace58 Mon Sep 17 00:00:00 2001
From: Jan Beulich <jbeul...@suse.com>
Date: Fri, 12 Dec 2014 10:24:13 +
Subject: [PATCH 13/44] domctl: fix IRQ permission granting/revocation


From 1965728cd5a1635859158f5800d844fc16774668 Mon Sep 17 00:00:00 2001
From: Konrad Rzeszutek Wilk <konrad.w...@oracle.com>
Date: Mon, 12 Jan 2015 11:29:33 -0500
Subject: [PATCH 42/44] Revert "dpci: add 'masked' as a gate for
 hvm_dirq_assist to process"


From 72f3c1e26e96686a41d2de1663e578538659f99a Mon Sep 17 00:00:00 2001
From: Konrad Rzeszutek Wilk <konrad.w...@oracle.com>
Date: Mon, 12 Jan 2015 11:30:00 -0500
Subject: [PATCH 43/44] Revert "dpci: replace tasklet with softirq"

rom a8ac2290ed95dbbc0dc1bdde86fc3a49fe784b28 Mon Sep 17 00:00:00 2001
From: Konrad Rzeszutek Wilk <konrad.w...@oracle.com>
Date: Mon, 12 Jan 2015 11:30:05 -0500
Subject: [PATCH 44/44] Revert "dpci: move from an hvm_irq_dpci (and struct
 domain) to an hvm_dirq_dpci model"

Quan


> Jan
> 
> > And then maybe Quan and Pengtao can give an estimation on how long it
> > takes to gather all necessary information and move on to next stage.
> >
> > Wei.
> >
> >> Jan
> >>
> 
> 


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [Patch v6 04/11] IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU mapping (top level ones)

[Xu, Quan]

On June 02, 2016 5:21 PM, Jan Beulich  wrote:
> I really don't know what else I
> could have done to make clear what exceptions are to be expected.

Sorry,  it may make you frustrated, but what you said is very clear. I really 
need to slow down and read your email along with code context carefully.
Maybe I want to save time for that sriov PT bug. Sorry again.


Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel