Re: [PATCH kernel 9/9] KVM: PPC: Add in-kernel acceleration for VFIO

[Alex Williamson]

On Wed, 14 Dec 2016 14:53:13 +1100
Alexey Kardashevskiy  wrote:

> On 10/12/16 02:35, Alex Williamson wrote:
> > On Fri, 9 Dec 2016 18:53:43 +1100
> > Alexey Kardashevskiy  wrote:
> >   
> >> On 09/12/16 04:55, Alex Williamson wrote:  
> >>> On Thu,  8 Dec 2016 19:19:56 +1100
> >>> Alexey Kardashevskiy  wrote:
> >>> 
>  This allows the host kernel to handle H_PUT_TCE, H_PUT_TCE_INDIRECT
>  and H_STUFF_TCE requests targeted an IOMMU TCE table used for VFIO
>  without passing them to user space which saves time on switching
>  to user space and back.
> 
>  This adds H_PUT_TCE/H_PUT_TCE_INDIRECT/H_STUFF_TCE handlers to KVM.
>  KVM tries to handle a TCE request in the real mode, if failed
>  it passes the request to the virtual mode to complete the operation.
>  If it a virtual mode handler fails, the request is passed to
>  the user space; this is not expected to happen though.
> 
>  To avoid dealing with page use counters (which is tricky in real mode),
>  this only accelerates SPAPR TCE IOMMU v2 clients which are required
>  to pre-register the userspace memory. The very first TCE request will
>  be handled in the VFIO SPAPR TCE driver anyway as the userspace view
>  of the TCE table (iommu_table::it_userspace) is not allocated till
>  the very first mapping happens and we cannot call vmalloc in real mode.
> 
>  This adds new attribute - KVM_DEV_VFIO_GROUP_SET_SPAPR_TCE - to
>  the VFIO KVM device. It takes a VFIO group fd and SPAPR TCE table fd
>  and associates a physical IOMMU table with the SPAPR TCE table (which
>  is a guest view of the hardware IOMMU table). The iommu_table object
>  is referenced so we do not have to retrieve in real mode when hypercall
>  happens.
> 
>  This does not implement the UNSET counterpart as there is no use for it -
>  once the acceleration is enabled, the existing userspace won't
>  disable it unless a VFIO container is detroyed so this adds necessary
>  cleanup to the KVM_DEV_VFIO_GROUP_DEL handler.
> 
>  This uses the kvm->lock mutex to protect against a race between
>  the VFIO KVM device's kvm_vfio_destroy() and SPAPR TCE table fd's
>  release() callback.
> 
>  This advertises the new KVM_CAP_SPAPR_TCE_VFIO capability to the user
>  space.
> 
>  This finally makes use of vfio_external_user_iommu_id() which was
>  introduced quite some time ago and was considered for removal.
> 
>  Tests show that this patch increases transmission speed from 220MB/s
>  to 750..1020MB/s on 10Gb network (Chelsea CXGB3 10Gb ethernet card).
> 
>  Signed-off-by: Alexey Kardashevskiy 
>  ---
>   Documentation/virtual/kvm/devices/vfio.txt |  21 +-
>   arch/powerpc/include/asm/kvm_host.h|   8 +
>   arch/powerpc/include/asm/kvm_ppc.h |   5 +
>   include/uapi/linux/kvm.h   |   8 +
>   arch/powerpc/kvm/book3s_64_vio.c   | 302 
>  +
>   arch/powerpc/kvm/book3s_64_vio_hv.c| 178 +
>   arch/powerpc/kvm/powerpc.c |   2 +
>   virt/kvm/vfio.c| 108 +++
>   8 files changed, 630 insertions(+), 2 deletions(-)
> 
>  diff --git a/Documentation/virtual/kvm/devices/vfio.txt 
>  b/Documentation/virtual/kvm/devices/vfio.txt
>  index ef51740c67ca..ddb5a6512ab3 100644
>  --- a/Documentation/virtual/kvm/devices/vfio.txt
>  +++ b/Documentation/virtual/kvm/devices/vfio.txt
>  @@ -16,7 +16,24 @@ Groups:
>   
>   KVM_DEV_VFIO_GROUP attributes:
> KVM_DEV_VFIO_GROUP_ADD: Add a VFIO group to VFIO-KVM device tracking
>  +kvm_device_attr.addr points to an int32_t file descriptor
>  +for the VFIO group.
> KVM_DEV_VFIO_GROUP_DEL: Remove a VFIO group from VFIO-KVM device 
>  tracking
>  +kvm_device_attr.addr points to an int32_t file descriptor
>  +for the VFIO group.
>  +  KVM_DEV_VFIO_GROUP_SET_SPAPR_TCE: attaches a guest visible TCE table
>  +allocated by sPAPR KVM.
>  +kvm_device_attr.addr points to a struct:
>   
>  -For each, kvm_device_attr.addr points to an int32_t file descriptor
>  -for the VFIO group.
>  +struct kvm_vfio_spapr_tce {
>  +__u32   argsz;
>  +__s32   groupfd;
>  +__s32   tablefd;
>  +__u8pad[4];
>  +};
>  +
>  +where
>  +@argsz is the size of kvm_vfio_spapr_tce_liobn;
>  +@groupfd is a file descriptor for a VFIO group;
>  +@tablefd is a file descriptor for a TCE table allocated via
>  +KVM_CREATE_SPAPR_TCE.
>  diff --git 

Re: [PATCH kernel 9/9] KVM: PPC: Add in-kernel acceleration for VFIO

[Alexey Kardashevskiy]

On 10/12/16 02:35, Alex Williamson wrote:
> On Fri, 9 Dec 2016 18:53:43 +1100
> Alexey Kardashevskiy  wrote:
> 
>> On 09/12/16 04:55, Alex Williamson wrote:
>>> On Thu,  8 Dec 2016 19:19:56 +1100
>>> Alexey Kardashevskiy  wrote:
>>>   
 This allows the host kernel to handle H_PUT_TCE, H_PUT_TCE_INDIRECT
 and H_STUFF_TCE requests targeted an IOMMU TCE table used for VFIO
 without passing them to user space which saves time on switching
 to user space and back.

 This adds H_PUT_TCE/H_PUT_TCE_INDIRECT/H_STUFF_TCE handlers to KVM.
 KVM tries to handle a TCE request in the real mode, if failed
 it passes the request to the virtual mode to complete the operation.
 If it a virtual mode handler fails, the request is passed to
 the user space; this is not expected to happen though.

 To avoid dealing with page use counters (which is tricky in real mode),
 this only accelerates SPAPR TCE IOMMU v2 clients which are required
 to pre-register the userspace memory. The very first TCE request will
 be handled in the VFIO SPAPR TCE driver anyway as the userspace view
 of the TCE table (iommu_table::it_userspace) is not allocated till
 the very first mapping happens and we cannot call vmalloc in real mode.

 This adds new attribute - KVM_DEV_VFIO_GROUP_SET_SPAPR_TCE - to
 the VFIO KVM device. It takes a VFIO group fd and SPAPR TCE table fd
 and associates a physical IOMMU table with the SPAPR TCE table (which
 is a guest view of the hardware IOMMU table). The iommu_table object
 is referenced so we do not have to retrieve in real mode when hypercall
 happens.

 This does not implement the UNSET counterpart as there is no use for it -
 once the acceleration is enabled, the existing userspace won't
 disable it unless a VFIO container is detroyed so this adds necessary
 cleanup to the KVM_DEV_VFIO_GROUP_DEL handler.

 This uses the kvm->lock mutex to protect against a race between
 the VFIO KVM device's kvm_vfio_destroy() and SPAPR TCE table fd's
 release() callback.

 This advertises the new KVM_CAP_SPAPR_TCE_VFIO capability to the user
 space.

 This finally makes use of vfio_external_user_iommu_id() which was
 introduced quite some time ago and was considered for removal.

 Tests show that this patch increases transmission speed from 220MB/s
 to 750..1020MB/s on 10Gb network (Chelsea CXGB3 10Gb ethernet card).

 Signed-off-by: Alexey Kardashevskiy 
 ---
  Documentation/virtual/kvm/devices/vfio.txt |  21 +-
  arch/powerpc/include/asm/kvm_host.h|   8 +
  arch/powerpc/include/asm/kvm_ppc.h |   5 +
  include/uapi/linux/kvm.h   |   8 +
  arch/powerpc/kvm/book3s_64_vio.c   | 302 
 +
  arch/powerpc/kvm/book3s_64_vio_hv.c| 178 +
  arch/powerpc/kvm/powerpc.c |   2 +
  virt/kvm/vfio.c| 108 +++
  8 files changed, 630 insertions(+), 2 deletions(-)

 diff --git a/Documentation/virtual/kvm/devices/vfio.txt 
 b/Documentation/virtual/kvm/devices/vfio.txt
 index ef51740c67ca..ddb5a6512ab3 100644
 --- a/Documentation/virtual/kvm/devices/vfio.txt
 +++ b/Documentation/virtual/kvm/devices/vfio.txt
 @@ -16,7 +16,24 @@ Groups:
  
  KVM_DEV_VFIO_GROUP attributes:
KVM_DEV_VFIO_GROUP_ADD: Add a VFIO group to VFIO-KVM device tracking
 +  kvm_device_attr.addr points to an int32_t file descriptor
 +  for the VFIO group.
KVM_DEV_VFIO_GROUP_DEL: Remove a VFIO group from VFIO-KVM device 
 tracking
 +  kvm_device_attr.addr points to an int32_t file descriptor
 +  for the VFIO group.
 +  KVM_DEV_VFIO_GROUP_SET_SPAPR_TCE: attaches a guest visible TCE table
 +  allocated by sPAPR KVM.
 +  kvm_device_attr.addr points to a struct:
  
 -For each, kvm_device_attr.addr points to an int32_t file descriptor
 -for the VFIO group.
 +  struct kvm_vfio_spapr_tce {
 +  __u32   argsz;
 +  __s32   groupfd;
 +  __s32   tablefd;
 +  __u8pad[4];
 +  };
 +
 +  where
 +  @argsz is the size of kvm_vfio_spapr_tce_liobn;
 +  @groupfd is a file descriptor for a VFIO group;
 +  @tablefd is a file descriptor for a TCE table allocated via
 +  KVM_CREATE_SPAPR_TCE.
 diff --git a/arch/powerpc/include/asm/kvm_host.h 
 b/arch/powerpc/include/asm/kvm_host.h
 index 28350a294b1e..94774503c70d 100644
 --- a/arch/powerpc/include/asm/kvm_host.h
 +++ b/arch/powerpc/include/asm/kvm_host.h
 @@ -191,6 +191,13 @@ struct kvmppc_pginfo {
atomic_t refcnt;
  };
  
 +struct kvmppc_spapr_tce_iommu_table {
 +  struct rcu_head rcu;
 +  struct list_head next;
 + 

Re: [PATCH kernel 9/9] KVM: PPC: Add in-kernel acceleration for VFIO

[Alex Williamson]

On Fri, 9 Dec 2016 18:53:43 +1100
Alexey Kardashevskiy  wrote:

> On 09/12/16 04:55, Alex Williamson wrote:
> > On Thu,  8 Dec 2016 19:19:56 +1100
> > Alexey Kardashevskiy  wrote:
> >   
> >> This allows the host kernel to handle H_PUT_TCE, H_PUT_TCE_INDIRECT
> >> and H_STUFF_TCE requests targeted an IOMMU TCE table used for VFIO
> >> without passing them to user space which saves time on switching
> >> to user space and back.
> >>
> >> This adds H_PUT_TCE/H_PUT_TCE_INDIRECT/H_STUFF_TCE handlers to KVM.
> >> KVM tries to handle a TCE request in the real mode, if failed
> >> it passes the request to the virtual mode to complete the operation.
> >> If it a virtual mode handler fails, the request is passed to
> >> the user space; this is not expected to happen though.
> >>
> >> To avoid dealing with page use counters (which is tricky in real mode),
> >> this only accelerates SPAPR TCE IOMMU v2 clients which are required
> >> to pre-register the userspace memory. The very first TCE request will
> >> be handled in the VFIO SPAPR TCE driver anyway as the userspace view
> >> of the TCE table (iommu_table::it_userspace) is not allocated till
> >> the very first mapping happens and we cannot call vmalloc in real mode.
> >>
> >> This adds new attribute - KVM_DEV_VFIO_GROUP_SET_SPAPR_TCE - to
> >> the VFIO KVM device. It takes a VFIO group fd and SPAPR TCE table fd
> >> and associates a physical IOMMU table with the SPAPR TCE table (which
> >> is a guest view of the hardware IOMMU table). The iommu_table object
> >> is referenced so we do not have to retrieve in real mode when hypercall
> >> happens.
> >>
> >> This does not implement the UNSET counterpart as there is no use for it -
> >> once the acceleration is enabled, the existing userspace won't
> >> disable it unless a VFIO container is detroyed so this adds necessary
> >> cleanup to the KVM_DEV_VFIO_GROUP_DEL handler.
> >>
> >> This uses the kvm->lock mutex to protect against a race between
> >> the VFIO KVM device's kvm_vfio_destroy() and SPAPR TCE table fd's
> >> release() callback.
> >>
> >> This advertises the new KVM_CAP_SPAPR_TCE_VFIO capability to the user
> >> space.
> >>
> >> This finally makes use of vfio_external_user_iommu_id() which was
> >> introduced quite some time ago and was considered for removal.
> >>
> >> Tests show that this patch increases transmission speed from 220MB/s
> >> to 750..1020MB/s on 10Gb network (Chelsea CXGB3 10Gb ethernet card).
> >>
> >> Signed-off-by: Alexey Kardashevskiy 
> >> ---
> >>  Documentation/virtual/kvm/devices/vfio.txt |  21 +-
> >>  arch/powerpc/include/asm/kvm_host.h|   8 +
> >>  arch/powerpc/include/asm/kvm_ppc.h |   5 +
> >>  include/uapi/linux/kvm.h   |   8 +
> >>  arch/powerpc/kvm/book3s_64_vio.c   | 302 
> >> +
> >>  arch/powerpc/kvm/book3s_64_vio_hv.c| 178 +
> >>  arch/powerpc/kvm/powerpc.c |   2 +
> >>  virt/kvm/vfio.c| 108 +++
> >>  8 files changed, 630 insertions(+), 2 deletions(-)
> >>
> >> diff --git a/Documentation/virtual/kvm/devices/vfio.txt 
> >> b/Documentation/virtual/kvm/devices/vfio.txt
> >> index ef51740c67ca..ddb5a6512ab3 100644
> >> --- a/Documentation/virtual/kvm/devices/vfio.txt
> >> +++ b/Documentation/virtual/kvm/devices/vfio.txt
> >> @@ -16,7 +16,24 @@ Groups:
> >>  
> >>  KVM_DEV_VFIO_GROUP attributes:
> >>KVM_DEV_VFIO_GROUP_ADD: Add a VFIO group to VFIO-KVM device tracking
> >> +  kvm_device_attr.addr points to an int32_t file descriptor
> >> +  for the VFIO group.
> >>KVM_DEV_VFIO_GROUP_DEL: Remove a VFIO group from VFIO-KVM device 
> >> tracking
> >> +  kvm_device_attr.addr points to an int32_t file descriptor
> >> +  for the VFIO group.
> >> +  KVM_DEV_VFIO_GROUP_SET_SPAPR_TCE: attaches a guest visible TCE table
> >> +  allocated by sPAPR KVM.
> >> +  kvm_device_attr.addr points to a struct:
> >>  
> >> -For each, kvm_device_attr.addr points to an int32_t file descriptor
> >> -for the VFIO group.
> >> +  struct kvm_vfio_spapr_tce {
> >> +  __u32   argsz;
> >> +  __s32   groupfd;
> >> +  __s32   tablefd;
> >> +  __u8pad[4];
> >> +  };
> >> +
> >> +  where
> >> +  @argsz is the size of kvm_vfio_spapr_tce_liobn;
> >> +  @groupfd is a file descriptor for a VFIO group;
> >> +  @tablefd is a file descriptor for a TCE table allocated via
> >> +  KVM_CREATE_SPAPR_TCE.
> >> diff --git a/arch/powerpc/include/asm/kvm_host.h 
> >> b/arch/powerpc/include/asm/kvm_host.h
> >> index 28350a294b1e..94774503c70d 100644
> >> --- a/arch/powerpc/include/asm/kvm_host.h
> >> +++ b/arch/powerpc/include/asm/kvm_host.h
> >> @@ -191,6 +191,13 @@ struct kvmppc_pginfo {
> >>atomic_t refcnt;
> >>  };
> >>  
> >> +struct kvmppc_spapr_tce_iommu_table {
> >> +  struct rcu_head rcu;
> >> +  struct list_head next;
> >> +  struct iommu_table *tbl;
> >> +  atomic_t refs;

Re: [PATCH kernel 9/9] KVM: PPC: Add in-kernel acceleration for VFIO

[Alexey Kardashevskiy]

On 09/12/16 04:55, Alex Williamson wrote:
> On Thu,  8 Dec 2016 19:19:56 +1100
> Alexey Kardashevskiy  wrote:
> 
>> This allows the host kernel to handle H_PUT_TCE, H_PUT_TCE_INDIRECT
>> and H_STUFF_TCE requests targeted an IOMMU TCE table used for VFIO
>> without passing them to user space which saves time on switching
>> to user space and back.
>>
>> This adds H_PUT_TCE/H_PUT_TCE_INDIRECT/H_STUFF_TCE handlers to KVM.
>> KVM tries to handle a TCE request in the real mode, if failed
>> it passes the request to the virtual mode to complete the operation.
>> If it a virtual mode handler fails, the request is passed to
>> the user space; this is not expected to happen though.
>>
>> To avoid dealing with page use counters (which is tricky in real mode),
>> this only accelerates SPAPR TCE IOMMU v2 clients which are required
>> to pre-register the userspace memory. The very first TCE request will
>> be handled in the VFIO SPAPR TCE driver anyway as the userspace view
>> of the TCE table (iommu_table::it_userspace) is not allocated till
>> the very first mapping happens and we cannot call vmalloc in real mode.
>>
>> This adds new attribute - KVM_DEV_VFIO_GROUP_SET_SPAPR_TCE - to
>> the VFIO KVM device. It takes a VFIO group fd and SPAPR TCE table fd
>> and associates a physical IOMMU table with the SPAPR TCE table (which
>> is a guest view of the hardware IOMMU table). The iommu_table object
>> is referenced so we do not have to retrieve in real mode when hypercall
>> happens.
>>
>> This does not implement the UNSET counterpart as there is no use for it -
>> once the acceleration is enabled, the existing userspace won't
>> disable it unless a VFIO container is detroyed so this adds necessary
>> cleanup to the KVM_DEV_VFIO_GROUP_DEL handler.
>>
>> This uses the kvm->lock mutex to protect against a race between
>> the VFIO KVM device's kvm_vfio_destroy() and SPAPR TCE table fd's
>> release() callback.
>>
>> This advertises the new KVM_CAP_SPAPR_TCE_VFIO capability to the user
>> space.
>>
>> This finally makes use of vfio_external_user_iommu_id() which was
>> introduced quite some time ago and was considered for removal.
>>
>> Tests show that this patch increases transmission speed from 220MB/s
>> to 750..1020MB/s on 10Gb network (Chelsea CXGB3 10Gb ethernet card).
>>
>> Signed-off-by: Alexey Kardashevskiy 
>> ---
>>  Documentation/virtual/kvm/devices/vfio.txt |  21 +-
>>  arch/powerpc/include/asm/kvm_host.h|   8 +
>>  arch/powerpc/include/asm/kvm_ppc.h |   5 +
>>  include/uapi/linux/kvm.h   |   8 +
>>  arch/powerpc/kvm/book3s_64_vio.c   | 302 
>> +
>>  arch/powerpc/kvm/book3s_64_vio_hv.c| 178 +
>>  arch/powerpc/kvm/powerpc.c |   2 +
>>  virt/kvm/vfio.c| 108 +++
>>  8 files changed, 630 insertions(+), 2 deletions(-)
>>
>> diff --git a/Documentation/virtual/kvm/devices/vfio.txt 
>> b/Documentation/virtual/kvm/devices/vfio.txt
>> index ef51740c67ca..ddb5a6512ab3 100644
>> --- a/Documentation/virtual/kvm/devices/vfio.txt
>> +++ b/Documentation/virtual/kvm/devices/vfio.txt
>> @@ -16,7 +16,24 @@ Groups:
>>  
>>  KVM_DEV_VFIO_GROUP attributes:
>>KVM_DEV_VFIO_GROUP_ADD: Add a VFIO group to VFIO-KVM device tracking
>> +kvm_device_attr.addr points to an int32_t file descriptor
>> +for the VFIO group.
>>KVM_DEV_VFIO_GROUP_DEL: Remove a VFIO group from VFIO-KVM device tracking
>> +kvm_device_attr.addr points to an int32_t file descriptor
>> +for the VFIO group.
>> +  KVM_DEV_VFIO_GROUP_SET_SPAPR_TCE: attaches a guest visible TCE table
>> +allocated by sPAPR KVM.
>> +kvm_device_attr.addr points to a struct:
>>  
>> -For each, kvm_device_attr.addr points to an int32_t file descriptor
>> -for the VFIO group.
>> +struct kvm_vfio_spapr_tce {
>> +__u32   argsz;
>> +__s32   groupfd;
>> +__s32   tablefd;
>> +__u8pad[4];
>> +};
>> +
>> +where
>> +@argsz is the size of kvm_vfio_spapr_tce_liobn;
>> +@groupfd is a file descriptor for a VFIO group;
>> +@tablefd is a file descriptor for a TCE table allocated via
>> +KVM_CREATE_SPAPR_TCE.
>> diff --git a/arch/powerpc/include/asm/kvm_host.h 
>> b/arch/powerpc/include/asm/kvm_host.h
>> index 28350a294b1e..94774503c70d 100644
>> --- a/arch/powerpc/include/asm/kvm_host.h
>> +++ b/arch/powerpc/include/asm/kvm_host.h
>> @@ -191,6 +191,13 @@ struct kvmppc_pginfo {
>>  atomic_t refcnt;
>>  };
>>  
>> +struct kvmppc_spapr_tce_iommu_table {
>> +struct rcu_head rcu;
>> +struct list_head next;
>> +struct iommu_table *tbl;
>> +atomic_t refs;
>> +};
>> +
>>  struct kvmppc_spapr_tce_table {
>>  struct list_head list;
>>  struct kvm *kvm;
>> @@ -199,6 +206,7 @@ struct kvmppc_spapr_tce_table {
>>  u32 page_shift;
>>  u64 offset; /* in pages */
>>  u64 size; 

Re: [PATCH kernel 9/9] KVM: PPC: Add in-kernel acceleration for VFIO

[Alex Williamson]

On Thu,  8 Dec 2016 19:19:56 +1100
Alexey Kardashevskiy  wrote:

> This allows the host kernel to handle H_PUT_TCE, H_PUT_TCE_INDIRECT
> and H_STUFF_TCE requests targeted an IOMMU TCE table used for VFIO
> without passing them to user space which saves time on switching
> to user space and back.
> 
> This adds H_PUT_TCE/H_PUT_TCE_INDIRECT/H_STUFF_TCE handlers to KVM.
> KVM tries to handle a TCE request in the real mode, if failed
> it passes the request to the virtual mode to complete the operation.
> If it a virtual mode handler fails, the request is passed to
> the user space; this is not expected to happen though.
> 
> To avoid dealing with page use counters (which is tricky in real mode),
> this only accelerates SPAPR TCE IOMMU v2 clients which are required
> to pre-register the userspace memory. The very first TCE request will
> be handled in the VFIO SPAPR TCE driver anyway as the userspace view
> of the TCE table (iommu_table::it_userspace) is not allocated till
> the very first mapping happens and we cannot call vmalloc in real mode.
> 
> This adds new attribute - KVM_DEV_VFIO_GROUP_SET_SPAPR_TCE - to
> the VFIO KVM device. It takes a VFIO group fd and SPAPR TCE table fd
> and associates a physical IOMMU table with the SPAPR TCE table (which
> is a guest view of the hardware IOMMU table). The iommu_table object
> is referenced so we do not have to retrieve in real mode when hypercall
> happens.
> 
> This does not implement the UNSET counterpart as there is no use for it -
> once the acceleration is enabled, the existing userspace won't
> disable it unless a VFIO container is detroyed so this adds necessary
> cleanup to the KVM_DEV_VFIO_GROUP_DEL handler.
> 
> This uses the kvm->lock mutex to protect against a race between
> the VFIO KVM device's kvm_vfio_destroy() and SPAPR TCE table fd's
> release() callback.
> 
> This advertises the new KVM_CAP_SPAPR_TCE_VFIO capability to the user
> space.
> 
> This finally makes use of vfio_external_user_iommu_id() which was
> introduced quite some time ago and was considered for removal.
> 
> Tests show that this patch increases transmission speed from 220MB/s
> to 750..1020MB/s on 10Gb network (Chelsea CXGB3 10Gb ethernet card).
> 
> Signed-off-by: Alexey Kardashevskiy 
> ---
>  Documentation/virtual/kvm/devices/vfio.txt |  21 +-
>  arch/powerpc/include/asm/kvm_host.h|   8 +
>  arch/powerpc/include/asm/kvm_ppc.h |   5 +
>  include/uapi/linux/kvm.h   |   8 +
>  arch/powerpc/kvm/book3s_64_vio.c   | 302 
> +
>  arch/powerpc/kvm/book3s_64_vio_hv.c| 178 +
>  arch/powerpc/kvm/powerpc.c |   2 +
>  virt/kvm/vfio.c| 108 +++
>  8 files changed, 630 insertions(+), 2 deletions(-)
> 
> diff --git a/Documentation/virtual/kvm/devices/vfio.txt 
> b/Documentation/virtual/kvm/devices/vfio.txt
> index ef51740c67ca..ddb5a6512ab3 100644
> --- a/Documentation/virtual/kvm/devices/vfio.txt
> +++ b/Documentation/virtual/kvm/devices/vfio.txt
> @@ -16,7 +16,24 @@ Groups:
>  
>  KVM_DEV_VFIO_GROUP attributes:
>KVM_DEV_VFIO_GROUP_ADD: Add a VFIO group to VFIO-KVM device tracking
> + kvm_device_attr.addr points to an int32_t file descriptor
> + for the VFIO group.
>KVM_DEV_VFIO_GROUP_DEL: Remove a VFIO group from VFIO-KVM device tracking
> + kvm_device_attr.addr points to an int32_t file descriptor
> + for the VFIO group.
> +  KVM_DEV_VFIO_GROUP_SET_SPAPR_TCE: attaches a guest visible TCE table
> + allocated by sPAPR KVM.
> + kvm_device_attr.addr points to a struct:
>  
> -For each, kvm_device_attr.addr points to an int32_t file descriptor
> -for the VFIO group.
> + struct kvm_vfio_spapr_tce {
> + __u32   argsz;
> + __s32   groupfd;
> + __s32   tablefd;
> + __u8pad[4];
> + };
> +
> + where
> + @argsz is the size of kvm_vfio_spapr_tce_liobn;
> + @groupfd is a file descriptor for a VFIO group;
> + @tablefd is a file descriptor for a TCE table allocated via
> + KVM_CREATE_SPAPR_TCE.
> diff --git a/arch/powerpc/include/asm/kvm_host.h 
> b/arch/powerpc/include/asm/kvm_host.h
> index 28350a294b1e..94774503c70d 100644
> --- a/arch/powerpc/include/asm/kvm_host.h
> +++ b/arch/powerpc/include/asm/kvm_host.h
> @@ -191,6 +191,13 @@ struct kvmppc_pginfo {
>   atomic_t refcnt;
>  };
>  
> +struct kvmppc_spapr_tce_iommu_table {
> + struct rcu_head rcu;
> + struct list_head next;
> + struct iommu_table *tbl;
> + atomic_t refs;
> +};
> +
>  struct kvmppc_spapr_tce_table {
>   struct list_head list;
>   struct kvm *kvm;
> @@ -199,6 +206,7 @@ struct kvmppc_spapr_tce_table {
>   u32 page_shift;
>   u64 offset; /* in pages */
>   u64 size;   /* window size in pages */
> + struct list_head iommu_tables;
>   struct page *pages[0];
>  };
>  
> diff