Re: [PATCH 2/2] armv7 initial device passthrough support

2013-06-25 Thread Mario Smarduch 
On 6/24/2013 10:01 PM, Christoffer Dall wrote:


 There are many other latency/perf. reqs for NFV related to RT,
 essentially Guest must run near native. In the end it may turn out this
 may need to be outside of main tree we'll see.

 It doesn't sound like this will be the end result.  Everything that you
 try to do in your patch set can be accomplished using VFIO and a more
 generic infrastructure for virtual IRQ integration with KVM and user
 space.

I mentioned in previous email we will pursue VFIO, but even
at that VFIO is a starting point for NFV.
 
 We should avoid creating an environment with important functionality
 outside of the main tree, if at all possible.

Of course that would be ideal but with NFV it may be more involved. 
This is similar Linux and TEM adaption  around 04/05. We  wanted 
to adapt Linux but it lacked required features that's when CGL specifications 
came into play to provide  guidance a lot of features (TIPC, OpenIMPI, 
preempt_rt, AEM) lived outside  mainline, supported by OS vendors delivering 
CGL compliant distro, while others decided to stick with IT, penetrating
some applications like HLR.

With NFV a likely scenario may evolve, TEMs need to start demonstrating
to operators fixed and wireless virtualization use cases. The only
significant difference is that unlike CGL for Linux, KVM has nor real
representation and understanding of NFV reqs (as opposed to proprietary 
vendors). 
I can't speak  for all TEMs but it's likely they will go off on their own
to demo/proto-type and worry about Open Source acceptance later.

 
 -Christoffer
 


--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 2/2] armv7 initial device passthrough support

2013-06-25 Thread Mario Smarduch 
On 6/25/2013 12:27 AM, Stuart Yoder wrote:


 We should avoid creating an environment with important functionality
 outside of the main tree, if at all possible.
 
 Also, as we architect that generic infrastructure we need to keep in mind that
 there are important use cases for doing I/O in user space that are not
 KVM guests-- just normal applications that need direct device
 access.

Yes that's a good point especially data plane NE, also LTE has 
these use cases at the radio side.

 
 Stuart
 


--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 2/2] armv7 initial device passthrough support

2013-06-24 Thread Mario Smarduch 


On 6/15/2013 5:47 PM, Paolo Bonzini wrote:
 Il 13/06/2013 11:19, Mario Smarduch ha scritto:
 Updated Device Passthrough Patch.
 - optimized IRQ-CPU-vCPU binding, irq is installed once
 - added dynamic IRQ affinity on schedule in
 - added documentation and few other coding recommendations.

 Per earlier discussion VFIO is our target but we like
 something earlier to work with to tackle performance
 latency issue (some ARM related) for device passthrough 
 while we migrate towards VFIO.
 
 I don't think this is acceptable upstream, unfortunately.  KVM device
 assignment is deprecated and we should not add more users.
That's fine we'll work our way towards dev-tree VFIO reusing what we can
working with the community.

At this point we're more concerned with numbers and best practices as 
opposed to mechanism this part will be time consuming. 
VFIO will be more background for us.

 
 What are the latency issues you have?

Our focus now is on IRQ latency and throughput. Right now it appears lowest 
latency
is 2x + exit/enter + IRQ injection overhead. We can't tolerate additional 
IPIs or deferred IRQ injection approaches. We're looking for numbers closer
to what IBMs ELI managed. Also high res timers which ARM Virt. Ext supports 
very well. Exitless interrupts which ARM handles very well too. There are
some future hw ARM interrupt enhancements coming up which may help a lot as 
well.

There are many other latency/perf. reqs for NFV related to RT,
essentially Guest must run near native. In the end it may turn out this
may need to be outside of main tree we'll see.

- Mario
 
 Paolo
 
 - Mario


--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 2/2] armv7 initial device passthrough support

2013-06-24 Thread Christoffer Dall 
On Mon, Jun 24, 2013 at 10:08:08AM +0200, Mario Smarduch wrote:
 
 
 On 6/15/2013 5:47 PM, Paolo Bonzini wrote:
  Il 13/06/2013 11:19, Mario Smarduch ha scritto:
  Updated Device Passthrough Patch.
  - optimized IRQ-CPU-vCPU binding, irq is installed once
  - added dynamic IRQ affinity on schedule in
  - added documentation and few other coding recommendations.
 
  Per earlier discussion VFIO is our target but we like
  something earlier to work with to tackle performance
  latency issue (some ARM related) for device passthrough 
  while we migrate towards VFIO.
  
  I don't think this is acceptable upstream, unfortunately.  KVM device
  assignment is deprecated and we should not add more users.
 That's fine we'll work our way towards dev-tree VFIO reusing what we can
 working with the community.
 
 At this point we're more concerned with numbers and best practices as 
 opposed to mechanism this part will be time consuming. 
 VFIO will be more background for us.
 
  
  What are the latency issues you have?
 
 Our focus now is on IRQ latency and throughput. Right now it appears lowest 
 latency
 is 2x + exit/enter + IRQ injection overhead. We can't tolerate additional 
 IPIs or deferred IRQ injection approaches. We're looking for numbers closer
 to what IBMs ELI managed. Also high res timers which ARM Virt. Ext supports 
 very well. Exitless interrupts which ARM handles very well too. There are
 some future hw ARM interrupt enhancements coming up which may help a lot as 
 well.
 
 There are many other latency/perf. reqs for NFV related to RT,
 essentially Guest must run near native. In the end it may turn out this
 may need to be outside of main tree we'll see.
 
It doesn't sound like this will be the end result.  Everything that you
try to do in your patch set can be accomplished using VFIO and a more
generic infrastructure for virtual IRQ integration with KVM and user
space.

We should avoid creating an environment with important functionality
outside of the main tree, if at all possible.

-Christoffer
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 2/2] armv7 initial device passthrough support

2013-06-24 Thread Stuart Yoder 
On Mon, Jun 24, 2013 at 3:01 PM, Christoffer Dall
christoffer.d...@linaro.org wrote:
 On Mon, Jun 24, 2013 at 10:08:08AM +0200, Mario Smarduch wrote:


 On 6/15/2013 5:47 PM, Paolo Bonzini wrote:
  Il 13/06/2013 11:19, Mario Smarduch ha scritto:
  Updated Device Passthrough Patch.
  - optimized IRQ-CPU-vCPU binding, irq is installed once
  - added dynamic IRQ affinity on schedule in
  - added documentation and few other coding recommendations.
 
  Per earlier discussion VFIO is our target but we like
  something earlier to work with to tackle performance
  latency issue (some ARM related) for device passthrough
  while we migrate towards VFIO.
 
  I don't think this is acceptable upstream, unfortunately.  KVM device
  assignment is deprecated and we should not add more users.
 That's fine we'll work our way towards dev-tree VFIO reusing what we can
 working with the community.

 At this point we're more concerned with numbers and best practices as
 opposed to mechanism this part will be time consuming.
 VFIO will be more background for us.

 
  What are the latency issues you have?

 Our focus now is on IRQ latency and throughput. Right now it appears lowest 
 latency
 is 2x + exit/enter + IRQ injection overhead. We can't tolerate additional
 IPIs or deferred IRQ injection approaches. We're looking for numbers closer
 to what IBMs ELI managed. Also high res timers which ARM Virt. Ext supports
 very well. Exitless interrupts which ARM handles very well too. There are
 some future hw ARM interrupt enhancements coming up which may help a lot as 
 well.

 There are many other latency/perf. reqs for NFV related to RT,
 essentially Guest must run near native. In the end it may turn out this
 may need to be outside of main tree we'll see.

 It doesn't sound like this will be the end result.  Everything that you
 try to do in your patch set can be accomplished using VFIO and a more
 generic infrastructure for virtual IRQ integration with KVM and user
 space.

 We should avoid creating an environment with important functionality
 outside of the main tree, if at all possible.

Also, as we architect that generic infrastructure we need to keep in mind that
there are important use cases for doing I/O in user space that are not
KVM guests-- just normal applications that need direct device
access.

Stuart
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 2/2] armv7 initial device passthrough support

2013-06-15 Thread Paolo Bonzini 
Il 13/06/2013 11:19, Mario Smarduch ha scritto:
 Updated Device Passthrough Patch.
 - optimized IRQ-CPU-vCPU binding, irq is installed once
 - added dynamic IRQ affinity on schedule in
 - added documentation and few other coding recommendations.
 
 Per earlier discussion VFIO is our target but we like
 something earlier to work with to tackle performance
 latency issue (some ARM related) for device passthrough 
 while we migrate towards VFIO.

I don't think this is acceptable upstream, unfortunately.  KVM device
assignment is deprecated and we should not add more users.

What are the latency issues you have?

Paolo

 - Mario
 
 
 Signed-off-by: Mario Smarduch mario.smard...@huawei.com
 ---
  arch/arm/include/asm/kvm_host.h |   31 +
  arch/arm/include/asm/kvm_vgic.h |   10 ++
  arch/arm/kvm/Makefile   |1 +
  arch/arm/kvm/arm.c  |   80 +
  arch/arm/kvm/assign-dev.c   |  248 
 +++
  arch/arm/kvm/vgic.c |  134 +
  include/linux/irqchip/arm-gic.h |1 +
  include/uapi/linux/kvm.h|   33 ++
  8 files changed, 538 insertions(+)
  create mode 100644 arch/arm/kvm/assign-dev.c
 
 diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
 index 57cb786..c85c3a0 100644
 --- a/arch/arm/include/asm/kvm_host.h
 +++ b/arch/arm/include/asm/kvm_host.h
 @@ -67,6 +67,10 @@ struct kvm_arch {
  
   /* Interrupt controller */
   struct vgic_distvgic;
 +
 + /* Device Passthrough Fields */
 + struct list_headassigned_dev_head;
 + struct mutexdev_passthrough_lock;
  };
  
  #define KVM_NR_MEM_OBJS 40
 @@ -146,6 +150,13 @@ struct kvm_vcpu_stat {
   u32 halt_wakeup;
  };
  
 +struct kvm_arm_assigned_dev_kernel {
 + struct list_head list;
 + struct kvm_arm_assigned_device dev;
 + irqreturn_t (*irq_handler)(int, void *);
 + unsigned long vcpuid_irq_arg;
 +};
 +
  struct kvm_vcpu_init;
  int kvm_vcpu_set_target(struct kvm_vcpu *vcpu,
   const struct kvm_vcpu_init *init);
 @@ -157,6 +168,26 @@ int kvm_arm_set_reg(struct kvm_vcpu *vcpu, const struct 
 kvm_one_reg *reg);
  u64 kvm_call_hyp(void *hypfn, ...);
  void force_vm_exit(const cpumask_t *mask);
  
 +#ifdef CONFIG_KVM_ARM_INT_PRIO_DROP
 +int kvm_arm_get_device_resources(struct kvm *,
 + struct kvm_arm_get_device_resources *);
 +int kvm_arm_assign_device(struct kvm *, struct kvm_arm_assigned_device *);
 +void kvm_arm_setdev_irq_affinity(struct kvm_vcpu *vcpu, int cpu);
 +#else
 +static inline int kvm_arm_get_device_resources(struct kvm *k, struct 
 kvm_arm_get_device_resources *r)
 +{
 + return -1;
 +}
 +static inline int kvm_arm_assign_device(struct kvm *k, struct 
 kvm_arm_assigned_device *d)
 +{
 + return -1;
 +}
 +
 +static inline void kvm_arm_setdev_irq_affinity(struct kvm_vcpu *vcpu, int 
 cpu)
 +{
 +}
 +#endif
 +
  #define KVM_ARCH_WANT_MMU_NOTIFIER
  struct kvm;
  int kvm_unmap_hva(struct kvm *kvm, unsigned long hva);
 diff --git a/arch/arm/include/asm/kvm_vgic.h b/arch/arm/include/asm/kvm_vgic.h
 index 343744e..fb6afd2 100644
 --- a/arch/arm/include/asm/kvm_vgic.h
 +++ b/arch/arm/include/asm/kvm_vgic.h
 @@ -107,6 +107,16 @@ struct vgic_dist {
  
   /* Bitmap indicating which CPU has something pending */
   unsigned long   irq_pending_on_cpu;
 +
 + /* Device passthrough  fields */
 + /* Host irq to guest irq mapping */
 + u8  guest_irq[VGIC_NR_SHARED_IRQS];
 +
 + /* Pending passthruogh irq */
 + struct vgic_bitmap  passthrough_spi_pending;
 +
 + /* At least one passthrough IRQ pending for some vCPU */
 + u32 passthrough_pending;
  #endif
  };
  
 diff --git a/arch/arm/kvm/Makefile b/arch/arm/kvm/Makefile
 index 53c5ed8..823fc38 100644
 --- a/arch/arm/kvm/Makefile
 +++ b/arch/arm/kvm/Makefile
 @@ -21,3 +21,4 @@ obj-y += arm.o handle_exit.o guest.o mmu.o emulate.o reset.o
  obj-y += coproc.o coproc_a15.o mmio.o psci.o perf.o
  obj-$(CONFIG_KVM_ARM_VGIC) += vgic.o
  obj-$(CONFIG_KVM_ARM_TIMER) += arch_timer.o
 +obj-$(CONFIG_KVM_ARM_INT_PRIO_DROP) += assign-dev.o
 diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
 index 37d216d..ba54c64 100644
 --- a/arch/arm/kvm/arm.c
 +++ b/arch/arm/kvm/arm.c
 @@ -26,6 +26,8 @@
  #include linux/mman.h
  #include linux/sched.h
  #include linux/kvm.h
 +#include linux/interrupt.h
 +#include linux/ioport.h
  #include trace/events/kvm.h
  
  #define CREATE_TRACE_POINTS
 @@ -43,6 +45,7 @@
  #include asm/kvm_emulate.h
  #include asm/kvm_coproc.h
  #include asm/kvm_psci.h
 +#include asm/kvm_host.h
  
  #ifdef REQUIRES_VIRT
  __asm__(.arch_extension virt);
 @@ -139,6 +142,11 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
  
   /* Mark the initial VMID generation invalid */
   kvm-arch.vmid_gen = 0;
 + /*
 +  * Initialize Dev Passthrough Fields
 +  */
 +

[PATCH 2/2] armv7 initial device passthrough support

2013-06-13 Thread Mario Smarduch 
Updated Device Passthrough Patch.
- optimized IRQ-CPU-vCPU binding, irq is installed once
- added dynamic IRQ affinity on schedule in
- added documentation and few other coding recommendations.

Per earlier discussion VFIO is our target but we like
something earlier to work with to tackle performance
latency issue (some ARM related) for device passthrough 
while we migrate towards VFIO.

- Mario


Signed-off-by: Mario Smarduch mario.smard...@huawei.com
---
 arch/arm/include/asm/kvm_host.h |   31 +
 arch/arm/include/asm/kvm_vgic.h |   10 ++
 arch/arm/kvm/Makefile   |1 +
 arch/arm/kvm/arm.c  |   80 +
 arch/arm/kvm/assign-dev.c   |  248 +++
 arch/arm/kvm/vgic.c |  134 +
 include/linux/irqchip/arm-gic.h |1 +
 include/uapi/linux/kvm.h|   33 ++
 8 files changed, 538 insertions(+)
 create mode 100644 arch/arm/kvm/assign-dev.c

diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
index 57cb786..c85c3a0 100644
--- a/arch/arm/include/asm/kvm_host.h
+++ b/arch/arm/include/asm/kvm_host.h
@@ -67,6 +67,10 @@ struct kvm_arch {
 
/* Interrupt controller */
struct vgic_distvgic;
+
+   /* Device Passthrough Fields */
+   struct list_headassigned_dev_head;
+   struct mutexdev_passthrough_lock;
 };
 
 #define KVM_NR_MEM_OBJS 40
@@ -146,6 +150,13 @@ struct kvm_vcpu_stat {
u32 halt_wakeup;
 };
 
+struct kvm_arm_assigned_dev_kernel {
+   struct list_head list;
+   struct kvm_arm_assigned_device dev;
+   irqreturn_t (*irq_handler)(int, void *);
+   unsigned long vcpuid_irq_arg;
+};
+
 struct kvm_vcpu_init;
 int kvm_vcpu_set_target(struct kvm_vcpu *vcpu,
const struct kvm_vcpu_init *init);
@@ -157,6 +168,26 @@ int kvm_arm_set_reg(struct kvm_vcpu *vcpu, const struct 
kvm_one_reg *reg);
 u64 kvm_call_hyp(void *hypfn, ...);
 void force_vm_exit(const cpumask_t *mask);
 
+#ifdef CONFIG_KVM_ARM_INT_PRIO_DROP
+int kvm_arm_get_device_resources(struct kvm *,
+   struct kvm_arm_get_device_resources *);
+int kvm_arm_assign_device(struct kvm *, struct kvm_arm_assigned_device *);
+void kvm_arm_setdev_irq_affinity(struct kvm_vcpu *vcpu, int cpu);
+#else
+static inline int kvm_arm_get_device_resources(struct kvm *k, struct 
kvm_arm_get_device_resources *r)
+{
+   return -1;
+}
+static inline int kvm_arm_assign_device(struct kvm *k, struct 
kvm_arm_assigned_device *d)
+{
+   return -1;
+}
+
+static inline void kvm_arm_setdev_irq_affinity(struct kvm_vcpu *vcpu, int cpu)
+{
+}
+#endif
+
 #define KVM_ARCH_WANT_MMU_NOTIFIER
 struct kvm;
 int kvm_unmap_hva(struct kvm *kvm, unsigned long hva);
diff --git a/arch/arm/include/asm/kvm_vgic.h b/arch/arm/include/asm/kvm_vgic.h
index 343744e..fb6afd2 100644
--- a/arch/arm/include/asm/kvm_vgic.h
+++ b/arch/arm/include/asm/kvm_vgic.h
@@ -107,6 +107,16 @@ struct vgic_dist {
 
/* Bitmap indicating which CPU has something pending */
unsigned long   irq_pending_on_cpu;
+
+   /* Device passthrough  fields */
+   /* Host irq to guest irq mapping */
+   u8  guest_irq[VGIC_NR_SHARED_IRQS];
+
+   /* Pending passthruogh irq */
+   struct vgic_bitmap  passthrough_spi_pending;
+
+   /* At least one passthrough IRQ pending for some vCPU */
+   u32 passthrough_pending;
 #endif
 };
 
diff --git a/arch/arm/kvm/Makefile b/arch/arm/kvm/Makefile
index 53c5ed8..823fc38 100644
--- a/arch/arm/kvm/Makefile
+++ b/arch/arm/kvm/Makefile
@@ -21,3 +21,4 @@ obj-y += arm.o handle_exit.o guest.o mmu.o emulate.o reset.o
 obj-y += coproc.o coproc_a15.o mmio.o psci.o perf.o
 obj-$(CONFIG_KVM_ARM_VGIC) += vgic.o
 obj-$(CONFIG_KVM_ARM_TIMER) += arch_timer.o
+obj-$(CONFIG_KVM_ARM_INT_PRIO_DROP) += assign-dev.o
diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
index 37d216d..ba54c64 100644
--- a/arch/arm/kvm/arm.c
+++ b/arch/arm/kvm/arm.c
@@ -26,6 +26,8 @@
 #include linux/mman.h
 #include linux/sched.h
 #include linux/kvm.h
+#include linux/interrupt.h
+#include linux/ioport.h
 #include trace/events/kvm.h
 
 #define CREATE_TRACE_POINTS
@@ -43,6 +45,7 @@
 #include asm/kvm_emulate.h
 #include asm/kvm_coproc.h
 #include asm/kvm_psci.h
+#include asm/kvm_host.h
 
 #ifdef REQUIRES_VIRT
 __asm__(.arch_extension   virt);
@@ -139,6 +142,11 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
 
/* Mark the initial VMID generation invalid */
kvm-arch.vmid_gen = 0;
+   /*
+* Initialize Dev Passthrough Fields
+*/
+   INIT_LIST_HEAD(kvm-arch.assigned_dev_head);
+   mutex_init(kvm-arch.dev_passthrough_lock);
 
return ret;
 out_free_stage2_pgd:
@@ -169,6 +177,40 @@ int kvm_arch_create_memslot(struct kvm_memory_slot *slot, 
unsigned long npages)
 void kvm_arch_destroy_vm(struct kvm *kvm)
 {
int i;
+   struct list_head