On 2017/07/27 20:53, Mooney, Sean K wrote:
-----Original Message-----
From: TETSURO NAKAMURA [mailto:nakamura.tets...@lab.ntt.co.jp]
Sent: Thursday, July 27, 2017 2:13 AM
To: Daniel P. Berrange <berra...@redhat.com>; Mooney, Sean K
<sean.k.moo...@intel.com>
Cc: Jay Pipes <jaypi...@gmail.com>; OpenStack Development Mailing List
(not for usage questions) <openstack-dev@lists.openstack.org>;
sfinu...@redhat.com; mriede...@gmail.com; 【社内】【ML】srv-apl-arch <srv-
apl-a...@lab.ntt.co.jp>
Subject: Re: [srv-apl-arch:7353] Re: [openstack-dev] [nova] Discussions
for ivshmem support in OpenStack Nova
On 2017/07/27 0:58, Daniel P. Berrange wrote:
On Wed, Jul 26, 2017 at 11:53:06AM -0400, Jay Pipes wrote:
On 07/26/2017 09:57 AM, Daniel P. Berrange wrote:
On Wed, Jul 26, 2017 at 09:50:23AM -0400, Jay Pipes wrote:
On 07/26/2017 03:06 AM, TETSURO NAKAMURA wrote:
Hi Nova team,
It has been quite a long time since the last discussion, but let
me make sure one thing about the thread below.
IIUC, Nova is not welcome ivshmem support because it is no longer
supported by DPDK+QEMU.
But how would you say if it is supported out of DPDK-tree and can
be used from the newest qemu version ?
We are now developing SPP, a DPDK-based vswitch, and thinking
about trying to implement ivshmem support under our SPP code tree
if nova (or at first libvirt community) is acceptable for ivshmem
configuration.
Your advice will be very helpful for our decision-making in our
project.
I think this is a question that the libvirt community would first
need to weigh in on since Nova is downstream from libvirt -- at
least in the sense of low-level hypervisor support.
Libvirt already supports ivshmem device config
http://libvirt.org/formatdomain.html#elementsShmem
Sorry, I suppose I should have said QEMU, not libvirt. Daniel, you
were the one that specifically discouraged doing anything on ivshmem
to Tetsuro:
http://lists.openstack.org/pipermail/openstack-dev/2017-
July/120136.h
tml
'ivshmem' was the original device in QEMU and that is indeed still
deprecated.
There are now two replacements 'ivshmem-plain' and 'ivshmem-doorbell'
which can be used instead, which are considered supported by QEMU,
though most people will still recommend using 'vhostuser' instead if
the use of ivshmem is at all network related.
Regards,
Daniel
Thank you very much for the information about current status of ivshmem
in QEMU.
I now understand that 'ivshmem', 'ivshmem-plain' and 'ivshmem-doorbell'
are different solutions, and libvirt already supports the latter two.
+ Mr. Sean Mooney
Did you mean that you caution against building new solutions ontop of
'ivshmem' or ontop of 'ivshmem-plain' and 'ivshmem-doorbell' too?
[Mooney, Sean K] I would caution against building any networking based uscases
ontop of ivshmem.
The move to using a memdev instead of directly specifying shm args to qemu for
ivshmem-plain/doorbell
Should allow hugepage memory to be used instead of posix sheared mem which
would be needed for spp.
That said just because you can use ivshmem-plain/doorbell with hugepages via a
memdev form the qemu commandline
does not mean it is supported by Libvirt
https://libvirt.org/formatdomain.html#elementsShmem or that it is the best
approach,
upstream development has shifted to virtio and vhost-user.
0 copy tx from a vm is already supported with vhost-user and ovs-dpdk.
0 copy rx Is under development which is the main delta in performance between
ivshmem and vhost-user that remains.
Both of these features could be ported to the spp.
Vhost-user does have some inherent overhead such as the descriptor rings etc
required to support virtio
Must be created which is required to support the virtio spec which give you
portability and performance when
Coupled with multi queue and the dpdk vhost pmd in the guest.
If vhost-user is really not sufficient for your usecase I would first suggest
extending Libvirt to allow
Passing a memdev name as a parameter to the shmem element.
At that point we could discuss how to request openstack to use those new
element to create the shmem device.
The best approach using what openstack support today for a generic shmem dev
would be via a Libvirt option in the image metatdata the same
Way that serial ports allocation or vGPU memory is configured. If this device
was to be used as a nic however
A different approach of creating a new vnic_type of ivsheme_plain and
ivshmem_doorbell would be more approiate with the
Port binding profile used to transport the memory mapping info similar to how
we expose info for sriov devices.
The issue with ivshsmem interfaces in a neutron context is its just a pci
device where bar2 points to a shared memory region
There is no way to pass info such as a mac address to a hypervisor for it to
enforce so the use of ivshmem deveices as
Nic means that you are enabling a adressless port that can transmit and recive
any packet unless you explicitly implement
Port security (mac/arp anti spoof) in the vswitch backend in this case spp.
That is why ivshmem was originally
Rejected in icehouse by nova.
--
Tetsuro Nakamura <nakamura.tets...@lab.ntt.co.jp> NTT Network Service
Systems Laboratories
TEL:0422 59 6914(National)/+81 422 59 6914(International) 3-9-11,
Midori-Cho Musashino-Shi, Tokyo 180-8585 Japan
Hi Sean,
Thank you very much for information.
Now it is clear that if we start working on ivshmem, the first target is
Libvirt to support memdev.
But it seems that before starting we need to check out the zero-copy
feature of vhost-user.
> 0 copy tx from a vm is already supported with vhost-user and ovs-dpdk.
> 0 copy rx Is under development which is the main delta in performance
between ivshmem and vhost-user that remains.
> Both of these features could be ported to the spp.
Could you inform me if there is a document or a reference where I can
get further info to enable these features and when these features are
merged?
We have only tested ovs-dpdk2.5 with dpdk16.04.
--
Tetsuro Nakamura <nakamura.tets...@lab.ntt.co.jp>
NTT Network Service Systems Laboratories
TEL:0422 59 6914(National)/+81 422 59 6914(International)
3-9-11, Midori-Cho Musashino-Shi, Tokyo 180-8585 Japan
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
