On Thu, 11 Apr 2019 09:30:12 +0000
"Benoit Ganne (bganne)" <[email protected]> wrote:
> Hi Stephen,
>
> > The rdma-core stuff is likely to be a problem on Azure. The Mellanox
> > device is always hidden as a secondary device behind a synthetic virtual
> > device based on VMBus. There are two DPDK different ways this is used. One
> > is with vdev_netvsc/failsafe/tap and the other is with netvsc PMD.
> > In either case, the virtual device expects to see the MLX device show up
> > as a VF if enabled.
> > So unless you want to write native VPP drivers for VMBus as well, I don't
> > see how having native rdma-core driver will help. It will only get
> > confused.
>
> I'd be interested to better understand how it works. My current understanding:
> - the Mellanox device shows up as a PCI SR-IOV VF in the Linux VM ("the VF")
> - the VF is associated with a VMBUS (not PCI) netvsc virtual device
> - the netvsc and the VF shares the same MAC. The netvsc is guaranteed to be
> always there whereas the VF can disappear between migrations etc.
> - control plane traffic (multicast eg. ARP, other?) always go through the
> netvsc
> - if the VF is present is gets all the traffic minus control plane. If it is
> not present, all traffic gets rerouted through netvsc as a fallback
>
> Am I correct? Could you elaborate a little bit more about what is the control
> plane traffic (basically what we should miss on the VF)?
> If so, I see at least 2 problems we need to tackle:
> - we need to receive control plane traffic from netvsc because of eg. ARP.
> However, being control-plane/fallback, it could be slow and use eg. TAP or
> even AF_PACKET. We will also need to use it to populate the neighbor entries
> of the VF
> - the VF can appear/disappear w/o notice in case of migration etc. That
> could be handled in different ways, eg. by the control plane agent (removing
> the rdma iface from VPP prior to migration, adding it after post migration)
> or by leveraging bonding or...
>
> Thanks for your help,
> Ben
Very close.
- In Azure the first packet of every flow arrives on the netvsc (slow path).
Packets arrive to FPGA, anything with non-matching flow goes to Azure host
flow control (VFP) which evaluates
it against rules. On success, it programs FPGA and forwards packet over
netvsc path.
- The presence of VF is reported as an event on VMBus
- VF is not enabled until netvsc sends message to host, acts as ack that VF
is ok.
- VF removal is reported to netvsc, and it switches datapath back
- the VF should not be manipulated directly, it is hidden in DPDK; in
Linux/FreeBSD it is marked as slave
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#12767): https://lists.fd.io/g/vpp-dev/message/12767
Mute This Topic: https://lists.fd.io/mt/30795618/21656
Group Owner: [email protected]
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
