Hi,
With this mail we would like to re-focus the long-pending discussions on how to
best support new tunnel stacks such as MPLS in GRE and new tunneling protocols
like NSH in OVS.
Contributions addressing individual aspects of the problem complex (L3
tunneling, NSH) have been in the make for more than a year now and yet there is
no acceptable overall solution in sight even for the known use cases.
Ericsson would like to engage more actively to help define and implement a
comprehensive solution in OVS for a flexible and extensible support of
tunneling protocols implemented either as tunnel ports, in OpenFlow, or both.
We would like to discuss our proposal on the OVS Conference next week and hope
that we can reach agreement on the way forward.
Best regards, Jan
Situation:
----------
1. OpenFlow up to version 1.4 only supports L2 (Ethernet) packets. At any point
in time a packet in the OF pipeline must have an Ethernet header. Particular
push/pop actions were defined to insert/delete certain "tunnel" headers within
the packet (VLAN, MPLS, PBB).
2. OpenFlow 1.5 standardized the "Packet type-aware pipeline" (PTAP, EXT-112)
[1], which allows an OpenFlow switch to handle non-Ethernet packets. It
introduces the new match field packet_type encoding the type of packet and new
pre-requisites for existing match fields. PTAP is not yet implemented in OVS.
Jean Tourrilhes did a prototype implementation in OVS at the time of
standardization [8,9].
3. OF extension "EXT-382: Generic tunnel Encap and Decap" [2] was proposed in
ONF as means for an OF controller to push/pop individual tunnel headers at the
front of a packet to handle arbitrary stacks of tunnel encapsulations within
OpenFlow. As the encap/decap actions would change the packet_type of the packet
in the pipeline, EXT-382 builds on PTAP. EXT-382 is candidate for inclusion in
upcoming OF 1.6.
4. Traditionally OVS implements tunnels as logical OF ports. Supported tunnel
types include GRE, VXLAN, GENEVE, STT. As the datapath and OF pipeline in OVS
are Ethernet-only, all tunnel ports are L2 tunnels, even though some of the
tunnel protocols would support other payload (GRE, GENEVE or VXLAN-GPE). The
concept of tunnel ports only works for IP encapsulation as OVS relies on IP
routing to select the outgoing interface for encapsulated packets.
5. The need in OVS for non-L2 tunnels such as MPLS in GRE or LISP to interwork
with external network equipment has driven the development of a series of
patches known as "L3 Tunneling support" by Lorand Jakub, Thomas Morin, Simon
Horman and Jiri Benc. For the latest version see [3,4]. This work has been
going on with interruptions for a long time.
In essence these patches make OVS pop the Ethernet header when sending a packet
to an "L3" port and to push a dummy Ethernet header with the correct Ethertype
when receiving a packet from an "L3" port.
However, the actual implementation carries the non-Ethernet packets received on
an L3 tunnel all the way through the datapath and the ofproto-dpif pipeline,
but without any visibility to the OF controller. The missing Ethernet header is
only pushed implicitly at transmission to a normal Ethernet port. Conversely,
an Ethernet header is stripped off just before sending to an L3 tunnel.
ofproto-dpif inserts internal push/pop_eth actions into the datapath when
needed.
This adds a lot of complexity to these patches without reaping the benefit of
supporting the packet type-aware pipeline. Without PTAP, however, it would be
sufficient to let the L3 tunnel vport push and pop Ethernet headers so that
datapath and ofproto-dpif would see Ethernet packets only.
The current patches also have conceptual problems. The controller can, for
example, match on and set MAC address fields of the non-existing Ethernet
header in an L3 packet, as these are cached in the packet's struct flow, but it
is prevented from pushing and popping VLAN headers and setting VLAN IDs.
Another issue with the patches is that a given tunnel port instance can either
carry Ethernet packets or non-Ethernet packets, but not both, even if the
tunnel protocol would allow that. This is a real limitation as it requires the
setup of two parallel tunnel meshes if both L2 and L3 payload is needed.
6. Since autumn 2015 Yi Yang and Johnson Li have been working on a set of
patches for OVS to support NSH encapsulation for SFC [5]. NSH is a service
tunnel header that can encapsulate different packet types, typically Ethernet
or IP. For transport it requires an outer Ethernet header or a "L3" transport
tunnel such as VXLAN-GPE or GRE that can carry NSH frames. A complication with
NSH is that in MD2 format the header includes optional TLV metadata fields,
similar but not quite identical to GENEVE.
The NSH work is stuck mainly because of the difficulty to handle the variable
encapsulations. For VXLAN-GPE transport it depends on the L3-Tunneling support.
Furthermore the OpenFlow pipeline must be able to push/pop an NSH header both
with and without Ethernet header. This calls for separate actions to push/pop
NSH and push/pop Ethernet headers, along the lines proposed in EXT-382, which
in turn requires the PTAP.
The alternative, integrated actions to push/pop NSH and Ethernet headers
together to not leave the Ethernet-only pipeline, was rejected by the OVS
maintainers [7].
Technical proposal:
-------------------
It seems that the only acceptable way forward to support NSH in OVS is to
implement PTAP and EXT-382 to push/pop Ethernet and NSH headers separately.
This would anyway be a good investment because PTAP is a major feature of OF
1.5 and EXT-382 is the key to implementing arbitrary encapsulation stacks in
OpenFlow. Together with the upcoming support for P4-specified protocols and
match fields, this would make OVS fit for whole new class of applications.
With PTAP in place it would be natural to let "versatile" tunnel ports (tunnels
which support multiple payloads, e.g. GRE, GENEVE, VXLAN-GPE) directly exchange
their payload packets with the PTAP, mapping the tunnel's protocol field to
packet_type and vice versa and dropping packets where no such mapping is
defined. A single versatile tunnel port could thus carry Ethernet and
non-Ethernet packets and there would be no need for an "L3" option to specify
the behavior.
It appears desirable to keep the legacy Ethernet-only mode of OVS bridges as an
option to maintain backward-compatibility with controllers that have not yet
implemented PTAP and EXT-382. The question is whether such L2-only bridges must
be limited to L2 tunnels or could support versatile tunnel ports as well. On
ingress from a versatile tunnel port, the vport could push a dummy Ethernet
header with an Ethertype derived from the tunnel's protocol field. On egress to
the tunnel, however, we'd need an "L3" option on the vport to determine whether
to pop the L2 header (including any VLAN tags) or not.
Suggested way forward:
----------------------
A) Implement PTAP based on Jean's prototype implementation [9]. PTAP introduces
the packet_type match field in struct flow to explicitly track the type of
packet. This is cleaner and more robust than the current implicit derivation
from base_layer, Ethertype and layer offsets.
B) Reduce the scope of the current L3 tunneling patches to the vport
implementation. Do the mapping between tunnel protocol field and packet_type
when connected to PTAP-enabled bridge. For non-PTAP bridges and "L3" tunnels
let the vport push/pop Ethernet/L2 headers as needed.
C) Implement VXLAN-GPE as a new "versatile" tunnel port.
D) Implement the generic parts of EXT-382 to handle generic encap/decap actions.
E) As part of NSH patches implement the new packet_type NSH and generic encap
and decap actions for NSH and ETH in the datapath. Use them to realize all
relevant NSH encapsulations.
F) OpenFlow support
PTAP is standard in OF 1.5, EXT-382 is not. It would be beneficial if both PTAP
and the generic encap/decap actions of EXT-382 would be made available in OF
1.3, 1.4 and 1.5 (as Nicira extensions, where necessary).
References:
-----------
[1] EXT-112 Packet Type-aware Pipeline
https://rs.opennetworking.org/bugs/browse/EXT-112
[2] EXT-382 Generic tunnel Encap and Decap
https://rs.opennetworking.org/bugs/browse/EXT-382
[3] [PATCH net-next v12 0/9] openvswitch: support for layer 3 encapsulated
packets
http://openvswitch.org/pipermail/dev/2016-October/080697.html
[4] [PATCH v12 rebased 0/3] userspace: Support for layer 3 encapsulated packets
http://openvswitch.org/pipermail/dev/2016-October/080828.html
[5] [RFC PATCH v2 00/13] Add Network Service Header Support
http://openvswitch.org/pipermail/dev/2016-July/074922.html
[7] Jesse Gross last mail regarding push/pop_nsh_eth
http://openvswitch.org/pipermail/dev/2016-September/079824.html
[8] Jean's post with link to his PTAP prototype
http://openvswitch.org/pipermail/dev/2016-February/066842.html
[9] Jean's latest PTAP prototype
https://github.com/jean2/openvswitch/commits/jean/ext-112-l3
_______________________________________________
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev
