On 6/25/25 6:25 PM, Tim Rozet wrote:
On Wed, Jun 25, 2025 at 5:42 PM Mark Michelson <mmich...@redhat.com
<mailto:mmich...@redhat.com>> wrote:
On 6/25/25 11:38 AM, Tim Rozet wrote:
>
> On Tue, Jun 24, 2025 at 5:14 PM Mark Michelson
<mmich...@redhat.com <mailto:mmich...@redhat.com>
> <mailto:mmich...@redhat.com <mailto:mmich...@redhat.com>>> wrote:
>
> On 6/24/25 1:31 PM, Tim Rozet wrote:
> > Thanks Mark for the detailed response and taking the time to
> review the
> > proposal. See inline.
> >
> > Tim Rozet
> > Red Hat OpenShift Networking Team
> >
> >
> > On Tue, Jun 24, 2025 at 12:04 PM Mark Michelson
> <mmich...@redhat.com <mailto:mmich...@redhat.com>
<mailto:mmich...@redhat.com <mailto:mmich...@redhat.com>>
> > <mailto:mmich...@redhat.com <mailto:mmich...@redhat.com>
<mailto:mmich...@redhat.com <mailto:mmich...@redhat.com>>>> wrote:
> >
> > On 6/18/25 8:26 PM, Numan Siddique wrote:
> > > On Fri, Jun 13, 2025 at 8:44 AM Tim Rozet via dev
> > > <ovs-dev@openvswitch.org <mailto:ovs-
d...@openvswitch.org> <mailto:ovs-dev@openvswitch.org <mailto:ovs-
d...@openvswitch.org>>
> <mailto:ovs-dev@openvswitch.org <mailto:ovs-
d...@openvswitch.org> <mailto:ovs-dev@openvswitch.org <mailto:ovs-
d...@openvswitch.org>>>>
> wrote:
> > >>
> > >> Hello,
> > >> In the OVN-Kubernetes we have been discussing and
> designing a way to
> > >> implement Service Function Chaining (SFC) for
various use
> cases.
> > Some of
> > >> these use cases are fairly complicated, involving
a DPU
> and multiple
> > >> clusters. However, we have tried to abstract the OVN
> design and
> > use case
> > >> into a generic implementation that is not specific
to our
> > particular use
> > >> cases. It follows SFC designs previously done
within other
> > projects like
> > >> OpenStack Neutron and OpenDaylight. Please see:
> > >>
> > >> https://docs.google.com/document/ <https://
docs.google.com/document/> <https://
> docs.google.com/document/ <http://docs.google.com/document/>>
> > d/1dLdpx_9ZCnjHHldbNZABIpJF_GXd69qb/
edit#bookmark=id.a7vfofkk8rj5
> > <https://docs.google.com/document/ <https://
docs.google.com/document/> <https://docs.google.com/ <https://
docs.google.com/>
> document/>
> > d/1dLdpx_9ZCnjHHldbNZABIpJF_GXd69qb/
> edit#bookmark=id.a7vfofkk8rj5>
> > >>
> > >> tl;dr the design includes new tables to declare
chains and
> > classifiers to
> > >> get traffic into that chain. There needs to be a
new stage in
> > the datapath
> > >> pipeline to evaluate this behavior upon port
ingress. We also
> > need these
> > >> flows to be hardware offloadable.
> > >>
> > >> For more details on the specific use cases we are
> targeting in the
> > >> OVN-Kubernetes project, please see:
> > >>
> > >> https://docs.google.com/document/
d/1MDZlu4oHL3RCWndbSgC- <https://docs.google.com/document/
d/1MDZlu4oHL3RCWndbSgC->
> <https://docs.google.com/document/d/1MDZlu4oHL3RCWndbSgC-
<https://docs.google.com/document/d/1MDZlu4oHL3RCWndbSgC->>
> > IGLgs1QfnB1l47nPqtM5iNo/edit?
tab=t.0#heading=h.g8u53k9ds9s5
> > <https://docs.google.com/document/
d/1MDZlu4oHL3RCWndbSgC- <https://docs.google.com/document/
d/1MDZlu4oHL3RCWndbSgC->
> <https://docs.google.com/document/d/1MDZlu4oHL3RCWndbSgC-
<https://docs.google.com/document/d/1MDZlu4oHL3RCWndbSgC->>
> > IGLgs1QfnB1l47nPqtM5iNo/edit?
tab=t.0#heading=h.g8u53k9ds9s5>
> > >>
> > >> Would appreciate feedback (either on the mailing
list or
> in the
> > design doc)
> > >> and thoughts from the OVN experts on how we can
> accomodate this
> > feature.
> > >>
> > >
> > > Hi Tim,
> > >
> > > There is a very similar proposal from @Sragdhara Datta
> Chaudhuri to
> > > add Network Functions support in OVN.
> > > Can you please take a look at it ? Looks like
there are many
> > > similarities in the requirements.
> > >
> > > https://mail.openvswitch.org/pipermail/ovs-
dev/2025- <https://mail.openvswitch.org/pipermail/ovs-dev/2025->
> <https://mail.openvswitch.org/pipermail/ovs-dev/2025-
<https://mail.openvswitch.org/pipermail/ovs-dev/2025->>
> > May/423586.html <https://mail.openvswitch.org/
pipermail/ovs- <https://mail.openvswitch.org/pipermail/ovs->
> <https://mail.openvswitch.org/pipermail/ovs- <https://
mail.openvswitch.org/pipermail/ovs->>
> > dev/2025-May/423586.html>
> > > https://mail.openvswitch.org/pipermail/ovs-
dev/2025- <https://mail.openvswitch.org/pipermail/ovs-dev/2025->
> <https://mail.openvswitch.org/pipermail/ovs-dev/2025-
<https://mail.openvswitch.org/pipermail/ovs-dev/2025->>
> > June/424102.html <https://mail.openvswitch.org/
pipermail/ovs- <https://mail.openvswitch.org/pipermail/ovs->
> <https://mail.openvswitch.org/pipermail/ovs- <https://
mail.openvswitch.org/pipermail/ovs->>
> > dev/2025-June/424102.html>
> > >
> > >
> > > Thanks
> > > Numan
> >
> > Hi Tim and Numan,
> >
> > I've looked at both the ovn-k proposal and the Nutanix
patch
> series. I
> > think the biggest differences between the proposals (aside
> from small
> > things, like naming) are the following:
> >
> > 1) Nutanix amends the ACL table to include a network
function
> group to
> > send the packet to if the packet matches. The ovn-k
proposal
> suggests a
> > new SFC_Classifier table that includes an ACL-like match.
> >
> > 2) ovn-k wants load balancing of the service
functions. The
> Nutanix
> > patch series has no load balancing.
> >
> > 3) ovn-k wants a Service_Function_Chain table, that allows
> for multiple
> > services to be chained. The Nutanix patch series
provides a
> > Network_Function_Group table that allows a single network
> function
> > to be
> > the active one. There is no concept of chaining in the
patch
> series.
> >
> > 4) ovn-k wants NSH-awareness. I don't 100% know what this
> entails, but
> > there is no NSH in the Nutanix patch series.
> >
> >
> > We don't necessarily require NSH. Some limited Cisco products
> support
> > NSH, but I'm not aware of other vendors. So for now the
majority
> of the
> > CNF use case would be proxied. However, we do need some
mechanism to
> > store metadata to know what chain the packet is currently
> on, especially
> > as packets go between nodes. This could be Geneve TLV
metadata. I'm
> > looking for feedback on this kind of stuff in the doc, as
I'm not
> sure
> > what is best suited for this and if it is offloadable.
> > >
> > IMO, items 2, 3, and 4 can be made as add-ons to the
Nutanix
> patch
> > series.
> >
> >
> > How do you envision it being added on? Would it be a separate
> feature,
> > or an extension of the Nutanix effort?
>
> These are great questions. My thought had been that it would
be an
> extension of the Nutanix feature.
>
> > I'm a bit concerned if it is the
> > latter, because I worry we will have boxed ourselves into
a certain
> > paradigm and be less flexible to accomodate the full SFC
RFC. For
> > example, in the Nutanix proposal it looks like the
functionality
> relies
> > on standard networking principles. The client ports are
connected
> to the
> > same subnet as the network function. In my proposal, there
is no
> concept
> > of this network connectivity. The new stage simply takes the
> packet and
> > delivers it to the port, without any requirement of layer 2 or
> layer 3
> > connectivity.
>
> I'm not 100% sure I understand what you mean about the
Nutanix proposal
> relying on standard network principles. For instance, my
reading of the
> Nutanix patches is that if the ACL matches, then the packet
is sent to
> the configured switch outport.
>
>
> What I mean is when the NF does not exist on the same switch as the
> client traffic. Looking at the proposal again I think the relevant
> section is "NF insertion across logical switches". In my
proposal there
> is no definition of needing a link between the switches. My
definition
> might be wrong in the OVN context, that's where I need feedback
and we
> need to discuss how it would work. To try to explain it in simple
terms.
> If I have a client on switch LS1 that sends traffic that is
classified
> to a chain with 1 NF (analogous to the Nutanix NF/NFG) on switch SVC
> LS...in Nutanix a child port is created by CMS to connect to the 2
> switches together, while in my proposal there is no concept of
that link:
>
> Nutanix proposal:
>
> --------
> | NF VM |
> --------
> | |
> ----- | | -----
> | VM1 | nfp1 nfp2 | VM2 |
> ---- - | | -------------- -----
| |
> | | | | SVC LS | |
| |
> p1| nfp1_ch1 nfp2_ch1 -------------- p3|
nfp1_ch2 nfp2_ch2
> --------------------
--------------------
> | LS1 | |
LS2 |
> --------------------
--------------------
>
> nfp1_ch1 is created by CMS to get the packet from the LS1 to SVC
LS. I'm
> guessing it doesn't matter in this case whether or not the SVC LS
is on
> the same OVN node? How would this work in IC? Would we need a
transit
> switch? How does nfp1_ch1 map to SVC LS? It isn't clear in the
proposal
> how the CMS configures these parts with examples. Should it
really be on
> the user to create these nfp1_ch1 ports? Or can they
automatically be
> inferred?
>
> In the OVNK proposal, I do not define that there needs to be links
> between switches. From the SFC perspective, it transcends standard
> networking. Therefore there is no reason to need a link between
> switches. Once it classifies the packet as needing to go to nfp1,
and if
> nfp1 is on the same host, it just sends the packet to its port.
If it's
> on a remote node, it adds header data and tunnels the packet to
the next
> node. This is how it can be implemented in raw openflow or in
other SDN
> controllers. That perspective may not be grounded in reality when it
> comes to how OVN ingress/egress pipelines and traffic forwarding
work.
> That's where I need your feedback and we need to figure out how
those
> pieces should work. IMHO it would be imperative to figure that out
> before the Nutanix stuff is merged.
Thanks for elucidating, Tim.
I think this has really made it more clear about the fundamental
differences between the SFC proposal and the Nutanix series.
The Nutanix series introduces a hook mechanism for packets that arrive
on a particular switch that is configured to send those packets out
to a
network function. And the series is
The ovn-k proposal essentially wants a new overlay on top of the
existing logical network for service function chaining. The SFC
proposal
is only using logical switches because that's the only thing that OVN
provides that is close enough to where an SFF should live in an OVN
topology.
>
> Then when the patch re-arrives on the
> configured switch inport, it uses conntrack information to
put the
> packet back on track to go to its intended destination. The
service
> function does not appear to require any sort of L2 switching
based
> solely on that.
>
> Even the final patch that introduces the health monitoring
doesn't rely
> on the switch subnet but instead uses NB_Global options to
determine
> the
> destination MAC to check. It doesn't seem to be necessary to
be on the
> same subnet as the switch on which the service is configured.
>
> I may be misinterpreting, though.
>
> > Furthermore in the Nutanix proposal there are requirements
> > around the packet not being modified, while in SFC it is
totally
> OK for
> > the packet to be modified. Once classified, the packet is
> identified by
> > its chain id and position in the chain (aforementioned
NSH/Geneve
> metadata).
>
> Can you refresh me on how the chain ID is determined in the SFC
> proposal? In the patch series, the function group ID is stored in
> conntrack, so when the packet rearrives into OVN, we use
conntrack to
> identify that the packet has come from a network function and
needs to
> be "resumed" as it were. Because the patches use conntrack, the
> packet's
> identifying info (src IP, dst IP, src port, dst port, l4
protocol)
> can't
> be changed, since it means that we won't be able to find the
packet in
> conntrack any longer.
>
>
> Sure. So in the SFC world when the packet is going to be sent to
the NF,
> the SFF (OVS switch) determines if the NF needs to be proxied or
not. If
> it does not need proxying, then the SFF sends the packet with the
> NSH header. This header describes to the NF the chain ID and the
current
> position in the chain (index). Note, this requires the NF to be NSH
> aware so that it can read the NSH header. At this point the NF will
> process the packet, and decrement the chain index, and send the
packet
> back to the SFF. Now when the packet arrives back in OVS, it can
read
> the NSH header and know where to send the packet to next. This way a
> single NF can actually be part of multiple chains. It can even
> reclassify packets to a different chain by itself. However, this all
> relies on NSH, which only a few NFs actually support.
>
> Now, when we look at NFs that do not support NSH and need
proxying. In
> this case the SFF "proxies" by "stripping the chain information" and
> sending the packet without any additional information to the NF.
In this
> model the NF can only be part of a single chain, because when the
packet
> is sent and comes back there would be no way to distinguish packets
> being one one chain or another. So what I have seen in the past
> implementations is you set OF registers to track the chain
internally in
> OVS. Let's take an example with a 2 NF chain, where the NFs are
split
> across nodes, and let's assume that we use Geneve with TLV to
hold our
> chain/index information. Let's define the chain as an ordered
list of
> NF1,NF2:
>
>
> NF1
NF2
> |
|
> |
|
> |
|
> +-----------+
+-----------+
> | |
| |
> client -----------|OVS node1 |---------------------------
|OVS node2 |
> | |---------------------------
| |
> | |
| |
> +-----------+
+-----------+ +
> |
> |
> server
>
>
>
> 1. client sends packet (let's assume to google, 8.8.8.8), it gets
classified and OF registers are stored with chain id, and index 255,
punted to chain processing OF table
>
> 2. OVS node 1 - SFC stage/table matches chain id, index 255, send
to NF1
>
> 3. NF1 receives raw packet, modifies dest IP address to be
*server*, sends packet back to OVS node1
>
> 4. OVS node1 - Recieves packet from in_port NF1, restores OF
register for chain id, stores register for index, now decremented to 254
At this point, I have questions about some implementation details,
and I
have bad news with regards to how OVN currently works.
You've stored the chain id and index in OF registers. for simplicity,
we'll assume two 8-bit registers, reg0 and reg1.
So packet0 arrives at OVS node 1 from the client. The chain id and
index
are stored in reg0 and reg1 respectively. packet0 is then sent to NF1
and we await the return of packet0 so that we can use the registers to
know what to do with it next.
Now at this point, let's say packet1 arrives at OVS node 1 from the
client. packet0 is still out at NF1. packet1 also needs to traverse the
NFs, but reg0 and reg1 are being used to track packet0. If we overwrite
the values, then when packet0 arrives back, reg0 and reg1 will have
packet1's chain state, not packet0's. How do you handle the collision?
AFAIK registers in openflow act in a per packet context. The registers
used to provide metadata to packet0 are per packet metadata that are
totally isolated from the registers used to handle packet1 context.
I checked with some knowledgeable folks, and you are correct that the
registers are scoped to a particular packet. But the flipside to this is
that the registers can't be preserved if that packet leaves OVS. In the
scenario that you outlined, did the packets leave OVS, or were the NFs
implemented on separate bridges within the same OVS instance?
When it comes to identifying the chain state, there are essentially
three options:
1) Use packet data. This is the NSH case, and based on what you stated
earlier, it sounds like we should not plan for this since most NFs don't
support NSH anyway.
2) Base our decisions based on the port on which the packet is received.
Assuming the chains are not dynamic, then we can set up each SF to use a
dedicated set of ports. This way, we can know that if, say, a packet
arrives on port 1, it means that it's a packet that hasn't gone through
any SFs yet, so we send to NF1 via port 2. Then when NF1 completes its
processing, it sends the packet to port 3. Receiving on this port means
we know to send to NF2 on port 4, e.g. We would still need to encode
information via Geneve in the case where the packet needs to reach an NF
on a separate node. This is essentially how the Nutanix series works,
but it does not support chaining multiple NFs together.
3) Daisy-chain the NFs. Instead of having OVN send the packet to each
NF, have OVN send the packet to NF1, then have NF1 send the packet to
NF2, etc. This likely is not a viable option since the NFs will have no
knowledge of how to reach each other, but I figured I'd throw it out
there anyway.
It sounds like (2) could be a reasonable way to go, assuming the chains
are static.
The same questions go for restoration of chain information at stage 8.
This becomes moot when dealing with OVN datapaths because a transition
out of a pipeline involves clearing all registers. So we can't rely on
registers to hold values once a packet has either moved from one
pipeline to another or has exited the OVS bridge. This is why conntrack
is relied on heavily for stateful data. But of course, if the packet
data is being changed by an NF, we can't use conntrack either.
The ingress/egress pipeline semantics in OVN is something we need to
figure out. Another way to look at this is when a packet is classified
to a chain, its ingress pipeline is essentially paused and the packet is
diverted to a special SFC pipeline. The packet is later returned to the
ingress pipeline to resume ingress and egress processing. Not sure if we
could introduce a new type of SFC pipeline into OVN?
This is essentially what I'm describing with the composable service
idea. The packet would, prior to hitting the switch, hit an alternate
SFC datapath. Once all NFs have executed, then the new datapath would
send the packet to the switch and it would be processed like normal.
Introducing a new SFC pipeline would be interesting, to say the least
:). Let's just say for now that if we could avoid this, I'd prefer that.
If it becomes a requirement, then we can try to work out exactly how
that would be implemented, and what the implications would be on all
levels of OVN.
If we don't use conntrack or OF registers, then we usually resort to
storing state in the ovn-controller runtime. But having to slowpath
every packet to send to ovn-controller multiple times to store and
retrieve chain state would likely be *terrible* for performance.
So ideally, the packet would always contain the chain id and index
information. But as you mentioned, not many NFs actually support
NSH, so
it's unlikely that we can rely on the packet to retain the information.
What do?
IMO we should be using OF registers. That's what we did in OpenDaylight.
Just to reiterate what I was saying above, were the packets ever leaving
the switch in this scenario?
> >
>
> 5. OVS node 1 - SFC stage/table matches chain id, index 254, send
to remote SFF OVS node 2. Enacapsulate in geneve, set Geneve
metadata chain id and index at 254.
>
> 6. OVS node 2 receives packet - SFC stage/table matches chain id,
index 254, send to NF2
>
> 7. NF2 receives raw packet, modifies something else in the
packet, sends back to OVS node2
>
> 8. OVS node 2 receives the packet from in_port NF2, restores OF
register for chain id, stores register for index, now decremented to 253
>
> 9. OVS node 2 - SFC stage/table matches on chain ID, index 253,
has reached the end of chain. Send packet back to original SFF to
resume datapath pipeline processing. Encapsulate in geneve, set
chain id and index at 253.
>
> 10. OVS node 1 receives packet. Processes chain id and determines
253 is end of chain. Continue to next stage of ingress datapath
pipeline.
>
> 11. Regular OVN datapath pipeline finishes, routes packet towards
server due to dest IP in packet.
>
>
> The chain has effectively rerouted the destination of the packet
to another server, without needing conntrack to store anything.
>
>
>
> In the SFC proposal, if the packet is modified, then that
means we
> would
> need to use something other than conntrack to track the chain
ID. Would
> we require NSH in order to track the chain ID properly? Or is
there
> some
> other way?
>
> >
> >
> > Item 1 is the biggest sticking point. From my point of
view,
> I prefer
> > the Nutanix approach of modifying the ACL table since,
> > * ACLs can be applied to switches or port groups. The
proposed
> > SFC_Classifier only applies to port groups.
> > * ACLs have things like logging and sampling that can be
> useful in this
> > scenario.
> > * ACLs can be tiered.
> > However, if there's a good reason why this will not
work for
> ovn-k's
> > scenario, then that would be good to know.
> >
> >
> > Using the ACLs I think would be fine for the OVNK use case as
> well. The
> > reason I didn't propose using ACLs were 2 fold:
> > 1. Trying to create a clear boundary for SFC. Since SFC
does not
> behave
> > like normal networking, I thought it would make sense to
make it
> its own
> > entity.
>
> This is where I really wish we had something like composable
> services in
> place, because it sounds like SFC is only being added to logical
> switches because that's the current best fit for them. They would
> really
> be better suited to their own datapath type.
>
> But for now, putting them on a logical switch is the best choice.
>
> The nice thing about ACL stages is that they are very early
in the
> logical switch pipelines. We perform FDB and mirror actions
before the
> ACL, but that's it.
>
> > 2. I didn't think OVN would be amenable to modifying ACL
to have
> a new
> > column to send to a chain.
>
> > In the Nutanix proposal it looks like the column is added
to send
> to a
> > NFG. Would we also add the ability to send to a SFC?
>
> The way I had thought about it, we could expand NFGs to
contain SFCs.
> Currently, an NFG has a list of network functions. But we could
> create a
> new column in the NFG table that could be one or more SFCs.
The idea
> would be that if you configure the network_functions column,
we use
> those. If you configure the service_function_chains column,
we use
> those
> instead. It would be a misconfiguration to use both at the
same time.
>
> >
> >
> > Currently, I would prefer to review and accept the Nutanix
> patch series
> > (for ovn25.09) and then add on the ovn-k features that are
> not present
> > in the series (for ovn26.03).
> >
> > Tim, what do you think?
> >
> >
> > I think first we should have a solid plan for how we will
add on
> the SFC
> > part. For example will we expand NFG so that we can load
balance
> across
> > it or only have 1 active at a time? If so, then it would
maybe make
> > sense now to add a new field to the NFG to indicate this
mode. Those
> > types of detail I would like to iron out and have a plan
for so
> we don't
> > find ourselves cornered when we try to add SFC later. wdyt?
>
> Yes, this is how my thought process was as well. The current NFG
> configuration allows for multiple network functions to be
configured,
> choosing a single one as the active one based on health checks.
>
> We have to consider that we want to:
> 1) Allow for multiple functions to be chained.
> 2) Allow for multiple functions/chains to be load balanced.
>
> There are many possibilities for how to implement these based
on the
> current patch series.
>
> For chaining, I think the best plan is to create a new
> Service_Function_Chain (or Network_Function_Chain if we want
to keep
> the
> same nomenclature) table. Then the NFG's network_function
column could
> allow for either singular functions or chains in the list of
> network_functions.
>
> Alternatively, we could get rid of the current Network_Function
> table in
> favor of replacing it with the Service_Function_Chain table. A
> Network_Function is nothing more than a
Service_Function_Chain with a
> single function, after all.
>
>
> +1
>
>
> For load balancing, we could either:
> a) Add a boolean to the NFG table, called load_balance. If set to
> false,
> then a single active network function or service function
chain is
> chosen from the list. If set to true, then all network
functions or
> service function chains are viable, and we use load balancing to
> determine which to use. We can still use health checks to
ensure we
> only
> try to load balance between live functions.
>
>
> +1 I think this is probably true but just want to also highlight
health
> checks should be optional as well.
I believe in the current patch series health checks are optional. If
you
do not set the destination MAC for health checks then they do not
happen. I can double-check to be sure though.
>
> b) Create a new Load_Balanced_Service_Function_Chain table that
> specifies lists of load balanced service function chains.
Then the NFG
> could place these in the network_functions as well.
> c) The same as B, but instead of adding a new table, add a new
> column to
> the existing Load_Balancer table that allows a list of
> network_functions
> (or chains) to be listed. Then these load balancers could be
applied to
> the NFG the same way as a network function.
>
At this point, this is my summary of the situation:
The patch series implements a hook system that allows packets to be
sent
out to non-mutating NFs. Packets may have traversed other switches and
routers in the network before arriving at the point that the hook is
invoked. Since the feature extends ACLs, packets can be sent to the NFs
during the ingress or egress pipeline. The NFs must be non-mutating
because we use conntrack to track the state of the packet. If the NFs
are chained, the chaining must be handled outside of OVN. NSH is never
involved.
The ovn-k proposal seeks to encapsulate packets with NSH info before
the
packet arrives in OVN.
We don't seek to use NSH. It's just the only real standard out there as
an SFC transport. I'm fine with not supporting NSH (at least initially)
and just using Geneve to carry the chain/index information.
OK great. I'll keep NSH out of discussions from this point since we can
save it for a later add-on.
Upon arrival in OVN, the packet should, as soon
as possible, be sent out to NFs. OVN may or may not need to proxy the
NSH information (though it likely will need to since most NFs are not
NSH-aware). Chaining may happen within the NFs or it can be handled by
OVN. The NFs may mutate the packet, meaning OVN cannot use conntrack to
track the chain id or index. Once all NFs have handled the packet, then
it is entered into the typical switch ingress pipeline and handled
as it
normally would be. AFAICT, sending the packets to NFs will *always*
happen ASAP, and cannot happen during the egress pipeline.
Yeah this is because we do not want the packet to be altered before it
is sent to the NF. For example, if the dest was a load balancer VIP, we
do not want the packet to get DNAT'ed by the LB and then sent to the
chain later. The classification and diversion of the packet should
happen as early as possible.
This is another reason why the composable service would be a good idea,
because the packet would have all SFC processing completed before ever
ingressing a switch.
If we're not using a composable service but instead using a logical
switch, though, then I have come around to your POV that we should do
SFC as the very first thing on ingress. Waiting until the ACL stage
means performing (potentially pointless and incorrect) FDB lookups on
packets that may be altered by NFs. Those FDB lookups should happen on
the altered packet instead.
Generally speaking, these are very different with regards to
implementation details. However, I think Numan and Tim are correct that
we could tailor the new tables to be able to work with both use
cases. I
can think through this and try to propose something that will work for
both. I had previously thought that the ovn-k case could bend to the
Nutanix's use case, but I think that's incorrect. I think they are
distinct enough to exist as separate features in OVN. I don't think
either use case is invalid, and aside from ensuring the tables can
accommodate both use cases, I don't think anything should block merging
of the Nutanix patch series.
Now more than ever, I think ovn-k SFC proposal would work best as a
composable service rather than in OVN logical switches. As a refresher,
the composable service idea is to essentially be able to insert new
hyper-specialized logical datapath types between VMs and switches, or
between switches and router distributed gateway ports. You could place
an "SFC classifier" datapath between the VM and the switch, allowing
for
the SFC processing to happen before the packet ever even enters the
logical switch, thereby not messing with the logical switch's
functionality at all. One thing I had always considered with the
composable services feature was that all composable services would
still
operate in OVN's br-int bridge. But if we want to be able to play fast
and loose with register behavior in composable services, it may be a
requirement to implement them within their own OVS bridges instead.
This
way they would have their own independent register space to use as they
see fit, including persisting register values after packets depart.
Depending on your answer to my questions above with regards to register
restoration, we may be able to implement identical logic to what you
illustrated within OVN. No composable services are actually implemented
in OVN yet, but ovn-northd refactoring efforts to allow for them to
exist are posted on patchwork currently. I had planned to try to
implement a simple NAT composable service as the first one, but SFC may
be a better way of proving their worth, especially if we need to be
able
to utilize secondary bridges.
Sounds interesting. Do you have a pointer to some links about composable
services and how to use them?
Sure, I have this:
https://docs.google.com/document/d/1hRdx9LTiquXoeKQNsTfq0LWGbGOHIqbhGTaxYVB4yUU/edit?tab=t.0
. From the document, everything up until the "Hook Services" section is
relevant. The "External" service is something I had come up with before
I had ever heard the term "SFC", but it kind of sought to have the same
goal. It proposes OVN dumbly sending packets to an OVS bridge and having
that OVS bridge do whatever it wants to the packet before sending the
packet back into OVN.
And there's also my presentation I gave last year at OVS+OVN conf:
https://www.youtube.com/watch?v=Gf4M-ZSmTz4 . This definitely is not
centered around SFC (I don't even recall if I mention the "external"
service in the talk), but it goes into detail about the intent of how
composable services will work and some use-cases they help with.
> >
> >
> > Thanks,
> > Mark Michelson
> > >
> > >> Thanks
> > >>
> > >> Tim Rozet
> > >> Red Hat OpenShift Networking Team
> > >>
> > >>>
> > >> _______________________________________________
> > >> dev mailing list
> > >> d...@openvswitch.org <mailto:d...@openvswitch.org>
<mailto:d...@openvswitch.org <mailto:d...@openvswitch.org>>
> <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org>
<mailto:d...@openvswitch.org <mailto:d...@openvswitch.org>>>
> > >> https://mail.openvswitch.org/mailman/listinfo/ovs-
dev <https://mail.openvswitch.org/mailman/listinfo/ovs-dev>
> <https://mail.openvswitch.org/mailman/listinfo/ovs-dev
<https://mail.openvswitch.org/mailman/listinfo/ovs-dev>> <https://
> > mail.openvswitch.org/mailman/listinfo/ovs-dev <http://
mail.openvswitch.org/mailman/listinfo/ovs-dev> <http://
> mail.openvswitch.org/mailman/listinfo/ovs-dev <http://
mail.openvswitch.org/mailman/listinfo/ovs-dev>>>
> > > _______________________________________________
> > > dev mailing list
> > > d...@openvswitch.org <mailto:d...@openvswitch.org>
<mailto:d...@openvswitch.org <mailto:d...@openvswitch.org>>
> <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org>
<mailto:d...@openvswitch.org <mailto:d...@openvswitch.org>>>
> > > https://mail.openvswitch.org/mailman/listinfo/ovs-
dev <https://mail.openvswitch.org/mailman/listinfo/ovs-dev>
> <https://mail.openvswitch.org/mailman/listinfo/ovs-dev
<https://mail.openvswitch.org/mailman/listinfo/ovs-dev>> <https://
> > mail.openvswitch.org/mailman/listinfo/ovs-dev <http://
mail.openvswitch.org/mailman/listinfo/ovs-dev> <http://
> mail.openvswitch.org/mailman/listinfo/ovs-dev <http://
mail.openvswitch.org/mailman/listinfo/ovs-dev>>>
> >
>
_______________________________________________
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
