Hi everyone,

First of all, thank you to those who could attend the conference and the
"ovn-local: A new proposed architecture for OVN" discussion panel:

https://www.openvswitch.org/support/ovscon2026/#t26

Some references, also for those who couldn't make it live (or online):

- Slides presented at the conference:
https://drive.google.com/file/d/1TBDrqoYk9EHseqSaif6xKt4ePYncSocb

- Recording of Mark's presentation (shared at the conference):
https://drive.google.com/file/d/1mwLmZQmUpZMZ3T-DMtkvo6xQaq867Ebc

- A snapshot of the document that lead after discussion between Red Hat
internal OVN team members to the proposal we shared at the conference:
https://drive.google.com/file/d/1w7IwB6UX-KbzUChBZb9yGi42yPIP7NZH/

As usual, it takes some time after the conference until we have
everything ready to share publicly so we don't (yet) have the recording
of the actual panel.  But I'll try to summarize as best as I can (at the
end of this email) what was discussed during the actual session but also
during the ad-hoc/hallway discussions after the official session.

I think at this point quite a few members of the community have
expressed interest in the initiative and it might be a good idea to
start planning some of the work that might be required for this.  As
you'll see below, I don't think there were any unsolvable technical
issues but there will definitely be challenges.

In order to get things going, I'd like to ask folks to think about
timelines regarding this new initiative.

I personally am quite confident that we can try to deliver some of the
changes suggested in the ovn-local document (e.g., phase 1) as early as
the next development cycle (27.09) and that there's a high chance to
implement the migration to the new architecture before our next LTS
release (28.03).  But that's my personal opinion, I'd like to hear
others too. :)

Please read further for the notes I took during the conference.

Best regards,
Dumitru
---

My notes from the panel and follow up discussion:
-------------------------------------------------
DISCLAIMER: I wrote this down after the fact so there might be inaccuracies.

The main take away from the discussion was that OVN users in the
audience were interested in moving towards such an architecture.  It’s
also in-line with other talks that were presented at the conference
(e.g., Crusoe is moving towards a custom ovn-ic deployment similar to
the OpenShift/ovn-kubernetes one - essentially one OVN per node, Stackit
has issues due to the large size of the SB database in their classic
centralized deployment).

There were a few questions/remarks from the audience and afterwards
during the ad-hoc discussion.  I'll list everything (including
answers/comments) here:

1. How can we deal with the "synchronization work that ovn-northd
currently does, e.g., allocating SB datapath and port_binding tunnel keys?

There were some options mentioned during the session:
a. fully distributed and predictable/consistent generation of the keys
by individual ovn-local instances.  This has IMO the risk of
complicating upgrades if we ever decide to change the algorithm used to
generate keys.

b. adding support to ovsdb to "run stored procedures" and move the key
allocation logic there

c. mimic what ovn-kubernetes does and have a "cluster manager", a
centralized component, essentially a way simpler ovn-northd that still
manages keys even for ovn-local deployments

2. Would moving to ovn-local shift the performance issues from the
central nodes to the chassis?  That’s because ovn-local would now
perform the work that ovn-northd does in the current deployment
(centralized).

I think we can't provide a definitive answer before experimenting with
the proposed new architecture but we expect that in the final ovn-local
deployment only locally relevant logical switches/routers would be
processed (reducing compute time to values likely similar to what
ovn-controller has to process today); the large SB contents would be
gone, reducing time to process those in the IDL (json handling).

3. Would moving to ovn-local cause upgrade incompatibility?  I.e., in
very large clusters upgrades can take days/months.

This has two aspects:
a. the first upgrade, from centralized OVN to ovn-local
b. subsequent upgrades, from ovn-local version x to ovn-local version x+1

"a" was discussed separately quite extensively during the ad-hoc session
after the conference, I'll detail that later.

for "b”, taking the ovn-kubernetes example, we probably have a chance of
incompatibility even with today’s architecture.  That’s because whenever
a packet needs to be tunneled to a remote chassis (i.e., it has been
received on a logical port on chassis-1 and needs to be sent out on a
logical port on chassis-2) we are splitting that datapath’s logical
pipeline across two nodes: ingress pipeline is executed on chassis-1
while egress pipeline of the same switch/router is executed on chassis-2.

So, if a change happens that involves, for example, moving stages from
one pipeline (ingress or egress) to the other then these may cause
inconsistent packet forwarding.

In general, this is handled today through feature flags (each chassis
advertises its "supported” feature set and ovn-northd only generates
logical flows that are compatible with the "oldest” ovn-controller).
Moving to ovn-local, we’d have to be even more careful with these kinds
of situations but, IMO, nothing new here.

4. What about the shared state that’s currently stored in the Southbound
(pushed up by ovn-controllers)?  E.g., mac-binding, fdb, learned routes.

That’s still needed in the ovn-local mode; either stored as new tables
in the (new) NB database or as a "Status/Operational/etc” database.

5. Numan suggested reviving his old RFC work that moved logical flow
generation to ovn-controller or at least first focusing on removing
Logical_Flow from the "new” ovn-local database.

https://mail.openvswitch.org/pipermail/ovs-dev/2021-June/384488.html

I think the ovn-local "Phase 2" part of the proposal presented at the
conference covers that and we could likely start with removing Logical
Flows but it's my personal opinion that it would require less work
overall if we first implement "Phase 1" of the proposal ("Delete the
Southbound database").

6. Is one of the problems with the current architecture that we end up
pushing "too detailed” information to the chassis from the central OVN node?

Yes, most likely, and that’s inline with other targeted optimizations we
did in the past, e.g., moving part of the LB flow generation directly to
ovn-controllers - those did come at a cost of duplicating some NB config
information (LBs synced to the SB).

NOTE: The following items were discussed in the ad-hoc session, after
the official end of the conference:

7. The "first" upgrade scenario: part of the cluster using ovn-local and
part of the cluster (including "old” central nodes) running the legacy
mode.  Some options were discussed:

a. Ilya suggested running two sets of databases: the "old" SB for
chassis that haven’t upgraded yet and the "new"
Status/Operational/Whatever database ovn-local needs moving forward.

In his suggestion ovn-northd (upgraded) would be the intermediate actor
forwarding contents written by the upgraded ovn-locals in the new
databases to the legacy SB

b. Another option would be to run a "SB forwarder/proxy" on the upgraded
nodes: ovn-local would be unaware of its existence, it would only write
to the new "Status" database; the "SB forwarder" would monitor the
"Status" and propagate the information to the "legacy SB" that’s still
running on the central node for the not-yet-upgraded chassis.

8. Taking things a step further and adding support for using other
databases for OVN config/status storage (e.g., etcd).

Numan argued that we may anyway need a translation layer in the new
ovn-local during one of the transition phases described in the talk in
order to abstract out database access (e.g., the OF generation part in
ovn-controller currently needs logical flows to be parsed in memory); we
could consider keeping that in place even after ovn-local is the only
way of deploying OVN; we would repurpose it to provide an abstraction
for fetching/pushing data to the NB/Status databases.

9. From my perspective one of the most important requirement for this
whole effort is maintaining OVN backwards compatibility from CMS
perspective.  There might be some small (hopefully) changes the CMS
would have to put in place for the first upgrade to ovn-local but
ideally we should avoid needing other changes in the CMS. This might
also mean:

a. transparently making ovn-sbctl work with ovn-local deployments
b. transparently making ovn-trace work with ovn-local deployments

10. It was raised that this might be a good moment to start using a
different programming language (probably Rust) for implementing the new
parts that will be required for ovn-local.

11. There was also a small debate whether we should keep logical flows
or not:
a. e.g., for debugging (easier to read than OpenFlow)
b. OTOH this would incur extra computation cost

_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

Reply via email to