On Mon, Nov 16, 2026 at 11:28 AM Mark Michelson <[email protected]> wrote:
>
> Hi Dumitru, thanks for the summary of everything. I don't want to
> comment too much on this since I helped contribute to the original
> document, but I do have one comment in-line below:

Sorry, I wrote this top paragraph before realizing I more to respond
to than just one item. I actually have three inline replies, so please
be sure to scroll down to see them all.

>
> On Mon, Nov 16, 2026 at 6:46 AM Dumitru Ceara <[email protected]> wrote:
> >
> > Hi everyone,
> >
> > First of all, thank you to those who could attend the conference and the
> > "ovn-local: A new proposed architecture for OVN" discussion panel:
> >
> > https://www.openvswitch.org/support/ovscon2026/#t26
> >
> > Some references, also for those who couldn't make it live (or online):
> >
> > - Slides presented at the conference:
> > https://drive.google.com/file/d/1TBDrqoYk9EHseqSaif6xKt4ePYncSocb
> >
> > - Recording of Mark's presentation (shared at the conference):
> > https://drive.google.com/file/d/1mwLmZQmUpZMZ3T-DMtkvo6xQaq867Ebc
> >
> > - A snapshot of the document that lead after discussion between Red Hat
> > internal OVN team members to the proposal we shared at the conference:
> > https://drive.google.com/file/d/1w7IwB6UX-KbzUChBZb9yGi42yPIP7NZH/
> >
> > As usual, it takes some time after the conference until we have
> > everything ready to share publicly so we don't (yet) have the recording
> > of the actual panel.  But I'll try to summarize as best as I can (at the
> > end of this email) what was discussed during the actual session but also
> > during the ad-hoc/hallway discussions after the official session.
> >
> > I think at this point quite a few members of the community have
> > expressed interest in the initiative and it might be a good idea to
> > start planning some of the work that might be required for this.  As
> > you'll see below, I don't think there were any unsolvable technical
> > issues but there will definitely be challenges.
> >
> > In order to get things going, I'd like to ask folks to think about
> > timelines regarding this new initiative.
> >
> > I personally am quite confident that we can try to deliver some of the
> > changes suggested in the ovn-local document (e.g., phase 1) as early as
> > the next development cycle (27.09) and that there's a high chance to
> > implement the migration to the new architecture before our next LTS
> > release (28.03).  But that's my personal opinion, I'd like to hear
> > others too. :)
>
> One vitally important thing to point out is that the phases in the
> linked Google Doc do NOT align with the phases that were laid out in
> the presentation at the conference. A quick translation:
>
> Doc phase 1 (create wrapper daemon): not in presentation
> Doc phase 2 (Delete southbound database)-> presentation phase 1
> Doc phase 3 (Eliminate unnecessary database tables)-> presentation phase 2
> Doc phase 4 (Bypass database writes for local data)-> presentation phase 4
> Doc phase 5 (Combine ovn-northd and ovn-controller)-> presentation phase 3
> Presentation phase 5 (Profile and optimize) -> Not in the document
>
> In summary, the doc phase 1 was not in the presentation since it was
> mostly a mechanical change, and it's really only usable by a subset of
> OVN deployments (specifically, those in an ovn-kubernetes-like
> deployment where ovn-northd and ovn-controller run on the same
> system). It could also completely be skipped over during development,
> potentially. Plus, deleting the southbound database makes for an
> impactful phase 1 in a presentation :). Doc phases 3 and 4 were
> swapped for the presentation based on feedback I received before
> giving the presentation. Profiling and optimizing were not mentioned
> in the doc, mainly because it is assumed. It was mentioned in the
> presentation just to ensure attendees that we would not just blindly
> make the changes and assume that they improved performance.
>
> Now to address what Dumitru mentioned :)
>
> For the most part, I think this work is all-or-nothing when it comes
> to releasing it. For instance, if we eliminate the southbound database
> but don't do any of the phases after, then the northbound database
> would be much larger than it needs to be, and it means keeping
> unnecessary tables around for backwards compatibility. Similarly, if
> we combine ovn-northd and ovn-controller, but we still write data to
> the database unnecessarily or don't optimize away unnecessary code, we
> end up with "centralized" logic being run redundantly on all nodes. It
> also means there could be awkward intermediate releases of OVN that
> have odd upgrade instructions that would not apply to future or past
> releases.
>
> I think that Doc phase 1 (create a wrapper daemon) is probably the
> only thing that we might release piecemeal. The other work likely
> needs to be completed in its entirety before we release it. Doc phase
> 1 is also the most trivial, relatively speaking, of all of the phases.
> I agree that it could be done in time for 27.09. I'm not so sure about
> hitting 28.03 with the rest of the work, but if we can do it, then
> great! It's hard for me to put an estimate on the work yet, since we
> don't have concrete work items defined yet. The phases in the document
> and presentation are high-level descriptions that don't get into the
> lower-level details of what needs to be changed.
>
> >
> > Please read further for the notes I took during the conference.
> >
> > Best regards,
> > Dumitru
> > ---
> >
> > My notes from the panel and follow up discussion:
> > -------------------------------------------------
> > DISCLAIMER: I wrote this down after the fact so there might be inaccuracies.
> >
> > The main take away from the discussion was that OVN users in the
> > audience were interested in moving towards such an architecture.  It’s
> > also in-line with other talks that were presented at the conference
> > (e.g., Crusoe is moving towards a custom ovn-ic deployment similar to
> > the OpenShift/ovn-kubernetes one - essentially one OVN per node, Stackit
> > has issues due to the large size of the SB database in their classic
> > centralized deployment).
> >
> > There were a few questions/remarks from the audience and afterwards
> > during the ad-hoc discussion.  I'll list everything (including
> > answers/comments) here:
> >
> > 1. How can we deal with the "synchronization work that ovn-northd
> > currently does, e.g., allocating SB datapath and port_binding tunnel keys?
> >
> > There were some options mentioned during the session:
> > a. fully distributed and predictable/consistent generation of the keys
> > by individual ovn-local instances.  This has IMO the risk of
> > complicating upgrades if we ever decide to change the algorithm used to
> > generate keys.
> >
> > b. adding support to ovsdb to "run stored procedures" and move the key
> > allocation logic there
> >
> > c. mimic what ovn-kubernetes does and have a "cluster manager", a
> > centralized component, essentially a way simpler ovn-northd that still
> > manages keys even for ovn-local deployments
> >
> > 2. Would moving to ovn-local shift the performance issues from the
> > central nodes to the chassis?  That’s because ovn-local would now
> > perform the work that ovn-northd does in the current deployment
> > (centralized).
> >
> > I think we can't provide a definitive answer before experimenting with
> > the proposed new architecture but we expect that in the final ovn-local
> > deployment only locally relevant logical switches/routers would be
> > processed (reducing compute time to values likely similar to what
> > ovn-controller has to process today); the large SB contents would be
> > gone, reducing time to process those in the IDL (json handling).
> >
> > 3. Would moving to ovn-local cause upgrade incompatibility?  I.e., in
> > very large clusters upgrades can take days/months.
> >
> > This has two aspects:
> > a. the first upgrade, from centralized OVN to ovn-local
> > b. subsequent upgrades, from ovn-local version x to ovn-local version x+1
> >
> > "a" was discussed separately quite extensively during the ad-hoc session
> > after the conference, I'll detail that later.
> >
> > for "b”, taking the ovn-kubernetes example, we probably have a chance of
> > incompatibility even with today’s architecture.  That’s because whenever
> > a packet needs to be tunneled to a remote chassis (i.e., it has been
> > received on a logical port on chassis-1 and needs to be sent out on a
> > logical port on chassis-2) we are splitting that datapath’s logical
> > pipeline across two nodes: ingress pipeline is executed on chassis-1
> > while egress pipeline of the same switch/router is executed on chassis-2.
> >
> > So, if a change happens that involves, for example, moving stages from
> > one pipeline (ingress or egress) to the other then these may cause
> > inconsistent packet forwarding.
> >
> > In general, this is handled today through feature flags (each chassis
> > advertises its "supported” feature set and ovn-northd only generates
> > logical flows that are compatible with the "oldest” ovn-controller).
> > Moving to ovn-local, we’d have to be even more careful with these kinds
> > of situations but, IMO, nothing new here.
> >
> > 4. What about the shared state that’s currently stored in the Southbound
> > (pushed up by ovn-controllers)?  E.g., mac-binding, fdb, learned routes.
> >
> > That’s still needed in the ovn-local mode; either stored as new tables
> > in the (new) NB database or as a "Status/Operational/etc” database.
> >
> > 5. Numan suggested reviving his old RFC work that moved logical flow
> > generation to ovn-controller or at least first focusing on removing
> > Logical_Flow from the "new” ovn-local database.
> >
> > https://mail.openvswitch.org/pipermail/ovs-dev/2021-June/384488.html
> >
> > I think the ovn-local "Phase 2" part of the proposal presented at the
> > conference covers that and we could likely start with removing Logical
> > Flows but it's my personal opinion that it would require less work
> > overall if we first implement "Phase 1" of the proposal ("Delete the
> > Southbound database").
> >
> > 6. Is one of the problems with the current architecture that we end up
> > pushing "too detailed” information to the chassis from the central OVN node?
> >
> > Yes, most likely, and that’s inline with other targeted optimizations we
> > did in the past, e.g., moving part of the LB flow generation directly to
> > ovn-controllers - those did come at a cost of duplicating some NB config
> > information (LBs synced to the SB).
> >
> > NOTE: The following items were discussed in the ad-hoc session, after
> > the official end of the conference:
> >
> > 7. The "first" upgrade scenario: part of the cluster using ovn-local and
> > part of the cluster (including "old” central nodes) running the legacy
> > mode.  Some options were discussed:
> >
> > a. Ilya suggested running two sets of databases: the "old" SB for
> > chassis that haven’t upgraded yet and the "new"
> > Status/Operational/Whatever database ovn-local needs moving forward.
>
> When you say "Status/Operational/Whatever database", is this just a
> rebranding of the current northbound database? Or are you referring to
> actually having a new database in addition to the current northbound
> database? If it's the former, then that aligns with what I was
> picturing when writing the docs. If it's the latter, then can you
> explain the need for an additional database?
>
> >
> > In his suggestion ovn-northd (upgraded) would be the intermediate actor
> > forwarding contents written by the upgraded ovn-locals in the new
> > databases to the legacy SB
> >
> > b. Another option would be to run a "SB forwarder/proxy" on the upgraded
> > nodes: ovn-local would be unaware of its existence, it would only write
> > to the new "Status" database; the "SB forwarder" would monitor the
> > "Status" and propagate the information to the "legacy SB" that’s still
> > running on the central node for the not-yet-upgraded chassis.
> >
> > 8. Taking things a step further and adding support for using other
> > databases for OVN config/status storage (e.g., etcd).
> >
> > Numan argued that we may anyway need a translation layer in the new
> > ovn-local during one of the transition phases described in the talk in
> > order to abstract out database access (e.g., the OF generation part in
> > ovn-controller currently needs logical flows to be parsed in memory); we
> > could consider keeping that in place even after ovn-local is the only
> > way of deploying OVN; we would repurpose it to provide an abstraction
> > for fetching/pushing data to the NB/Status databases.
> >
> > 9. From my perspective one of the most important requirement for this
> > whole effort is maintaining OVN backwards compatibility from CMS
> > perspective.  There might be some small (hopefully) changes the CMS
> > would have to put in place for the first upgrade to ovn-local but
> > ideally we should avoid needing other changes in the CMS. This might
> > also mean:
> >
> > a. transparently making ovn-sbctl work with ovn-local deployments
> > b. transparently making ovn-trace work with ovn-local deployments
> >
> > 10. It was raised that this might be a good moment to start using a
> > different programming language (probably Rust) for implementing the new
> > parts that will be required for ovn-local.
> >
> > 11. There was also a small debate whether we should keep logical flows
> > or not:
> > a. e.g., for debugging (easier to read than OpenFlow)
> > b. OTOH this would incur extra computation cost
>
> Even without the Logical_Flow database table, there is still some need
> for the use of logical flows since users can provide matches as part
> of an ACL or Logical_Router_Policy definition.
>
> Other than that, the only real use for logical flows in a standard OVN
> deployment would be for debugging. If we think of this as only a tool
> for debugging, I think trying to keep logical flows around long-term
> is going to lead to frustration and inaccuracy.
>
> As we accustom ourselves to writing OpenFlow directly, we eventually
> will reach a point where it makes little sense to define new logical
> flow actions that align with the OpenFlow we create. For example, we
> currently have logical flow actions that translate to a resubmit to
> some high numbered OpenFlow table to look up a specific value and set
> that value in some register (lookup_arp() for example). Now let's
> imagine that we no longer need to write logical flows, and we do
> something similar directly in OpenFlow. We would need to artificially
> define and document a new logical flow action solely for the purposes
> of debugging, even though that logical flow action would never
> actually be used anywhere in the code. Similar things are likely to
> happen with specific registers that we use in OpenFlow matches.
> Currently, we define certain shortcuts for logical flow matches and
> actions (e.g. "flags.loopback" translates to "reg10[0]"). Eventually,
> once we are defining everything in OpenFlow, such shortcuts will be
> unnecessary to define. Logical flows will eventually bitrot and
> they'll be unreliable to use for debugging since they haven't actually
> kept up with reality. If our actual development path is to define
> OpenFlow directly, then the accompanying "fake" logical flows could
> become inaccurate in the case where we update an OpenFlow flow but
> forget to update the accompanying logical flow. The code will still
> work, but now the debugging aid will show inaccurate information.
>
> I think that any steps we take to try to preserve logical flows will
> be counter to the goals that the ovn-local project is trying to
> achieve. Deletion of the Logical_Flow database table is one of the
> biggest wins for this project since it is massive. Logical flows
> really only make sense as a centralized concept that describes the
> relationship of the logical datapaths. I think instead of trying to
> preserve logical flows as a debugging tool, we could focus instead on
> providing new debugging tools that work for ovn-local.
> >

_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

Reply via email to