On Mon, Nov 16, 2026 at 11:28 AM Mark Michelson <[email protected]> wrote: > > Hi Dumitru, thanks for the summary of everything. I don't want to > comment too much on this since I helped contribute to the original > document, but I do have one comment in-line below:
Sorry, I wrote this top paragraph before realizing I more to respond to than just one item. I actually have three inline replies, so please be sure to scroll down to see them all. > > On Mon, Nov 16, 2026 at 6:46 AM Dumitru Ceara <[email protected]> wrote: > > > > Hi everyone, > > > > First of all, thank you to those who could attend the conference and the > > "ovn-local: A new proposed architecture for OVN" discussion panel: > > > > https://www.openvswitch.org/support/ovscon2026/#t26 > > > > Some references, also for those who couldn't make it live (or online): > > > > - Slides presented at the conference: > > https://drive.google.com/file/d/1TBDrqoYk9EHseqSaif6xKt4ePYncSocb > > > > - Recording of Mark's presentation (shared at the conference): > > https://drive.google.com/file/d/1mwLmZQmUpZMZ3T-DMtkvo6xQaq867Ebc > > > > - A snapshot of the document that lead after discussion between Red Hat > > internal OVN team members to the proposal we shared at the conference: > > https://drive.google.com/file/d/1w7IwB6UX-KbzUChBZb9yGi42yPIP7NZH/ > > > > As usual, it takes some time after the conference until we have > > everything ready to share publicly so we don't (yet) have the recording > > of the actual panel. But I'll try to summarize as best as I can (at the > > end of this email) what was discussed during the actual session but also > > during the ad-hoc/hallway discussions after the official session. > > > > I think at this point quite a few members of the community have > > expressed interest in the initiative and it might be a good idea to > > start planning some of the work that might be required for this. As > > you'll see below, I don't think there were any unsolvable technical > > issues but there will definitely be challenges. > > > > In order to get things going, I'd like to ask folks to think about > > timelines regarding this new initiative. > > > > I personally am quite confident that we can try to deliver some of the > > changes suggested in the ovn-local document (e.g., phase 1) as early as > > the next development cycle (27.09) and that there's a high chance to > > implement the migration to the new architecture before our next LTS > > release (28.03). But that's my personal opinion, I'd like to hear > > others too. :) > > One vitally important thing to point out is that the phases in the > linked Google Doc do NOT align with the phases that were laid out in > the presentation at the conference. A quick translation: > > Doc phase 1 (create wrapper daemon): not in presentation > Doc phase 2 (Delete southbound database)-> presentation phase 1 > Doc phase 3 (Eliminate unnecessary database tables)-> presentation phase 2 > Doc phase 4 (Bypass database writes for local data)-> presentation phase 4 > Doc phase 5 (Combine ovn-northd and ovn-controller)-> presentation phase 3 > Presentation phase 5 (Profile and optimize) -> Not in the document > > In summary, the doc phase 1 was not in the presentation since it was > mostly a mechanical change, and it's really only usable by a subset of > OVN deployments (specifically, those in an ovn-kubernetes-like > deployment where ovn-northd and ovn-controller run on the same > system). It could also completely be skipped over during development, > potentially. Plus, deleting the southbound database makes for an > impactful phase 1 in a presentation :). Doc phases 3 and 4 were > swapped for the presentation based on feedback I received before > giving the presentation. Profiling and optimizing were not mentioned > in the doc, mainly because it is assumed. It was mentioned in the > presentation just to ensure attendees that we would not just blindly > make the changes and assume that they improved performance. > > Now to address what Dumitru mentioned :) > > For the most part, I think this work is all-or-nothing when it comes > to releasing it. For instance, if we eliminate the southbound database > but don't do any of the phases after, then the northbound database > would be much larger than it needs to be, and it means keeping > unnecessary tables around for backwards compatibility. Similarly, if > we combine ovn-northd and ovn-controller, but we still write data to > the database unnecessarily or don't optimize away unnecessary code, we > end up with "centralized" logic being run redundantly on all nodes. It > also means there could be awkward intermediate releases of OVN that > have odd upgrade instructions that would not apply to future or past > releases. > > I think that Doc phase 1 (create a wrapper daemon) is probably the > only thing that we might release piecemeal. The other work likely > needs to be completed in its entirety before we release it. Doc phase > 1 is also the most trivial, relatively speaking, of all of the phases. > I agree that it could be done in time for 27.09. I'm not so sure about > hitting 28.03 with the rest of the work, but if we can do it, then > great! It's hard for me to put an estimate on the work yet, since we > don't have concrete work items defined yet. The phases in the document > and presentation are high-level descriptions that don't get into the > lower-level details of what needs to be changed. > > > > > Please read further for the notes I took during the conference. > > > > Best regards, > > Dumitru > > --- > > > > My notes from the panel and follow up discussion: > > ------------------------------------------------- > > DISCLAIMER: I wrote this down after the fact so there might be inaccuracies. > > > > The main take away from the discussion was that OVN users in the > > audience were interested in moving towards such an architecture. It’s > > also in-line with other talks that were presented at the conference > > (e.g., Crusoe is moving towards a custom ovn-ic deployment similar to > > the OpenShift/ovn-kubernetes one - essentially one OVN per node, Stackit > > has issues due to the large size of the SB database in their classic > > centralized deployment). > > > > There were a few questions/remarks from the audience and afterwards > > during the ad-hoc discussion. I'll list everything (including > > answers/comments) here: > > > > 1. How can we deal with the "synchronization work that ovn-northd > > currently does, e.g., allocating SB datapath and port_binding tunnel keys? > > > > There were some options mentioned during the session: > > a. fully distributed and predictable/consistent generation of the keys > > by individual ovn-local instances. This has IMO the risk of > > complicating upgrades if we ever decide to change the algorithm used to > > generate keys. > > > > b. adding support to ovsdb to "run stored procedures" and move the key > > allocation logic there > > > > c. mimic what ovn-kubernetes does and have a "cluster manager", a > > centralized component, essentially a way simpler ovn-northd that still > > manages keys even for ovn-local deployments > > > > 2. Would moving to ovn-local shift the performance issues from the > > central nodes to the chassis? That’s because ovn-local would now > > perform the work that ovn-northd does in the current deployment > > (centralized). > > > > I think we can't provide a definitive answer before experimenting with > > the proposed new architecture but we expect that in the final ovn-local > > deployment only locally relevant logical switches/routers would be > > processed (reducing compute time to values likely similar to what > > ovn-controller has to process today); the large SB contents would be > > gone, reducing time to process those in the IDL (json handling). > > > > 3. Would moving to ovn-local cause upgrade incompatibility? I.e., in > > very large clusters upgrades can take days/months. > > > > This has two aspects: > > a. the first upgrade, from centralized OVN to ovn-local > > b. subsequent upgrades, from ovn-local version x to ovn-local version x+1 > > > > "a" was discussed separately quite extensively during the ad-hoc session > > after the conference, I'll detail that later. > > > > for "b”, taking the ovn-kubernetes example, we probably have a chance of > > incompatibility even with today’s architecture. That’s because whenever > > a packet needs to be tunneled to a remote chassis (i.e., it has been > > received on a logical port on chassis-1 and needs to be sent out on a > > logical port on chassis-2) we are splitting that datapath’s logical > > pipeline across two nodes: ingress pipeline is executed on chassis-1 > > while egress pipeline of the same switch/router is executed on chassis-2. > > > > So, if a change happens that involves, for example, moving stages from > > one pipeline (ingress or egress) to the other then these may cause > > inconsistent packet forwarding. > > > > In general, this is handled today through feature flags (each chassis > > advertises its "supported” feature set and ovn-northd only generates > > logical flows that are compatible with the "oldest” ovn-controller). > > Moving to ovn-local, we’d have to be even more careful with these kinds > > of situations but, IMO, nothing new here. > > > > 4. What about the shared state that’s currently stored in the Southbound > > (pushed up by ovn-controllers)? E.g., mac-binding, fdb, learned routes. > > > > That’s still needed in the ovn-local mode; either stored as new tables > > in the (new) NB database or as a "Status/Operational/etc” database. > > > > 5. Numan suggested reviving his old RFC work that moved logical flow > > generation to ovn-controller or at least first focusing on removing > > Logical_Flow from the "new” ovn-local database. > > > > https://mail.openvswitch.org/pipermail/ovs-dev/2021-June/384488.html > > > > I think the ovn-local "Phase 2" part of the proposal presented at the > > conference covers that and we could likely start with removing Logical > > Flows but it's my personal opinion that it would require less work > > overall if we first implement "Phase 1" of the proposal ("Delete the > > Southbound database"). > > > > 6. Is one of the problems with the current architecture that we end up > > pushing "too detailed” information to the chassis from the central OVN node? > > > > Yes, most likely, and that’s inline with other targeted optimizations we > > did in the past, e.g., moving part of the LB flow generation directly to > > ovn-controllers - those did come at a cost of duplicating some NB config > > information (LBs synced to the SB). > > > > NOTE: The following items were discussed in the ad-hoc session, after > > the official end of the conference: > > > > 7. The "first" upgrade scenario: part of the cluster using ovn-local and > > part of the cluster (including "old” central nodes) running the legacy > > mode. Some options were discussed: > > > > a. Ilya suggested running two sets of databases: the "old" SB for > > chassis that haven’t upgraded yet and the "new" > > Status/Operational/Whatever database ovn-local needs moving forward. > > When you say "Status/Operational/Whatever database", is this just a > rebranding of the current northbound database? Or are you referring to > actually having a new database in addition to the current northbound > database? If it's the former, then that aligns with what I was > picturing when writing the docs. If it's the latter, then can you > explain the need for an additional database? > > > > > In his suggestion ovn-northd (upgraded) would be the intermediate actor > > forwarding contents written by the upgraded ovn-locals in the new > > databases to the legacy SB > > > > b. Another option would be to run a "SB forwarder/proxy" on the upgraded > > nodes: ovn-local would be unaware of its existence, it would only write > > to the new "Status" database; the "SB forwarder" would monitor the > > "Status" and propagate the information to the "legacy SB" that’s still > > running on the central node for the not-yet-upgraded chassis. > > > > 8. Taking things a step further and adding support for using other > > databases for OVN config/status storage (e.g., etcd). > > > > Numan argued that we may anyway need a translation layer in the new > > ovn-local during one of the transition phases described in the talk in > > order to abstract out database access (e.g., the OF generation part in > > ovn-controller currently needs logical flows to be parsed in memory); we > > could consider keeping that in place even after ovn-local is the only > > way of deploying OVN; we would repurpose it to provide an abstraction > > for fetching/pushing data to the NB/Status databases. > > > > 9. From my perspective one of the most important requirement for this > > whole effort is maintaining OVN backwards compatibility from CMS > > perspective. There might be some small (hopefully) changes the CMS > > would have to put in place for the first upgrade to ovn-local but > > ideally we should avoid needing other changes in the CMS. This might > > also mean: > > > > a. transparently making ovn-sbctl work with ovn-local deployments > > b. transparently making ovn-trace work with ovn-local deployments > > > > 10. It was raised that this might be a good moment to start using a > > different programming language (probably Rust) for implementing the new > > parts that will be required for ovn-local. > > > > 11. There was also a small debate whether we should keep logical flows > > or not: > > a. e.g., for debugging (easier to read than OpenFlow) > > b. OTOH this would incur extra computation cost > > Even without the Logical_Flow database table, there is still some need > for the use of logical flows since users can provide matches as part > of an ACL or Logical_Router_Policy definition. > > Other than that, the only real use for logical flows in a standard OVN > deployment would be for debugging. If we think of this as only a tool > for debugging, I think trying to keep logical flows around long-term > is going to lead to frustration and inaccuracy. > > As we accustom ourselves to writing OpenFlow directly, we eventually > will reach a point where it makes little sense to define new logical > flow actions that align with the OpenFlow we create. For example, we > currently have logical flow actions that translate to a resubmit to > some high numbered OpenFlow table to look up a specific value and set > that value in some register (lookup_arp() for example). Now let's > imagine that we no longer need to write logical flows, and we do > something similar directly in OpenFlow. We would need to artificially > define and document a new logical flow action solely for the purposes > of debugging, even though that logical flow action would never > actually be used anywhere in the code. Similar things are likely to > happen with specific registers that we use in OpenFlow matches. > Currently, we define certain shortcuts for logical flow matches and > actions (e.g. "flags.loopback" translates to "reg10[0]"). Eventually, > once we are defining everything in OpenFlow, such shortcuts will be > unnecessary to define. Logical flows will eventually bitrot and > they'll be unreliable to use for debugging since they haven't actually > kept up with reality. If our actual development path is to define > OpenFlow directly, then the accompanying "fake" logical flows could > become inaccurate in the case where we update an OpenFlow flow but > forget to update the accompanying logical flow. The code will still > work, but now the debugging aid will show inaccurate information. > > I think that any steps we take to try to preserve logical flows will > be counter to the goals that the ovn-local project is trying to > achieve. Deletion of the Logical_Flow database table is one of the > biggest wins for this project since it is massive. Logical flows > really only make sense as a centralized concept that describes the > relationship of the logical datapaths. I think instead of trying to > preserve logical flows as a debugging tool, we could focus instead on > providing new debugging tools that work for ovn-local. > > _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
