Am Tue, Nov 17, 2026 at 11:40:15AM -0500 schrieb Numan Siddique: > On Mon, Nov 16, 2026 at 4:06 PM Mark Michelson <[email protected]> wrote: > > > > Hi Han > > > > On Mon, Nov 16, 2026 at 3:44 PM Han Zhou <[email protected]> wrote: > > > > > > On Mon, Nov 16, 2026 at 8:28 AM Mark Michelson <[email protected]> > > > wrote: > > > > > > > > Hi Dumitru, thanks for the summary of everything. I don't want to > > > > comment too much on this since I helped contribute to the original > > > > document, but I do have one comment in-line below: > > > > > > > > On Mon, Nov 16, 2026 at 6:46 AM Dumitru Ceara <[email protected]> wrote: > > > > > > > > > > Hi everyone, > > > > > > > > > > First of all, thank you to those who could attend the conference and > > > > > the > > > > > "ovn-local: A new proposed architecture for OVN" discussion panel: > > > > > > > > > > https://www.openvswitch.org/support/ovscon2026/#t26 > > > > > > > > > > Some references, also for those who couldn't make it live (or online): > > > > > > > > > > - Slides presented at the conference: > > > > > https://drive.google.com/file/d/1TBDrqoYk9EHseqSaif6xKt4ePYncSocb > > > > > > > > > > - Recording of Mark's presentation (shared at the conference): > > > > > https://drive.google.com/file/d/1mwLmZQmUpZMZ3T-DMtkvo6xQaq867Ebc > > > > > > > > > > - A snapshot of the document that lead after discussion between Red > > > > > Hat > > > > > internal OVN team members to the proposal we shared at the conference: > > > > > https://drive.google.com/file/d/1w7IwB6UX-KbzUChBZb9yGi42yPIP7NZH/ > > > > > > > > > > As usual, it takes some time after the conference until we have > > > > > everything ready to share publicly so we don't (yet) have the > > > > > recording > > > > > of the actual panel. But I'll try to summarize as best as I can (at > > > > > the > > > > > end of this email) what was discussed during the actual session but > > > > > also > > > > > during the ad-hoc/hallway discussions after the official session. > > > > > > > > > > I think at this point quite a few members of the community have > > > > > expressed interest in the initiative and it might be a good idea to > > > > > start planning some of the work that might be required for this. As > > > > > you'll see below, I don't think there were any unsolvable technical > > > > > issues but there will definitely be challenges. > > > > > > > > > > In order to get things going, I'd like to ask folks to think about > > > > > timelines regarding this new initiative. > > > > > > > > > > I personally am quite confident that we can try to deliver some of the > > > > > changes suggested in the ovn-local document (e.g., phase 1) as early > > > > > as > > > > > the next development cycle (27.09) and that there's a high chance to > > > > > implement the migration to the new architecture before our next LTS > > > > > release (28.03). But that's my personal opinion, I'd like to hear > > > > > others too. :) > > > > > > > > One vitally important thing to point out is that the phases in the > > > > linked Google Doc do NOT align with the phases that were laid out in > > > > the presentation at the conference. A quick translation: > > > > > > > > Doc phase 1 (create wrapper daemon): not in presentation > > > > Doc phase 2 (Delete southbound database)-> presentation phase 1 > > > > Doc phase 3 (Eliminate unnecessary database tables)-> presentation > > > > phase 2 > > > > Doc phase 4 (Bypass database writes for local data)-> presentation > > > > phase 4 > > > > Doc phase 5 (Combine ovn-northd and ovn-controller)-> presentation > > > > phase 3 > > > > Presentation phase 5 (Profile and optimize) -> Not in the document > > > > > > > > In summary, the doc phase 1 was not in the presentation since it was > > > > mostly a mechanical change, and it's really only usable by a subset of > > > > OVN deployments (specifically, those in an ovn-kubernetes-like > > > > deployment where ovn-northd and ovn-controller run on the same > > > > system). > > > > > > A quick question here: > > > > > > In the document phase1 it says: > > > "This mostly benefits the use case of running all OVN components on a > > > single node. Clustered > > > OVN deployments will get the upgrade and deployment benefits, but ... " > > > > > > Sorry I didn't understand how clustered deployments would be supported > > > by this, and how they would benefit. > > > Does it mean we would be running all processes on both central and > > > distributed nodes? Would the connection between ovn-controller on > > > chassis and SB-DB on central still work? Maybe it is not difficult to > > > make iit work one way or another, but at least I don't see any benefit > > > here for the clustered (i.e. central) deployments. > > > > > > Did I misunderstand anything? > > > > You did not misunderstand anything. The rest of the sentence after the > > "but" says: "but since identical ovn-northd processes and identical > > southbound databases will be running on multiple compute nodes, there > > will be redundant processing on each compute node." So no, northd and > > the southbound database would not be centralized. > > > > The idea would be that ovn-northd, the southbound database and > > ovn-controller would still exist as individual processes, but could be > > managed by a single software package. The actual ovn-local package at > > this stage would likely be a simple systemd service that wraps the > > processes. The reason the upgrade benefits are mentioned is that > > instead of having to upgrade ovn-central and ovn-host packages, there > > would be a single package to upgrade. However, as the doc mentions > > doing this on a clustered system also introduces the problem that > > running ovn-northd and the southbound database on every node creates > > redundant processing, as well as completely ballooning the amount of > > southbound data that is present on every system. The idea is that this > > then transitions into the later phases of the document that eliminate > > the redundant elements (remove southbound database, combine ovn-northd > > and ovn-controller processes, etc.). > > > > Honestly, trying to sell this as any sort of benefit for a clustered > > system is flimsy. It is only a benefit for ovn-kubernetes-like systems > > where everything is run on a single node. When writing the doc, the > > idea of introducing the ovn-local package as an early phase made sense > > to me, since it provided a concrete step towards moving towards the > > other phases. But I think that when it comes to actually implementing > > this, imposing an ovn-local process on everyone can wait until close > > to the end of the work. > > > > I think there could be a benefit to having an OPTIONAL ovn-local > > package as an early goal. This way, ovn-kubernetes-like deployments > > could switch over to the ovn-local package early, but clustered > > systems could ignore its existence entirely and keep on doing what > > they do until the later phases are implemented. Once the other phases > > are implemented, then everyone would benefit from the rearchitected > > OVN. > > > > > > > > Best, > > > Han > > > > > > > It could also completely be skipped over during development, > > > > potentially. Plus, deleting the southbound database makes for an > > > > impactful phase 1 in a presentation :). Doc phases 3 and 4 were > > > > swapped for the presentation based on feedback I received before > > > > giving the presentation. Profiling and optimizing were not mentioned > > > > in the doc, mainly because it is assumed. It was mentioned in the > > > > presentation just to ensure attendees that we would not just blindly > > > > make the changes and assume that they improved performance. > > > > > > > > Now to address what Dumitru mentioned :) > > > > > > > > For the most part, I think this work is all-or-nothing when it comes > > > > to releasing it. For instance, if we eliminate the southbound database > > > > but don't do any of the phases after, then the northbound database > > > > would be much larger than it needs to be, and it means keeping > > > > unnecessary tables around for backwards compatibility. Similarly, if > > > > we combine ovn-northd and ovn-controller, but we still write data to > > > > the database unnecessarily or don't optimize away unnecessary code, we > > > > end up with "centralized" logic being run redundantly on all nodes. It > > > > also means there could be awkward intermediate releases of OVN that > > > > have odd upgrade instructions that would not apply to future or past > > > > releases. > > > > > > > > I think that Doc phase 1 (create a wrapper daemon) is probably the > > > > only thing that we might release piecemeal. The other work likely > > > > needs to be completed in its entirety before we release it. Doc phase > > > > 1 is also the most trivial, relatively speaking, of all of the phases. > > > > I agree that it could be done in time for 27.09. I'm not so sure about > > > > hitting 28.03 with the rest of the work, but if we can do it, then > > > > great! It's hard for me to put an estimate on the work yet, since we > > > > don't have concrete work items defined yet. The phases in the document > > > > and presentation are high-level descriptions that don't get into the > > > > lower-level details of what needs to be changed. > > > > > > > > > > > > > > Please read further for the notes I took during the conference. > > > > > > > > > > Best regards, > > > > > Dumitru > > > > > --- > > > > > > > > > > My notes from the panel and follow up discussion: > > > > > ------------------------------------------------- > > > > > DISCLAIMER: I wrote this down after the fact so there might be > > > > > inaccuracies. > > > > > > > > > > The main take away from the discussion was that OVN users in the > > > > > audience were interested in moving towards such an architecture. It’s > > > > > also in-line with other talks that were presented at the conference > > > > > (e.g., Crusoe is moving towards a custom ovn-ic deployment similar to > > > > > the OpenShift/ovn-kubernetes one - essentially one OVN per node, > > > > > Stackit > > > > > has issues due to the large size of the SB database in their classic > > > > > centralized deployment). > > > > > > > > > > There were a few questions/remarks from the audience and afterwards > > > > > during the ad-hoc discussion. I'll list everything (including > > > > > answers/comments) here: > > > > > > > > > > 1. How can we deal with the "synchronization work that ovn-northd > > > > > currently does, e.g., allocating SB datapath and port_binding tunnel > > > > > keys? > > > > > > > > > > There were some options mentioned during the session: > > > > > a. fully distributed and predictable/consistent generation of the keys > > > > > by individual ovn-local instances. This has IMO the risk of > > > > > complicating upgrades if we ever decide to change the algorithm used > > > > > to > > > > > generate keys. > > > > > > > > > > b. adding support to ovsdb to "run stored procedures" and move the key > > > > > allocation logic there > > > > > > > > > > c. mimic what ovn-kubernetes does and have a "cluster manager", a > > > > > centralized component, essentially a way simpler ovn-northd that still > > > > > manages keys even for ovn-local deployments > > > > > > > > > > 2. Would moving to ovn-local shift the performance issues from the > > > > > central nodes to the chassis? That’s because ovn-local would now > > > > > perform the work that ovn-northd does in the current deployment > > > > > (centralized). > > > > > > > > > > I think we can't provide a definitive answer before experimenting with > > > > > the proposed new architecture but we expect that in the final > > > > > ovn-local > > > > > deployment only locally relevant logical switches/routers would be > > > > > processed (reducing compute time to values likely similar to what > > > > > ovn-controller has to process today); the large SB contents would be > > > > > gone, reducing time to process those in the IDL (json handling). > > > > > > > > > > 3. Would moving to ovn-local cause upgrade incompatibility? I.e., in > > > > > very large clusters upgrades can take days/months. > > > > > > > > > > This has two aspects: > > > > > a. the first upgrade, from centralized OVN to ovn-local > > > > > b. subsequent upgrades, from ovn-local version x to ovn-local version > > > > > x+1 > > > > > > > > > > "a" was discussed separately quite extensively during the ad-hoc > > > > > session > > > > > after the conference, I'll detail that later. > > > > > > > > > > for "b”, taking the ovn-kubernetes example, we probably have a chance > > > > > of > > > > > incompatibility even with today’s architecture. That’s because > > > > > whenever > > > > > a packet needs to be tunneled to a remote chassis (i.e., it has been > > > > > received on a logical port on chassis-1 and needs to be sent out on a > > > > > logical port on chassis-2) we are splitting that datapath’s logical > > > > > pipeline across two nodes: ingress pipeline is executed on chassis-1 > > > > > while egress pipeline of the same switch/router is executed on > > > > > chassis-2. > > > > > > > > > > So, if a change happens that involves, for example, moving stages from > > > > > one pipeline (ingress or egress) to the other then these may cause > > > > > inconsistent packet forwarding. > > > > > > > > > > In general, this is handled today through feature flags (each chassis > > > > > advertises its "supported” feature set and ovn-northd only generates > > > > > logical flows that are compatible with the "oldest” ovn-controller). > > > > > Moving to ovn-local, we’d have to be even more careful with these > > > > > kinds > > > > > of situations but, IMO, nothing new here. > > > > > > > > > > 4. What about the shared state that’s currently stored in the > > > > > Southbound > > > > > (pushed up by ovn-controllers)? E.g., mac-binding, fdb, learned > > > > > routes. > > > > > > > > > > That’s still needed in the ovn-local mode; either stored as new tables > > > > > in the (new) NB database or as a "Status/Operational/etc” database. > > > > > > > > > > 5. Numan suggested reviving his old RFC work that moved logical flow > > > > > generation to ovn-controller or at least first focusing on removing > > > > > Logical_Flow from the "new” ovn-local database. > > > > > > > > > > https://mail.openvswitch.org/pipermail/ovs-dev/2021-June/384488.html > > > > > > > > > > I think the ovn-local "Phase 2" part of the proposal presented at the > > > > > conference covers that and we could likely start with removing Logical > > > > > Flows but it's my personal opinion that it would require less work > > > > > overall if we first implement "Phase 1" of the proposal ("Delete the > > > > > Southbound database"). > > Thanks for putting everything together and sharing the details. > > If I understand correctly phase 1 will eliminate SB DB and the logical > flows will > be stored in the NB DB ? Please correct me if I'm wrong. > > IMHO both can go in parallel. If one gets accepted before the other, > it should not be > hard to adjust. > > > > > > > > > > > > > 6. Is one of the problems with the current architecture that we end up > > > > > pushing "too detailed” information to the chassis from the central > > > > > OVN node? > > > > > > > > > > Yes, most likely, and that’s inline with other targeted optimizations > > > > > we > > > > > did in the past, e.g., moving part of the LB flow generation directly > > > > > to > > > > > ovn-controllers - those did come at a cost of duplicating some NB > > > > > config > > > > > information (LBs synced to the SB). > > > > > > > > > > NOTE: The following items were discussed in the ad-hoc session, after > > > > > the official end of the conference: > > > > > > > > > > 7. The "first" upgrade scenario: part of the cluster using ovn-local > > > > > and > > > > > part of the cluster (including "old” central nodes) running the legacy > > > > > mode. Some options were discussed: > > > > > > > > > > a. Ilya suggested running two sets of databases: the "old" SB for > > > > > chassis that haven’t upgraded yet and the "new" > > > > > Status/Operational/Whatever database ovn-local needs moving forward. > > > > > > > > When you say "Status/Operational/Whatever database", is this just a > > > > rebranding of the current northbound database? Or are you referring to > > > > actually having a new database in addition to the current northbound > > > > database? If it's the former, then that aligns with what I was > > > > picturing when writing the docs. If it's the latter, then can you > > > > explain the need for an additional database? > > > > > > > > > > > > > > In his suggestion ovn-northd (upgraded) would be the intermediate > > > > > actor > > > > > forwarding contents written by the upgraded ovn-locals in the new > > > > > databases to the legacy SB > > > > > > > > > > b. Another option would be to run a "SB forwarder/proxy" on the > > > > > upgraded > > > > > nodes: ovn-local would be unaware of its existence, it would only > > > > > write > > > > > to the new "Status" database; the "SB forwarder" would monitor the > > > > > "Status" and propagate the information to the "legacy SB" that’s still > > > > > running on the central node for the not-yet-upgraded chassis. > > > > > > > > > > 8. Taking things a step further and adding support for using other > > > > > databases for OVN config/status storage (e.g., etcd). > > > > > > > > > > Numan argued that we may anyway need a translation layer in the new > > > > > ovn-local during one of the transition phases described in the talk in > > > > > order to abstract out database access (e.g., the OF generation part in > > > > > ovn-controller currently needs logical flows to be parsed in memory); > > > > > we > > > > > could consider keeping that in place even after ovn-local is the only > > > > > way of deploying OVN; we would repurpose it to provide an abstraction > > > > > for fetching/pushing data to the NB/Status databases. > > > > > > > > > > 9. From my perspective one of the most important requirement for this > > > > > whole effort is maintaining OVN backwards compatibility from CMS > > > > > perspective. There might be some small (hopefully) changes the CMS > > > > > would have to put in place for the first upgrade to ovn-local but > > > > > ideally we should avoid needing other changes in the CMS. This might > > > > > also mean: > > > > > > > > > > a. transparently making ovn-sbctl work with ovn-local deployments > > > > > b. transparently making ovn-trace work with ovn-local deployments > > > > > > > > > > 10. It was raised that this might be a good moment to start using a > > > > > different programming language (probably Rust) for implementing the > > > > > new > > > > > parts that will be required for ovn-local. > > > > > > > > > > 11. There was also a small debate whether we should keep logical flows > > > > > or not: > > > > > a. e.g., for debugging (easier to read than OpenFlow) > > > > > b. OTOH this would incur extra computation cost > > > > > > > > Even without the Logical_Flow database table, there is still some need > > > > for the use of logical flows since users can provide matches as part > > > > of an ACL or Logical_Router_Policy definition. > > > > > > > > Other than that, the only real use for logical flows in a standard OVN > > > > deployment would be for debugging. If we think of this as only a tool > > > > for debugging, I think trying to keep logical flows around long-term > > > > is going to lead to frustration and inaccuracy. > > > > > > > > As we accustom ourselves to writing OpenFlow directly, we eventually > > > > will reach a point where it makes little sense to define new logical > > > > flow actions that align with the OpenFlow we create. For example, we > > > > currently have logical flow actions that translate to a resubmit to > > > > some high numbered OpenFlow table to look up a specific value and set > > > > that value in some register (lookup_arp() for example). Now let's > > > > imagine that we no longer need to write logical flows, and we do > > > > something similar directly in OpenFlow. We would need to artificially > > > > define and document a new logical flow action solely for the purposes > > > > of debugging, even though that logical flow action would never > > > > actually be used anywhere in the code. Similar things are likely to > > > > happen with specific registers that we use in OpenFlow matches. > > > > Currently, we define certain shortcuts for logical flow matches and > > > > actions (e.g. "flags.loopback" translates to "reg10[0]"). Eventually, > > > > once we are defining everything in OpenFlow, such shortcuts will be > > > > unnecessary to define. Logical flows will eventually bitrot and > > > > they'll be unreliable to use for debugging since they haven't actually > > > > kept up with reality. If our actual development path is to define > > > > OpenFlow directly, then the accompanying "fake" logical flows could > > > > become inaccurate in the case where we update an OpenFlow flow but > > > > forget to update the accompanying logical flow. The code will still > > > > work, but now the debugging aid will show inaccurate information. > > > I have a different take on the logical flows. I don't think we should > eliminate the use of > logical flows. Logical flows provide an excellent wrapper for Openflow rules. > IMO expressing a pipeline as logical flows is easier than openflow rules > and I think it will be easier to add new features. > > How I'd use logical flows is similar to the RFC which I had proposed earlier. > (https://github.com/numansiddique/ovn/blob/rfc_lflows_in_controller_v2_p16/lib/lflow.c#L148) > > Lets a logical switch/datapath - sw0 is relevant to a node 'A'. > ovn-local on that node > will call a library function (something like below) which generates > all the logical flows for > this logical switch and returns. And then the same logical flows > returned will be converted > to openflow rules using the expr parser.
Hi Numan, We did some profiling in our environment on a heavily loaded gateway chassis. We did profile the recompute of the lflow_output engine node in ovn-controller (without the lflow cache). In there we found that we take around 7% of the total runtime for ovnacts_parse_string and 3.5% of the total time for expr_parse_string (for matches). I am not sure how much of the time in there really is spend on parsing strings. But we should definately keep in mind the cpu time such conversion will take and if we gain benefits in skipping lflows. Thanks, Felix > > static void build_dp_lflows(struct hmap *lflows, struct local_datapath *) > > And when a user runs the command - "ovn-sbctl dump-flows <datapath>" > , ovn-sbctl (or ovn-nbctl) > utility will call the same function - build_lswitch_dp_lflows() and > dumps the logical flows. > > > Thanks > Numan > > > > > > > > > > > I think that any steps we take to try to preserve logical flows will > > > > be counter to the goals that the ovn-local project is trying to > > > > achieve. Deletion of the Logical_Flow database table is one of the > > > > biggest wins for this project since it is massive. Logical flows > > > > really only make sense as a centralized concept that describes the > > > > relationship of the logical datapaths. I think instead of trying to > > > > preserve logical flows as a debugging tool, we could focus instead on > > > > providing new debugging tools that work for ovn-local. > > > > > > > > > > > > > > _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
