On Fri, Nov 20, 2026 at 3:56 PM Mark Michelson <[email protected]> wrote: > > Thanks for this detailed plan, Dumitru. > > On a high level, I really like the idea of an ovnd process that has > various modes that it can run under. This gives us lots of ways to > gradually roll out the changes. I have some inline notes below. > > On Fri, Nov 20, 2026 at 4:58 AM Dumitru Ceara <[email protected]> wrote: > > > > On 11/19/26 4:54 PM, Dumitru Ceara wrote: > > > Hi all, > > > > > > I'll try to reply inline to the various ongoing conversations. > > > > > > I'll also try to present a more detailed view of how I personally see > > > the ovn-local transition work happening (in the end of the email). I'll > > > try to address Felix's comments [0] in that part as well. > > > > > > [0] > > > https://mail.openvswitch.org/pipermail/ovs-dev/2026-November/438023.html > > > > > > On 11/18/26 8:33 AM, Han Zhou via dev wrote: > > >> On Tue, Nov 17, 2026 at 8:40 AM Numan Siddique <[email protected]> wrote: > > >>> > > >>> On Mon, Nov 16, 2026 at 4:06 PM Mark Michelson <[email protected]> > > >>> wrote: > > >>>> > > >>>> Hi Han > > >>>> > > >>>> On Mon, Nov 16, 2026 at 3:44 PM Han Zhou <[email protected]> wrote: > > >>>>> > > >>>>> On Mon, Nov 16, 2026 at 8:28 AM Mark Michelson <[email protected]> > > >>>>> wrote: > > >>>>>> > > >>>>>> Hi Dumitru, thanks for the summary of everything. I don't want to > > >>>>>> comment too much on this since I helped contribute to the original > > >>>>>> document, but I do have one comment in-line below: > > >>>>>> > > >>>>>> On Mon, Nov 16, 2026 at 6:46 AM Dumitru Ceara <[email protected]> > > >>>>>> wrote: > > >>>>>>> > > >>>>>>> Hi everyone, > > >>>>>>> > > >>>>>>> First of all, thank you to those who could attend the conference > > >>>>>>> and the > > >>>>>>> "ovn-local: A new proposed architecture for OVN" discussion panel: > > >>>>>>> > > >>>>>>> https://www.openvswitch.org/support/ovscon2026/#t26 > > >>>>>>> > > >>>>>>> Some references, also for those who couldn't make it live (or > > >>>>>>> online): > > >>>>>>> > > >>>>>>> - Slides presented at the conference: > > >>>>>>> https://drive.google.com/file/d/1TBDrqoYk9EHseqSaif6xKt4ePYncSocb > > >>>>>>> > > >>>>>>> - Recording of Mark's presentation (shared at the conference): > > >>>>>>> https://drive.google.com/file/d/1mwLmZQmUpZMZ3T-DMtkvo6xQaq867Ebc > > >>>>>>> > > >>>>>>> - A snapshot of the document that lead after discussion between Red > > >>>>>>> Hat > > >>>>>>> internal OVN team members to the proposal we shared at the > > >>>>>>> conference: > > >>>>>>> https://drive.google.com/file/d/1w7IwB6UX-KbzUChBZb9yGi42yPIP7NZH/ > > >>>>>>> > > >>>>>>> As usual, it takes some time after the conference until we have > > >>>>>>> everything ready to share publicly so we don't (yet) have the > > >>>>>>> recording > > >>>>>>> of the actual panel. But I'll try to summarize as best as I can > > >>>>>>> (at the > > >>>>>>> end of this email) what was discussed during the actual session but > > >>>>>>> also > > >>>>>>> during the ad-hoc/hallway discussions after the official session. > > >>>>>>> > > >>>>>>> I think at this point quite a few members of the community have > > >>>>>>> expressed interest in the initiative and it might be a good idea to > > >>>>>>> start planning some of the work that might be required for this. As > > >>>>>>> you'll see below, I don't think there were any unsolvable technical > > >>>>>>> issues but there will definitely be challenges. > > >>>>>>> > > >>>>>>> In order to get things going, I'd like to ask folks to think about > > >>>>>>> timelines regarding this new initiative. > > >>>>>>> > > >>>>>>> I personally am quite confident that we can try to deliver some of > > >>>>>>> the > > >>>>>>> changes suggested in the ovn-local document (e.g., phase 1) as > > >>>>>>> early as > > >>>>>>> the next development cycle (27.09) and that there's a high chance to > > >>>>>>> implement the migration to the new architecture before our next LTS > > >>>>>>> release (28.03). But that's my personal opinion, I'd like to hear > > >>>>>>> others too. :) > > >>>>>> > > >>>>>> One vitally important thing to point out is that the phases in the > > >>>>>> linked Google Doc do NOT align with the phases that were laid out in > > >>>>>> the presentation at the conference. A quick translation: > > >>>>>> > > >>>>>> Doc phase 1 (create wrapper daemon): not in presentation > > >>>>>> Doc phase 2 (Delete southbound database)-> presentation phase 1 > > >>>>>> Doc phase 3 (Eliminate unnecessary database tables)-> presentation > > >>>>>> phase 2 > > >>>>>> Doc phase 4 (Bypass database writes for local data)-> presentation > > >>>>>> phase 4 > > >>>>>> Doc phase 5 (Combine ovn-northd and ovn-controller)-> presentation > > >>>>>> phase 3 > > >>>>>> Presentation phase 5 (Profile and optimize) -> Not in the document > > >>>>>> > > >>>>>> In summary, the doc phase 1 was not in the presentation since it was > > >>>>>> mostly a mechanical change, and it's really only usable by a subset > > >>>>>> of > > >>>>>> OVN deployments (specifically, those in an ovn-kubernetes-like > > >>>>>> deployment where ovn-northd and ovn-controller run on the same > > >>>>>> system). > > >>>>> > > >>>>> A quick question here: > > >>>>> > > >>>>> In the document phase1 it says: > > >>>>> "This mostly benefits the use case of running all OVN components on a > > >>>>> single node. Clustered > > >>>>> OVN deployments will get the upgrade and deployment benefits, but ... > > >>>>> " > > >>>>> > > >>>>> Sorry I didn't understand how clustered deployments would be supported > > >>>>> by this, and how they would benefit. > > >>>>> Does it mean we would be running all processes on both central and > > >>>>> distributed nodes? Would the connection between ovn-controller on > > >>>>> chassis and SB-DB on central still work? Maybe it is not difficult to > > >>>>> make iit work one way or another, but at least I don't see any benefit > > >>>>> here for the clustered (i.e. central) deployments. > > >>>>> > > >>>>> Did I misunderstand anything? > > >>>> > > >>>> You did not misunderstand anything. The rest of the sentence after the > > >>>> "but" says: "but since identical ovn-northd processes and identical > > >>>> southbound databases will be running on multiple compute nodes, there > > >>>> will be redundant processing on each compute node." So no, northd and > > >>>> the southbound database would not be centralized. > > >>>> > > >>>> The idea would be that ovn-northd, the southbound database and > > >>>> ovn-controller would still exist as individual processes, but could be > > >>>> managed by a single software package. The actual ovn-local package at > > >>>> this stage would likely be a simple systemd service that wraps the > > >>>> processes. The reason the upgrade benefits are mentioned is that > > >>>> instead of having to upgrade ovn-central and ovn-host packages, there > > >>>> would be a single package to upgrade. However, as the doc mentions > > >>>> doing this on a clustered system also introduces the problem that > > >>>> running ovn-northd and the southbound database on every node creates > > >>>> redundant processing, as well as completely ballooning the amount of > > >>>> southbound data that is present on every system. The idea is that this > > >>>> then transitions into the later phases of the document that eliminate > > >>>> the redundant elements (remove southbound database, combine ovn-northd > > >>>> and ovn-controller processes, etc.). > > >>>> > > >>>> Honestly, trying to sell this as any sort of benefit for a clustered > > >>>> system is flimsy. It is only a benefit for ovn-kubernetes-like systems > > >>>> where everything is run on a single node. When writing the doc, the > > >>>> idea of introducing the ovn-local package as an early phase made sense > > >>>> to me, since it provided a concrete step towards moving towards the > > >>>> other phases. But I think that when it comes to actually implementing > > >>>> this, imposing an ovn-local process on everyone can wait until close > > >>>> to the end of the work. > > >>>> > > >>>> I think there could be a benefit to having an OPTIONAL ovn-local > > >>>> package as an early goal. This way, ovn-kubernetes-like deployments > > >>>> could switch over to the ovn-local package early, but clustered > > >>>> systems could ignore its existence entirely and keep on doing what > > >>>> they do until the later phases are implemented. Once the other phases > > >>>> are implemented, then everyone would benefit from the rearchitected > > >>>> OVN. > > >> > > >> Thanks Mark for clarifying. I did have a misunderstanding, because the > > >> phase 1 in the document didn't make it clear for clustered deployment > > >> that the northd and SB will be moved from central to distributed. I > > >> thought we'd keep the northd and SB DB centralized in phase 1, which > > >> means we need the wrapper daemon on both central nodes and HV nodes, > > >> each using different part of the daemon, and that's why I was > > >> confused. Now it is clear, and I agree with you the phase 1 alone is > > >> more meaningful for ovn-k8s-like deployments, and less practical for > > >> clustered deployments. > > >> > > > > > > The way I've been imagining this since we started thinking about > > > ovn-local internally was kind of similar to what ovn-kubernetes did. > > > They also used to have a "central" component and a "node" component. > > > > > > They merged them into a single ovnkube-controller process which can be > > > run in multiple modes: > > > > > > a. central mode > > > b. node mode > > > c. central+node mode (this is the default now, with their one OVN per > > > node model) > > > > > > But more on that at the end of the email. :) > > > > > >>>> > > >>>>> > > >>>>> Best, > > >>>>> Han > > >>>>> > > >>>>>> It could also completely be skipped over during development, > > >>>>>> potentially. Plus, deleting the southbound database makes for an > > >>>>>> impactful phase 1 in a presentation :). Doc phases 3 and 4 were > > >>>>>> swapped for the presentation based on feedback I received before > > >>>>>> giving the presentation. Profiling and optimizing were not mentioned > > >>>>>> in the doc, mainly because it is assumed. It was mentioned in the > > >>>>>> presentation just to ensure attendees that we would not just blindly > > >>>>>> make the changes and assume that they improved performance. > > >>>>>> > > >>>>>> Now to address what Dumitru mentioned :) > > >>>>>> > > >>>>>> For the most part, I think this work is all-or-nothing when it comes > > >>>>>> to releasing it. For instance, if we eliminate the southbound > > >>>>>> database > > >>>>>> but don't do any of the phases after, then the northbound database > > >>>>>> would be much larger than it needs to be, and it means keeping > > >>>>>> unnecessary tables around for backwards compatibility. Similarly, if > > >>>>>> we combine ovn-northd and ovn-controller, but we still write data to > > >>>>>> the database unnecessarily or don't optimize away unnecessary code, > > >>>>>> we > > >>>>>> end up with "centralized" logic being run redundantly on all nodes. > > >>>>>> It > > >>>>>> also means there could be awkward intermediate releases of OVN that > > >>>>>> have odd upgrade instructions that would not apply to future or past > > >>>>>> releases. > > > > > > Mark, I don't necessarily agree, especially on the "all-or-nothing when > > > it comes to releasing" part. I think there might be a viable way to > > > incrementally implement what's needed for ovn-local while still > > > delivering sane releases to our users. Please see below, at the end of > > > the email. > > > > > >>>>>> > > >>>>>> I think that Doc phase 1 (create a wrapper daemon) is probably the > > >>>>>> only thing that we might release piecemeal. The other work likely > > >>>>>> needs to be completed in its entirety before we release it. Doc phase > > >>>>>> 1 is also the most trivial, relatively speaking, of all of the > > >>>>>> phases. > > >>>>>> I agree that it could be done in time for 27.09. I'm not so sure > > >>>>>> about > > > > > > I didn't mean this "phase 1" but the "phase 1" in the actual > > > presentation. IIUC the wrapper package (as I think you called it later > > > in this email) is just that, a (systemd) wrapper to start all daemons at > > > once. > > > > > > But for some (most?) of our users that will probably not be very useful. > > > E.g., OpenShift ovn-kubernetes manages the lifecycle of the components > > > on their own as they're different containers of the same pod. I think > > > OpenStack does that too for some components. > > > > > >>>>>> hitting 28.03 with the rest of the work, but if we can do it, then > > >>>>>> great! It's hard for me to put an estimate on the work yet, since we > > >>>>>> don't have concrete work items defined yet. The phases in the > > >>>>>> document > > >>>>>> and presentation are high-level descriptions that don't get into the > > >>>>>> lower-level details of what needs to be changed. > > >>>>>> > > > > > > I agree it's hard to estimate work that's "defined" only at such a high > > > level. I'll try to address this with a more detailed plan (or at least > > > what I currently think the plan should be) in the end of this email. > > > > > >>>>>>> > > >>>>>>> Please read further for the notes I took during the conference. > > >>>>>>> > > >>>>>>> Best regards, > > >>>>>>> Dumitru > > >>>>>>> --- > > >>>>>>> > > >>>>>>> My notes from the panel and follow up discussion: > > >>>>>>> ------------------------------------------------- > > >>>>>>> DISCLAIMER: I wrote this down after the fact so there might be > > >>>>>>> inaccuracies. > > >>>>>>> > > >>>>>>> The main take away from the discussion was that OVN users in the > > >>>>>>> audience were interested in moving towards such an architecture. > > >>>>>>> It’s > > >>>>>>> also in-line with other talks that were presented at the conference > > >>>>>>> (e.g., Crusoe is moving towards a custom ovn-ic deployment similar > > >>>>>>> to > > >>>>>>> the OpenShift/ovn-kubernetes one - essentially one OVN per node, > > >>>>>>> Stackit > > >>>>>>> has issues due to the large size of the SB database in their classic > > >>>>>>> centralized deployment). > > >>>>>>> > > >>>>>>> There were a few questions/remarks from the audience and afterwards > > >>>>>>> during the ad-hoc discussion. I'll list everything (including > > >>>>>>> answers/comments) here: > > >>>>>>> > > >>>>>>> 1. How can we deal with the "synchronization work that ovn-northd > > >>>>>>> currently does, e.g., allocating SB datapath and port_binding > > >>>>>>> tunnel keys? > > >>>>>>> > > >>>>>>> There were some options mentioned during the session: > > >>>>>>> a. fully distributed and predictable/consistent generation of the > > >>>>>>> keys > > >>>>>>> by individual ovn-local instances. This has IMO the risk of > > >>>>>>> complicating upgrades if we ever decide to change the algorithm > > >>>>>>> used to > > >>>>>>> generate keys. > > >>>>>>> > > >>>>>>> b. adding support to ovsdb to "run stored procedures" and move the > > >>>>>>> key > > >>>>>>> allocation logic there > > >>>>>>> > > >>>>>>> c. mimic what ovn-kubernetes does and have a "cluster manager", a > > >>>>>>> centralized component, essentially a way simpler ovn-northd that > > >>>>>>> still > > >>>>>>> manages keys even for ovn-local deployments > > >>>>>>> > > >>>>>>> 2. Would moving to ovn-local shift the performance issues from the > > >>>>>>> central nodes to the chassis? That’s because ovn-local would now > > >>>>>>> perform the work that ovn-northd does in the current deployment > > >>>>>>> (centralized). > > >>>>>>> > > >>>>>>> I think we can't provide a definitive answer before experimenting > > >>>>>>> with > > >>>>>>> the proposed new architecture but we expect that in the final > > >>>>>>> ovn-local > > >>>>>>> deployment only locally relevant logical switches/routers would be > > >>>>>>> processed (reducing compute time to values likely similar to what > > >>>>>>> ovn-controller has to process today); the large SB contents would be > > >>>>>>> gone, reducing time to process those in the IDL (json handling). > > >>>>>>> > > >>>>>>> 3. Would moving to ovn-local cause upgrade incompatibility? I.e., > > >>>>>>> in > > >>>>>>> very large clusters upgrades can take days/months. > > >>>>>>> > > >>>>>>> This has two aspects: > > >>>>>>> a. the first upgrade, from centralized OVN to ovn-local > > >>>>>>> b. subsequent upgrades, from ovn-local version x to ovn-local > > >>>>>>> version x+1 > > >>>>>>> > > >>>>>>> "a" was discussed separately quite extensively during the ad-hoc > > >>>>>>> session > > >>>>>>> after the conference, I'll detail that later. > > >>>>>>> > > >>>>>>> for "b”, taking the ovn-kubernetes example, we probably have a > > >>>>>>> chance of > > >>>>>>> incompatibility even with today’s architecture. That’s because > > >>>>>>> whenever > > >>>>>>> a packet needs to be tunneled to a remote chassis (i.e., it has been > > >>>>>>> received on a logical port on chassis-1 and needs to be sent out on > > >>>>>>> a > > >>>>>>> logical port on chassis-2) we are splitting that datapath’s logical > > >>>>>>> pipeline across two nodes: ingress pipeline is executed on chassis-1 > > >>>>>>> while egress pipeline of the same switch/router is executed on > > >>>>>>> chassis-2. > > >>>>>>> > > >>>>>>> So, if a change happens that involves, for example, moving stages > > >>>>>>> from > > >>>>>>> one pipeline (ingress or egress) to the other then these may cause > > >>>>>>> inconsistent packet forwarding. > > >>>>>>> > > >>>>>>> In general, this is handled today through feature flags (each > > >>>>>>> chassis > > >>>>>>> advertises its "supported” feature set and ovn-northd only generates > > >>>>>>> logical flows that are compatible with the "oldest” ovn-controller). > > >>>>>>> Moving to ovn-local, we’d have to be even more careful with these > > >>>>>>> kinds > > >>>>>>> of situations but, IMO, nothing new here. > > >>>>>>> > > >>>>>>> 4. What about the shared state that’s currently stored in the > > >>>>>>> Southbound > > >>>>>>> (pushed up by ovn-controllers)? E.g., mac-binding, fdb, learned > > >>>>>>> routes. > > >>>>>>> > > >>>>>>> That’s still needed in the ovn-local mode; either stored as new > > >>>>>>> tables > > >>>>>>> in the (new) NB database or as a "Status/Operational/etc” database. > > >>>>>>> > > >>>>>>> 5. Numan suggested reviving his old RFC work that moved logical flow > > >>>>>>> generation to ovn-controller or at least first focusing on removing > > >>>>>>> Logical_Flow from the "new” ovn-local database. > > >>>>>>> > > >>>>>>> https://mail.openvswitch.org/pipermail/ovs-dev/2021-June/384488.html > > >>>>>>> > > >>>>>>> I think the ovn-local "Phase 2" part of the proposal presented at > > >>>>>>> the > > >>>>>>> conference covers that and we could likely start with removing > > >>>>>>> Logical > > >>>>>>> Flows but it's my personal opinion that it would require less work > > >>>>>>> overall if we first implement "Phase 1" of the proposal ("Delete the > > >>>>>>> Southbound database"). > > >>> > > >>> Thanks for putting everything together and sharing the details. > > >>> > > >>> If I understand correctly phase 1 will eliminate SB DB and the logical > > >>> flows will > > >>> be stored in the NB DB ? Please correct me if I'm wrong. > > > > > > Numan, that's definitely not what I was imagining but I might be > > > interpreting things differently than others. I thought that "SB DB > > > elimination" would mean we don't generate logical flows anymore. I'd > > > definitely vote against storing logical flows in the NB. > > > > > > But, with the idea in mind that we can try to deliver ovn-local > > > functionality gradually, across multiple OVN releases in a sane way, I > > > think my original statement above doesn't stand anymore. Please see > > > below for a potential way of dealing with logical flows. > > > > > >>> > > >>> IMHO both can go in parallel. If one gets accepted before the other, > > >>> it should not be > > >>> hard to adjust. > > >>> > > >>> > > >>> > > >>>>>>> > > >>>>>>> 6. Is one of the problems with the current architecture that we end > > >>>>>>> up > > >>>>>>> pushing "too detailed” information to the chassis from the central > > >>>>>>> OVN node? > > >>>>>>> > > >>>>>>> Yes, most likely, and that’s inline with other targeted > > >>>>>>> optimizations we > > >>>>>>> did in the past, e.g., moving part of the LB flow generation > > >>>>>>> directly to > > >>>>>>> ovn-controllers - those did come at a cost of duplicating some NB > > >>>>>>> config > > >>>>>>> information (LBs synced to the SB). > > >>>>>>> > > >>>>>>> NOTE: The following items were discussed in the ad-hoc session, > > >>>>>>> after > > >>>>>>> the official end of the conference: > > >>>>>>> > > >>>>>>> 7. The "first" upgrade scenario: part of the cluster using > > >>>>>>> ovn-local and > > >>>>>>> part of the cluster (including "old” central nodes) running the > > >>>>>>> legacy > > >>>>>>> mode. Some options were discussed: > > >>>>>>> > > >>>>>>> a. Ilya suggested running two sets of databases: the "old" SB for > > >>>>>>> chassis that haven’t upgraded yet and the "new" > > >>>>>>> Status/Operational/Whatever database ovn-local needs moving forward. > > >>>>>> > > >>>>>> When you say "Status/Operational/Whatever database", is this just a > > >>>>>> rebranding of the current northbound database? Or are you referring > > >>>>>> to > > >>>>>> actually having a new database in addition to the current northbound > > >>>>>> database? If it's the former, then that aligns with what I was > > >>>>>> picturing when writing the docs. If it's the latter, then can you > > >>>>>> explain the need for an additional database? > > >>>>>> > > > > > > No matter what we do, for "centralized" deployments (like OpenStack with > > > NB + ovn-northd + SB running on central nodes today), we will need a > > > shared store for all operational data. That includes Port_Binding.up, > > > MAC_Bindings, FDBs, IGMP_Groups, Learned_Routes, etc. > > > > > > In the proposal you/we presented at the conference those would end up > > > being moved to the NB database. But the feedback we got at the > > > conference was that there's not a lot of benefit if we do that and that > > > it might be cleaner if we store all that in a separate database, a > > > "smaller SB" if you prefer that name :). > > > > > > I think Felix's reply to this thread also suggests that a separate > > > database would be preferable. It also means the NB can keep the same > > > schema as it currently has. > > > > > > From > > > https://mail.openvswitch.org/pipermail/ovs-dev/2026-November/438023.html : > > > > > > I would not see a benefit of having it in the same database. > > > People can still just run one ovsdb cluster and run both DBs on that. > > > That would give everyone more flexibility and we would not loose > > > anything. > > > > > > But for me the biggest question there would probably be the Chassis > > > (and related) tables. Since the CMS needs to build e.g. > > > HA_Chassis_Groups and similar things for them, maybe they make more > > > sense in NB than in the other db. > > > > > > I don't really agree with the last part in Felix's reply, I think. For > > > the Chassis part, the CMS should have out-of-band information about what > > > Chassis configuration it provisioned on the actual hypervisors (e.g., > > > system-ids/hostnames are all configured in the HV local OVSDB anyway by > > > the CMS). > > > > > >>>>>>> > > >>>>>>> In his suggestion ovn-northd (upgraded) would be the intermediate > > >>>>>>> actor > > >>>>>>> forwarding contents written by the upgraded ovn-locals in the new > > >>>>>>> databases to the legacy SB > > >>>>>>> > > >>>>>>> b. Another option would be to run a "SB forwarder/proxy" on the > > >>>>>>> upgraded > > >>>>>>> nodes: ovn-local would be unaware of its existence, it would only > > >>>>>>> write > > >>>>>>> to the new "Status" database; the "SB forwarder" would monitor the > > >>>>>>> "Status" and propagate the information to the "legacy SB" that’s > > >>>>>>> still > > >>>>>>> running on the central node for the not-yet-upgraded chassis. > > >>>>>>> > > >>>>>>> 8. Taking things a step further and adding support for using other > > >>>>>>> databases for OVN config/status storage (e.g., etcd). > > >>>>>>> > > >>>>>>> Numan argued that we may anyway need a translation layer in the new > > >>>>>>> ovn-local during one of the transition phases described in the talk > > >>>>>>> in > > >>>>>>> order to abstract out database access (e.g., the OF generation part > > >>>>>>> in > > >>>>>>> ovn-controller currently needs logical flows to be parsed in > > >>>>>>> memory); we > > >>>>>>> could consider keeping that in place even after ovn-local is the > > >>>>>>> only > > >>>>>>> way of deploying OVN; we would repurpose it to provide an > > >>>>>>> abstraction > > >>>>>>> for fetching/pushing data to the NB/Status databases. > > >>>>>>> > > >>>>>>> 9. From my perspective one of the most important requirement for > > >>>>>>> this > > >>>>>>> whole effort is maintaining OVN backwards compatibility from CMS > > >>>>>>> perspective. There might be some small (hopefully) changes the CMS > > >>>>>>> would have to put in place for the first upgrade to ovn-local but > > >>>>>>> ideally we should avoid needing other changes in the CMS. This might > > >>>>>>> also mean: > > >>>>>>> > > >>>>>>> a. transparently making ovn-sbctl work with ovn-local deployments > > >>>>>>> b. transparently making ovn-trace work with ovn-local deployments > > >>>>>>> > > >>>>>>> 10. It was raised that this might be a good moment to start using a > > >>>>>>> different programming language (probably Rust) for implementing the > > >>>>>>> new > > >>>>>>> parts that will be required for ovn-local. > > >>>>>>> > > >>>>>>> 11. There was also a small debate whether we should keep logical > > >>>>>>> flows > > >>>>>>> or not: > > >>>>>>> a. e.g., for debugging (easier to read than OpenFlow) > > >>>>>>> b. OTOH this would incur extra computation cost > > >>>>>> > > >>>>>> Even without the Logical_Flow database table, there is still some > > >>>>>> need > > >>>>>> for the use of logical flows since users can provide matches as part > > >>>>>> of an ACL or Logical_Router_Policy definition. > > >>>>>> > > >>>>>> Other than that, the only real use for logical flows in a standard > > >>>>>> OVN > > >>>>>> deployment would be for debugging. If we think of this as only a tool > > >>>>>> for debugging, I think trying to keep logical flows around long-term > > >>>>>> is going to lead to frustration and inaccuracy. > > >>>>>> > > >>>>>> As we accustom ourselves to writing OpenFlow directly, we eventually > > >>>>>> will reach a point where it makes little sense to define new logical > > >>>>>> flow actions that align with the OpenFlow we create. For example, we > > >>>>>> currently have logical flow actions that translate to a resubmit to > > >>>>>> some high numbered OpenFlow table to look up a specific value and set > > >>>>>> that value in some register (lookup_arp() for example). Now let's > > >>>>>> imagine that we no longer need to write logical flows, and we do > > >>>>>> something similar directly in OpenFlow. We would need to artificially > > >>>>>> define and document a new logical flow action solely for the purposes > > >>>>>> of debugging, even though that logical flow action would never > > >>>>>> actually be used anywhere in the code. Similar things are likely to > > >>>>>> happen with specific registers that we use in OpenFlow matches. > > >>>>>> Currently, we define certain shortcuts for logical flow matches and > > >>>>>> actions (e.g. "flags.loopback" translates to "reg10[0]"). Eventually, > > >>>>>> once we are defining everything in OpenFlow, such shortcuts will be > > >>>>>> unnecessary to define. Logical flows will eventually bitrot and > > >>>>>> they'll be unreliable to use for debugging since they haven't > > >>>>>> actually > > >>>>>> kept up with reality. If our actual development path is to define > > >>>>>> OpenFlow directly, then the accompanying "fake" logical flows could > > >>>>>> become inaccurate in the case where we update an OpenFlow flow but > > >>>>>> forget to update the accompanying logical flow. The code will still > > >>>>>> work, but now the debugging aid will show inaccurate information. > > >>> > > >>> > > >>> I have a different take on the logical flows. I don't think we should > > >>> eliminate the use of > > >>> logical flows. Logical flows provide an excellent wrapper for Openflow > > >>> rules. > > >>> IMO expressing a pipeline as logical flows is easier than openflow rules > > >>> and I think it will be easier to add new features. > > >>> > > >>> How I'd use logical flows is similar to the RFC which I had proposed > > >>> earlier. > > >>> (https://github.com/numansiddique/ovn/blob/rfc_lflows_in_controller_v2_p16/lib/lflow.c#L148) > > >>> > > >>> Lets a logical switch/datapath - sw0 is relevant to a node 'A'. > > >>> ovn-local on that node > > >>> will call a library function (something like below) which generates > > >>> all the logical flows for > > >>> this logical switch and returns. And then the same logical flows > > >>> returned will be converted > > >>> to openflow rules using the expr parser. > > >>> > > >>> static void build_dp_lflows(struct hmap *lflows, struct local_datapath > > >>> *) > > >>> > > >>> And when a user runs the command - "ovn-sbctl dump-flows <datapath>" > > >>> , ovn-sbctl (or ovn-nbctl) > > >>> utility will call the same function - build_lswitch_dp_lflows() and > > >>> dumps the logical flows. > > >>> > > >> > > >> It seems the proposal document didn't make it clear whether we want to > > >> keep logical flows or not. Phase 4 will remove Logical_Flow table, but > > >> there can be two approaches: > > >> 1). Keep the logical flow generation phase and then translation to > > >> OpenFlow, but only eliminate the storage in the DB - the approach that > > >> Numan mentioned > > >> 2). Completely remove the logical flow generation phase and directly > > >> generate OpenFlow rules > > >> > > >> In fact it may make sense to do above as two sequential steps. > > >> I agree with the benefits of only doing 1) mentioned by Numan, but we > > >> would lose the huge performance benefit of 2), because logical flow > > >> parsing is the heaviest part of today's ovn-controller processing. > > >> Given that it has always been a scale bottleneck, personally I am > > >> leaning towards 2) - avoiding the logical flow generation phase > > >> completely. For implementation I think doing it in the above two > > >> sub-phases would help evaluating the pros and cons. > > >> > > > > > > I completely agree with this. I don't think we should spend a lot of > > > time debating whether we should remove logical flow infrastructure (even > > > if we don't have logical flows in the SB anymore) at this point. > > > > > > It does feel like a natural way to incrementally implement this. Using > > > Han's notation, first do "1)" (with logical flows in memory) and then > > > evaluate if we need "2)" (logical flow generation replaced with direct > > > OpenFlow generation). > > > > > > However, it's interesting to see that logical flows being the "scale > > > bottleneck" is not really the pictures all our users have. Quoting from > > > Felix's other reply: > > > > > > https://mail.openvswitch.org/pipermail/ovs-dev/2026-November/438101.html > > > > > > We did some profiling in our environment on a heavily loaded gateway > > > chassis. We did profile the recompute of the lflow_output engine node > > > in ovn-controller (without the lflow cache). > > > In there we found that we take around 7% of the total runtime for > > > ovnacts_parse_string and 3.5% of the total time for expr_parse_string > > > (for matches). > > > > > > 10.5% doesn't seem like "that much" to me but, of course, if we can > > > optimize that maybe we should. > > > > > >> Best, > > >> Han > > >> > > >>> > > >>> Thanks > > >>> Numan > > >>> > > >>> > > >>> > > >>>>>> > > >>>>>> I think that any steps we take to try to preserve logical flows will > > >>>>>> be counter to the goals that the ovn-local project is trying to > > >>>>>> achieve. Deletion of the Logical_Flow database table is one of the > > >>>>>> biggest wins for this project since it is massive. Logical flows > > >>>>>> really only make sense as a centralized concept that describes the > > > > > > Does it though? I know for sure I really appreciate the existence of > > > logical flows when I debug ovn-kubernetes setups. And those are already > > > "one OVN per node" deployments. > > Let me clarify my point. Let's imagine a non-ovn-kubernetes-like > system. In such a system, the logical flows are something that are > stored in the SB DB and are completely divorced from the physical > hypervisors. Viewing the logical flows shows the entire logical > network, illustrating how packets flow between each of the logical > datapaths. It doesn't matter where ports are bound, how many > hypervisors exist, etc. > > Now, when flow generation is moved to each individual hypervisor, each > hypervisor is only going to be monitoring datapaths/ports that are > local to that hypervisor. This means that if that hypervisor emits > logical flows, then we can only get a partial picture of the network. > We wouldn't be able to see where packets go once they leave that > hypervisor. We also can't get a complete picture of what the > datapath's ingress and egress pipelines do based on one hypervisor's > view of the network. > > As a simple example, let's say we have a logical switch LS1. It has > two ports, LSP1 and LSP2. LSP1 is bound on hypervisor HV1, and LSP2 is > bound on hypervisor HV2. In such a case, if we ask HV1 for its logical > flows, what would we see? What about HV2? Would we see the same set of > flows, or would the fact that we are only seeing the locally bound > ports make it so that each set of logical flows is different? > > This is why I think that logical flows are most useful for a > centralized holistic view of the network.
Please take a look at my comment here - https://mail.openvswitch.org/pipermail/ovs-dev/2026-November/438091.html on the logical flows. If we go with the approach I suggested there, then we generate the logical flows on the fly and ovn-sbctl/ovn-nbctl can generate and dump the logical flows. Thanks Numan > > We could change what logical flows are supposed to represent. We could > say they provide an abstraction of the underlying OpenFlow on a > particular hypervisor. It means reframing logical flows not as a > complete set of flows pertaining to the ingress and egress pipelines > of a datapath, since a hypervisor may not have a complete view of the > datapath. > > > > > > Note: I'm not saying we shouldn't remove logical flows from the SB, I > > > still think we should do that. I think the discussion above was just > > > about using the logical flow generation/parsing infrastructure in the > > > new ovn-local daemon. > > > > > >>>>>> relationship of the logical datapaths. I think instead of trying to > > >>>>>> preserve logical flows as a debugging tool, we could focus instead on > > >>>>>> providing new debugging tools that work for ovn-local. > > > > > > I think that is something we should consider anyway, what can we do to > > > improve debugging experience in the "ovn-local world". > > > > > > Finally, here's my _personal_ view on how we could incrementally > > > implement ovn-local such that we can also have sane releases in the > > > meantime. I'll be using letters to denote the different phases to avoid > > > the confusion with the phases in Mark's original doc and presentation. > > > > > > Phase A: > > > -------- > > > - Description: create a unified binary and package containing the > > > current ovn-northd AND ovn-controller code. > > > > > > In this phase we don't change any functionality, we just factor out the > > > ovn-controller and ovn-northd main entry points into libraries. We > > > adapt the ovn-controller and ovn-northd code to use the new libraries. > > > > > > We _also_ create a new binary, maybe call it ovnd (or ovn-local if > > > that's nicer). > > > > > > This new binary takes as command line arguments the union of the command > > > line arguments ovn-northd and ovn-controller take. > > > > > > It also takes as argument the mode it should be started in, two options > > > that can also be combined: > > > - northd: should start exactly the same thing as regular ovn-northd > > > - controller: should start exactly the same thing as regular > > > ovn-controller > > > - northd,controller: should start two separate threads, one for the > > > northd code and the other for the ovn-controller code. > > > > > > Note: as a syntactic sugar we could also support --mode=all which would > > > be equivalent to --mode=northd,controller > > Other syntactic sugar could be to have an `ovn-northd` symlink that > aliases ovnd --mode=northd and `ovn-controller` that aliases ovnd > --mode=controller. > > > > > > > In this phase we have: > > > - no major functional changes > > > - NB stays the same as today > > > - SB stays the same as today > > > - users can migrate towards using ovnd (with the appropriate mode set) > > > without being impacted in any way > > > - no performance change > > > > > > Pros: > > > - ovn-kubernetes-like deployments (single OVN stack per node) can remove > > > the need to start separate containers for ovn-northd and ovn-controller > > > and instead only create a single container for ovnd with --mode=all. > > > > > > Phase B: > > > -------- > > > - Description: remove the SB database for ovnd --mode=northd,controller > > > > > > In this phase we'd like to improve the lives of ovn-kubernetes-like > > > users. To do that, the first thing we can do is remove the need for a > > > separate SB database process. > > > > > > I think we have two options. > > > > > > B1. Add another abstraction layer in OVN to choose whether all database > > > interaction (connection, getting updates, committing transactions) > > > should actually go towards a real OVS SB DB or just to an in-memory store. > > > > > > Cons: > > > - seems like a huge amount of work TBH > > > - what about ovn-sbctl commands? > > > > > > B2. Add a new mode to ovsdb, allowing any process to run an OVS database > > > in memory. This would likely be a new feature in OVS as it would > > > require moving parts of ovsdb-server.c to a library ovnd can link to. > > > It would also likely require a new type of database connection stream, > > > e.g., "direct-access-stream". With such in-memory connections we can > > > skip the json serialization/deserialization part that currently happens > > > when clients interact with the database. We could just had the json > > > objects (e.g., updates or transactions) from one side to the other. > > I think B2 makes the most sense to me here. This keeps the API the > same but changes what happens on the backend. > > > > > > > Cons: needs a non-trivial change in OVSDB. > > > Pros: > > > - probably less work than option B1. > > > - ovn-sbctl still works, the DB is just running inside of ovnd. > > > > > > In this phase (regardless if we choose B1 or B2) we have: > > > - no OVN functional changes > > > - NB stays the same as today > > > - SB _schema_ stays the same as today but users can decide (ovnd cmdline > > > argument) to start the DB inside ovnd > > > - ovn-sbctl still works out of the box > > > - reduced latency/cpu usage because we remove some of the unnecessary > > > json serialization/deserialization + unix socket between ovnd and the SB. > > > - ovn-kubernetes-like users can migrate to the new SB mode > > > - neutron-like users can still deploy the SB as they're used to, on the > > > central node > > > > > > Phase C: > > > -------- > > > - Description: add a new mode to ovnd, --mode=allocator (or better name). > > > > > > In this phase we factor out the parts of ovn-northd we already > > > identified as things that need to be run on a central node. That > > > includes: datapath tunnel key allocation, port_binding tunnel key > > > allocation, IPAM, etc. > > > > > > This makes the supported ovnd modes become: > > > - allocator: starts only this new component that connects to the NB and > > > allocates the required keys/IPs and writes them to the SB > > > - northd: should start the trimmed down ovn-northd component (without IP > > > allocation > > > - controller: should start exactly the same thing as regular > > > ovn-controller > > > - any combination of the values above, for example: > > > --mode=allocator,northd,controller starts three separate threads, one > > > for each component, in the same process > > > - the --mode=all syntactic sugar becomes equivalent to: > > > --mode=allocator,northd,controller > > > > > > Note: if --mode combines "allocator" & "northd" then the keys/IPs can be > > > directly made available to the northd module without having to go > > > through the additional SB roundtrip. > > > > > > Note: the ovn-northd and ovn-controller binaries still exist and are > > > equivalent to running: > > > - For ovn-northd: ovnd --mode=allocator,northd > > > - For ovn-controller: ovnd --mode=controller > > > > > > In this phase we have (compared to phase B): > > > - no NB changes > > > - no SB changes > > > - no major functional changes > > > - no performance changes > > > - ovn-kubernetes-like users can keep using --mode=all, no change required > > > - neutron-like users can decide to migrate to running two separate ovnd > > > processes on the central node (--mode=allocator AND --mode=northd) but > > > they can also just keep using the ovn-northd binary, that still works. > > > > > > Phase D: > > > -------- > > > - Description: allow --mode=controller to connect to the NB database and > > > determine the set of local logical switches/routers (equivalent to the > > > local_datapaths set ovn-controller builds today). > > > > > > In this phase, if ovnd --mode=controller is started with a NB connection > > > string (--ovnnb-db), the controller module opens a connection to the NB > > > database. > > > > > > It then monitors the minimum set of required tables. Similar to what > > > ovn-controller does today to get updates about local datapaths, we'd > > > have to set the monitor condition to include: > > > - LSPs of type router/switch (equivalent to Sb.port_binding.type=patch) > > > - LSPs of type l2gateway,external > > > - LSPs of type VIF whose name matches local OVS interfaces with > > > external_ids:iface-id set. > > > > > > And then build the set of local switches/routers. > > > > > > Note: we might want to make this opt-in, TBD. > > > Note: these connections are read-only from --mode=controller perspective > > > and I'm assuming scale concerns related to all controllers connecting to > > > the NB can potentially be addressed by using ovsdb-relays for the NB. > > > > > > Phase E: > > > -------- > > > - Description: Stop writing to SB tables that are read-only from > > > ovn-controller perspective. > > > > > > For each table we essentially move the processing that happens in the > > > --mode=northd component to --mode=controller (only for local > > > switches/routers). > > > > > > This is the first tricky stage because we really don't want to break > > > neutron-like users (or ovn-kubernetes-like users that still use a real > > > SB database). > > > > > > This can be a prolonged phase, we don't have to get rid of all tables at > > > once. > > > > > > Note: we need to keep backwards compatibility so tables we get rid of > > > _cannot_ and _should not_ be removed from the SB schema. > > > > > > Note: during this phase, --mode=controller still maintains a connection > > > to SB for tables not yet migrated (including Datapath_Binding and > > > Port_Binding for tunnel keys). > > > > > > Let's assume we're removing the SB.Address_Set. The changes that need > > > to happen are probably: > > > > > > E1. In --mode=controller, if --ovnnb-db is set: > > > - add a chassis-specific feature flag to inform --mode=northd nodes that > > > this chassis is able to work with both NB/SB Address_set records. > > > - stop monitoring the SB database Address_Set table > > > - start monitoring the NB database Address_Set table > > > - change the controller incremental processing engine so that it can > > > handle both inputs, SB.Address_Set and Nb.Address_Set > > > > > > E2. In --mode=northd: > > > - if _ALL_ registered chassis have the Address Set feature flag set, > > > stop writing to SB.Address_Set > > > - in consequence, if at least one registered chassis doesn't have the > > > feature flag, we keep populating SB.Address_Set as we do today. > > > > > > E3: Update debug tools that currently require the contents of the SB > > > table we stopped writing to so that they fallback to using the NB, e.g.: > > > ovn-detrace. > > One thing to point out is that some of the tables that are "identical" > between the NB and SB database actually have some tiny difference. For > example the Mirror table in the NB database has an extra column > (mirror_rules) that is not in the SB database. In this case, since the > SB database has a subset of the columns in the NB database, this is > fine. However, as a separate example, the Load_Balancer table in the > SB database has fields like datapath_group that don't exist in the NB > database. So you can't just seamlessly move ovn-controller over to the > NB version. It likely means that ovn-controller would need to derive > the SB-only values somehow from the NB table. It's doable, but I > figured it's worth pointing out. > > > > > > > Let's take a slightly more complex example, the SB.Multicast_Group table. > > > > > > Today ovn-controller registers for SB.Multicat_Group updates > > > conditionally, only for records whose datapaths are local. The changes > > > that need to happen are probably: > > > > > > E1. In --mode=controller, if --ovnnb-db is set: > > > - add a chassis-specific feature flag to inform --mode=northd nodes that > > > this chassis is able to work _without_ SB Multicast_group records. > > > - we already monitor the set of locally relevant logical > > > switches/routers, factor out the code that creates SB Multicast_groups > > > in northd today into a library; use that library to generate the > > > equivalents of the SB Multicast groups (but locally) > > > - update the incremental processing graph to add new inputs (based on > > > the state computed above) to other nodes that depend on > > > en_sb_multicast_group today. > > > > > > E2. In --mode=northd: > > > - change the code to use the multicast group library we built in E1 above. > > > - if _ALL_ registered chassis have the Multicast Group feature flag set, > > > stop writing to SB.Multicast_group > > > - in consequence, if at least one registered chassis doesn't have the > > > feature flag, we keep populating SB.Multicast_group as we do today. > > > > > > In this phase we have (compared to phase D): > > > - no NB changes > > > - no SB schema changes > > > - functional changes in both northd and controller components but it's > > > all still backwards compatible > > > - performance improvements due to not having to populate SB records > > > - ovn-kubernetes-like users can keep using --mode=all, no change > > > required, they get the performance boost automatically > > > - neutron-like users can safely upgrade compute and central nodes, > > > thanks to the feature flag there's no functionality change > > > > > > This phase can continue until we handled all tables that are read-only > > > from ovn-controller perspective. > > > > > > Once this phase E is completed, in theory, the --mode=northd component > > > will have no work left to do. The only centralized computation is > > > already handled by --mode=allocator. That component still writes (among > > > other things) tunnel keys and IPs to SB.Datapath_Binding and > > > SB.Port_Binding. > > > > > > Phase F: > > > -------- > > > - Description: Stop writing to SB tables that are read-write from > > > --mode=controller perspective. > > > > > > This is the next tricky phase. This is also the phase where we would > > > start needing the "Status" (operational, whatever) database. > > > > > > Here's an incomplete list of tables/columns ovn-controller writes to: > > > - SB.Chassis/Chassis_Private > > > - SB.Port_Binding.up > > > - SB.IGMP_Group > > > - SB.MAC_Binding > > > - SB.FDB > > > - SB.Learned_Routes > > > - other tables required to store allocator generated keys/IPs > > > > > > All the information that is written to those tables/columns today should > > > be set in the Status database after this Phase. (Partial) upgrades > > > should still function properly. > > > > > > Required changes: > > > > > > F1. Define the new Status database schema (see tables above) > > > - also update utilities like ovn-ctl to be able to start the database > > > (either along side NB or as a separate ovsdb-server instance) - the > > > Status database is supposed to run on the central node of the cluster. > > > > > > F2. For upgrades to work properly we need to ensure that partially > > > upgraded clusters (only part of chassis upgraded to the new version and > > > the central node not upgraded yet) are still in a sane state: > > > - old --mode=northd instances expect information to still live in the SB > > > - new --mode=controller instances expect information to live in the > > > Status DB. > > > > > > A potential solution is to have a "temporary" proxy daemon running on a > > > central node that reconciles old SB with new Status databases. This > > > proxy would: > > > - make sure SB records exist and are in sync for all tables that have a > > > corresponding Status table. > > > - e.g., it will have to "forward" port.up information from Status to SB > > > and "forward" mac_binding information from SB to status (in case a mac > > > binding is learned on a not-yet upgraded chassis). > > > - forward tunnel keys/IPs generated by the allocator in the SB to the > > > Status database > > > > > > An alternative solution could be to have the controller component write > > > to both SB/Status databases as long as it doesn't get notified (how?) by > > > the central component that all chassis have been upgraded. > > > > > > F3. Change --mode=controller code to monitor from/write to the Status > > > database. > > > > > > Phase G: > > > -------- > > > In this phase we make the --mode=allocator component write directly to > > > the Status database. This can only happen only once phase F has been > > > completed. Alternatively we can just keep both methods and if a SB > > > connection is configured along side to the Status connection, the > > > component could just write to both databases. > > > > > > Phase H: > > > -------- > > > - Description: remove --mode=northd. > > > > > > As mentioned at the end of Phase E, --mode=northd doesn't need to do > > > anything anymore. Upgraded chassis already can access the Status > > > database and the only remaining centralized processing is handled by > > > --mode=allocator. > > > > > > We also don't need the proxy from phase F anymore. > > > > > > Phase I: > > > -------- > > > - Description: profile and optimize. > > > > > > As you may have noticed I didn't mention Logical_Flows anywhere, that's > > > because I assumed that in phase E we'd be still internally generating > > > logical flows (in --mode=controller). > > > > > > At this point, because most processing happens in --mode=controller we > > > can safely consider removing the logical flow intermediate step and > > > directly generating OpenFlow rules. > > > > > > Other similar optimizations can happen here, without risking major > > > breakages. > > > > > > NOTE: After writing the overview of the phases above I realized there's > > > some more "centralized" work that needs to be handled today: MAC_Binding > > > and FDB aging (flushing), RBAC. Today that's in ovn-northd, in the new > > > proposal this could be part of the small component that continues to run > > > on the central node, --mode=allocator. We might need a better name. > > > > > > I'm sure I over-simplified things but I hope this helps us move the > > > discussion forward and that it conveys why I think we can do the move to > > > "ovn-local" in a sane manner even across multiple releases. > > > > > > > After sleeping on it, I decided to send a follow up with a partial > > alternative. :) > > > > In the above, after Phase H, the Status DB is nothing much than a > > trimmed down Southbound DB. We can maybe reduce some of the upgrade > > complexity if we just keep using the Southbound DB and leave > > empty/unused tables there. > > > > As Mark already mentioned in the ovn-local doc, doing that would mean > > that we'd have to keep the unused tables in the SB schema for ever. But > > that may be an acceptable compromise. And later we could try to figure > > out a way to migrate from the SB schema (with unused tables) to a new > > Status schema. > > > > Regards, > > Dumitru > > > _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
