Hi all,

I'll try to reply inline to the various ongoing conversations.

I'll also try to present a more detailed view of how I personally see
the ovn-local transition work happening (in the end of the email).  I'll
try to address Felix's comments [0] in that part as well.

[0] https://mail.openvswitch.org/pipermail/ovs-dev/2026-November/438023.html

On 11/18/26 8:33 AM, Han Zhou via dev wrote:
> On Tue, Nov 17, 2026 at 8:40 AM Numan Siddique <[email protected]> wrote:
>>
>> On Mon, Nov 16, 2026 at 4:06 PM Mark Michelson <[email protected]> wrote:
>>>
>>> Hi Han
>>>
>>> On Mon, Nov 16, 2026 at 3:44 PM Han Zhou <[email protected]> wrote:
>>>>
>>>> On Mon, Nov 16, 2026 at 8:28 AM Mark Michelson <[email protected]> wrote:
>>>>>
>>>>> Hi Dumitru, thanks for the summary of everything. I don't want to
>>>>> comment too much on this since I helped contribute to the original
>>>>> document, but I do have one comment in-line below:
>>>>>
>>>>> On Mon, Nov 16, 2026 at 6:46 AM Dumitru Ceara <[email protected]> wrote:
>>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> First of all, thank you to those who could attend the conference and the
>>>>>> "ovn-local: A new proposed architecture for OVN" discussion panel:
>>>>>>
>>>>>> https://www.openvswitch.org/support/ovscon2026/#t26
>>>>>>
>>>>>> Some references, also for those who couldn't make it live (or online):
>>>>>>
>>>>>> - Slides presented at the conference:
>>>>>> https://drive.google.com/file/d/1TBDrqoYk9EHseqSaif6xKt4ePYncSocb
>>>>>>
>>>>>> - Recording of Mark's presentation (shared at the conference):
>>>>>> https://drive.google.com/file/d/1mwLmZQmUpZMZ3T-DMtkvo6xQaq867Ebc
>>>>>>
>>>>>> - A snapshot of the document that lead after discussion between Red Hat
>>>>>> internal OVN team members to the proposal we shared at the conference:
>>>>>> https://drive.google.com/file/d/1w7IwB6UX-KbzUChBZb9yGi42yPIP7NZH/
>>>>>>
>>>>>> As usual, it takes some time after the conference until we have
>>>>>> everything ready to share publicly so we don't (yet) have the recording
>>>>>> of the actual panel.  But I'll try to summarize as best as I can (at the
>>>>>> end of this email) what was discussed during the actual session but also
>>>>>> during the ad-hoc/hallway discussions after the official session.
>>>>>>
>>>>>> I think at this point quite a few members of the community have
>>>>>> expressed interest in the initiative and it might be a good idea to
>>>>>> start planning some of the work that might be required for this.  As
>>>>>> you'll see below, I don't think there were any unsolvable technical
>>>>>> issues but there will definitely be challenges.
>>>>>>
>>>>>> In order to get things going, I'd like to ask folks to think about
>>>>>> timelines regarding this new initiative.
>>>>>>
>>>>>> I personally am quite confident that we can try to deliver some of the
>>>>>> changes suggested in the ovn-local document (e.g., phase 1) as early as
>>>>>> the next development cycle (27.09) and that there's a high chance to
>>>>>> implement the migration to the new architecture before our next LTS
>>>>>> release (28.03).  But that's my personal opinion, I'd like to hear
>>>>>> others too. :)
>>>>>
>>>>> One vitally important thing to point out is that the phases in the
>>>>> linked Google Doc do NOT align with the phases that were laid out in
>>>>> the presentation at the conference. A quick translation:
>>>>>
>>>>> Doc phase 1 (create wrapper daemon): not in presentation
>>>>> Doc phase 2 (Delete southbound database)-> presentation phase 1
>>>>> Doc phase 3 (Eliminate unnecessary database tables)-> presentation phase 2
>>>>> Doc phase 4 (Bypass database writes for local data)-> presentation phase 4
>>>>> Doc phase 5 (Combine ovn-northd and ovn-controller)-> presentation phase 3
>>>>> Presentation phase 5 (Profile and optimize) -> Not in the document
>>>>>
>>>>> In summary, the doc phase 1 was not in the presentation since it was
>>>>> mostly a mechanical change, and it's really only usable by a subset of
>>>>> OVN deployments (specifically, those in an ovn-kubernetes-like
>>>>> deployment where ovn-northd and ovn-controller run on the same
>>>>> system).
>>>>
>>>> A quick question here:
>>>>
>>>> In the document phase1 it says:
>>>> "This mostly benefits the use case of running all OVN components on a
>>>> single node. Clustered
>>>> OVN deployments will get the upgrade and deployment benefits, but ... "
>>>>
>>>> Sorry I didn't understand how clustered deployments would be supported
>>>> by this, and how they would benefit.
>>>> Does it mean we would be running all processes on both central and
>>>> distributed nodes? Would the connection between ovn-controller on
>>>> chassis and SB-DB on central still work? Maybe it is not difficult to
>>>> make iit work one way or another, but at least I don't see any benefit
>>>> here for the clustered (i.e. central) deployments.
>>>>
>>>> Did I misunderstand anything?
>>>
>>> You did not misunderstand anything. The rest of the sentence after the
>>> "but" says: "but since identical ovn-northd processes and identical
>>> southbound databases will be running on multiple compute nodes, there
>>> will be redundant processing on each compute node." So no, northd and
>>> the southbound database would not be centralized.
>>>
>>> The idea would be that ovn-northd, the southbound database and
>>> ovn-controller would still exist as individual processes, but could be
>>> managed by a single software package. The actual ovn-local package at
>>> this stage would likely be a simple systemd service that wraps the
>>> processes. The reason the upgrade benefits are mentioned is that
>>> instead of having to upgrade ovn-central and ovn-host packages, there
>>> would be a single package to upgrade. However, as the doc mentions
>>> doing this on a clustered system also introduces the problem that
>>> running ovn-northd and the southbound database on every node creates
>>> redundant processing, as well as completely ballooning the amount of
>>> southbound data that is present on every system. The idea is that this
>>> then transitions into the later phases of the document that eliminate
>>> the redundant elements (remove southbound database, combine ovn-northd
>>> and ovn-controller processes, etc.).
>>>
>>> Honestly, trying to sell this as any sort of benefit for a clustered
>>> system is flimsy. It is only a benefit for ovn-kubernetes-like systems
>>> where everything is run on a single node. When writing the doc, the
>>> idea of introducing the ovn-local package as an early phase made sense
>>> to me, since it provided a concrete step towards moving towards the
>>> other phases. But I think that when it comes to actually implementing
>>> this, imposing an ovn-local process on everyone can wait until close
>>> to the end of the work.
>>>
>>> I think there could be a benefit to having an OPTIONAL ovn-local
>>> package as an early goal. This way, ovn-kubernetes-like deployments
>>> could switch over to the ovn-local package early, but clustered
>>> systems could ignore its existence entirely and keep on doing what
>>> they do until the later phases are implemented. Once the other phases
>>> are implemented, then everyone would benefit from the rearchitected
>>> OVN.
> 
> Thanks Mark for clarifying. I did have a misunderstanding, because the
> phase 1 in the document didn't make it clear for clustered deployment
> that the northd and SB will be moved from central to distributed. I
> thought we'd keep the northd and SB DB centralized in phase 1, which
> means we need the wrapper daemon on both central nodes and HV nodes,
> each using different part of the daemon, and that's why I was
> confused. Now it is clear, and I agree with you the phase 1 alone is
> more meaningful for ovn-k8s-like deployments, and less practical for
> clustered deployments.
> 

The way I've been imagining this since we started thinking about
ovn-local internally was kind of similar to what ovn-kubernetes did.
They also used to have a "central" component and a "node" component.

They merged them into a single ovnkube-controller process which can be
run in multiple modes:

a. central mode
b. node mode
c. central+node mode (this is the default now, with their one OVN per
node model)

But more on that at the end of the email. :)

>>>
>>>>
>>>> Best,
>>>> Han
>>>>
>>>>> It could also completely be skipped over during development,
>>>>> potentially. Plus, deleting the southbound database makes for an
>>>>> impactful phase 1 in a presentation :). Doc phases 3 and 4 were
>>>>> swapped for the presentation based on feedback I received before
>>>>> giving the presentation. Profiling and optimizing were not mentioned
>>>>> in the doc, mainly because it is assumed. It was mentioned in the
>>>>> presentation just to ensure attendees that we would not just blindly
>>>>> make the changes and assume that they improved performance.
>>>>>
>>>>> Now to address what Dumitru mentioned :)
>>>>>
>>>>> For the most part, I think this work is all-or-nothing when it comes
>>>>> to releasing it. For instance, if we eliminate the southbound database
>>>>> but don't do any of the phases after, then the northbound database
>>>>> would be much larger than it needs to be, and it means keeping
>>>>> unnecessary tables around for backwards compatibility. Similarly, if
>>>>> we combine ovn-northd and ovn-controller, but we still write data to
>>>>> the database unnecessarily or don't optimize away unnecessary code, we
>>>>> end up with "centralized" logic being run redundantly on all nodes. It
>>>>> also means there could be awkward intermediate releases of OVN that
>>>>> have odd upgrade instructions that would not apply to future or past
>>>>> releases.

Mark, I don't necessarily agree, especially on the "all-or-nothing when
it comes to releasing" part.  I think there might be a viable way to
incrementally implement what's needed for ovn-local while still
delivering sane releases to our users.  Please see below, at the end of
the email.

>>>>>
>>>>> I think that Doc phase 1 (create a wrapper daemon) is probably the
>>>>> only thing that we might release piecemeal. The other work likely
>>>>> needs to be completed in its entirety before we release it. Doc phase
>>>>> 1 is also the most trivial, relatively speaking, of all of the phases.
>>>>> I agree that it could be done in time for 27.09. I'm not so sure about

I didn't mean this "phase 1" but the "phase 1" in the actual
presentation.  IIUC the wrapper package (as I think you called it later
in this email) is just that, a (systemd) wrapper to start all daemons at
once.

But for some (most?) of our users that will probably not be very useful.
 E.g., OpenShift ovn-kubernetes manages the lifecycle of the components
on their own as they're different containers of the same pod.  I think
OpenStack does that too for some components.

>>>>> hitting 28.03 with the rest of the work, but if we can do it, then
>>>>> great! It's hard for me to put an estimate on the work yet, since we
>>>>> don't have concrete work items defined yet. The phases in the document
>>>>> and presentation are high-level descriptions that don't get into the
>>>>> lower-level details of what needs to be changed.
>>>>>

I agree it's hard to estimate work that's "defined" only at such a high
level.  I'll try to address this with a more detailed plan (or at least
what I currently think the plan should be) in the end of this email.

>>>>>>
>>>>>> Please read further for the notes I took during the conference.
>>>>>>
>>>>>> Best regards,
>>>>>> Dumitru
>>>>>> ---
>>>>>>
>>>>>> My notes from the panel and follow up discussion:
>>>>>> -------------------------------------------------
>>>>>> DISCLAIMER: I wrote this down after the fact so there might be 
>>>>>> inaccuracies.
>>>>>>
>>>>>> The main take away from the discussion was that OVN users in the
>>>>>> audience were interested in moving towards such an architecture.  It’s
>>>>>> also in-line with other talks that were presented at the conference
>>>>>> (e.g., Crusoe is moving towards a custom ovn-ic deployment similar to
>>>>>> the OpenShift/ovn-kubernetes one - essentially one OVN per node, Stackit
>>>>>> has issues due to the large size of the SB database in their classic
>>>>>> centralized deployment).
>>>>>>
>>>>>> There were a few questions/remarks from the audience and afterwards
>>>>>> during the ad-hoc discussion.  I'll list everything (including
>>>>>> answers/comments) here:
>>>>>>
>>>>>> 1. How can we deal with the "synchronization work that ovn-northd
>>>>>> currently does, e.g., allocating SB datapath and port_binding tunnel 
>>>>>> keys?
>>>>>>
>>>>>> There were some options mentioned during the session:
>>>>>> a. fully distributed and predictable/consistent generation of the keys
>>>>>> by individual ovn-local instances.  This has IMO the risk of
>>>>>> complicating upgrades if we ever decide to change the algorithm used to
>>>>>> generate keys.
>>>>>>
>>>>>> b. adding support to ovsdb to "run stored procedures" and move the key
>>>>>> allocation logic there
>>>>>>
>>>>>> c. mimic what ovn-kubernetes does and have a "cluster manager", a
>>>>>> centralized component, essentially a way simpler ovn-northd that still
>>>>>> manages keys even for ovn-local deployments
>>>>>>
>>>>>> 2. Would moving to ovn-local shift the performance issues from the
>>>>>> central nodes to the chassis?  That’s because ovn-local would now
>>>>>> perform the work that ovn-northd does in the current deployment
>>>>>> (centralized).
>>>>>>
>>>>>> I think we can't provide a definitive answer before experimenting with
>>>>>> the proposed new architecture but we expect that in the final ovn-local
>>>>>> deployment only locally relevant logical switches/routers would be
>>>>>> processed (reducing compute time to values likely similar to what
>>>>>> ovn-controller has to process today); the large SB contents would be
>>>>>> gone, reducing time to process those in the IDL (json handling).
>>>>>>
>>>>>> 3. Would moving to ovn-local cause upgrade incompatibility?  I.e., in
>>>>>> very large clusters upgrades can take days/months.
>>>>>>
>>>>>> This has two aspects:
>>>>>> a. the first upgrade, from centralized OVN to ovn-local
>>>>>> b. subsequent upgrades, from ovn-local version x to ovn-local version x+1
>>>>>>
>>>>>> "a" was discussed separately quite extensively during the ad-hoc session
>>>>>> after the conference, I'll detail that later.
>>>>>>
>>>>>> for "b”, taking the ovn-kubernetes example, we probably have a chance of
>>>>>> incompatibility even with today’s architecture.  That’s because whenever
>>>>>> a packet needs to be tunneled to a remote chassis (i.e., it has been
>>>>>> received on a logical port on chassis-1 and needs to be sent out on a
>>>>>> logical port on chassis-2) we are splitting that datapath’s logical
>>>>>> pipeline across two nodes: ingress pipeline is executed on chassis-1
>>>>>> while egress pipeline of the same switch/router is executed on chassis-2.
>>>>>>
>>>>>> So, if a change happens that involves, for example, moving stages from
>>>>>> one pipeline (ingress or egress) to the other then these may cause
>>>>>> inconsistent packet forwarding.
>>>>>>
>>>>>> In general, this is handled today through feature flags (each chassis
>>>>>> advertises its "supported” feature set and ovn-northd only generates
>>>>>> logical flows that are compatible with the "oldest” ovn-controller).
>>>>>> Moving to ovn-local, we’d have to be even more careful with these kinds
>>>>>> of situations but, IMO, nothing new here.
>>>>>>
>>>>>> 4. What about the shared state that’s currently stored in the Southbound
>>>>>> (pushed up by ovn-controllers)?  E.g., mac-binding, fdb, learned routes.
>>>>>>
>>>>>> That’s still needed in the ovn-local mode; either stored as new tables
>>>>>> in the (new) NB database or as a "Status/Operational/etc” database.
>>>>>>
>>>>>> 5. Numan suggested reviving his old RFC work that moved logical flow
>>>>>> generation to ovn-controller or at least first focusing on removing
>>>>>> Logical_Flow from the "new” ovn-local database.
>>>>>>
>>>>>> https://mail.openvswitch.org/pipermail/ovs-dev/2021-June/384488.html
>>>>>>
>>>>>> I think the ovn-local "Phase 2" part of the proposal presented at the
>>>>>> conference covers that and we could likely start with removing Logical
>>>>>> Flows but it's my personal opinion that it would require less work
>>>>>> overall if we first implement "Phase 1" of the proposal ("Delete the
>>>>>> Southbound database").
>>
>> Thanks for putting everything together and sharing the details.
>>
>> If I understand correctly phase 1 will eliminate SB DB and the logical
>> flows will
>> be stored in the NB DB ?  Please correct me if I'm wrong.

Numan, that's definitely not what I was imagining but I might be
interpreting things differently than others.  I thought that "SB DB
elimination" would mean we don't generate logical flows anymore.  I'd
definitely vote against storing logical flows in the NB.

But, with the idea in mind that we can try to deliver ovn-local
functionality gradually, across multiple OVN releases in a sane way, I
think my original statement above doesn't stand anymore.  Please see
below for a potential way of dealing with logical flows.

>>
>> IMHO both can go in parallel.  If one gets accepted before the other,
>> it should not be
>> hard to adjust.
>>
>>
>>
>>>>>>
>>>>>> 6. Is one of the problems with the current architecture that we end up
>>>>>> pushing "too detailed” information to the chassis from the central OVN 
>>>>>> node?
>>>>>>
>>>>>> Yes, most likely, and that’s inline with other targeted optimizations we
>>>>>> did in the past, e.g., moving part of the LB flow generation directly to
>>>>>> ovn-controllers - those did come at a cost of duplicating some NB config
>>>>>> information (LBs synced to the SB).
>>>>>>
>>>>>> NOTE: The following items were discussed in the ad-hoc session, after
>>>>>> the official end of the conference:
>>>>>>
>>>>>> 7. The "first" upgrade scenario: part of the cluster using ovn-local and
>>>>>> part of the cluster (including "old” central nodes) running the legacy
>>>>>> mode.  Some options were discussed:
>>>>>>
>>>>>> a. Ilya suggested running two sets of databases: the "old" SB for
>>>>>> chassis that haven’t upgraded yet and the "new"
>>>>>> Status/Operational/Whatever database ovn-local needs moving forward.
>>>>>
>>>>> When you say "Status/Operational/Whatever database", is this just a
>>>>> rebranding of the current northbound database? Or are you referring to
>>>>> actually having a new database in addition to the current northbound
>>>>> database? If it's the former, then that aligns with what I was
>>>>> picturing when writing the docs. If it's the latter, then can you
>>>>> explain the need for an additional database?
>>>>>

No matter what we do, for "centralized" deployments (like OpenStack with
NB + ovn-northd + SB running on central nodes today), we will need a
shared store for all operational data.  That includes Port_Binding.up,
MAC_Bindings, FDBs, IGMP_Groups, Learned_Routes, etc.

In the proposal you/we presented at the conference those would end up
being moved to the NB database.  But the feedback we got at the
conference was that there's not a lot of benefit if we do that and that
it might be cleaner if we store all that in a separate database, a
"smaller SB" if you prefer that name :).

I think Felix's reply to this thread also suggests that a separate
database would be preferable.  It also means the NB can keep the same
schema as it currently has.

From
https://mail.openvswitch.org/pipermail/ovs-dev/2026-November/438023.html :

  I would not see a benefit of having it in the same database.
  People can still just run one ovsdb cluster and run both DBs on that.
  That would give everyone more flexibility and we would not loose
  anything.

  But for me the biggest question there would probably be the Chassis
  (and related) tables. Since the CMS needs to build e.g.
  HA_Chassis_Groups and similar things for them, maybe they make more
  sense in NB than in the other db.

I don't really agree with the last part in Felix's reply, I think.  For
the Chassis part, the CMS should have out-of-band information about what
Chassis configuration it provisioned on the actual hypervisors (e.g.,
system-ids/hostnames are all configured in the HV local OVSDB anyway by
the CMS).

>>>>>>
>>>>>> In his suggestion ovn-northd (upgraded) would be the intermediate actor
>>>>>> forwarding contents written by the upgraded ovn-locals in the new
>>>>>> databases to the legacy SB
>>>>>>
>>>>>> b. Another option would be to run a "SB forwarder/proxy" on the upgraded
>>>>>> nodes: ovn-local would be unaware of its existence, it would only write
>>>>>> to the new "Status" database; the "SB forwarder" would monitor the
>>>>>> "Status" and propagate the information to the "legacy SB" that’s still
>>>>>> running on the central node for the not-yet-upgraded chassis.
>>>>>>
>>>>>> 8. Taking things a step further and adding support for using other
>>>>>> databases for OVN config/status storage (e.g., etcd).
>>>>>>
>>>>>> Numan argued that we may anyway need a translation layer in the new
>>>>>> ovn-local during one of the transition phases described in the talk in
>>>>>> order to abstract out database access (e.g., the OF generation part in
>>>>>> ovn-controller currently needs logical flows to be parsed in memory); we
>>>>>> could consider keeping that in place even after ovn-local is the only
>>>>>> way of deploying OVN; we would repurpose it to provide an abstraction
>>>>>> for fetching/pushing data to the NB/Status databases.
>>>>>>
>>>>>> 9. From my perspective one of the most important requirement for this
>>>>>> whole effort is maintaining OVN backwards compatibility from CMS
>>>>>> perspective.  There might be some small (hopefully) changes the CMS
>>>>>> would have to put in place for the first upgrade to ovn-local but
>>>>>> ideally we should avoid needing other changes in the CMS. This might
>>>>>> also mean:
>>>>>>
>>>>>> a. transparently making ovn-sbctl work with ovn-local deployments
>>>>>> b. transparently making ovn-trace work with ovn-local deployments
>>>>>>
>>>>>> 10. It was raised that this might be a good moment to start using a
>>>>>> different programming language (probably Rust) for implementing the new
>>>>>> parts that will be required for ovn-local.
>>>>>>
>>>>>> 11. There was also a small debate whether we should keep logical flows
>>>>>> or not:
>>>>>> a. e.g., for debugging (easier to read than OpenFlow)
>>>>>> b. OTOH this would incur extra computation cost
>>>>>
>>>>> Even without the Logical_Flow database table, there is still some need
>>>>> for the use of logical flows since users can provide matches as part
>>>>> of an ACL or Logical_Router_Policy definition.
>>>>>
>>>>> Other than that, the only real use for logical flows in a standard OVN
>>>>> deployment would be for debugging. If we think of this as only a tool
>>>>> for debugging, I think trying to keep logical flows around long-term
>>>>> is going to lead to frustration and inaccuracy.
>>>>>
>>>>> As we accustom ourselves to writing OpenFlow directly, we eventually
>>>>> will reach a point where it makes little sense to define new logical
>>>>> flow actions that align with the OpenFlow we create. For example, we
>>>>> currently have logical flow actions that translate to a resubmit to
>>>>> some high numbered OpenFlow table to look up a specific value and set
>>>>> that value in some register (lookup_arp() for example). Now let's
>>>>> imagine that we no longer need to write logical flows, and we do
>>>>> something similar directly in OpenFlow. We would need to artificially
>>>>> define and document a new logical flow action solely for the purposes
>>>>> of debugging, even though that logical flow action would never
>>>>> actually be used anywhere in the code. Similar things are likely to
>>>>> happen with specific registers that we use in OpenFlow matches.
>>>>> Currently, we define certain shortcuts for logical flow matches and
>>>>> actions (e.g. "flags.loopback" translates to "reg10[0]"). Eventually,
>>>>> once we are defining everything in OpenFlow, such shortcuts will be
>>>>> unnecessary to define. Logical flows will eventually bitrot and
>>>>> they'll be unreliable to use for debugging since they haven't actually
>>>>> kept up with reality. If our actual development path is to define
>>>>> OpenFlow directly, then the accompanying "fake" logical flows could
>>>>> become inaccurate in the case where we update an OpenFlow flow but
>>>>> forget to update the accompanying logical flow. The code will still
>>>>> work, but now the debugging aid will show inaccurate information.
>>
>>
>> I have a different take on the logical flows.  I don't think we should
>> eliminate the use of
>> logical flows.  Logical flows provide an excellent wrapper for Openflow 
>> rules.
>> IMO expressing a pipeline as logical flows is easier than openflow rules
>> and I think it will be easier to add new features.
>>
>> How I'd use logical flows is similar to the RFC which I had proposed earlier.
>> (https://github.com/numansiddique/ovn/blob/rfc_lflows_in_controller_v2_p16/lib/lflow.c#L148)
>>
>> Lets a logical switch/datapath - sw0 is relevant to a node 'A'.
>> ovn-local on that node
>> will call a library function (something like below) which generates
>> all the logical flows for
>> this logical switch and returns.  And then the same logical flows
>> returned will be converted
>> to openflow rules using the expr parser.
>>
>> static void build_dp_lflows(struct hmap *lflows, struct local_datapath *)
>>
>> And when a  user runs the command - "ovn-sbctl dump-flows <datapath>"
>> , ovn-sbctl (or ovn-nbctl)
>> utility will call the same function -  build_lswitch_dp_lflows() and
>> dumps the logical flows.
>>
> 
> It seems the proposal document didn't make it clear whether we want to
> keep logical flows or not. Phase 4 will remove Logical_Flow table, but
> there can be two approaches:
> 1). Keep the logical flow generation phase and then translation to
> OpenFlow, but only eliminate the storage in the DB - the approach that
> Numan mentioned
> 2). Completely remove the logical flow generation phase and directly
> generate OpenFlow rules
> 
> In fact it may make sense to do above as two sequential steps.
> I agree with the benefits of only doing 1) mentioned by Numan, but we
> would lose the huge performance benefit of 2), because logical flow
> parsing is the heaviest part of today's ovn-controller processing.
> Given that it has always been a scale bottleneck, personally I am
> leaning towards 2) - avoiding the logical flow generation phase
> completely. For implementation I think doing it in the above two
> sub-phases would help evaluating the pros and cons.
> 

I completely agree with this.  I don't think we should spend a lot of
time debating whether we should remove logical flow infrastructure (even
if we don't have logical flows in the SB anymore) at this point.

It does feel like a natural way to incrementally implement this.  Using
Han's notation, first do "1)" (with logical flows in memory) and then
evaluate if we need "2)" (logical flow generation replaced with direct
OpenFlow generation).

However, it's interesting to see that logical flows being the "scale
bottleneck" is not really the pictures all our users have.  Quoting from
Felix's other reply:

https://mail.openvswitch.org/pipermail/ovs-dev/2026-November/438101.html

  We did some profiling in our environment on a heavily loaded gateway
  chassis. We did profile the recompute of the lflow_output engine node
  in ovn-controller (without the lflow cache).
  In there we found that we take around 7% of the total runtime for
  ovnacts_parse_string and 3.5% of the total time for expr_parse_string
  (for matches).

10.5% doesn't seem like "that much" to me but, of course, if we can
optimize that maybe we should.

> Best,
> Han
> 
>>
>> Thanks
>> Numan
>>
>>
>>
>>>>>
>>>>> I think that any steps we take to try to preserve logical flows will
>>>>> be counter to the goals that the ovn-local project is trying to
>>>>> achieve. Deletion of the Logical_Flow database table is one of the
>>>>> biggest wins for this project since it is massive. Logical flows
>>>>> really only make sense as a centralized concept that describes the

Does it though?  I know for sure I really appreciate the existence of
logical flows when I debug ovn-kubernetes setups.  And those are already
"one OVN per node" deployments.

Note: I'm not saying we shouldn't remove logical flows from the SB, I
still think we should do that.  I think the discussion above was just
about using the logical flow generation/parsing infrastructure in the
new ovn-local daemon.

>>>>> relationship of the logical datapaths. I think instead of trying to
>>>>> preserve logical flows as a debugging tool, we could focus instead on
>>>>> providing new debugging tools that work for ovn-local.

I think that is something we should consider anyway, what can we do to
improve debugging experience in the "ovn-local world".

Finally, here's my _personal_ view on how we could incrementally
implement ovn-local such that we can also have sane releases in the
meantime.  I'll be using letters to denote the different phases to avoid
the confusion with the phases in Mark's original doc and presentation.

Phase A:
--------
- Description: create a unified binary and package containing the
current ovn-northd AND ovn-controller code.

In this phase we don't change any functionality, we just factor out the
ovn-controller and ovn-northd main entry points into libraries.  We
adapt the ovn-controller and ovn-northd code to use the new libraries.

We _also_ create a new binary, maybe call it ovnd (or ovn-local if
that's nicer).

This new binary takes as command line arguments the union of the command
line arguments ovn-northd and ovn-controller take.

It also takes as argument the mode it should be started in, two options
that can also be combined:
- northd: should start exactly the same thing as regular ovn-northd
- controller: should start exactly the same thing as regular ovn-controller
- northd,controller: should start two separate threads, one for the
northd code and the other for the ovn-controller code.

Note: as a syntactic sugar we could also support --mode=all which would
be equivalent to --mode=northd,controller

In this phase we have:
- no major functional changes
- NB stays the same as today
- SB stays the same as today
- users can migrate towards using ovnd (with the appropriate mode set)
without being impacted in any way
- no performance change

Pros:
- ovn-kubernetes-like deployments (single OVN stack per node) can remove
the need to start separate containers for ovn-northd and ovn-controller
and instead only create a single container for ovnd with --mode=all.

Phase B:
--------
- Description: remove the SB database for ovnd --mode=northd,controller

In this phase we'd like to improve the lives of ovn-kubernetes-like
users.  To do that, the first thing we can do is remove the need for a
separate SB database process.

I think we have two options.

B1. Add another abstraction layer in OVN to choose whether all database
interaction (connection, getting updates, committing transactions)
should actually go towards a real OVS SB DB or just to an in-memory store.

Cons:
- seems like a huge amount of work TBH
- what about ovn-sbctl commands?

B2. Add a new mode to ovsdb, allowing any process to run an OVS database
in memory.  This would likely be a new feature in OVS as it would
require moving parts of ovsdb-server.c to a library ovnd can link to.
It would also likely require a new type of database connection stream,
e.g., "direct-access-stream".  With such in-memory connections we can
skip the json serialization/deserialization part that currently happens
when clients interact with the database.  We could just had the json
objects (e.g., updates or transactions) from one side to the other.

Cons: needs a non-trivial change in OVSDB.
Pros:
- probably less work than option B1.
- ovn-sbctl still works, the DB is just running inside of ovnd.

In this phase (regardless if we choose B1 or B2) we have:
- no OVN functional changes
- NB stays the same as today
- SB _schema_ stays the same as today but users can decide (ovnd cmdline
argument) to start the DB inside ovnd
- ovn-sbctl still works out of the box
- reduced latency/cpu usage because we remove some of the unnecessary
json serialization/deserialization + unix socket between ovnd and the SB.
- ovn-kubernetes-like users can migrate to the new SB mode
- neutron-like users can still deploy the SB as they're used to, on the
central node

Phase C:
--------
- Description: add a new mode to ovnd, --mode=allocator (or better name).

In this phase we factor out the parts of ovn-northd we already
identified as things that need to be run on a central node.  That
includes: datapath tunnel key allocation, port_binding tunnel key
allocation, IPAM, etc.

This makes the supported ovnd modes become:
- allocator: starts only this new component that connects to the NB and
allocates the required keys/IPs and writes them to the SB
- northd: should start the trimmed down ovn-northd component (without IP
allocation
- controller: should start exactly the same thing as regular ovn-controller
- any combination of the values above, for example:
  --mode=allocator,northd,controller starts three separate threads, one
for each component, in the same process
- the --mode=all syntactic sugar becomes equivalent to:
  --mode=allocator,northd,controller

Note: if --mode combines "allocator" & "northd" then the keys/IPs can be
directly made available to the northd module without having to go
through the additional SB roundtrip.

Note: the ovn-northd and ovn-controller binaries still exist and are
equivalent to running:
- For ovn-northd: ovnd --mode=allocator,northd
- For ovn-controller: ovnd --mode=controller

In this phase we have (compared to phase B):
- no NB changes
- no SB changes
- no major functional changes
- no performance changes
- ovn-kubernetes-like users can keep using --mode=all, no change required
- neutron-like users can decide to migrate to running two separate ovnd
processes on the central node (--mode=allocator AND --mode=northd) but
they can also just keep using the ovn-northd binary, that still works.

Phase D:
--------
- Description: allow --mode=controller to connect to the NB database and
determine the set of local logical switches/routers (equivalent to the
local_datapaths set ovn-controller builds today).

In this phase, if ovnd --mode=controller is started with a NB connection
string (--ovnnb-db), the controller module opens a connection to the NB
database.

It then monitors the minimum set of required tables.  Similar to what
ovn-controller does today to get updates about local datapaths, we'd
have to set the monitor condition to include:
- LSPs of type router/switch (equivalent to Sb.port_binding.type=patch)
- LSPs of type l2gateway,external
- LSPs of type VIF whose name matches local OVS interfaces with
external_ids:iface-id set.

And then build the set of local switches/routers.

Note: we might want to make this opt-in, TBD.
Note: these connections are read-only from --mode=controller perspective
and I'm assuming scale concerns related to all controllers connecting to
the NB can potentially be addressed by using ovsdb-relays for the NB.

Phase E:
--------
- Description: Stop writing to SB tables that are read-only from
ovn-controller perspective.

For each table we essentially move the processing that happens in the
--mode=northd component to --mode=controller (only for local
switches/routers).

This is the first tricky stage because we really don't want to break
neutron-like users (or ovn-kubernetes-like users that still use a real
SB database).

This can be a prolonged phase, we don't have to get rid of all tables at
once.

Note: we need to keep backwards compatibility so tables we get rid of
_cannot_ and _should not_ be removed from the SB schema.

Note: during this phase, --mode=controller still maintains a connection
to SB for tables not yet migrated (including Datapath_Binding and
Port_Binding for tunnel keys).

Let's assume we're removing the SB.Address_Set.  The changes that need
to happen are probably:

E1. In --mode=controller, if --ovnnb-db is set:
- add a chassis-specific feature flag to inform --mode=northd nodes that
this chassis is able to work with both NB/SB Address_set records.
- stop monitoring the SB database Address_Set table
- start monitoring the NB database Address_Set table
- change the controller incremental processing engine so that it can
handle both inputs, SB.Address_Set and Nb.Address_Set

E2. In --mode=northd:
- if _ALL_ registered chassis have the Address Set feature flag set,
stop writing to SB.Address_Set
- in consequence, if at least one registered chassis doesn't have the
feature flag, we keep populating SB.Address_Set as we do today.

E3: Update debug tools that currently require the contents of the SB
table we stopped writing to so that they fallback to using the NB, e.g.:
ovn-detrace.

Let's take a slightly more complex example, the SB.Multicast_Group table.

Today ovn-controller registers for SB.Multicat_Group updates
conditionally, only for records whose datapaths are local.  The changes
that need to happen are probably:

E1. In --mode=controller, if --ovnnb-db is set:
- add a chassis-specific feature flag to inform --mode=northd nodes that
this chassis is able to work _without_ SB Multicast_group records.
- we already monitor the set of locally relevant logical
switches/routers, factor out the code that creates SB Multicast_groups
in northd today into a library; use that library to generate the
equivalents of the SB Multicast groups (but locally)
- update the incremental processing graph to add new inputs (based on
the state computed above) to other nodes that depend on
en_sb_multicast_group today.

E2. In --mode=northd:
- change the code to use the multicast group library we built in E1 above.
- if _ALL_ registered chassis have the Multicast Group feature flag set,
stop writing to SB.Multicast_group
- in consequence, if at least one registered chassis doesn't have the
feature flag, we keep populating SB.Multicast_group as we do today.

In this phase we have (compared to phase D):
- no NB changes
- no SB schema changes
- functional changes in both northd and controller components but it's
all still backwards compatible
- performance improvements due to not having to populate SB records
- ovn-kubernetes-like users can keep using --mode=all, no change
required, they get the performance boost automatically
- neutron-like users can safely upgrade compute and central nodes,
thanks to the feature flag there's no functionality change

This phase can continue until we handled all tables that are read-only
from ovn-controller perspective.

Once this phase E is completed, in theory, the --mode=northd component
will have no work left to do.  The only centralized computation is
already handled by --mode=allocator.  That component still writes (among
other things) tunnel keys and IPs to SB.Datapath_Binding and
SB.Port_Binding.

Phase F:
--------
- Description: Stop writing to SB tables that are read-write from
  --mode=controller perspective.

This is the next tricky phase.  This is also the phase where we would
start needing the "Status" (operational, whatever) database.

Here's an incomplete list of tables/columns ovn-controller writes to:
- SB.Chassis/Chassis_Private
- SB.Port_Binding.up
- SB.IGMP_Group
- SB.MAC_Binding
- SB.FDB
- SB.Learned_Routes
- other tables required to store allocator generated keys/IPs

All the information that is written to those tables/columns today should
be set in the Status database after this Phase.  (Partial) upgrades
should still function properly.

Required changes:

F1. Define the new Status database schema (see tables above)
- also update utilities like ovn-ctl to be able to start the database
(either along side NB or as a separate ovsdb-server instance) - the
Status database is supposed to run on the central node of the cluster.

F2. For upgrades to work properly we need to ensure that partially
upgraded clusters (only part of chassis upgraded to the new version and
the central node not upgraded yet) are still in a sane state:
- old --mode=northd instances expect information to still live in the SB
- new --mode=controller instances expect information to live in the
Status DB.

A potential solution is to have a "temporary" proxy daemon running on a
central node that reconciles old SB with new Status databases.  This
proxy would:
- make sure SB records exist and are in sync for all tables that have a
corresponding Status table.
- e.g., it will have to "forward" port.up information from Status to SB
and "forward" mac_binding information from SB to status (in case a mac
binding is learned on a not-yet upgraded chassis).
- forward tunnel keys/IPs generated by the allocator in the SB to the
Status database

An alternative solution could be to have the controller component write
to both SB/Status databases as long as it doesn't get notified (how?) by
the central component that all chassis have been upgraded.

F3. Change --mode=controller code to monitor from/write to the Status
database.

Phase G:
--------
In this phase we make the --mode=allocator component write directly to
the Status database.  This can only happen only once phase F has been
completed.  Alternatively we can just keep both methods and if a SB
connection is configured along side to the Status connection, the
component could just write to both databases.

Phase H:
--------
- Description: remove --mode=northd.

As mentioned at the end of Phase E, --mode=northd doesn't need to do
anything anymore.  Upgraded chassis already can access the Status
database and the only remaining centralized processing is handled by
--mode=allocator.

We also don't need the proxy from phase F anymore.

Phase I:
--------
- Description: profile and optimize.

As you may have noticed I didn't mention Logical_Flows anywhere, that's
because I assumed that in phase E we'd be still internally generating
logical flows (in --mode=controller).

At this point, because most processing happens in --mode=controller we
can safely consider removing the logical flow intermediate step and
directly generating OpenFlow rules.

Other similar optimizations can happen here, without risking major
breakages.

NOTE: After writing the overview of the phases above I realized there's
some more "centralized" work that needs to be handled today: MAC_Binding
and FDB aging (flushing), RBAC.  Today that's in ovn-northd, in the new
proposal this could be part of the small component that continues to run
on the central node, --mode=allocator.  We might need a better name.

I'm sure I over-simplified things but I hope this helps us move the
discussion forward and that it conveys why I think we can do the move to
"ovn-local" in a sane manner even across multiple releases.

Regards,
Dumitru




_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

Reply via email to