Hi Numan, On 6/18/25 11:38 PM, Numan Siddique wrote: > On Wed, Jun 18, 2025 at 5:00 PM Dumitru Ceara <dce...@redhat.com> wrote: >> >> Hi Han, >> >> On 6/18/25 7:40 PM, Han Zhou wrote: >>> Thanks Numan for the proposal. >>> >>> On Wed, Jun 18, 2025 at 4:27 AM Dumitru Ceara <dce...@redhat.com> wrote: >>>> >>>> Hi Numan, Ales, >>>> >>>> On 6/10/25 4:24 PM, Numan Siddique wrote: >>>>> On Tue, Jun 10, 2025 at 2:04 AM Ales Musil <amu...@redhat.com> wrote: >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Jun 5, 2025 at 9:44 PM Numan Siddique <num...@ovn.org> wrote: >>>>>>> >>>>>>> On Thu, Jun 5, 2025 at 11:33 AM Dumitru Ceara <dce...@redhat.com> >>> wrote: >>>>>>>> >>>>>>>> On 6/4/25 6:24 PM, Numan Siddique wrote: >>>>>>>>> Hello, >>>>>>>>> >>>>>>>> >>>>>>>> Hi Numan, >>>>>> >>>>>> >>>>>> Hi Dumitru and Numan, >>>>>> >>>>>> I think the project might be a good idea in general, just adding >>>>>> my opinion on some parts below. >>>>>> >>>>>>>> >>>>>>>>> There's a need to configure the provider bridge with specific >>> OpenFlow >>>>>>>>> rules after packets leave the OVN pipeline and enter via the patch >>>>>>>>> port. >>>>>>>>> >>>>>>>>> To simplify this for CMS, I propose utilizing OVN logical flows. >>> This >>>>>>>>> would eliminate the need for CMS to manage direct OpenFlow >>> connections >>>>>>>>> and programming. >>>>>>>>> >>>>>>>> >>>>>>>> This seems very interesting! >>>>>>>> >>>>>>>>> To achieve this, I've developed a new service within OVN called >>>>>>>>> `ovn-pr-controller` (pr = provider). Here's a high-level overview: >>>>>>>>> >>>>>>>>> A new database, `OVN_Provider`, is created with two main tables: >>>>>>>>> `PR_Bridge` and `Logical_Flow`. >>>>>>>>> >>>>>>>>> `ovn-pr-controller` connects to this database, translates logical >>>>>>>>> flows to OpenFlow rules, and programs the bridges. >>>>>>>>> >>>>>>>>> CMS adds logical flows for managed provider bridges by connecting to >>>>>>>>> the `OVN_Provider` database. >>>>>>>>> >>>>>>>>> CMS can define the pipeline as needed. >>>>>>>>> >>>>>>>>> An `ovn-prctl` utility (similar to `ovn-nbctl`) is used to program >>>>>>>>> logical flows. >>>>>>>>> >>>>>>>>> Example Usage: >>>>>>>>> >>>>>>>>> # Add a provider bridge >>>>>>>>> `ovn-prctl add-br br-ext` >>>>>>>>> >>>>>>>>> # Add logical flows >>>>>>>>> `ovn-prctl add-flow br-ext 0 100 "inport == \\"patch-port\\"" >>>>>>>>> "ct_snat_zone = 1000; next;"` >>>>>>>>> `ovn-prctl add-flow br-ext 0 0 "1” "next;"` >>>>>>>>> >>>>>>>>> `ovn-prctl add-flow br-ext 1 1000 "ip4" "ct_snat;"` >>>>>>>>> >>>>>>>>> `ovn-prctl add-flow br-ext 2 1000 "ip4 && ct.new && ct.trk && >>> ip4.src >>>>>>>>> == 10.0.0.11" "ct_snat(100.64.0.11); next;"` >>>>>>>>> >>>>>>>>> `ovn-prctl add-flow br-ext 2 0 "ip4" "next;"` >>>>>>>>> >>>>>>>>> `ovn-prctl add-flow br-ext 3 100 "ip4 && ip4.dst == 52.92.128.0/17" >>>>>>>>> "tun.id = 1000; tun_ip4.dst = 10.100.100.1; eth.dst = >>>>>>>>> 4c:96:14:14:01:b0; outport = \\"vxlan0\\"; output;"` >>>>>>>>> >>>>>>>>> `ovn-prctl add-flow br-ext 3 0 “1” “output;”` >>>>>>>>> >>>>>>>>> I'd like to get the community's feedback on whether this service >>> would >>>>>>>>> be a valuable addition to OVN. >>>>>>>>> >>>>>>>> >>>>>>>> I wonder if we need these to be separate databases/processes. >>> Wouldn't >>>>>>>> it make sense to expose this in the NB database directly? >>>>>>>> >>>>>>>> With your proposal, these new ovn-pr-controllers would have to >>> connect >>>>>>>> to the central OVN_Provider database from all chassis. But we >>> already >>>>>>>> have the ovn-controller connection to the Southbound. Could we >>> reuse that? >>>>>>> >>>>>>> I forgot to mention that the OVN provider ovsdb-server and the >>>>>>> ovn-pr-controller needs >>>>>>> to be run on each chassis. >>>>>>> >>>> >>>> I see. However, I think it gives even more flexibility if the OVN >>>> provider database is centralized. Then the CMS central component (e.g., >>>> neutron for OpenStack) could just write "logical flows" there and rely >>>> on per-chassis ovn-pr-controllers to translate that to OpenFlow. >>>> >>> While it brings flexibility, we should also keep in mind that the central >>> approach could introduce another choke point for scale. Even if we reuse SB >>> DB as the central component, the new ovn-pr-controllers process would >>> require new connections to the SB DB. >>> >> >> True, but I was not really thinking of using the SB DB as the central >> component for ovn-pr-controllers but rather a new database server. And >> I really hope it wouldn't require storing as many logical flows (and >> data in general) as we store in the SB today. >> > > Thanks for the comments and the discussion. > > Just FYI - I thought we are in favour of having a separate project. > So I reworked a bit and renamed the project to - ovnbr controller (ovn > bridge controller) > - https://github.com/numansiddique/ovnbr/tree/initial_commits > > But I can move back the code to ovn project. So not a problem. > > > >>> Supporting both central and distributed may be an overkill at least as a >>> start, if we could decide which one is more practical from a CMS point of >>> view. >>> >> >> Maybe, but the "distributed" case would probably mean more work for the >> CMS on the node side (a per chassis CMS component is definitely required >> then to provision the per chassis database). >> >> I do understand however that we'd need a way to abstract >> chassis-specific resources (e.g., OVS ports) in the other, >> "centralized", case. And that might be a significant disadvantage. >> >>>> On second thought, if designed properly, there's probably no need to add >>>> a restriction to colocate the new database and ovn-pr-controller. We >>>> could just support both types of deployments: > > @Dumitru Ceara If I understand correctly, what you're suggesting is that > CMS can decide to run ovnbr database either central or locally - for > each chassis. >
Yes, that's what I was thinking of. > I think that would work too. Only disadvantage is that CMS has to > ensure that the bridge names and physical interface names > have to be the same. Although we can definitely find a way to handle this. > > The present proposal schema - > https://github.com/numansiddique/ovnbr/blob/initial_commits/ovn-br.ovsschema#L66 > has only a Bridge table and no ports defined. In the present proposal > CMS can add > logical flows like - "inport == eth1 && ....." and ovnbr-controller > expects an OVS interface - eth1 in the OVS bridge. > Right, we'd need a way to represent chassis specific information if we support a central ovnbr database. Probably at least: OVS ports, conntrack zones, provider bridge names. > > > >>>> - ovn-pr-controller & database on the same chassis >>>> - (multiple) ovn-pr-controller connecting to a database on a different >>>> node (e.g., colocated with the Southbound) >>>> >>>>>>> This is basically to allow a CMS agent running on each chassis to use >>>>>>> logical flows >>>>>>> and program them in the OVN provider database, instead of talking >>>>>>> OpenFlow directly. >>>>>>> >>>>>>> OVN has a very nice way of abstracting OpenFlows using logical flows >>>>>>> and my idea is to leverage it. >>>>>>> >>>>>>> ovnkube-node, as you know, programs the OpenFlow rules to the >>>>>>> provider bridge directly by execing >>>>>>> "ovs-ofctl". The OVN kubernetes community could possibly use OVN >>>>>>> provider controller. >>>> >>>> Another thing the CMS node component (ovnkube-node in the example above) >>>> needs to do is reconcile flows on the "provider" bridge on restart. >>>> >>>> This new OVN provider controller would also remove the need for the CMS >>>> to do that, which is an advantage. >>>> >>> >>> It is definitely valuable, and no objection from me. But I think it is >>> better to ask opinion from CMS component maintainers if the OVS pipeline >>> for the provider bridge is complex enough to justify this. Also, how much >>> would it help by managing logical flows through an OVSDB versus managing >>> open-flow rules though ovs-ofctl. Keep in mind that the abstraction here is >>> not at the level of the NB DB which focuses on logical topologies - here >>> the abstraction is logical-flows (something at the SB level), so the CMS >>> still needs to manage the pipeline rather than just logical topology. > > That's correct. Expectation is that CMS knows what it is doing, > And we have the requirement to add flows in the provider bridges and > hence thought > of this abstraction to > - leverage the OVN logical flows > - and to leverage the openflow handling of ovn-controller. > > > My personal preference is also to have it part of OVN so that it is > easier to manage. > > But as Ales mentioned, we have many logical actions like - > put_dhcp_opts etc (which use controller action) > which will not work with the ovn-bridge-controller. Probably > ovn-bridge-controller can maintain its own symbol table. > > > Thanks > Numan > > >>> >>> Adding @Tim Rozet <tro...@redhat.com> @Girish Moodalbail >>> <gmoodalb...@gmail.com> from ovn-k8s to comment. >>> >> >> Adding Jakub Libosvar too, from the neutron side, as we briefly touched >> on the content of this proposal in a discussion while trying to onboard >> the OVN BGP support in neutron. IMO a centralized "provider network >> database" might fit that specific use case quite well because the open >> flow rules that needed to be added to the provider bridge were quite >> generic. >> > > > > >>>>>>> I'm not suggesting this, but just giving an example or a potential >>> user. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> I believe it could be useful, but I'm unsure if it should be >>>>>>>>> integrated into OVN directly or be a separate project within >>>>>>>>> `ovn-org`. >>>>>> >>>>>> >>>>>> In my opinion it would be better as a standalone project within >>>>>> ovn-org. There is couple of reasons why: >>>>>> >>>>>> 1) Separation between ovn actions and the new ones. I suppose the >>>>>> ovn-pr-controller won't support like put_arp etc. in other words >>>>>> actions that require userspace handling. >>>>>> >>>>>> 2) There are certain expectations about parameters, mostly ct actions >>>>>> and it feels a little strange to populate a specific register in order >>>>>> to get the action working. >>>>>> >>>>>> 3) Maintenance cost to some extent, standalone project might be >>>>>> easier to maintain, it can be written in different language if it >>>>>> would be better fit. >>>>>> >>>>>> 4) The abstraction can be generated automatically to some extent >>>>>> so most of the code might be actually a way to generate the parsing >>>>>> etc. >>>>>> >>>>>> I'm fully aware that this would require some duplication. With that >>>>>> said, this just my opinion and I won't block any other decision. >>>>> >>>>> >>>>> Hi Ales, >>>>> >>>>> Thanks for the replies. I agree with your points. I think it makes >>>>> sense to have >>>>> a separate project even if the code is duplicated. >>>>> >>>> >>>> I'm not sure I see anything blocking the ovn-pr-controller (or whatever >>>> we decide to call it) code from being part of the ovn-org/ovn project. >>>> >>>> It can be a standalone daemon (similar to controller/ovn-controller or >>>> controller-vtep/ovn-controller-vtep). >>>> >>>> It can even be written in a different language (e.g., rust) if we wish. >>>> >>>> Keeping it in the same project actually reduces maintenance cost because >>>> we can just reuse the expr parsing library from OVN directly. We can >>>> also reuse all the build/test infra (if we wish). >>>> >>>> It would also mean we wouldn't have to come up with different release >>>> strategies, contribution guidelines, etc. >>>> >>>> Also, with downstream in mind, it might be (slightly) easier for distros >>>> to package this new binary if its code lived in ovn-org/ovn. >>>> >>> >>> In my option it is really not good to duplicate components in different >>> repos. Unless we find ways to expose logical flow translation as a library >>> and share between the projects, I am inclined to use OVN repo directly. >>> >>> Thanks, >>> Han >>> >>>>> Implementing in a different language would be nice but I think it will >>>>> take a lot >>>>> of effort to implement expr parsing and openflow encoding. >>>>> >>>>> Thanks >>>>> Numan >>>>> >>>> >>>> Thanks, >>>> Dumitru >>>> >>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> I guess it depends a bit on the approach chosen for implementation >>> but >>>>>>>> at a first glance this seems as a good candidate for ovn-org to me. >>>>>>>> >>>>>>>>> Three possible options are: >>>>>>>>> >>>>>>>>> - Integrate the `OVN provider controller` into `ovn-org/ovn`. >>>>>>>>> >>>>>>>>> - Create a separate project within `ovn-org` (which would require >>>>>>>>> duplicating some files like `lib/actions.c`). >>>>>>>>> >>>>>>>>> - Do not pursue this. >>>>>>>>> >>>>>>>> >>>>>>>> Listing it here because it's on the "con" side of things. :) I >>> wonder >>>>>>>> if we'll end up duplicating all OVS actions into OVN logical flow >>> actions. >>>>>>> >>>>>>> Actually that's my goal. Instead of using OpenFlow rules directly, a >>>>>>> user can express >>>>>>> their pipeline as logical flows and not worry about the intricacies of >>>>>>> openflow syntax. >>>>>>> It does add another layer of abstraction for sure. >>>>>>> >>>>>>> Thanks >>>>>>> Numan >>>>>>> >>>>>>> >>>>>>>> >>>>>>>>> I welcome your thoughts and would like to know if other OVN users >>> have >>>>>>>>> similar requirements. >>>>>>>>> >>>>>>>>> >>>>>>>>> A proof-of-concept is available here: >>>>>>>>> >>> https://github.com/numansiddique/ovn/tree/provider_controller_support. >>>>>>>>> >>>>>>>>> >>>>>>>>> If there is a consensus in pursuing this further, I'll work on >>>>>>>>> refining the patches and submit them as RFC to start with. >>>>>>>>> >>>>>>>> >>>>>>>> In general I have no objections to an RFC. (CC the rest of the >>> maintainers). >>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Numan >>>>>>>> >>>>>>>> Regards, >>>>>>>> Dumitru >>>>>>>> >>>>>>> >>>>>> >>>>>> Thanks, >>>>>> Ales >>>>> >>> >> Regards, Dumitru _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
