Hi Frode and Gurpreet,
On Sun, 1 Sep 2024 at 10:29 Gurpreet Singh <gurps...@redhat.com> wrote: > +1 on using VRFs for tenant isolation. We do not want to use separate BGP > daemon for the purpose of isolation but use VRFs. There should be 1 BGP > daemon at the system level that caters to multiple tenants. > Sure, I completely agree with you about this. From the point of view of the ovn-controller, the VRF provides the isolation that we’re looking for. However, I’m not sure about the isolation of the failure domain since we’ve all the VRF’s running in the same BGP daemon for a single router (let’s try to explain based on Frode’s comment). > Regards > Gurpreet > > On Sep 1, 2024, at 6:07 AM, Frode Nordahl <fnord...@ubuntu.com> wrote: > > > > fre. 30. aug. 2024, 19:48 skrev Roberto Bartzen Acosta < > roberto.aco...@luizalabs.com>: > >> Hello Frode, >> >> Thanks for working on this. >> > > Hello, Roberto, > > Thank you for your interest in the work. > > >> Em sex., 30 de ago. de 2024 às 12:37, Frode Nordahl <fnord...@ubuntu.com> >> escreveu: >> >>> Hello, Tiago, >>> >>> Please find my response in-line below. >>> >>> fre. 30. aug. 2024, 17:09 skrev Tiago Pires <tiago.pi...@luizalabs.com>: >>> >>> > Hi all, >>> > >>> > I did test your Routing protocol port redirection patch and I was >>> > wondering how you guys are planning to make the learning and >>> > advertisement of the routes between the Logical router's routing table >>> and >>> > the bgp daemon. >>> > >>> >>> Thank you for your interest in this work. The answer to your question is >>> in >>> the thread you are quoting, I'll highlight it with comments below. >>> >>> Thanks >>> > >>> > Regards, >>> > >>> > Tiago Pires >>> > >>> > On Tue, Aug 6, 2024 at 9:03 AM Frode Nordahl <fnord...@ubuntu.com> >>> wrote: >>> > >>> >> On Mon, Aug 5, 2024 at 12:02 PM Ales Musil <amu...@redhat.com> wrote: >>> >> > On Thu, Aug 1, 2024 at 6:04 PM Frode Nordahl <fnord...@ubuntu.com> >>> >> wrote: >>> >> >> >>> >> >> Hello, Ales, >>> >> >> >>> >> >> This is a fork of the thread to go back to discuss some of the >>> items >>> >> >> raised in the most recent instance of the OVN A/V Community meeting >>> >> >> [6]. >>> >> > >>> >> > >>> >> > >>> >> > Hi Frode, >>> >> > >>> >> > thank you for the followup discussion. >>> >> > >>> >> >> >>> >> >> On Fri, Jun 28, 2024 at 11:03 AM Ales Musil <amu...@redhat.com> >>> wrote: >>> >> >> > On Tue, Jun 25, 2024 at 6:52 PM Frode Nordahl < >>> fnord...@ubuntu.com> >>> >> wrote: >>> >> >> >> >>> >> >> >> Hello, >>> >> >> >> >>> >> >> >> We are increasingly seeing requests for integration between OVN >>> >> >> >> powered CMSs/workloads and the fabric. >>> >> >> >> >>> >> >> >> As a side note, this is a very interesting topic to me >>> personally, >>> >> and >>> >> >> >> I think there are opportunities in the long term for this class >>> of >>> >> >> >> software to potentially fill a void for more automated and >>> SDN-like >>> >> >> >> ways of managing the physical network, as previously closed >>> physical >>> >> >> >> switch hardware is increasingly opening up to programmatic >>> extension >>> >> >> >> and control. >>> >> >> >> >>> >> >> >> While very exciting, it will take a while, both in terms of >>> evolving >>> >> >> >> how networking teams are organized, in terms of the longevity of >>> >> >> >> networking gear making entity wide refresh cycles very long, >>> not to >>> >> >> >> mention gathering agreement and momentum to build such a thing >>> from >>> >> >> >> the pieces we have. >>> >> >> >> >>> >> >> >> >>> >> >> >> So to be pragmatic, we need to integrate with something that >>> fabric >>> >> >> >> network engineers are comfortable with, and already available on >>> >> most >>> >> >> >> networking hardware, be it closed or open, today. >>> >> >> >> >>> >> >> >> The most ubiquitous routing protocol, which has prevailed in >>> modern >>> >> >> >> layer 3 only data center designs [0], is BGP. >>> >> >> >> >>> >> >> >> Use cases: >>> >> >> >> * Allow fabric to locate and direct traffic to reroutable >>> resources >>> >> >> >> such as IPv4/IPv6 prefixes, Floating IPs (FIPs) and Load >>> Balancer >>> >> >> >> VIPs. >>> >> >> >> >>> >> >> >> * Use the fabric as a load balancer, announcing the same >>> service IP >>> >> on >>> >> >> >> multiple hosts (anycast). >>> >> >> >> >>> >> >> >> * Aggregate announcements from stacked CMSes (i.e. Kubernetes >>> >> running >>> >> >> >> on top of OpenStack). >>> >> >> >> >>> >> >> >> >>> >> >> >> Requirements: >>> >> >> >> * Data path must be hardware offloaded, i.e. the next hop >>> address >>> >> the >>> >> >> >> peer resolves for announcements of OVN resources needs to be an >>> LRP >>> >> >> >> IP. >>> >> >> >> >>> >> >> >> * Minimize configuration overhead through the use of IPv6 LLAs >>> for >>> >> >> >> peering routing both IPv4 and IPv6 prefixes over a IPv6 BGP >>> session >>> >> >> >> [1] (aka. “BGP Unnumbered”). >>> >> >> >> >>> >> >> >> * Support ECMP out of the host, i.e. use L3 interfaces >>> potentially >>> >> >> >> connecting to two different ToRs, instead of bonds, avoiding the >>> >> >> >> additional complexity of multi-chassis bonds. >>> >> >> >> >>> >> >> >> * Support BGP authentication [2][3], i.e. the source, >>> destination >>> >> >> >> address and ports in packet headers can not be changed. >>> >> >> >> >>> >> >> >> * Compatibility >>> >> >> >> * Running a BGP protocol suite on the host is becoming a >>> thing in >>> >> >> >> its own right, and our users may have requirements of their own >>> that >>> >> >> >> influence their choice of implementation. We need to take this >>> into >>> >> >> >> account and choose integration methods that allow OVN to work >>> with >>> >> >> >> multiple protocol suite implementations. >>> >> >> >> >>> >> >> >> * While we have the power to change and fix issues in popular >>> >> >> >> routing protocol suites, such as FRR, we need to be able to >>> >> integrate >>> >> >> >> with versions that exist on networking hardware out there today. >>> >> >> >> >>> >> >> >> Limitations that influence/dictate implementation choices: >>> >> >> >> * Peering with IPv6 LLAs to meet the configuration overhead >>> >> >> >> requirement makes the peering relationship point to point. >>> >> >> >> >>> >> >> >> * Popular BGP implementations, such as FRR which is used as >>> routing >>> >> >> >> protocol suite by many ToR open source NOSes, does not accept >>> >> >> >> sending/receiving IPv6 LLA next hop with the route, so the BGP >>> peer >>> >> >> >> address will be used as next hop. (There are even mentions of >>> 3rd >>> >> >> >> party nexthop currently not being supported, but not sure if >>> that is >>> >> >> >> accurate [4]). >>> >> >> >> >>> >> >> >> * As mentioned above, BGP authentication requires IP headers to >>> be >>> >> >> >> unchanged for the BGP TCP packets going to/from the BGP speaker. >>> >> >> >> >>> >> >> >> >>> >> >> >> Proposed implementation: >>> >> >> >> >>> >> >> >> We are in the process of preparing some RFC/PoC patches that at >>> a >>> >> high >>> >> >> >> level will: >>> >> >> >> * Manage a VRF in the system serving two purposes: >>> >> >> >> * Leaking of route information from ovn-controller to the VRF >>> >> >> >> routing table, which a routing protocol suite can redistribute >>> >> subject >>> >> >> >> to configuration. >>> >> >> >> >>> >> >> >> * Provide an IP endpoint that a VRF aware application, such >>> as >>> >> FRR, >>> >> >> >> can bind to serving as a BGP speaker on behalf of a OVN LRP IP. >>> >> >> >> >>> >> >> >> * We will attach a OVN VIF to this VRF that has data path rules >>> >> that: >>> >> >> >> >>> >> >> >> * Forward required traffic destined to the OVN LRP IP to the >>> VRF. >>> >> >> >> >>> >> >> >> * Forward required traffic from the application bound to the >>> VRF >>> >> as >>> >> >> >> if it originated from the OVN LRP IP. >>> >> >>> > >>> The above bullets give an overview of the proposed implementation. >>> >> >> Just to validate the understanding and align expectations, the main >> purpose of this implementation/module developed for route-exchange-netlink >> is to export addresses that are typically "external" from the point of view >> of the OVN router / SDN. In this case dnat_and_snat rules for FIPs and LB >> VIP's configured on the Logical Router, right? >> > > Development happens in iterations, so this is indeed where we start. We > also want to get to learning of routes, as that would simplify > configuration, and hopefully we will. > > So, it's out of the scope to advertise/learn routes from the router's >> internal networks, as well as the router's static route table, directly >> connected LS subnets, etc. Therefore, it's out of the scope to make the >> service and integration of the BGP daemon multi-tenant, since there is no >> plan for segmentation by namespaces to run different BGP daemons, right? I >> imagine that in the multi-tenant use case we would have a BGP daemon for >> each logical router, in addition to playing with the router's complete >> route table. >> > > The BGP redirect option and route redistribute options are per LRP, and > there are no restrictions on how the LSP redirected to is terminated in the > system. Likewise, the use of VRFs to exchange route information give you > isolation. We view the BGP daemon as a system/admin level entity and the > isolation you seek can be configured in it? > > We have large users that do not use NAT and would be interested in > redistributing LR networks attached to distributed gateways (the OpenStack > use case). > > However, before diving into that we need to figure out how to connect a > distributed topology with the per chassis gateway router, as the answer to > that will impact what resources it makes sense to redistribute where. > > Maybe this is the most important part to discuss related to the BGP daemon as a system/admin level. We know that a logical router is attacked to only one gateway chassis, so what will happen when we need to perform maintenance on the BGP daemon of the chassis where the router's LRP resides? My only concern about having only one daemon is related to this, maintenance caused by problems in other tenants or in the OS itself can cause traffic disruption due to reconnection of all BGP sessions. Regarding the implementation itself, I see at least two main applications for integrating BGP with OVN. The first one is the integration with the fabric ToR’s, which is the target of this proposal. Something similar to what Neutron-Dynamic-Routing aims to do in the OpenStack world (advertise everything that is public and needs to be integrated into the fabric routing). The second one is the integration with the SDN itself, and this is something that large public clouds have been doing for quite some time, that is, allowing a BGP instance to connect to router networks at the overlay/SDN level. So, this doesn’t depend on cloud technology, it’s something that transcends the use of OpenStack or OVN-Kubernetes for example. There are temporary solutions with many limitations such as OVN-BGP-Agent, but they do not meet the complete integration requirements. This is the point that we are most interested in, giving OVN the ability to learn routes and inject them into the router's route tables at the overlay level and segmented by tenant. We would like to discuss with you further the vision of this second use case of BGP integration with OVN, and of course, knowing how we can help with this initiative. Best regards, Roberto > -- > Frode Nordahl > > > > Best regards, >> Roberto >> >> >>> >> >> >>> >> >> >> >>> >> >> >> Hopefully we'll have something up on the list before the end of >>> this >>> >> >> >> week, which makes it real and easier to reason about for further >>> >> >> >> discussion. >>> >> >> >> >>> >> >> >> >>> >> >> >> Prior art: >>> >> >> >> >>> >> >> >> We recognize that there already exists a third party approach to >>> >> this >>> >> >> >> in the ovn-bgp-agent [5] governed by OpenStack, and our goal >>> with >>> >> this >>> >> >> >> work is to provide a tighter integration that might cater >>> >> generically >>> >> >> >> for other CMSes and use cases. >>> >> >> >> >>> >> >> >> >>> >> >> >> 0: https://datatracker.ietf.org/doc/html/rfc7938 >>> >> >> >> 1: https://datatracker.ietf.org/doc/html/rfc5549 >>> >> >> >> 2: https://datatracker.ietf.org/doc/html/rfc2385 >>> >> >> >> 3: https://datatracker.ietf.org/doc/html/rfc5925 >>> >> >> >> 4: >>> >> >>> https://github.com/FRRouting/frr/blob/cc3519f3e6eaa06f762e0d447202df32df66e129/bgpd/bgp_route.c#L2719 >>> >> >> >> 5: https://docs.openstack.org/ovn-bgp-agent/latest/ >>> >> >> >>> >> >> >>> >> >> 6: >>> >> >>> https://mail.openvswitch.org/pipermail/ovs-dev/2024-August/416209.html >>> >> >> >>> >> >> > >>> >> >> > Hi Frode, >>> >> >> > >>> >> >> > looking forward to the RFC. >>> >> >> >>> >> >> As we agreed, the current set of patches that we have >>> >> >> [7][8][9][10][11][12][13] will not be considered for the 24.09 >>> release >>> >> >> as we would like to make it more feature complete and target the >>> 25.03 >>> >> >> release instead. In that context I guess they serve as the RFC >>> >> >> patches. >>> >> >> >>> >> >> 7: >>> >> https://mail.openvswitch.org/pipermail/ovs-dev/2024-July/416038.html >>> >> >> 8: >>> >> https://mail.openvswitch.org/pipermail/ovs-dev/2024-July/416039.html >>> >> >> 9: >>> >> https://mail.openvswitch.org/pipermail/ovs-dev/2024-July/416040.html >>> >> >> 10: >>> >> https://mail.openvswitch.org/pipermail/ovs-dev/2024-July/416042.html >>> >> >> 11: >>> >> https://mail.openvswitch.org/pipermail/ovs-dev/2024-July/416041.html >>> >> >> 12: >>> >> https://mail.openvswitch.org/pipermail/ovs-dev/2024-July/416043.html >>> >> >> 13: >>> >> https://mail.openvswitch.org/pipermail/ovs-dev/2024-July/416044.html >>> >> >> >>> >> >>> > >>> The above patches are the current state of the work and we will continue >>> it >>> the coming cycle with the intention to get it into the 25.03 release. If >>> you have cycles to try it out and provide feedback, that would be most >>> welcome. >>> >>> -- >>> Frode Nordahl >>> >>> >> In addition to the above there is the LRP BGP redirect patch from >>> >> >> Martin [14], which could be useful independently. >>> >> >> >>> >> >> 14: >>> >> https://mail.openvswitch.org/pipermail/ovs-dev/2024-July/416095.html >>> >> >> >>> >> >> Discussion points from meeting: >>> >> >> 1) OVN and Netlink code >>> >> >> In the meeting you raised some concerns about introducing Netlink >>> code >>> >> >> in the OVN repository. While I agree with you 100% that the part of >>> >> >> [10] that vendors code from OVS (contents of >>> >> >> route-exchange-netlink-private.h), should instead be a patch for >>> OVS. >>> >> >> >>> >> >> The parts of [10] that provide higher layer helper functions, >>> >> >> consuming OVS library code, do not naturally fit in OVS as OVS >>> itself >>> >> >> has no use for them. >>> >> > >>> >> > >>> >> > >>> >> > After the discussion that we had during the meeting I agree that we >>> >> should reuse >>> >> > as much OvS code as possible. >>> >> > >>> >> >> >>> >> >> As a quick reminder, we are looking at Netlink because it provides >>> a >>> >> >> simple and established API for exchange of this type of information >>> >> >> which is already supported by all routing protocol suites out >>> there. >>> >> >> It is not tied to any particular data path type, we could >>> >> >> theoretically even use it as IPC between two userspace processes, >>> >> >> removing the kernel from the picture with support on the routing >>> >> >> protocol suite side. (There has been some discussion on this for >>> BIRD: >>> >> >> >>> >> >>> https://bird.network.cz/pipermail/bird-users/2021-September/015707.html >>> ). >>> >> >> >>> >> >> Would it be possible to reach some compromise to include only the >>> >> >> parts that consume OVS library code (route-exchange-netlink.{c|h})? >>> >> >> >>> >> >> While a plugin based approach was also suggested, and we have prior >>> >> >> examples of successfully using that, it does not come without >>> >> >> substantial cost. So I want to explore what options there are to >>> host >>> >> >> this inside the main repository. >>> >> > >>> >> > >>> >> > We could maintain the netlink plugin in the OVN codebase as an >>> example >>> >> > and still have the plugin system in place. The plugin system has >>> >> potential >>> >> > benefit of not locking ourselves to just netlink, but we could >>> >> potentially use >>> >> > API regardless of the OVN code. Would that be enough of >>> justification >>> >> > and to add the plugin system + netlink as default plugin housed >>> >> > in the OVN codebase? >>> >> >>> >> This makes sense and provides a path forward, thanks! >>> >> >>> >> We are in a point of the development cycle where we need to tend to >>> >> some downstream stuff, but we will continue this work. I'll try to get >>> >> patches to make the OVS route-table module consumable/reusable for >>> >> this work posted as soon as possible and start thinking about what's >>> >> needed in the route-exchange provider interface. >>> >> >>> >> >> 2) OVS OpenFlow extensions >>> >> >> One of the counter proposals you brought up was to add OVS OpenFlow >>> >> >> extensions to allow OVN instruct OVS to insert routes into a system >>> >> >> routing table. >>> >> >> >>> >> >> While I see this could be a clear separation of concerns between >>> OVN >>> >> >> and OVS, and OpenFlow being OVN's native integration language, I >>> >> >> struggle a bit with the general usefulness of such an extension. >>> >> >> >>> >> >> Our use case for inserting routes into a system routing table is >>> >> >> purely for exchange of control plane information with some external >>> >> >> system, such as a routing protocol suite, and we have no interest >>> in >>> >> >> using it for actual data path control. This is in contrast to how >>> OVN >>> >> >> uses OpenFlow generally, which as far as I understand is to control >>> >> >> the data path. >>> >> > >>> >> > >>> >> > In the light of the other discussion it doesn't really make sense to >>> >> have >>> >> > plugin when most of the netlink code wouldn't be in the OvS codebase >>> >> > anyway. >>> >> >>> >> Assuming you are referring to the OpenFlow extension here, and yes, I >>> >> agree. >>> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> [ snip ] >>> >> >> >>> >> >> > Another important part that we should keep in mind if possible, >>> is >>> >> the EVNP >>> >> >> > use case. To be able to configure VXLAN tunnels based on the info >>> >> that we >>> >> >> > will receive. >>> >> >> > >>> >> >> > I'm not sure how far/deep in the actual design you are but maybe >>> the >>> >> following >>> >> >> > might be helpful in some way. What I had in mind was sort of >>> plugin >>> >> that would >>> >> >> > expose the info of bound entities that are interesting in terms >>> of >>> >> BGP >>> >> >> > (it could be configurable), so mainly FIPS, LBs, GW router IPs. >>> For >>> >> the import >>> >> >> > part (which would be applicable only for GW LR) we would create >>> >> entries in SB >>> >> >> > DB similarly as we do currently with Multicast_Group >>> >> >> > (BGP_Routes? EVPN_Tunnels?). Northd could consume those values >>> and >>> >> configure >>> >> >> > logical flows and encaps as needed. >>> >> >> >>> >> >> While the EVPN part is not a priority for us at this point in >>> time, we >>> >> >> will of course be interested in making sure the work we put into >>> stage >>> >> >> 1 (ovn-controller redistributing FIPS, LBs and GW router IPs), >>> stage 2 >>> >> >> (ovn-controller learning routes) will be consumable for a stage 3. >>> >> > >>> >> > >>> >> > Great, yeah my point during the discussion wasn't about >>> >> > making it available right away just to reiterate that it would be of >>> >> > interest for us and if needed we can of course help with the >>> development >>> >> > process. >>> >> >>> >> Cool stuff, let's do it! >>> >> >>> >> -- >>> >> Frode Nordahl >>> >> >>> >> >> >>> >> >> >>> >> >> -- >>> >> >> Frode Nordahl >>> >> >> >>> >> >> > Let me know if that makes sense. >>> >> >> > >>> >> >> > Thanks, >>> >> >> > Ales >>> >> >> > >>> >> >> > >>> >> >> > -- >>> >> >> > >>> >> >> > Ales Musil >>> >> >> > >>> >> >> > Senior Software Engineer - OVN Core >>> >> >> > >>> >> >> > Red Hat EMEA >>> >> >> > >>> >> >> > amu...@redhat.com >>> >> >> >>> >> > >>> >> > Thanks, >>> >> > Ales >>> >> > >>> >> > -- >>> >> > >>> >> > Ales Musil >>> >> > >>> >> > Senior Software Engineer - OVN Core >>> >> > >>> >> > Red Hat EMEA >>> >> > >>> >> > amu...@redhat.com >>> >> _______________________________________________ >>> >> dev mailing list >>> >> d...@openvswitch.org >>> >> https://mail.openvswitch.org/mailman/listinfo/ovs-dev >>> >> >>> > >>> > >>> > *‘Esta mensagem é direcionada apenas para os endereços constantes no >>> > cabeçalho inicial. Se você não está listado nos endereços constantes no >>> > cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa >>> > mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas >>> estão >>> > imediatamente anuladas e proibidas’.* >>> > >>> > *‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para >>> > assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não >>> > poderá aceitar a responsabilidade por quaisquer perdas ou danos >>> causados >>> > por esse e-mail ou por seus anexos’.* >>> > >>> _______________________________________________ >>> dev mailing list >>> d...@openvswitch.org >>> https://mail.openvswitch.org/mailman/listinfo/ovs-dev >>> >> >> >> *‘Esta mensagem é direcionada apenas para os endereços constantes no >> cabeçalho inicial. Se você não está listado nos endereços constantes no >> cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa >> mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão >> imediatamente anuladas e proibidas’.* >> >> *‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para >> assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não >> poderá aceitar a responsabilidade por quaisquer perdas ou danos causados >> por esse e-mail ou por seus anexos’.* >> > > -- _‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’._ * **‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.* _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
