On 11/16/26 12:10 PM, Rukomoinikova Aleksandra wrote: > On 12.11.2026 20:31, Numan Siddique wrote: >> On Thu, Nov 12, 2026 at 12:12 PM Dumitru Ceara <[email protected]> wrote: >>> On 11/12/26 4:51 PM, Rukomoinikova Aleksandra wrote: >>>> On 12.11.2026 16:04, Dumitru Ceara wrote: >>>>> Внимание: ВНЕШНИЙ отправитель! >>>>> >>>>> >>>>> Будьте осторожны с вложениями и ссылками. >>>>> >>>>> >>>>> On 11/12/26 1:31 PM, Ales Musil wrote: >>>>>> On Thu, Nov 12, 2026 at 12:56 PM Rukomoinikova Aleksandra >>>>>> <[email protected]> wrote: >>>>>> >>>>>>> On 12.11.2026 10:20, Ales Musil wrote: >>>>>>>> On Wed, Nov 11, 2026 at 2:34 PM Dumitru Ceara <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>> On 11/11/26 2:07 PM, Rukomoinikova Aleksandra wrote: >>>>>>>> > On 11.11.2026 13:58, Dumitru Ceara wrote: >>>>>>>> >> On 11/11/26 11:32 AM, Rukomoinikova Aleksandra wrote: >>>>>>>> >>> On 11.11.2026 01:48, Numan Siddique wrote: >>>>>>>> >>>> On Fri, Aug 28, 2026 at 6:18 AM Alexandra Rukomoinikova via >>>>>>>> dev >>>>>>>> >>>> <[email protected]> wrote: >>>>>>>> >>>>> Until now the CMS had to create the EVPN interface stack >>>>>>>> (bridge, >>>>>>>> >>>>> VXLAN and advertise devices) on every chassis itself and >>>>>>>> pass the >>>>>>>> >>>>> interface names via the "dynamic-routing-bridge-ifname", >>>>>>>> >>>>> "dynamic-routing-vxlan-ifname" and >>>>>>>> "dynamic-routing-advertise-ifname" >>>>>>>> >>>>> options of the Logical_Switch. >>>>>>>> >>>>> >>>>>>>> >>>>> Add a new "dynamic-routing-maintain-evpn" option, which >>>>>>>> makes >>>>>>>> >>>>> ovn-controller create and maintain those interfaces on its >>>>>>>> own on >>>>>>>> >>>>> every chassis where the logical switch is local. The only >>>>>>>> supported >>>>>>>> >>>>> value is "mvd", i.e. one VXLAN device per VNI. The devices >>>>>>>> are named >>>>>>>> >>>>> after the VNI: ovnbr<vni>, ovnvxlan<vni> and ovnlo<vni>. >>>>>>>> >>>>> >>>>>>>> >>>>> If "dynamic-routing-redistribute" includes "ip", the >>>>>>>> logical >>>>>>>> router >>>>>>>> >>>>> port has to be specified with >>>>>>>> "dynamic-routing-maintain-evpn-lrp". >>>>>>>> >>>>> In that case ovn-controller also creates the ovnvrf<vni> >>>>>>>> VRF, enslaves >>>>>>>> >>>>> the bridge to it and assigns the router port MAC to the >>>>>>>> bridge, so >>>>>>>> >>>>> that the logical router addresses can be advertised over >>>>>>>> EVPN. >>>>>>>> >>>>> The interfaces are removed on graceful shutdown of >>>>>>>> ovn-controller. >>>>>>>> >>>>> >>>>>>>> >>>>> The UDP destination port of the VXLAN device is >>>>>>>> configurable >>>>>>>> with >>>>>>>> >>>>> the "dynamic-routing-maintain-evpn-vxlan-port" option and >>>>>>>> defaults >>>>>>>> >>>>> to 4790. It has to differ from the port used by the OVS >>>>>>>> datapath >>>>>>>> >>>>> VXLAN device, because the maintained device only carries >>>>>>>> EVPN state >>>>>>>> >>>>> for the routing daemon. >>>>>>>> >>>>> >>>>>>>> >>>>> Signed-off-by: Alexandra Rukomoinikova >>>>>>>> <[email protected] >>>>>>>> >>>>>>>> >>>> Hi Alexandra, >>>>>>>> >>>> >>>>>>>> >> Hi Alexandra, Numan, >>>>>>>> >> >>>>>>>> >>>> Thanks for working on this; having ovn-controller own the >>>>>>>> EVPN >>>>>>>> >>>> device stack is a nice improvement. I built the series (it >>>>>>>> >>>> compiles cleanly) and went through it, and found a few >>>>>>>> issues >>>>>>>> that >>>>>>>> >>>> I think need addressing before it can go in. >>>>>>>> >>>> >>>>>>>> >> Indeed, thanks for working on this! I can see how this would >>>>>>>> make the >>>>>>>> >> CMS life easier. However, I have some more concerns, please >>>>>>>> see below. >>>>>>>> >> >>>>>>>> >>>> 1) Interface names can exceed IFNAMSIZ. >>>>>>>> >>>> >>>>>>>> >>>> The devices are named ovnvxlan<vni>, ovnbr<vni>, >>>>>>>> ovnlo<vni> and >>>>>>>> >>>> ovnvrf<vni>. A VNI can have 8 digits, so for VNI >= >>>>>>>> 10000000 >>>>>>>> >>>> "ovnvxlan<vni>" is 16 characters, but Linux allows at >>>>>>>> most 15. >>>>>>>> >>>> The buffers are IFNAMSIZ + 1, so snprintf() doesn't >>>>>>>> truncate, >>>>>>>> >>>> and the kernel then rejects the name on every run. >>>>>>>> Could we use >>>>>>>> >>>> a shorter prefix (e.g. "ovnvx"), or at least reject >>>>>>>> such >>>>>>>> VNIs >>>>>>>> >>>> with a clear warning? >>>>>>>> >>>> >>>>>>>> >>>> 2) VRF names and table ids can collide with route-exchange. >>>>>>>> >>>> >>>>>>>> >>>> route-exchange already creates VRFs named >>>>>>> "ovnvrf<table_id>" >>>>>>>> >>>> (controller/route.c). This patch creates "ovnvrf<vni>" >>>>>>>> with the >>>>>>>> >>>> VNI as the table id. If a router's dynamic-routing >>>>>>>> VRF id >>>>>>>> >>>> happens to equal a switch's VNI, both features end up >>>>>>>> managing >>>>>>>> >>>> the same device. route-exchange creates it with >>>>>>>> NLM_F_EXCL and >>>>>>>> >>>> treats EEXIST as an error, the EVPN code treats EEXIST >>>>>>>> as >>>>>>>> >>>> success, and whichever side tears down first deletes >>>>>>>> the >>>>>>>> >>>> other's VRF. Using the VNI as the table id also >>>>>>>> clashes >>>>>>>> with any >>>>>>>> >>>> other user of that table number. >>>>>>>> >>>> >>>>>>>> >>>> 3) VNI 253/254 maps onto the default/main routing table. >>>>>>>> >>>> >>>>>>>> >>>> Related to the above: TABLE_ID_VALID() doesn't exclude >>>>>>>> >>>> RT_TABLE_DEFAULT or RT_TABLE_MAIN, so a VNI of 254 >>>>>>>> creates a VRF >>>>>>>> >>>> bound to the host's main table. I think we need a >>>>>>>> separate >>>>>>>> >>>> table id (configurable, or derived in a way that can't >>>>>>>> hit >>>>>>>> >>>> reserved or already-used tables) rather than using the >>>>>>>> VNI >>>>>>>> >>>> directly. >>>>>>>> >>>> >>>>>>>> >>>> 4) The link device is hard-coded to vxlan_sys_4789. >>>>>>>> >>>> >>>>>>>> >>>> ovn-controller creates the OVS EVPN VXLAN ports from >>>>>>>> the >>>>>>>> >>>> chassis's ovn-evpn-vxlan-ports, which can be any port, >>>>>>>> so the >>>>>>>> >>>> kernel device is vxlan_sys_<port>. On a chassis that >>>>>>>> isn't >>>>>>>> >>>> using port 4789 (or has no EVPN VXLAN port yet) the >>>>>>>> lookup >>>>>>>> >>>> fails. evpn_create_devices() then returns false, the >>>>>>>> entry >>>>>>>> >>>> never becomes synced, and every engine run re-runs the >>>>>>> whole >>>>>>>> >>>> create path, sets the netlink error status (which keeps >>>>>>>> waking >>>>>>>> >>>> the engine) and logs a warning. The system test works >>>>>>>> around >>>>>>>> >>>> this by creating vxlan_sys_4789 by hand. >>>>>>>> >>>> >>>>>>>> >>>> Along the same lines, the documentation says the >>>>>>>> maintained >>>>>>>> >>>> device's port (default 4790) must differ from "the OVS >>>>>>>> datapath >>>>>>>> >>>> VXLAN device (4789)". It really has to differ from >>>>>>>> whatever is >>>>>>>> >>>> in ovn-evpn-vxlan-ports; if that includes 4790, the two >>>>>>>> devices >>>>>>>> >>>> compete for the same UDP port. Can we derive both the >>>>>>>> link >>>>>>>> >>>> device and the conflict check from >>>>>>>> ovn-evpn-vxlan-ports? >>>>>>>> >>>> >>>>>>>> >>>> 5) Existing devices are adopted without checking them. >>>>>>>> >>>> >>>>>>>> >>>> EEXIST is treated as success and the reconcile state >>>>>>>> lives only >>>>>>>> >>>> in memory, so after "ovn-controller exit --restart": >>>>>>>> >>>> >>>>>>>> >>>> - if the local IP, VXLAN port, LRP MAC or redistribute >>>>>>>> mode of a >>>>>>>> >>>> VNI changed while ovn-controller was down, the old >>>>>>>> devices are >>>>>>>> >>>> kept with the stale configuration; >>>>>>>> >>>> - if a VNI was removed while ovn-controller was down, >>>>>>>> its devices >>>>>>>> >>>> are never cleaned up; >>>>>>>> >>>> - a CMS-created device that happens to be named >>>>>>>> ovnbr<vni> etc. >>>>>>>> >>>> is taken over, and later deleted. >>>>>>>> >>>> >>>>>>>> >>>> The restart test passes because the VNI is still >>>>>>>> present >>>>>>>> after >>>>>>>> >>>> the restart, but removing it while ovn-controller is >>>>>>>> down isn't >>>>>>>> >>>> covered. I think we need to verify the kind and >>>>>>>> attributes of >>>>>>>> >>>> an existing device before trusting EEXIST, and to sweep >>>>>>>> stale >>>>>>>> >>>> ovn*<vni> devices on startup. >>>>>>>> >>> Hi Numan and Dumitru and Ales, >>>>>>>> >>> >>>>>>>> >>> I’ve thought some more about why I implemented it this way, >>>>>>>> and I’d like >>>>>>>> >>> to discuss one point. >>>>>>>> >>> >>>>>>>> >>> Some context for Dumitru and Ales: I added you to the thread >>>>>>>> because I’d >>>>>>>> >>> be interested in your opinion. Currently, the code that >>>>>>>> creates the EVPN >>>>>>>> >>> interfaces (|ovnbr<vni>|, |ovnvxlan<vni>|, |ovnlo<vni>|, and >>>>>>>> >>> |ovnvrf<vni>|) does not check their actual state in the >>>>>>>> kernel. It only >>>>>>>> >>> reacts to changes in the desired configuration: options, >>>>>>>> local >>>>>>>> IP, and >>>>>>>> >>> so on. If an interface is modified or deleted externally, the >>>>>>>> controller >>>>>>>> >>> won’t notice. It won’t even recreate a deleted interface >>>>>>>> until >>>>>>>> something >>>>>>>> >>> changes in the configuration. >>>>>>>> >>> >>>>>>>> >>> I implemented it this way for two reasons: >>>>>>>> >>> >>>>>>>> >>> 1. >>>>>>>> >>> >>>>>>>> >>> I thought it made sense that, once we give OVN >>>>>>>> responsibility for >>>>>>>> >>> these interfaces, the CMS has neither a need nor a >>>>>>>> reason >>>>>>>> to touch >>>>>>>> >>> them. So the controller shouldn’t need to expect changes >>>>>>> made >>>>>>>> >>> outside its control. But I agree that this reasoning >>>>>>>> breaks down as >>>>>>>> >>> soon as a person gets involved: someone could easily >>>>>>>> modify >>>>>>>> >>> interfaces created by OVN, either by mistake or while >>>>>>>> debugging. My >>>>>>>> >>> assumption here was that, if the CMS or a person changes >>>>>>>> something, >>>>>>>> >>> OVN can already break things even without my code, >>>>>>>> because the >>>>>>>> >>> controller currently retries failed netlink transactions >>>>>>>> in a loop. >>>>>>>> >>> >>>>>>>> >>> 2. >>>>>>>> >>> >>>>>>>> >>> I didn’t want to check kernel state on every controller >>>>>>>> iteration, >>>>>>>> >>> as that seemed excessive. However, >>>>>>>> |neighbor_exchange_run()|already >>>>>>>> >>> dumps the neighbor table for every tracked interface on >>>>>>> every >>>>>>>> >>> iteration, which is considerably more expensive than a >>>>>>>> few >>>>>>>> >>> |RTM_GETLINK|requests. So this argument doesn’t seem to >>>>>>>> hold up. >>>>>>>> >> You make a good point here but that is something we should >>>>>>>> address in >>>>>>>> >> general anyway, we should add proper incremental processing >>>>>>>> for >>>>>>>> >> monitoring kernel state (routes, neighbors, links, etc). >>>>>>>> >> >>>>>>>> >>> There is another case the current approach doesn’t cover: an >>>>>>>> >>> |ovn-controller|restart (|exit --restart|or a crash), as >>>>>>>> Numan >>>>>>>> pointed >>>>>>>> >>> out in the review above. After startup, the internal state is >>>>>>>> empty, >>>>>>>> >>> interface creation returns |EEXIST|, and we consider the >>>>>>>> stack >>>>>>>> >>> synchronized. If, for example, the VXLAN port or local IP >>>>>>>> changed while >>>>>>>> >>> the controller was down, the device will retain the old >>>>>>>> values, and the >>>>>>>> >>> controller won’t know. >>>>>>>> >>> >>>>>>>> >>> So the question is which approach would be better: >>>>>>>> >>> >>>>>>>> >>> 1. >>>>>>>> >>> >>>>>>>> >>> *Check on every iteration.*Send |RTM_GETLINK|requests >>>>>>>> for >>>>>>>> each VNI’s >>>>>>>> >>> interfaces on every iteration. Check that each interface >>>>>>>> exists and >>>>>>>> >>> verify its type, master, MAC address, and UP state; for >>>>>>>> VXLAN, also >>>>>>>> >>> check the VNI, local IP, destination port, and link. >>>>>>>> Compare these >>>>>>>> >>> against the desired configuration. Fix attributes that >>>>>>>> the kernel >>>>>>>> >>> doesn’t allow changing on an existing device (port, >>>>>>>> local >>>>>>>> IP, VNI) >>>>>>>> >>> by recreating the interface, and update everything else >>>>>>>> in place. >>>>>>>> >>> The advantage is that this is simple and reliable: it >>>>>>>> doesn’t depend >>>>>>>> >>> on what woke the controller up, and the same code >>>>>>>> handles >>>>>>>> restarts, >>>>>>>> >>> lost events, and manual changes. The downside I’m not >>>>>>>> happy about is >>>>>>>> >>> the amount of code needed to parse and compare >>>>>>>> attributes. >>>>>>>> >>> >>>>>>>> >>> 2. >>>>>>>> >>> >>>>>>>> >>> *Rely on *|*if_notifier_cb*|*.*Check kernel state only >>>>>>>> when >>>>>>>> >>> |host-if-monitor|reports a change to our interfaces. To >>>>>>>> make this >>>>>>>> >>> reliable, we would need to: >>>>>>>> >>> >>>>>>>> >>> * >>>>>>>> >>> >>>>>>>> >>> Keep a module-level dirty flag and clear it only >>>>>>>> after >>>>>>>> >>> successful reconciliation here in evpn devices code. >>>>>>>> >>> >>>>>>>> >>> * >>>>>>>> >>> >>>>>>>> >>> Handle |ENOBUFS|on the netlink socket, indicating >>>>>>>> lost events, >>>>>>>> >>> by marking everything dirty. Currently, the callback >>>>>>>> simply >>>>>>>> >>> returns in this case. >>>>>>>> >>> >>>>>>>> >>> * >>>>>>>> >>> >>>>>>>> >>> Remove the |IFF_UP|filter in |if_notifier_cb|so that >>>>>>>> we also >>>>>>>> >>> catch interfaces going down, and add |ovnvrf*|to the >>>>>>>> tracked >>>>>>>> >>> interfaces. >>>>>>>> >>> >>>>>>>> >>> * >>>>>>>> >>> >>>>>>>> >>> Always reconcile with the kernel when ovn-controller >>>>>>>> starts, >>>>>>>> >>> since devices left over from a previous run don’t >>>>>>>> generate any >>>>>>>> >>> events. >>>>>>>> >>> The advantage is that we query the kernel only when >>>>>>>> something >>>>>>>> >>> has actually changed. The downside is more state and >>>>>>>> more edge >>>>>>>> >>> cases that are easy to miss. It doesn’t really seem >>>>>>>> worth it. >>>>>>>> >>> >>>>>>>> >>> We need the attribute parsing and comparison code in both >>>>>>>> approaches, at >>>>>>>> >>> least to handle restarts. The only difference is /when/we >>>>>>>> call >>>>>>>> it. So >>>>>>>> >>> I’m leaning toward the first approach: it’s simpler and more >>>>>>>> reliable. >>>>>>>> >>> But maybe you have different opinions. >>>>>>>> >>> >>>>>>>> >> This is, IIRC, the main reason why in internal discussions I >>>>>>>> had with >>>>>>>> >> Ales when trying to figure out how to integrate OVN with EVPN >>>>>>>> >> deployments we decided against maintaining the required host >>>>>>>> side >>>>>>>> >> devices ourselves - it's quite a lot of (arguably >>>>>>>> uninteresting) work. >>>>>>>> >> And probably with a lot of added code and the bugs that come >>>>>>>> with it. >>>>>>>> >> >>>>>>>> >>> What do you think? >>>>>>>> >>> >>>>>>>> >> I'm wondering now if it isn't better to just provide a helper >>>>>>>> utility as >>>>>>>> >> part of OVN utilities that the CMS can run to ensure that the >>>>>>>> host side >>>>>>>> >> interfaces are in the correct state (matching what's in the NB >>>>>>>> and SB >>>>>>>> >> database). It doesn't have to but it could be a simple script >>>>>>>> I guess. >>>>>>>> >> And it could reconcile the host side interfaces automatically >>>>>>>> if needed. >>>>>>>> >> >>>>>>>> >> It would still be up to the CMS to figure out the best way to >>>>>>>> run the >>>>>>>> >> utility, e.g., triggered when specific interface changes are >>>>>>>> detected. >>>>>>>> >> >>>>>>>> >> Would it make sense to reconsider this as a different >>>>>>>> approach? >>>>>>>> > >>>>>>>> > Hi, thanks for the reply! >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Hi Dumitru, Alexandra and Numan, >>>>>>> Hi Ales, >>>>>>> >>>>>>> Thanks for sharing your view! Please see my replies inline. >>>>>>>> > >>>>>>>> > To be honest, I’m not entirely sure the script is the right way >>>>>>>> to go. >>>>>>>> > Let me describe how I picture it, so we can check it matches >>>>>>>> what you >>>>>>>> > have in mind: >>>>>>>> > >>>>>>>> > 1. ovn-controller has a handler that validates all the >>>>>>>> interface >>>>>>>> > attributes against the current configuration and recreates >>>>>>>> the >>>>>>>> > interfaces if needed. >>>>>>>> >>>>>>>> The way I was imagining it (maybe over-simplified) was that >>>>>>>> ovn-controller doesn't do any of this work. It would be the >>>>>>>> responsibility of the CMS to call the script whenever it >>>>>>>> (re)starts >>>>>>>> ovn-controller (e.g., as part of the systemd scripts). >>>>>>>> >>>>>>>> >>>>>>>> I also don't think that this should be OVN's responsibility >>>>>>>> at all. Sure we have "maintain-vrf", but it shouldn't be there >>>>>>>> IMO. The benefit vs. maintenance cost and code size doesn't >>>>>>>> seem to favor those changes. And the discussion here >>>>>>>> solidifies that even more for me. >>>>>>> I understand you don't like it, but I'd like to try once more to argue >>>>>>> why this isn't such a really bad feature: >>>>>>> >>>>>>> 1. The devices are part of OVN, not of the routing daemon or something >>>>>>> else. OVN writes >>>>>>> into them: FDB and neighbor entries on the bridge and VXLAN >>>>>>> devices, >>>>>>> route-exchange installs routes into >>>>>>> the VRF. Their configuration is derived purely from OVN state >>>>>>> (VNI, >>>>>>> per-VNI local IP, router port MAC, the VXLAN port derived from the >>>>>>> VNI). Nothing depends on something external like routing deaemon >>>>>>> or >>>>>>> etc, >>>>>>> so switching to another routing daemon wouldn't change anything >>>>>>> here. >>>>>>> >>>>>> Well not really, the expectation is for CMS to provide them. If they >>>>>> don't >>>>>> it's their fault. So I don't see them as part of OVN. >>>>>> >>>>> Sure, but Alexandra is trying to change this restriction in this patch >>>>> set. >>>>> >>>>>>> 2. ovn-controller knows better then CMS where and when the devices are >>>>>>> needed. >>>>>>> They are needed on chassis where the switch is local, and that >>>>>>> changes at runtime, e.g. when a VM migrates to another hypervisor >>>>>>> so >>>>>>> ovn-controller sees that quite fast but a CMS has to track port >>>>>>> bindings itself and race to configure the new host in time. With >>>>>>> this feature, for L2 EVPN the CMS doesn't have to do anything once >>>>>>> FRR is running; for L3 EVPN it only has to configure the VRF in >>>>>>> FRR. >>>>>>> >>>>>> OVN does, but the same logic can be applied to FRR. Should OVN >>>>>> start FRR after failover? If CMS already has the FRR ready on all >>>>>> chassis where it is important why couldn't it have all the needed >>>>>> interfaces? That's not any different, setting them up once on those >>>>>> chassis which will make potential failover even quicker than >>>>>> OVN catching up and adding the interfaces only after they are needed. >>>>>> >>>>> This is a good point I didn't think of before. The routing control >>>>> plane needs the interfaces in order to function properly. And the >>>>> routing control plane is under control of the CMS. >>>> >>>> My example probably wasn't the best one. Let me clarify two points: >>>> >>>> 1. >>>> >>>> Migration may not be the best example, but what about simply >>>> attaching a port to a subnet? Would the CMS have to create >>>> interfaces for every VNI on every hypervisor? I'm not sure I >>>> understand what you meant. >>>> >>> Yes, that's the current expectation that the CMS configures interfaces >>> for every required VNI on the required hypervisors. The routing control >>> plane (e.g., FRR) must be running on the required hypervisors too and >>> must anyway be configured to handle the VNI. That's also outside the >>> scope of OVN and most likely configured by the CMS. >>> >>>> 2. >>>> >>>> Regarding failover, I disagree with your argument that the CMS would >>>> be faster. The CMS and the controller would receive the updates at >>>> essentially the same time, when the old controller releases the >>>> chassisredirect port and the new one claims it. By that point, the >>>> new controller would already have recomputed the OpenFlow rules and >>>> routing state locally and would be ready to receive traffic. Isn't >>>> that the case? >>>> >>> I think Ales was thinking of the potential fast-failover use case - in >>> that case FRR would have to already be running on both chassis a DGP >>> might be scheduled on. So the absence of the Linux interfaces on the >>> not-yet-active chassis would make it that FRR can't immediately change >>> the routing information it may be advertising. >>> >>>>>>> I hope I’ve made it clear what I mean. >>>>>>> >>>>>>> About the maintenance cost, I'd suggest we define once, and document (in >>>>>>> ovn user documentation), >>>>>>> what OVN is responsible for and then rely on that: >>>>>>> >>>>>>> - OVN creates and deletes the devices and makes sure they are up. >>>>>>> - OVN owns them. If someone modifies or deletes them after OVN created >>>>>>> them, that's a misconfiguration. >>>>>>> - On startup, OVN dumps the devices in its group, deletes the ones >>>>>>> that are no longer in the configuration, and checks the attributes >>>>>>> of the remaining ones once against the desired config, recreating >>>>>>> them on mismatch. That covers configuration changes made while >>>>>>> ovn-controller was down. No extra state needs to be stored: the >>>>>>> kernel already holds what is currently configured. >>>>>>> - No periodic validation of attributes at runtime. >>>>>>> >>>>>>> The same startup cleanup would also fix stale VRFs from >>>>>>> dynamic-routing-maintain-vrf. >>>>>>> >>>>>>> Does this sound like an acceptable scope? >>>>>>> >>>>>> That's still quite a bit of code to maintain. >>>>>> >>>>>> >>>>>>> So in my patch I'd propose adding a pass on startup that dumps all >>>>>>> the interfaces (Further down in the email, there’s a comment from me >>>>>>> about this.) and deletes the stale ones, but doesn't validate their >>>>>>> attributes. >>>>>>> >>>>>>> But this proposal still leaves one case open: if someone changes the >>>>>>> configuration (not specifically, not changing the VNI, but rather >>>>>>> parameters like MAC/IP) >>>>>>> >>>>>>> while ovn-controller is stopped. Maybe I could store >>>>>>> what is currently configured in the ovsdb? Then on restart we >>>>>>> validate it, and if the user changed anything, even the MAC address, >>>>>>> OVN recreates the interfaces, the same way it already does when the >>>>>>> change happens while the controller is running. >>>>>>> What do you think? >>>>>>> >>>>>> That broadens the scope again, now we would have to maintain >>>>>> what was there previously. >>>>>> >>>>>> >>>>> I'm on the fence when it comes to this discussion in general. This >>>>> argument is making me sway again into the direction of "let the CMS do >>>>> it". It feels like we will be adding a lot of code just to make the >>>>> life of the CMS easier while the CMS can probably already do that in a >>>>> simpler way (as it has more domain-specific knowledge). >>>> I'm still struggling to understand why the CMS is considered better >>>> placed to manage these interfaces...The CMS expresses the desired >>>> logical configuration through NB. Why should it also need to track which >>>> hypervisor currently hosts a particular lsp and manage the corresponding >>>> local interfaces? To me, that seems to duplicate placement information >>>> that OVN already maintains. >>>> >>> Well, we have a very simple counter example: LSPs and the OVS >>> interfaces. The CMS needs to create the LSP (in the NB) but also need >>> to create the OVS interface (with correct external_ids) on the >>> hypervisor where the workload actually runs. >>> >>>>>>>> > 2. We call it on ovn-controller startup, which solves the >>>>>>>> problem Numan >>>>>>>> > described (more on that below). >>>>>>>> >>>>>>>> See above. >>>>>>>> >>>>>>>> > 3. If the CMS isn’t sure the interfaces are valid, it can call >>>>>>>> the same >>>>>>>> > entry point. >>>>>>>> > >>>>>>>> > The upside is that ovn-controller doesn’t have to check every >>>>>>>> attribute >>>>>>>> > on every iteration. But I have some doubts about it, because it >>>>>>>> means >>>>>>>> > one more knob and one more entry point. The whole idea of >>>>>>>> maintain-evpn >>>>>>>> > is that OVN owns these interfaces and the CMS doesn’t have to >>>>>>>> think >>>>>>>> > about them. A “check it if you’re not sure” knob hands that >>>>>>>> > responsibility back to the CMS. That said, if you think this >>>>>>>> approach >>>>>>>> > sounds good, I’m happy to add it. I will cast aside all my >>>>>>>> doubts >>>>>>>> > >>>>>>>> >>>>>>>> I'm not sure it's the best thing, I was just trying to reconsider >>>>>>>> it as >>>>>>>> an alternative. :) >>>>>>>> >>>>>>>> > About the problem Numan described: it also applies to the VRFs >>>>>>>> in >>>>>>>> > dynamic routing. If ovn-controller is stopped without cleanup >>>>>>>> and >>>>>>>> > someone changes the configuration in the meantime (removes the >>>>>>>> VNI or >>>>>>>> > the maintain-vrf option), then after the restart ovn-controller >>>>>>>> will >>>>>>>> > never clean up the leftover interfaces. >>>>>>>> > >>>>>>>> > I think this can be solved separately from this patch, for EVPN >>>>>>>> and VRF >>>>>>>> > at the same time. My idea is: >>>>>>>> > >>>>>>>> > * Put all interfaces created by OVN into the same device >>>>>>>> group >>>>>>>> > (IFLA_GROUP). It can be set right when the interface is >>>>>>> created. >>>>>>>> > * On ovn-controller startup, dump the interfaces and delete >>>>>>>> the ones >>>>>>>> > in our group that are no longer in the current >>>>>>>> configuration. >>>>>>>> > * Do this only once, on startup. >>>>>>>> > * Interfaces created by the CMS (e.g. a VRF without >>>>>>>> maintain-vrf) >>>>>>>> > aren’t in our group, so we won’t touch them. >>>>>>>> > >>>>>>>> > The group number should probably be configurable, in case >>>>>>>> groups >>>>>>>> are >>>>>>>> > already used for something else on the host. >>>>>>>> > >>>>>>>> >>>>>>>> This is a nice idea! I'd argue however that the group number >>>>>>>> should be >>>>>>>> mandatory if we go this way. >>>>>>>> >>>>>>>> >>>>>>>> I mean is this different from a name check? Unfortunately, the kernel >>>>>>>> doesn't allow us to filter by device group so we need to dump >>>>>>>> everything anyway. >>>>>>> This would only happen once, on startup, so does it really matter? I >>>>>>> don't think filtering interfaces by group on startup is a big >>>>>>> problem to do it in ovn-controller on startup once. And how is it >>>>>>> different from dumping all interfaces and >>>>>>> filtering them by name, i.e. ovnvrf*, ovnvx*, ovnbr*, ovnlo*? That >>>>>>> would work too, but it looks kind of ugly, doesn't it? I'd still dump >>>>>>> all the interfaces, filter them by name and then parse the VNI out of >>>>>>> >>>>>>> the name. I don't really like that... >>>>>>> >>>>>> I mean what is the difference between filtering by name and group? >>>>>> Besides the fact that filtering by name doesn't require extra setup >>>>>> while the group does as we don't currently do that. >>>>>> >>>>> I missed the lack of kernel side filtering yesterday. That does change >>>>> the context, I agree, today we can already by name already in >>>>> ovn-controller, no need for the group then. >>>> >>>> But then, when dumping interfaces, I would have to compare each >>>> interface name against at least four strings (ovnbr, ovnvrf, ovnvx, >>>> ovnlo). I understand your point about introducing a group ID when we >>>> could rely on existing information, but I still think the code would be >>>> cleaner with a group ID. >>>> >>>>>>> Anyway, I’ll look forward to hearing your thoughts on how I should go >>>>>>> about this. >>>>>>> >>>>>>>> > With this approach, on ovn-controller startup we can call the >>>>>>>> routine >>>>>>>> > that syncs the attributes and cleans up the stale interfaces. >>>>>>>> > >>>>>>>> > What do you think? >>>>>>>> > >>>>>>>> >>>>>>>> I guess as long as we document it well that CMS touching anything >>>>>>>> that's >>>>>>>> in that group results in a misconfiguration it's probably fine. >>>>>>>> >>>>>>>> I'm curious what others think about this (Ales, Numan, ..) >>>>>>>> though. >>>>>>>> >>>>>>>> >>>>>>>> As I said above I'm not sure we should maintain this in OVN. >>>>>>>> Especially when the contract is a on the weak side, reconcile >>>>>>>> only on restart and if anything happens in between you might >>>>>>>> have a problem. >>>>>>>> >>>>>>>> However, if Dumitru and Numan are fine with this feature >>>>>>>> I won't block it. >>>>>>>> >>>>>>>> >>>>>>>> > Thanks, Sasha >>>>>>>> > >>>>>>>> >>>>>>>> Thanks for the discussion! >>>>>>>> >>>>>>>> Dumitru >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Ales >>>>>>>> >> Hi Sasha, Ales, Dumitru, >> >> My 2 cents. >> >> Thanks for the discussion. I think the answer is different for L2 >> and L3 VNIs, and that this may let us agree on a smaller scope. >> >> The deciding question for me is whether enabling or disabling EVPN on >> a logical switch needs any per-VNI routing daemon configuration. >> >> Dumitru, you wrote that FRR "must anyway be configured to handle the >> VNI". For L3 VNIs that's true: each one needs "vrf X / vni N", a >> "router bgp ... vrf X" instance and an export policy. For L2 VNIs, >> as far as I understand FRR, it isn't: with a one-time configuration >> (the BGP instance, the EVPN sessions and "advertise-all-vni" under >> "address-family l2vpn evpn"), zebra discovers a new L2 VNI from the >> VXLAN device attached to a bridge, and BGP starts advertising its >> Type-2 and Type-3 routes without any change to FRR. The one >> condition is that the route targets line up across VTEPs: the >> default "<ASN>:<VNI>" only works with a shared ASN, so deployments >> with per-node ASNs need a fabric-wide route-target scheme in that >> one-time configuration. (Please correct me here if my understanding is wrong >> here). >> >> So: >> >> - For L2 VNIs, once FRR is set up, the kernel devices are the only >> per-VNI, per-chassis work left for the CMS. That's where I think >> it makes sense for ovn-controller to manage them. >> >> - For L3 VNIs, the CMS has to configure FRR per VRF anyway, and >> Dumitru's fast-failover point applies: the devices have to exist >> on every chassis a distributed gateway port may move to, so that >> FRR is ready before the port moves. There the CMS might as well >> create the devices too. For what it's worth, that's what we do: >> our agent creates the L3 VNI devices together with the FRR >> configuration for each VRF. >> >> >> IMO, it makes sense for ovn-controller to manage the vni interfaces >> for L2 VNIs. However if it complicates the code in ovn-controller, >> I'm fine passing on the responsibility to CMS. >> >> I have another suggestion - is to take the approach along the lines >> of the VIF plug >> providers: ovn-controller computes the desired per-VNI devices for the >> logical switches, and a provider (or an external agent reading that >> state from the local OVS database) creates them. OVN would keep the >> placement logic and the device code would live outside the tree. >> >> Is this approach we can explore ? >> >> Thanks >> Numan > Hi everyone!
Hi Sasha, > Numan, thanks for the suggestion to use a separate provider, similar to > VIF plug providers. Before we go down that way, I’d like to give the > built-in approach another look =). I think we can keep the > implementation fairly small if we agree on what OVN should handle. I > also want to explain why I think L3 fits well into the existing > mechanism one more time. > > 1. I agree with your explanation of L2, and I’d like to keep L3 support > too. In the L2 case you described, once the routing daemon and route > targets are set up, adding a VNI doesn’t require any changes to the > daemon’s configuration. We just need to create the Linux interfaces on > the right chassis. As you said, having OVN automate this would be > useful. The same applies to L3, and most of the device management code > would be shared with L2. I checked the L3 fast failover case you > described: OVN already creates the necessary infrastructure on both > gateway chassis, primary and standby. So the CMS just needs to configure > and start the routing daemon on both nodes ahead of time, with nothing > else to do during failover. I’ll also add a test to cover the switchover > and check that routing works correctly afterwards. I think a test for that would be great nevertheless (even today without support for auto-creating EVPN interfaces). > 2. I think we should clearly document what the controller does and > doesn’t handle. OVN creates, configures, and brings up its devices. When > the OVN configuration changes, it updates or recreates them and removes > anything that’s no longer needed. At startup, it picks up configuration > changes made while it was stopped and cleans up devices left over from > the previous run that are no longer needed. But it won’t keep checking > every device attribute at runtime, and we won’t guarantee that it > automatically fixes changes made outside OVN. The CMS and administrators > shouldn’t modify or delete these devices themselves; we’d treat that as > a misconfiguration. I think these are reasonable limits, and they should > help keep the code small and straightforward. > 3. I’ll move all the device management code into a separate file. > > What do you think? Would this approach work for you? > After reading the details below, and thinking a bit more about the vif provider alternative I have some more thoughts, please see below. :) > Below are a few more details on what I’d like to finish in this patch, > including how I’d implement the behavior described in point 2. Feel free > to skip this part :) > There are three things I’d like to address: > 1. The VRF conflict I’d like to rework this part of the patch. Right > now, for an L3VNI, we specify the LRP that connects the EVPN interfaces > to the OVN router, but we create ovnvrf<VNI> independently of the > router’s VRF. We then expect the CMS to set the router’s VRF to > ovnvrf<VNI>. That’s where the issues Numan pointed out in the review > come from. My suggestion is to make the explicit VRF setting optional in > this mode. If no VRF is set, the controller will create one > automatically, using the tunnel_key of the router associated with the > LRP as the table ID, just as dynamic routing does today. The evpn-device > code will handle creating and deleting this VRF, and route-exchange will > use the same VRF for route advertisements. This way, we’ll no longer tie > the table ID to the VNI, and the router and its EVPN interfaces will > share one VRF. > 2. Handling devices at startup I’d still like to consider saving the > last successfully applied device configuration in the local OVSDB. We’d > store the actual computed values: MAC, local VTEP IP, VNI, UDP port, So, essentially we'd store in the local OVSDB the "desired state" for the EVPN required linux interfaces. That's good. > LRP, and so on. At startup, the controller would compute these values > again from the current OVN configuration, compare them with the saved > values, and apply any changes. For example, if the LRP name stays the > same but its MAC changes, we’ll catch that and recreate the interface. > This would let us handle OVN configuration changes within the limits > described above. But now that we have the "desired state" in OVSDB, it really feels like a pity to half implement a controller that syncs this as Linux host interfaces.. Briefly looking at the dummy vif provider and at the official representor provider [0] there are some things to take into account: The vif-plug implementation is really designed to "plug" VIFs corresponding to LSPs. Also, in this case there's no LSP representing the EVPN interfaces. I guess we could change the vif-plug implementation to handle EVPN interface plugging - not sure how nice/ugly that'd be. But if we do have that covered somehow then I think it would actually be relatively easy to have an evpn vif provider that implements a proper reconciliation loop between the OVSDB "desired state" and host interfaces "actual state". The alternative to all this would still be a separate controller reading the OVSDB "desired state". Just to keep that logic outside of OVN. [0] https://github.com/ovn-org/ovn-vif/blob/main/lib/vif-plug-providers/representor/vif-plug-representor.c > 3. Cleaning up old interfaces I’ll use your suggestion to dump all > interfaces and identify the old ones by name, since the group_id > approach didn’t work out for you. > It’d be nice to hear your thoughts on this. > > Thanks, Sasha > Regards, Dumitru >> >>>>> Thanks, >>>>> Dumitru >>>>> >>>>> Внимание: ВНЕШНИЙ отправитель! >>>>> >>>>> >>>>> Будьте осторожны с вложениями и ссылками. >>> Regards, >>> Dumitru >>> > _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
