On 18.11.2026 16:17, Dumitru Ceara wrote:
> On 11/16/26 12:10 PM, Rukomoinikova Aleksandra wrote:
>> On 12.11.2026 20:31, Numan Siddique wrote:
>>> On Thu, Nov 12, 2026 at 12:12 PM Dumitru Ceara <[email protected]> wrote:
>>>> On 11/12/26 4:51 PM, Rukomoinikova Aleksandra wrote:
>>>>> On 12.11.2026 16:04, Dumitru Ceara wrote:
>>>>>> Внимание: ВНЕШНИЙ отправитель!
>>>>>>
>>>>>>
>>>>>> Будьте осторожны с вложениями и ссылками.
>>>>>>
>>>>>>
>>>>>> On 11/12/26 1:31 PM, Ales Musil wrote:
>>>>>>> On Thu, Nov 12, 2026 at 12:56 PM Rukomoinikova Aleksandra
>>>>>>> <[email protected]> wrote:
>>>>>>>
>>>>>>>> On 12.11.2026 10:20, Ales Musil wrote:
>>>>>>>>> On Wed, Nov 11, 2026 at 2:34 PM Dumitru Ceara <[email protected]> 
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>        On 11/11/26 2:07 PM, Rukomoinikova Aleksandra wrote:
>>>>>>>>>        > On 11.11.2026 13:58, Dumitru Ceara wrote:
>>>>>>>>>        >> On 11/11/26 11:32 AM, Rukomoinikova Aleksandra wrote:
>>>>>>>>>        >>> On 11.11.2026 01:48, Numan Siddique wrote:
>>>>>>>>>        >>>> On Fri, Aug 28, 2026 at 6:18 AM Alexandra Rukomoinikova 
>>>>>>>>> via dev
>>>>>>>>>        >>>> <[email protected]> wrote:
>>>>>>>>>        >>>>> Until now the CMS had to create the EVPN interface stack
>>>>>>>>>        (bridge,
>>>>>>>>>        >>>>> VXLAN and advertise devices) on every chassis itself and
>>>>>>>>>        pass the
>>>>>>>>>        >>>>> interface names via the "dynamic-routing-bridge-ifname",
>>>>>>>>>        >>>>> "dynamic-routing-vxlan-ifname" and
>>>>>>>>>        "dynamic-routing-advertise-ifname"
>>>>>>>>>        >>>>> options of the Logical_Switch.
>>>>>>>>>        >>>>>
>>>>>>>>>        >>>>> Add a new "dynamic-routing-maintain-evpn" option, which 
>>>>>>>>> makes
>>>>>>>>>        >>>>> ovn-controller create and maintain those interfaces on 
>>>>>>>>> its
>>>>>>>>>        own on
>>>>>>>>>        >>>>> every chassis where the logical switch is local.  The 
>>>>>>>>> only
>>>>>>>>>        supported
>>>>>>>>>        >>>>> value is "mvd", i.e. one VXLAN device per VNI.  The 
>>>>>>>>> devices
>>>>>>>>>        are named
>>>>>>>>>        >>>>> after the VNI: ovnbr<vni>, ovnvxlan<vni> and ovnlo<vni>.
>>>>>>>>>        >>>>>
>>>>>>>>>        >>>>> If "dynamic-routing-redistribute" includes "ip", the 
>>>>>>>>> logical
>>>>>>>>>        router
>>>>>>>>>        >>>>> port has to be specified with
>>>>>>>>>        "dynamic-routing-maintain-evpn-lrp".
>>>>>>>>>        >>>>> In that case ovn-controller also creates the ovnvrf<vni>
>>>>>>>>>        VRF, enslaves
>>>>>>>>>        >>>>> the bridge to it and assigns the router port MAC to the
>>>>>>>>>        bridge, so
>>>>>>>>>        >>>>> that the logical router addresses can be advertised over 
>>>>>>>>> EVPN.
>>>>>>>>>        >>>>> The interfaces are removed on graceful shutdown of
>>>>>>>>>        ovn-controller.
>>>>>>>>>        >>>>>
>>>>>>>>>        >>>>> The UDP destination port of the VXLAN device is 
>>>>>>>>> configurable
>>>>>>>>>        with
>>>>>>>>>        >>>>> the "dynamic-routing-maintain-evpn-vxlan-port" option and
>>>>>>>>>        defaults
>>>>>>>>>        >>>>> to 4790.  It has to differ from the port used by the OVS
>>>>>>>>>        datapath
>>>>>>>>>        >>>>> VXLAN device, because the maintained device only carries
>>>>>>>>>        EVPN state
>>>>>>>>>        >>>>> for the routing daemon.
>>>>>>>>>        >>>>>
>>>>>>>>>        >>>>> Signed-off-by: Alexandra Rukomoinikova 
>>>>>>>>> <[email protected]
>>>>>>>>>
>>>>>>>>>        >>>> Hi Alexandra,
>>>>>>>>>        >>>>
>>>>>>>>>        >> Hi Alexandra, Numan,
>>>>>>>>>        >>
>>>>>>>>>        >>>> Thanks for working on this; having ovn-controller own the 
>>>>>>>>> EVPN
>>>>>>>>>        >>>> device stack is a nice improvement.  I built the series 
>>>>>>>>> (it
>>>>>>>>>        >>>> compiles cleanly) and went through it, and found a few 
>>>>>>>>> issues
>>>>>>>>>        that
>>>>>>>>>        >>>> I think need addressing before it can go in.
>>>>>>>>>        >>>>
>>>>>>>>>        >> Indeed, thanks for working on this!  I can see how this 
>>>>>>>>> would
>>>>>>>>>        make the
>>>>>>>>>        >> CMS life easier.  However, I have some more concerns, please
>>>>>>>>>        see below.
>>>>>>>>>        >>
>>>>>>>>>        >>>> 1) Interface names can exceed IFNAMSIZ.
>>>>>>>>>        >>>>
>>>>>>>>>        >>>>      The devices are named ovnvxlan<vni>, ovnbr<vni>,
>>>>>>>>>        ovnlo<vni> and
>>>>>>>>>        >>>>      ovnvrf<vni>.  A VNI can have 8 digits, so for VNI >=
>>>>>>>>>        10000000
>>>>>>>>>        >>>>      "ovnvxlan<vni>" is 16 characters, but Linux allows at
>>>>>>>>>        most 15.
>>>>>>>>>        >>>>      The buffers are IFNAMSIZ + 1, so snprintf() doesn't
>>>>>>>>>        truncate,
>>>>>>>>>        >>>>      and the kernel then rejects the name on every run.
>>>>>>>>>        Could we use
>>>>>>>>>        >>>>      a shorter prefix (e.g. "ovnvx"), or at least reject 
>>>>>>>>> such
>>>>>>>>>        VNIs
>>>>>>>>>        >>>>      with a clear warning?
>>>>>>>>>        >>>>
>>>>>>>>>        >>>> 2) VRF names and table ids can collide with 
>>>>>>>>> route-exchange.
>>>>>>>>>        >>>>
>>>>>>>>>        >>>>      route-exchange already creates VRFs named
>>>>>>>> "ovnvrf<table_id>"
>>>>>>>>>        >>>>      (controller/route.c).  This patch creates 
>>>>>>>>> "ovnvrf<vni>"
>>>>>>>>>        with the
>>>>>>>>>        >>>>      VNI as the table id.  If a router's dynamic-routing 
>>>>>>>>> VRF id
>>>>>>>>>        >>>>      happens to equal a switch's VNI, both features end up
>>>>>>>>>        managing
>>>>>>>>>        >>>>      the same device.  route-exchange creates it with
>>>>>>>>>        NLM_F_EXCL and
>>>>>>>>>        >>>>      treats EEXIST as an error, the EVPN code treats 
>>>>>>>>> EEXIST as
>>>>>>>>>        >>>>      success, and whichever side tears down first deletes 
>>>>>>>>> the
>>>>>>>>>        >>>>      other's VRF.  Using the VNI as the table id also 
>>>>>>>>> clashes
>>>>>>>>>        with any
>>>>>>>>>        >>>>      other user of that table number.
>>>>>>>>>        >>>>
>>>>>>>>>        >>>> 3) VNI 253/254 maps onto the default/main routing table.
>>>>>>>>>        >>>>
>>>>>>>>>        >>>>      Related to the above: TABLE_ID_VALID() doesn't 
>>>>>>>>> exclude
>>>>>>>>>        >>>>      RT_TABLE_DEFAULT or RT_TABLE_MAIN, so a VNI of 254
>>>>>>>>>        creates a VRF
>>>>>>>>>        >>>>      bound to the host's main table.  I think we need a 
>>>>>>>>> separate
>>>>>>>>>        >>>>      table id (configurable, or derived in a way that 
>>>>>>>>> can't hit
>>>>>>>>>        >>>>      reserved or already-used tables) rather than using 
>>>>>>>>> the VNI
>>>>>>>>>        >>>>      directly.
>>>>>>>>>        >>>>
>>>>>>>>>        >>>> 4) The link device is hard-coded to vxlan_sys_4789.
>>>>>>>>>        >>>>
>>>>>>>>>        >>>>      ovn-controller creates the OVS EVPN VXLAN ports from 
>>>>>>>>> the
>>>>>>>>>        >>>>      chassis's ovn-evpn-vxlan-ports, which can be any 
>>>>>>>>> port,
>>>>>>>>>        so the
>>>>>>>>>        >>>>      kernel device is vxlan_sys_<port>.  On a chassis 
>>>>>>>>> that isn't
>>>>>>>>>        >>>>      using port 4789 (or has no EVPN VXLAN port yet) the 
>>>>>>>>> lookup
>>>>>>>>>        >>>>      fails.  evpn_create_devices() then returns false, 
>>>>>>>>> the entry
>>>>>>>>>        >>>>      never becomes synced, and every engine run re-runs 
>>>>>>>>> the
>>>>>>>> whole
>>>>>>>>>        >>>>      create path, sets the netlink error status (which 
>>>>>>>>> keeps
>>>>>>>>>        waking
>>>>>>>>>        >>>>      the engine) and logs a warning.  The system test 
>>>>>>>>> works
>>>>>>>>>        around
>>>>>>>>>        >>>>      this by creating vxlan_sys_4789 by hand.
>>>>>>>>>        >>>>
>>>>>>>>>        >>>>      Along the same lines, the documentation says the 
>>>>>>>>> maintained
>>>>>>>>>        >>>>      device's port (default 4790) must differ from "the 
>>>>>>>>> OVS
>>>>>>>>>        datapath
>>>>>>>>>        >>>>      VXLAN device (4789)".  It really has to differ from
>>>>>>>>>        whatever is
>>>>>>>>>        >>>>      in ovn-evpn-vxlan-ports; if that includes 4790, the 
>>>>>>>>> two
>>>>>>>>>        devices
>>>>>>>>>        >>>>      compete for the same UDP port.  Can we derive both 
>>>>>>>>> the link
>>>>>>>>>        >>>>      device and the conflict check from 
>>>>>>>>> ovn-evpn-vxlan-ports?
>>>>>>>>>        >>>>
>>>>>>>>>        >>>> 5) Existing devices are adopted without checking them.
>>>>>>>>>        >>>>
>>>>>>>>>        >>>>      EEXIST is treated as success and the reconcile state
>>>>>>>>>        lives only
>>>>>>>>>        >>>>      in memory, so after "ovn-controller exit --restart":
>>>>>>>>>        >>>>
>>>>>>>>>        >>>>      - if the local IP, VXLAN port, LRP MAC or 
>>>>>>>>> redistribute
>>>>>>>>>        mode of a
>>>>>>>>>        >>>>        VNI changed while ovn-controller was down, the old
>>>>>>>>>        devices are
>>>>>>>>>        >>>>        kept with the stale configuration;
>>>>>>>>>        >>>>      - if a VNI was removed while ovn-controller was down,
>>>>>>>>>        its devices
>>>>>>>>>        >>>>        are never cleaned up;
>>>>>>>>>        >>>>      - a CMS-created device that happens to be named
>>>>>>>>>        ovnbr<vni> etc.
>>>>>>>>>        >>>>        is taken over, and later deleted.
>>>>>>>>>        >>>>
>>>>>>>>>        >>>>      The restart test passes because the VNI is still 
>>>>>>>>> present
>>>>>>>>>        after
>>>>>>>>>        >>>>      the restart, but removing it while ovn-controller is
>>>>>>>>>        down isn't
>>>>>>>>>        >>>>      covered.  I think we need to verify the kind and
>>>>>>>>>        attributes of
>>>>>>>>>        >>>>      an existing device before trusting EEXIST, and to 
>>>>>>>>> sweep
>>>>>>>>>        stale
>>>>>>>>>        >>>>      ovn*<vni> devices on startup.
>>>>>>>>>        >>> Hi Numan and Dumitru and Ales,
>>>>>>>>>        >>>
>>>>>>>>>        >>> I’ve thought some more about why I implemented it this way,
>>>>>>>>>        and I’d like
>>>>>>>>>        >>> to discuss one point.
>>>>>>>>>        >>>
>>>>>>>>>        >>> Some context for Dumitru and Ales: I added you to the 
>>>>>>>>> thread
>>>>>>>>>        because I’d
>>>>>>>>>        >>> be interested in your opinion. Currently, the code that
>>>>>>>>>        creates the EVPN
>>>>>>>>>        >>> interfaces (|ovnbr<vni>|, |ovnvxlan<vni>|, |ovnlo<vni>|, 
>>>>>>>>> and
>>>>>>>>>        >>> |ovnvrf<vni>|) does not check their actual state in the
>>>>>>>>>        kernel. It only
>>>>>>>>>        >>> reacts to changes in the desired configuration: options, 
>>>>>>>>> local
>>>>>>>>>        IP, and
>>>>>>>>>        >>> so on. If an interface is modified or deleted externally, 
>>>>>>>>> the
>>>>>>>>>        controller
>>>>>>>>>        >>> won’t notice. It won’t even recreate a deleted interface 
>>>>>>>>> until
>>>>>>>>>        something
>>>>>>>>>        >>> changes in the configuration.
>>>>>>>>>        >>>
>>>>>>>>>        >>> I implemented it this way for two reasons:
>>>>>>>>>        >>>
>>>>>>>>>        >>>   1.
>>>>>>>>>        >>>
>>>>>>>>>        >>>      I thought it made sense that, once we give OVN
>>>>>>>>>        responsibility for
>>>>>>>>>        >>>      these interfaces, the CMS has neither a need nor a 
>>>>>>>>> reason
>>>>>>>>>        to touch
>>>>>>>>>        >>>      them. So the controller shouldn’t need to expect 
>>>>>>>>> changes
>>>>>>>> made
>>>>>>>>>        >>>      outside its control. But I agree that this reasoning
>>>>>>>>>        breaks down as
>>>>>>>>>        >>>      soon as a person gets involved: someone could easily 
>>>>>>>>> modify
>>>>>>>>>        >>>      interfaces created by OVN, either by mistake or while
>>>>>>>>>        debugging. My
>>>>>>>>>        >>>      assumption here was that, if the CMS or a person 
>>>>>>>>> changes
>>>>>>>>>        something,
>>>>>>>>>        >>>      OVN can already break things even without my code,
>>>>>>>>>        because the
>>>>>>>>>        >>>      controller currently retries failed netlink 
>>>>>>>>> transactions
>>>>>>>>>        in a loop.
>>>>>>>>>        >>>
>>>>>>>>>        >>>   2.
>>>>>>>>>        >>>
>>>>>>>>>        >>>      I didn’t want to check kernel state on every 
>>>>>>>>> controller
>>>>>>>>>        iteration,
>>>>>>>>>        >>>      as that seemed excessive. However,
>>>>>>>>>        |neighbor_exchange_run()|already
>>>>>>>>>        >>>      dumps the neighbor table for every tracked interface 
>>>>>>>>> on
>>>>>>>> every
>>>>>>>>>        >>>      iteration, which is considerably more expensive than 
>>>>>>>>> a few
>>>>>>>>>        >>>      |RTM_GETLINK|requests. So this argument doesn’t seem 
>>>>>>>>> to
>>>>>>>>>        hold up.
>>>>>>>>>        >> You make a good point here but that is something we should
>>>>>>>>>        address in
>>>>>>>>>        >> general anyway, we should add proper incremental processing 
>>>>>>>>> for
>>>>>>>>>        >> monitoring kernel state (routes, neighbors, links, etc).
>>>>>>>>>        >>
>>>>>>>>>        >>> There is another case the current approach doesn’t cover: 
>>>>>>>>> an
>>>>>>>>>        >>> |ovn-controller|restart (|exit --restart|or a crash), as 
>>>>>>>>> Numan
>>>>>>>>>        pointed
>>>>>>>>>        >>> out in the review above. After startup, the internal state 
>>>>>>>>> is
>>>>>>>>>        empty,
>>>>>>>>>        >>> interface creation returns |EEXIST|, and we consider the 
>>>>>>>>> stack
>>>>>>>>>        >>> synchronized. If, for example, the VXLAN port or local IP
>>>>>>>>>        changed while
>>>>>>>>>        >>> the controller was down, the device will retain the old
>>>>>>>>>        values, and the
>>>>>>>>>        >>> controller won’t know.
>>>>>>>>>        >>>
>>>>>>>>>        >>> So the question is which approach would be better:
>>>>>>>>>        >>>
>>>>>>>>>        >>>   1.
>>>>>>>>>        >>>
>>>>>>>>>        >>>      *Check on every iteration.*Send |RTM_GETLINK|requests 
>>>>>>>>> for
>>>>>>>>>        each VNI’s
>>>>>>>>>        >>>      interfaces on every iteration. Check that each 
>>>>>>>>> interface
>>>>>>>>>        exists and
>>>>>>>>>        >>>      verify its type, master, MAC address, and UP state; 
>>>>>>>>> for
>>>>>>>>>        VXLAN, also
>>>>>>>>>        >>>      check the VNI, local IP, destination port, and link.
>>>>>>>>>        Compare these
>>>>>>>>>        >>>      against the desired configuration. Fix attributes that
>>>>>>>>>        the kernel
>>>>>>>>>        >>>      doesn’t allow changing on an existing device (port, 
>>>>>>>>> local
>>>>>>>>>        IP, VNI)
>>>>>>>>>        >>>      by recreating the interface, and update everything 
>>>>>>>>> else
>>>>>>>>>        in place.
>>>>>>>>>        >>>      The advantage is that this is simple and reliable: it
>>>>>>>>>        doesn’t depend
>>>>>>>>>        >>>      on what woke the controller up, and the same code 
>>>>>>>>> handles
>>>>>>>>>        restarts,
>>>>>>>>>        >>>      lost events, and manual changes. The downside I’m not
>>>>>>>>>        happy about is
>>>>>>>>>        >>>      the amount of code needed to parse and compare 
>>>>>>>>> attributes.
>>>>>>>>>        >>>
>>>>>>>>>        >>>   2.
>>>>>>>>>        >>>
>>>>>>>>>        >>>      *Rely on *|*if_notifier_cb*|*.*Check kernel state 
>>>>>>>>> only when
>>>>>>>>>        >>>      |host-if-monitor|reports a change to our interfaces. 
>>>>>>>>> To
>>>>>>>>>        make this
>>>>>>>>>        >>>      reliable, we would need to:
>>>>>>>>>        >>>
>>>>>>>>>        >>>        *
>>>>>>>>>        >>>
>>>>>>>>>        >>>          Keep a module-level dirty flag and clear it only 
>>>>>>>>> after
>>>>>>>>>        >>>          successful reconciliation here in evpn devices 
>>>>>>>>> code.
>>>>>>>>>        >>>
>>>>>>>>>        >>>        *
>>>>>>>>>        >>>
>>>>>>>>>        >>>          Handle |ENOBUFS|on the netlink socket, indicating
>>>>>>>>>        lost events,
>>>>>>>>>        >>>          by marking everything dirty. Currently, the 
>>>>>>>>> callback
>>>>>>>>>        simply
>>>>>>>>>        >>>          returns in this case.
>>>>>>>>>        >>>
>>>>>>>>>        >>>        *
>>>>>>>>>        >>>
>>>>>>>>>        >>>          Remove the |IFF_UP|filter in |if_notifier_cb|so 
>>>>>>>>> that
>>>>>>>>>        we also
>>>>>>>>>        >>>          catch interfaces going down, and add |ovnvrf*|to 
>>>>>>>>> the
>>>>>>>>>        tracked
>>>>>>>>>        >>>          interfaces.
>>>>>>>>>        >>>
>>>>>>>>>        >>>        *
>>>>>>>>>        >>>
>>>>>>>>>        >>>          Always reconcile with the kernel when 
>>>>>>>>> ovn-controller
>>>>>>>>>        starts,
>>>>>>>>>        >>>          since devices left over from a previous run don’t
>>>>>>>>>        generate any
>>>>>>>>>        >>>          events.
>>>>>>>>>        >>>          The advantage is that we query the kernel only 
>>>>>>>>> when
>>>>>>>>>        something
>>>>>>>>>        >>>          has actually changed. The downside is more state 
>>>>>>>>> and
>>>>>>>>>        more edge
>>>>>>>>>        >>>          cases that are easy to miss. It doesn’t really 
>>>>>>>>> seem
>>>>>>>>>        worth it.
>>>>>>>>>        >>>
>>>>>>>>>        >>> We need the attribute parsing and comparison code in both
>>>>>>>>>        approaches, at
>>>>>>>>>        >>> least to handle restarts. The only difference is /when/we 
>>>>>>>>> call
>>>>>>>>>        it. So
>>>>>>>>>        >>> I’m leaning toward the first approach: it’s simpler and 
>>>>>>>>> more
>>>>>>>>>        reliable.
>>>>>>>>>        >>> But maybe you have different opinions.
>>>>>>>>>        >>>
>>>>>>>>>        >> This is, IIRC, the main reason why in internal discussions I
>>>>>>>>>        had with
>>>>>>>>>        >> Ales when trying to figure out how to integrate OVN with 
>>>>>>>>> EVPN
>>>>>>>>>        >> deployments we decided against maintaining the required 
>>>>>>>>> host side
>>>>>>>>>        >> devices ourselves - it's quite a lot of (arguably
>>>>>>>>>        uninteresting) work.
>>>>>>>>>        >> And probably with a lot of added code and the bugs that come
>>>>>>>>>        with it.
>>>>>>>>>        >>
>>>>>>>>>        >>> What do you think?
>>>>>>>>>        >>>
>>>>>>>>>        >> I'm wondering now if it isn't better to just provide a 
>>>>>>>>> helper
>>>>>>>>>        utility as
>>>>>>>>>        >> part of OVN utilities that the CMS can run to ensure that 
>>>>>>>>> the
>>>>>>>>>        host side
>>>>>>>>>        >> interfaces are in the correct state (matching what's in the 
>>>>>>>>> NB
>>>>>>>>>        and SB
>>>>>>>>>        >> database).  It doesn't have to but it could be a simple 
>>>>>>>>> script
>>>>>>>>>        I guess.
>>>>>>>>>        >> And it could reconcile the host side interfaces 
>>>>>>>>> automatically
>>>>>>>>>        if needed.
>>>>>>>>>        >>
>>>>>>>>>        >> It would still be up to the CMS to figure out the best way 
>>>>>>>>> to
>>>>>>>>>        run the
>>>>>>>>>        >> utility, e.g., triggered when specific interface changes are
>>>>>>>>>        detected.
>>>>>>>>>        >>
>>>>>>>>>        >> Would it make sense to reconsider this as a different 
>>>>>>>>> approach?
>>>>>>>>>        >
>>>>>>>>>        > Hi, thanks for the reply!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi Dumitru, Alexandra and Numan,
>>>>>>>> Hi Ales,
>>>>>>>>
>>>>>>>> Thanks for sharing your view! Please see my replies inline.
>>>>>>>>>        >
>>>>>>>>>        > To be honest, I’m not entirely sure the script is the right 
>>>>>>>>> way
>>>>>>>>>        to go.
>>>>>>>>>        > Let me describe how I picture it, so we can check it matches
>>>>>>>>>        what you
>>>>>>>>>        > have in mind:
>>>>>>>>>        >
>>>>>>>>>        >  1. ovn-controller has a handler that validates all the 
>>>>>>>>> interface
>>>>>>>>>        >     attributes against the current configuration and 
>>>>>>>>> recreates the
>>>>>>>>>        >     interfaces if needed.
>>>>>>>>>
>>>>>>>>>        The way I was imagining it (maybe over-simplified) was that
>>>>>>>>>        ovn-controller doesn't do any of this work.  It would be the
>>>>>>>>>        responsibility of the CMS to call the script whenever it 
>>>>>>>>> (re)starts
>>>>>>>>>        ovn-controller (e.g., as part of the systemd scripts).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I also don't think that this should be OVN's responsibility
>>>>>>>>> at all. Sure we have "maintain-vrf", but it shouldn't be there
>>>>>>>>> IMO. The benefit vs. maintenance cost and code size doesn't
>>>>>>>>> seem to favor those changes. And the discussion here
>>>>>>>>> solidifies that even more for me.
>>>>>>>> I understand you don't like it, but I'd like to try once more to argue
>>>>>>>> why this isn't such a really bad feature:
>>>>>>>>
>>>>>>>> 1. The devices are part of OVN, not of the routing daemon or something
>>>>>>>> else. OVN writes
>>>>>>>>        into them: FDB and neighbor entries on the bridge and VXLAN 
>>>>>>>> devices,
>>>>>>>> route-exchange installs routes into
>>>>>>>>        the VRF. Their configuration is derived purely from OVN state 
>>>>>>>> (VNI,
>>>>>>>>        per-VNI local IP, router port MAC, the VXLAN port derived from 
>>>>>>>> the
>>>>>>>>        VNI). Nothing depends on something external like routing 
>>>>>>>> deaemon or
>>>>>>>> etc,
>>>>>>>>        so switching to another routing daemon wouldn't change anything 
>>>>>>>> here.
>>>>>>>>
>>>>>>> Well not really, the expectation is for CMS to provide them. If they 
>>>>>>> don't
>>>>>>> it's their fault. So I don't see them as part of OVN.
>>>>>>>
>>>>>> Sure, but Alexandra is trying to change this restriction in this patch 
>>>>>> set.
>>>>>>
>>>>>>>> 2. ovn-controller knows better then CMS where and when the devices are
>>>>>>>> needed.
>>>>>>>>        They are needed on chassis where the switch is local, and that
>>>>>>>>        changes at runtime, e.g. when a VM migrates to another 
>>>>>>>> hypervisor so
>>>>>>>>        ovn-controller sees that quite fast but a CMS has to track port
>>>>>>>>        bindings itself and race to configure the new host in time. With
>>>>>>>>        this feature, for L2 EVPN the CMS doesn't have to do anything 
>>>>>>>> once
>>>>>>>>        FRR is running; for L3 EVPN it only has to configure the VRF in 
>>>>>>>> FRR.
>>>>>>>>
>>>>>>> OVN does, but the same logic can be applied to FRR. Should OVN
>>>>>>> start FRR after failover? If CMS already has the FRR ready on all
>>>>>>> chassis where it is important why couldn't it have all the needed
>>>>>>> interfaces? That's not any different, setting them up once on those
>>>>>>> chassis which will make potential failover even quicker than
>>>>>>> OVN catching up and adding the interfaces only after they are needed.
>>>>>>>
>>>>>> This is a good point I didn't think of before.  The routing control
>>>>>> plane needs the interfaces in order to function properly.  And the
>>>>>> routing control plane is under control of the CMS.
>>>>> My example probably wasn't the best one. Let me clarify two points:
>>>>>
>>>>>    1.
>>>>>
>>>>>       Migration may not be the best example, but what about simply
>>>>>       attaching a port to a subnet? Would the CMS have to create
>>>>>       interfaces for every VNI on every hypervisor? I'm not sure I
>>>>>       understand what you meant.
>>>>>
>>>> Yes, that's the current expectation that the CMS configures interfaces
>>>> for every required VNI on the required hypervisors.  The routing control
>>>> plane (e.g., FRR) must be running on the required hypervisors too and
>>>> must anyway be configured to handle the VNI.  That's also outside the
>>>> scope of OVN and most likely configured by the CMS.
>>>>
>>>>>    2.
>>>>>
>>>>>       Regarding failover, I disagree with your argument that the CMS would
>>>>>       be faster. The CMS and the controller would receive the updates at
>>>>>       essentially the same time, when the old controller releases the
>>>>>       chassisredirect port and the new one claims it. By that point, the
>>>>>       new controller would already have recomputed the OpenFlow rules and
>>>>>       routing state locally and would be ready to receive traffic. Isn't
>>>>>       that the case?
>>>>>
>>>> I think Ales was thinking of the potential fast-failover use case - in
>>>> that case FRR would have to already be running on both chassis a DGP
>>>> might be scheduled on.  So the absence of the Linux interfaces on the
>>>> not-yet-active chassis would make it that FRR can't immediately change
>>>> the routing information it may be advertising.
>>>>
>>>>>>>> I hope I’ve made it clear what I mean.
>>>>>>>>
>>>>>>>> About the maintenance cost, I'd suggest we define once, and document 
>>>>>>>> (in
>>>>>>>> ovn user documentation),
>>>>>>>> what OVN is responsible for and then rely on that:
>>>>>>>>
>>>>>>>> - OVN creates and deletes the devices and makes sure they are up.
>>>>>>>> - OVN owns them. If someone modifies or deletes them after OVN created
>>>>>>>>       them, that's a misconfiguration.
>>>>>>>> - On startup, OVN dumps the devices in its group, deletes the ones
>>>>>>>>       that are no longer in the configuration, and checks the 
>>>>>>>> attributes
>>>>>>>>       of the remaining ones once against the desired config, recreating
>>>>>>>>       them on mismatch. That covers configuration changes made while
>>>>>>>>       ovn-controller was down. No extra state needs to be stored: the
>>>>>>>>       kernel already holds what is currently configured.
>>>>>>>> - No periodic validation of attributes at runtime.
>>>>>>>>
>>>>>>>> The same startup cleanup would also fix stale VRFs from
>>>>>>>> dynamic-routing-maintain-vrf.
>>>>>>>>
>>>>>>>> Does this sound like an acceptable scope?
>>>>>>>>
>>>>>>> That's still quite a bit of code to maintain.
>>>>>>>
>>>>>>>
>>>>>>>> So in my patch I'd propose adding a pass on startup that dumps all
>>>>>>>> the interfaces (Further down in the email, there’s a comment from me
>>>>>>>> about this.) and deletes the stale ones, but doesn't validate their
>>>>>>>> attributes.
>>>>>>>>
>>>>>>>> But this proposal still leaves one case open: if someone changes the
>>>>>>>> configuration (not specifically, not changing the VNI, but rather
>>>>>>>> parameters like MAC/IP)
>>>>>>>>
>>>>>>>>      while ovn-controller is stopped. Maybe I could store
>>>>>>>> what is currently configured in the ovsdb? Then on restart we
>>>>>>>> validate it, and if the user changed anything, even the MAC address,
>>>>>>>> OVN recreates the interfaces, the same way it already does when the
>>>>>>>> change happens while the controller is running.
>>>>>>>> What do you think?
>>>>>>>>
>>>>>>> That broadens the scope again, now we would have to maintain
>>>>>>> what was there previously.
>>>>>>>
>>>>>>>
>>>>>> I'm on the fence when it comes to this discussion in general.  This
>>>>>> argument is making me sway again into the direction of "let the CMS do
>>>>>> it".  It feels like we will be adding a lot of code just to make the
>>>>>> life of the CMS easier while the CMS can probably already do that in a
>>>>>> simpler way (as it has more domain-specific knowledge).
>>>>> I'm still struggling to understand why the CMS is considered better
>>>>> placed to manage these interfaces...The CMS expresses the desired
>>>>> logical configuration through NB. Why should it also need to track which
>>>>> hypervisor currently hosts a particular lsp and manage the corresponding
>>>>> local interfaces? To me, that seems to duplicate placement information
>>>>> that OVN already maintains.
>>>>>
>>>> Well, we have a very simple counter example: LSPs and the OVS
>>>> interfaces.  The CMS needs to create the LSP (in the NB) but also need
>>>> to create the OVS interface (with correct external_ids) on the
>>>> hypervisor where the workload actually runs.
>>>>
>>>>>>>>>        >  2. We call it on ovn-controller startup, which solves the
>>>>>>>>>        problem Numan
>>>>>>>>>        >     described (more on that below).
>>>>>>>>>
>>>>>>>>>        See above.
>>>>>>>>>
>>>>>>>>>        >  3. If the CMS isn’t sure the interfaces are valid, it can 
>>>>>>>>> call
>>>>>>>>>        the same
>>>>>>>>>        >     entry point.
>>>>>>>>>        >
>>>>>>>>>        > The upside is that ovn-controller doesn’t have to check every
>>>>>>>>>        attribute
>>>>>>>>>        > on every iteration. But I have some doubts about it, because 
>>>>>>>>> it
>>>>>>>>>        means
>>>>>>>>>        > one more knob and one more entry point. The whole idea of
>>>>>>>>>        maintain-evpn
>>>>>>>>>        > is that OVN owns these interfaces and the CMS doesn’t have 
>>>>>>>>> to think
>>>>>>>>>        > about them. A “check it if you’re not sure” knob hands that
>>>>>>>>>        > responsibility back to the CMS. That said, if you think this
>>>>>>>>>        approach
>>>>>>>>>        > sounds good, I’m happy to add it. I will cast aside all my 
>>>>>>>>> doubts
>>>>>>>>>        >
>>>>>>>>>
>>>>>>>>>        I'm not sure it's the best thing, I was just trying to 
>>>>>>>>> reconsider
>>>>>>>>>        it as
>>>>>>>>>        an alternative. :)
>>>>>>>>>
>>>>>>>>>        > About the problem Numan described: it also applies to the 
>>>>>>>>> VRFs in
>>>>>>>>>        > dynamic routing. If ovn-controller is stopped without 
>>>>>>>>> cleanup and
>>>>>>>>>        > someone changes the configuration in the meantime (removes 
>>>>>>>>> the
>>>>>>>>>        VNI or
>>>>>>>>>        > the maintain-vrf option), then after the restart 
>>>>>>>>> ovn-controller
>>>>>>>>>        will
>>>>>>>>>        > never clean up the leftover interfaces.
>>>>>>>>>        >
>>>>>>>>>        > I think this can be solved separately from this patch, for 
>>>>>>>>> EVPN
>>>>>>>>>        and VRF
>>>>>>>>>        > at the same time. My idea is:
>>>>>>>>>        >
>>>>>>>>>        >   * Put all interfaces created by OVN into the same device 
>>>>>>>>> group
>>>>>>>>>        >     (IFLA_GROUP). It can be set right when the interface is
>>>>>>>> created.
>>>>>>>>>        >   * On ovn-controller startup, dump the interfaces and delete
>>>>>>>>>        the ones
>>>>>>>>>        >     in our group that are no longer in the current 
>>>>>>>>> configuration.
>>>>>>>>>        >   * Do this only once, on startup.
>>>>>>>>>        >   * Interfaces created by the CMS (e.g. a VRF without 
>>>>>>>>> maintain-vrf)
>>>>>>>>>        >     aren’t in our group, so we won’t touch them.
>>>>>>>>>        >
>>>>>>>>>        > The group number should probably be configurable, in case 
>>>>>>>>> groups
>>>>>>>>>        are
>>>>>>>>>        > already used for something else on the host.
>>>>>>>>>        >
>>>>>>>>>
>>>>>>>>>        This is a nice idea!  I'd argue however that the group number
>>>>>>>>>        should be
>>>>>>>>>        mandatory if we go this way.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I mean is this different from a name check? Unfortunately, the kernel
>>>>>>>>> doesn't allow us to filter by device group so we need to dump
>>>>>>>>> everything anyway.
>>>>>>>> This would only happen once, on startup, so does it really matter? I
>>>>>>>> don't think filtering interfaces by group on startup is a big
>>>>>>>> problem to do it in ovn-controller on startup once. And how is it
>>>>>>>> different from dumping all interfaces and
>>>>>>>> filtering them by name, i.e. ovnvrf*, ovnvx*, ovnbr*, ovnlo*? That
>>>>>>>> would work too, but it looks kind of ugly, doesn't it? I'd still dump
>>>>>>>> all the interfaces, filter them by name and then parse the VNI out of
>>>>>>>>
>>>>>>>> the name. I don't really like that...
>>>>>>>>
>>>>>>> I mean what is the difference between filtering by name and group?
>>>>>>> Besides the fact that filtering by name doesn't require extra setup
>>>>>>> while the group does as we don't currently do that.
>>>>>>>
>>>>>> I missed the lack of kernel side filtering yesterday.  That does change
>>>>>> the context, I agree, today we can already by name already in
>>>>>> ovn-controller, no need for the group then.
>>>>> But then, when dumping interfaces, I would have to compare each
>>>>> interface name against at least four strings (ovnbr, ovnvrf, ovnvx,
>>>>> ovnlo). I understand your point about introducing a group ID when we
>>>>> could rely on existing information, but I still think the code would be
>>>>> cleaner with a group ID.
>>>>>
>>>>>>>> Anyway, I’ll look forward to hearing your thoughts on how I should go
>>>>>>>> about this.
>>>>>>>>
>>>>>>>>>        > With this approach, on ovn-controller startup we can call the
>>>>>>>>>        routine
>>>>>>>>>        > that syncs the attributes and cleans up the stale interfaces.
>>>>>>>>>        >
>>>>>>>>>        > What do you think?
>>>>>>>>>        >
>>>>>>>>>
>>>>>>>>>        I guess as long as we document it well that CMS touching 
>>>>>>>>> anything
>>>>>>>>>        that's
>>>>>>>>>        in that group results in a misconfiguration it's probably fine.
>>>>>>>>>
>>>>>>>>>        I'm curious what others think about this (Ales, Numan, ..) 
>>>>>>>>> though.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> As I said above I'm not sure we should maintain this in OVN.
>>>>>>>>> Especially when the contract is a on the weak side, reconcile
>>>>>>>>> only on restart and if anything happens in between you might
>>>>>>>>> have a problem.
>>>>>>>>>
>>>>>>>>> However, if Dumitru and Numan are fine with this feature
>>>>>>>>> I won't block it.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>        > Thanks, Sasha
>>>>>>>>>        >
>>>>>>>>>
>>>>>>>>>        Thanks for the discussion!
>>>>>>>>>
>>>>>>>>>        Dumitru
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Ales
>>>>>>>>>
>>> Hi Sasha, Ales, Dumitru,
>>>
>>> My 2 cents.
>>>
>>> Thanks for the discussion.  I think the answer is different for L2
>>> and L3 VNIs, and that this may let us agree on a smaller scope.
>>>
>>> The deciding question for me is whether enabling or disabling EVPN on
>>> a logical switch needs any per-VNI routing daemon configuration.
>>>
>>> Dumitru, you wrote that FRR "must anyway be configured to handle the
>>> VNI".  For L3 VNIs that's true: each one needs "vrf X / vni N", a
>>> "router bgp ... vrf X" instance and an export policy.  For L2 VNIs,
>>> as far as I understand FRR, it isn't: with a one-time configuration
>>> (the BGP instance, the EVPN sessions and "advertise-all-vni" under
>>> "address-family l2vpn evpn"), zebra discovers a new L2 VNI from the
>>> VXLAN device attached to a bridge, and BGP starts advertising its
>>> Type-2 and Type-3 routes without any change to FRR.  The one
>>> condition is that the route targets line up across VTEPs: the
>>> default "<ASN>:<VNI>" only works with a shared ASN, so deployments
>>> with per-node ASNs need a fabric-wide route-target scheme in that
>>> one-time configuration.  (Please correct me here if my understanding is 
>>> wrong
>>> here).
>>>
>>> So:
>>>
>>> - For L2 VNIs, once FRR is set up, the kernel devices are the only
>>>     per-VNI, per-chassis work left for the CMS.  That's where I think
>>>     it makes sense for ovn-controller to manage them.
>>>
>>> - For L3 VNIs, the CMS has to configure FRR per VRF anyway, and
>>>     Dumitru's fast-failover point applies: the devices have to exist
>>>     on every chassis a distributed gateway port may move to, so that
>>>     FRR is ready before the port moves.  There the CMS might as well
>>>     create the devices too.  For what it's worth, that's what we do:
>>>     our agent creates the L3 VNI devices together with the FRR
>>>     configuration for each VRF.
>>>
>>>
>>> IMO,  it makes sense for ovn-controller to manage the vni interfaces
>>> for L2 VNIs.  However if it complicates the code in ovn-controller,
>>> I'm fine passing on the responsibility to CMS.
>>>
>>> I have another suggestion -  is to take the approach along the lines
>>> of the VIF plug
>>> providers: ovn-controller computes the desired per-VNI devices for the
>>> logical switches, and a provider (or an external agent reading that
>>> state from the local OVS database) creates them.  OVN would keep the
>>> placement logic and the device code would live outside the tree.
>>>
>>> Is this approach we can explore ?
>>>
>>> Thanks
>>> Numan
>> Hi everyone!
> Hi Sasha,
>
>> Numan, thanks for the suggestion to use a separate provider, similar to
>> VIF plug providers. Before we go down that way, I’d like to give the
>> built-in approach another look =). I think we can keep the
>> implementation fairly small if we agree on what OVN should handle. I
>> also want to explain why I think L3 fits well into the existing
>> mechanism one more time.
>>
>> 1. I agree with your explanation of L2, and I’d like to keep L3 support
>> too. In the L2 case you described, once the routing daemon and route
>> targets are set up, adding a VNI doesn’t require any changes to the
>> daemon’s configuration. We just need to create the Linux interfaces on
>> the right chassis. As you said, having OVN automate this would be
>> useful. The same applies to L3, and most of the device management code
>> would be shared with L2. I checked the L3 fast failover case you
>> described: OVN already creates the necessary infrastructure on both
>> gateway chassis, primary and standby. So the CMS just needs to configure
>> and start the routing daemon on both nodes ahead of time, with nothing
>> else to do during failover. I’ll also add a test to cover the switchover
>> and check that routing works correctly afterwards.
> I think a test for that would be great nevertheless (even today without
> support for auto-creating EVPN interfaces).
>
>> 2. I think we should clearly document what the controller does and
>> doesn’t handle. OVN creates, configures, and brings up its devices. When
>> the OVN configuration changes, it updates or recreates them and removes
>> anything that’s no longer needed. At startup, it picks up configuration
>> changes made while it was stopped and cleans up devices left over from
>> the previous run that are no longer needed. But it won’t keep checking
>> every device attribute at runtime, and we won’t guarantee that it
>> automatically fixes changes made outside OVN. The CMS and administrators
>> shouldn’t modify or delete these devices themselves; we’d treat that as
>> a misconfiguration. I think these are reasonable limits, and they should
>> help keep the code small and straightforward.
>> 3. I’ll move all the device management code into a separate file.
>>
>> What do you think? Would this approach work for you?
>>
> After reading the details below, and thinking a bit more about the vif
> provider alternative I have some more thoughts, please see below. :)
>
>> Below are a few more details on what I’d like to finish in this patch,
>> including how I’d implement the behavior described in point 2. Feel free
>> to skip this part :)
>> There are three things I’d like to address:
>> 1. The VRF conflict
I’d like to rework this part of the patch. Right
>> now, for an L3VNI, we specify the LRP that connects the EVPN interfaces
>> to the OVN router, but we create ovnvrf<VNI> independently of the
>> router’s VRF. We then expect the CMS to set the router’s VRF to
>> ovnvrf<VNI>. That’s where the issues Numan pointed out in the review
>> come from.
My suggestion is to make the explicit VRF setting optional in
>> this mode. If no VRF is set, the controller will create one
>> automatically, using the tunnel_key of the router associated with the
>> LRP as the table ID, just as dynamic routing does today.
The evpn-device
>> code will handle creating and deleting this VRF, and route-exchange will
>> use the same VRF for route advertisements. This way, we’ll no longer tie
>> the table ID to the VNI, and the router and its EVPN interfaces will
>> share one VRF.
>> 2. Handling devices at startup
I’d still like to consider saving the
>> last successfully applied device configuration in the local OVSDB. We’d
>> store the actual computed values: MAC, local VTEP IP, VNI, UDP port,
> So, essentially we'd store in the local OVSDB the "desired state" for
> the EVPN required linux interfaces.  That's good.
>
>> LRP, and so on.
At startup, the controller would compute these values
>> again from the current OVN configuration, compare them with the saved
>> values, and apply any changes. For example, if the LRP name stays the
>> same but its MAC changes, we’ll catch that and recreate the interface.
>> This would let us handle OVN configuration changes within the limits
>> described above.
> But now that we have the "desired state" in OVSDB, it really feels like
> a pity to half implement a controller that syncs this as Linux host
> interfaces..
>
> Briefly looking at the dummy vif provider and at the official
> representor provider [0] there are some things to take into account:
>
> The vif-plug implementation is really designed to "plug" VIFs
> corresponding to LSPs.  Also, in this case there's no LSP representing
> the EVPN interfaces.
>
> I guess we could change the vif-plug implementation to handle EVPN
> interface plugging - not sure how nice/ugly that'd be.  But if we do
> have that covered somehow then I think it would actually be relatively
> easy to have an evpn vif provider that implements a proper
> reconciliation loop between the OVSDB "desired state" and host
> interfaces "actual state".
>
> The alternative to all this would still be a separate controller reading
> the OVSDB "desired state".  Just to keep that logic outside of OVN.
>
> [0]
> https://github.com/ovn-org/ovn-vif/blob/main/lib/vif-plug-providers/representor/vif-plug-representor.c
>
>> 3. Cleaning up old interfaces
I’ll use your suggestion to dump all
>> interfaces and identify the old ones by name, since the group_id
>> approach didn’t work out for you.
>> It’d be nice to hear your thoughts on this.
>>
>> Thanks, Sasha
>>
> Regards,
> Dumitru
Hi Dumitru,
Thanks for the reply! I’ll add a failover test.
I like the provider idea when different deployments need to do something 
differently. With OVN EVPN, the controller already determines which 
bridges, VXLAN interfaces, and VRFs are needed on each chassis. I’m 
still trying to see what a second provider would do differently with 
those devices.
I can see a clearer use for providers when configuring the routing 
daemon. For example, OVN could create the VRF, and an FRR provider could 
configure FRR to use it. A provider for another daemon, such as BIRD, 
could handle that daemon’s configuration while reusing the same Linux 
devices. I think it is such a good idea and we can do it in future.
I’m also not sure moving device management into a provider would save 
much code in OVN. The controller would still decide which devices are 
needed on each chassis and when to remove them. It would also need to 
pass that information to the provider and find out whether the devices 
are ready. After looking at the VIF plug code, I see two ways to do this.
1. A separate agent using OVSDB. ovn-controller writes the desired 
device configuration to the local OVSDB. The agent watches for changes 
and creates, updates, or removes the Linux devices. The agent can live 
outside OVN, while the controller keeps the records up to date and needs 
a way to tell whether the agent has applied the current configuration.
2. Extend VIF plug or add a dedicated provider interface in OVN. As you 
pointed out, the EVPN devices have no corresponding LSP or another 
entity. They’re derived from settings on the logical switch or router. 
We’d need to define when the controller calls the provider, what 
configuration it passes in, and how the provider reports that the 
devices are ready.
Both approaches move the Linux device operations out of the controller, 
but they also add integration code on the OVN side.
What do you think?

Thanks,


Sasha


>
>>>>>> Thanks,
>>>>>> Dumitru
>>>>>>
>>>>>> Внимание: ВНЕШНИЙ отправитель!
>>>>>>
>>>>>>
>>>>>> Будьте осторожны с вложениями и ссылками.
>>>> Regards,
>>>> Dumitru
>>>>

_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

Reply via email to