On 11/16/26 12:10 PM, Rukomoinikova Aleksandra wrote:
> On 12.11.2026 20:31, Numan Siddique wrote:
>> On Thu, Nov 12, 2026 at 12:12 PM Dumitru Ceara <[email protected]> wrote:
>>> On 11/12/26 4:51 PM, Rukomoinikova Aleksandra wrote:
>>>> On 12.11.2026 16:04, Dumitru Ceara wrote:
>>>>> Внимание: ВНЕШНИЙ отправитель!
>>>>>
>>>>>
>>>>> Будьте осторожны с вложениями и ссылками.
>>>>>
>>>>>
>>>>> On 11/12/26 1:31 PM, Ales Musil wrote:
>>>>>> On Thu, Nov 12, 2026 at 12:56 PM Rukomoinikova Aleksandra
>>>>>> <[email protected]> wrote:
>>>>>>
>>>>>>> On 12.11.2026 10:20, Ales Musil wrote:
>>>>>>>> On Wed, Nov 11, 2026 at 2:34 PM Dumitru Ceara <[email protected]> 
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>       On 11/11/26 2:07 PM, Rukomoinikova Aleksandra wrote:
>>>>>>>>       > On 11.11.2026 13:58, Dumitru Ceara wrote:
>>>>>>>>       >> On 11/11/26 11:32 AM, Rukomoinikova Aleksandra wrote:
>>>>>>>>       >>> On 11.11.2026 01:48, Numan Siddique wrote:
>>>>>>>>       >>>> On Fri, Aug 28, 2026 at 6:18 AM Alexandra Rukomoinikova via 
>>>>>>>> dev
>>>>>>>>       >>>> <[email protected]> wrote:
>>>>>>>>       >>>>> Until now the CMS had to create the EVPN interface stack
>>>>>>>>       (bridge,
>>>>>>>>       >>>>> VXLAN and advertise devices) on every chassis itself and
>>>>>>>>       pass the
>>>>>>>>       >>>>> interface names via the "dynamic-routing-bridge-ifname",
>>>>>>>>       >>>>> "dynamic-routing-vxlan-ifname" and
>>>>>>>>       "dynamic-routing-advertise-ifname"
>>>>>>>>       >>>>> options of the Logical_Switch.
>>>>>>>>       >>>>>
>>>>>>>>       >>>>> Add a new "dynamic-routing-maintain-evpn" option, which 
>>>>>>>> makes
>>>>>>>>       >>>>> ovn-controller create and maintain those interfaces on its
>>>>>>>>       own on
>>>>>>>>       >>>>> every chassis where the logical switch is local.  The only
>>>>>>>>       supported
>>>>>>>>       >>>>> value is "mvd", i.e. one VXLAN device per VNI.  The devices
>>>>>>>>       are named
>>>>>>>>       >>>>> after the VNI: ovnbr<vni>, ovnvxlan<vni> and ovnlo<vni>.
>>>>>>>>       >>>>>
>>>>>>>>       >>>>> If "dynamic-routing-redistribute" includes "ip", the 
>>>>>>>> logical
>>>>>>>>       router
>>>>>>>>       >>>>> port has to be specified with
>>>>>>>>       "dynamic-routing-maintain-evpn-lrp".
>>>>>>>>       >>>>> In that case ovn-controller also creates the ovnvrf<vni>
>>>>>>>>       VRF, enslaves
>>>>>>>>       >>>>> the bridge to it and assigns the router port MAC to the
>>>>>>>>       bridge, so
>>>>>>>>       >>>>> that the logical router addresses can be advertised over 
>>>>>>>> EVPN.
>>>>>>>>       >>>>> The interfaces are removed on graceful shutdown of
>>>>>>>>       ovn-controller.
>>>>>>>>       >>>>>
>>>>>>>>       >>>>> The UDP destination port of the VXLAN device is 
>>>>>>>> configurable
>>>>>>>>       with
>>>>>>>>       >>>>> the "dynamic-routing-maintain-evpn-vxlan-port" option and
>>>>>>>>       defaults
>>>>>>>>       >>>>> to 4790.  It has to differ from the port used by the OVS
>>>>>>>>       datapath
>>>>>>>>       >>>>> VXLAN device, because the maintained device only carries
>>>>>>>>       EVPN state
>>>>>>>>       >>>>> for the routing daemon.
>>>>>>>>       >>>>>
>>>>>>>>       >>>>> Signed-off-by: Alexandra Rukomoinikova 
>>>>>>>> <[email protected]
>>>>>>>>
>>>>>>>>       >>>> Hi Alexandra,
>>>>>>>>       >>>>
>>>>>>>>       >> Hi Alexandra, Numan,
>>>>>>>>       >>
>>>>>>>>       >>>> Thanks for working on this; having ovn-controller own the 
>>>>>>>> EVPN
>>>>>>>>       >>>> device stack is a nice improvement.  I built the series (it
>>>>>>>>       >>>> compiles cleanly) and went through it, and found a few 
>>>>>>>> issues
>>>>>>>>       that
>>>>>>>>       >>>> I think need addressing before it can go in.
>>>>>>>>       >>>>
>>>>>>>>       >> Indeed, thanks for working on this!  I can see how this would
>>>>>>>>       make the
>>>>>>>>       >> CMS life easier.  However, I have some more concerns, please
>>>>>>>>       see below.
>>>>>>>>       >>
>>>>>>>>       >>>> 1) Interface names can exceed IFNAMSIZ.
>>>>>>>>       >>>>
>>>>>>>>       >>>>      The devices are named ovnvxlan<vni>, ovnbr<vni>,
>>>>>>>>       ovnlo<vni> and
>>>>>>>>       >>>>      ovnvrf<vni>.  A VNI can have 8 digits, so for VNI >=
>>>>>>>>       10000000
>>>>>>>>       >>>>      "ovnvxlan<vni>" is 16 characters, but Linux allows at
>>>>>>>>       most 15.
>>>>>>>>       >>>>      The buffers are IFNAMSIZ + 1, so snprintf() doesn't
>>>>>>>>       truncate,
>>>>>>>>       >>>>      and the kernel then rejects the name on every run.
>>>>>>>>       Could we use
>>>>>>>>       >>>>      a shorter prefix (e.g. "ovnvx"), or at least reject 
>>>>>>>> such
>>>>>>>>       VNIs
>>>>>>>>       >>>>      with a clear warning?
>>>>>>>>       >>>>
>>>>>>>>       >>>> 2) VRF names and table ids can collide with route-exchange.
>>>>>>>>       >>>>
>>>>>>>>       >>>>      route-exchange already creates VRFs named
>>>>>>> "ovnvrf<table_id>"
>>>>>>>>       >>>>      (controller/route.c).  This patch creates "ovnvrf<vni>"
>>>>>>>>       with the
>>>>>>>>       >>>>      VNI as the table id.  If a router's dynamic-routing 
>>>>>>>> VRF id
>>>>>>>>       >>>>      happens to equal a switch's VNI, both features end up
>>>>>>>>       managing
>>>>>>>>       >>>>      the same device.  route-exchange creates it with
>>>>>>>>       NLM_F_EXCL and
>>>>>>>>       >>>>      treats EEXIST as an error, the EVPN code treats EEXIST 
>>>>>>>> as
>>>>>>>>       >>>>      success, and whichever side tears down first deletes 
>>>>>>>> the
>>>>>>>>       >>>>      other's VRF.  Using the VNI as the table id also 
>>>>>>>> clashes
>>>>>>>>       with any
>>>>>>>>       >>>>      other user of that table number.
>>>>>>>>       >>>>
>>>>>>>>       >>>> 3) VNI 253/254 maps onto the default/main routing table.
>>>>>>>>       >>>>
>>>>>>>>       >>>>      Related to the above: TABLE_ID_VALID() doesn't exclude
>>>>>>>>       >>>>      RT_TABLE_DEFAULT or RT_TABLE_MAIN, so a VNI of 254
>>>>>>>>       creates a VRF
>>>>>>>>       >>>>      bound to the host's main table.  I think we need a 
>>>>>>>> separate
>>>>>>>>       >>>>      table id (configurable, or derived in a way that can't 
>>>>>>>> hit
>>>>>>>>       >>>>      reserved or already-used tables) rather than using the 
>>>>>>>> VNI
>>>>>>>>       >>>>      directly.
>>>>>>>>       >>>>
>>>>>>>>       >>>> 4) The link device is hard-coded to vxlan_sys_4789.
>>>>>>>>       >>>>
>>>>>>>>       >>>>      ovn-controller creates the OVS EVPN VXLAN ports from 
>>>>>>>> the
>>>>>>>>       >>>>      chassis's ovn-evpn-vxlan-ports, which can be any port,
>>>>>>>>       so the
>>>>>>>>       >>>>      kernel device is vxlan_sys_<port>.  On a chassis that 
>>>>>>>> isn't
>>>>>>>>       >>>>      using port 4789 (or has no EVPN VXLAN port yet) the 
>>>>>>>> lookup
>>>>>>>>       >>>>      fails.  evpn_create_devices() then returns false, the 
>>>>>>>> entry
>>>>>>>>       >>>>      never becomes synced, and every engine run re-runs the
>>>>>>> whole
>>>>>>>>       >>>>      create path, sets the netlink error status (which keeps
>>>>>>>>       waking
>>>>>>>>       >>>>      the engine) and logs a warning.  The system test works
>>>>>>>>       around
>>>>>>>>       >>>>      this by creating vxlan_sys_4789 by hand.
>>>>>>>>       >>>>
>>>>>>>>       >>>>      Along the same lines, the documentation says the 
>>>>>>>> maintained
>>>>>>>>       >>>>      device's port (default 4790) must differ from "the OVS
>>>>>>>>       datapath
>>>>>>>>       >>>>      VXLAN device (4789)".  It really has to differ from
>>>>>>>>       whatever is
>>>>>>>>       >>>>      in ovn-evpn-vxlan-ports; if that includes 4790, the two
>>>>>>>>       devices
>>>>>>>>       >>>>      compete for the same UDP port.  Can we derive both the 
>>>>>>>> link
>>>>>>>>       >>>>      device and the conflict check from 
>>>>>>>> ovn-evpn-vxlan-ports?
>>>>>>>>       >>>>
>>>>>>>>       >>>> 5) Existing devices are adopted without checking them.
>>>>>>>>       >>>>
>>>>>>>>       >>>>      EEXIST is treated as success and the reconcile state
>>>>>>>>       lives only
>>>>>>>>       >>>>      in memory, so after "ovn-controller exit --restart":
>>>>>>>>       >>>>
>>>>>>>>       >>>>      - if the local IP, VXLAN port, LRP MAC or redistribute
>>>>>>>>       mode of a
>>>>>>>>       >>>>        VNI changed while ovn-controller was down, the old
>>>>>>>>       devices are
>>>>>>>>       >>>>        kept with the stale configuration;
>>>>>>>>       >>>>      - if a VNI was removed while ovn-controller was down,
>>>>>>>>       its devices
>>>>>>>>       >>>>        are never cleaned up;
>>>>>>>>       >>>>      - a CMS-created device that happens to be named
>>>>>>>>       ovnbr<vni> etc.
>>>>>>>>       >>>>        is taken over, and later deleted.
>>>>>>>>       >>>>
>>>>>>>>       >>>>      The restart test passes because the VNI is still 
>>>>>>>> present
>>>>>>>>       after
>>>>>>>>       >>>>      the restart, but removing it while ovn-controller is
>>>>>>>>       down isn't
>>>>>>>>       >>>>      covered.  I think we need to verify the kind and
>>>>>>>>       attributes of
>>>>>>>>       >>>>      an existing device before trusting EEXIST, and to sweep
>>>>>>>>       stale
>>>>>>>>       >>>>      ovn*<vni> devices on startup.
>>>>>>>>       >>> Hi Numan and Dumitru and Ales,
>>>>>>>>       >>>
>>>>>>>>       >>> I’ve thought some more about why I implemented it this way,
>>>>>>>>       and I’d like
>>>>>>>>       >>> to discuss one point.
>>>>>>>>       >>>
>>>>>>>>       >>> Some context for Dumitru and Ales: I added you to the thread
>>>>>>>>       because I’d
>>>>>>>>       >>> be interested in your opinion. Currently, the code that
>>>>>>>>       creates the EVPN
>>>>>>>>       >>> interfaces (|ovnbr<vni>|, |ovnvxlan<vni>|, |ovnlo<vni>|, and
>>>>>>>>       >>> |ovnvrf<vni>|) does not check their actual state in the
>>>>>>>>       kernel. It only
>>>>>>>>       >>> reacts to changes in the desired configuration: options, 
>>>>>>>> local
>>>>>>>>       IP, and
>>>>>>>>       >>> so on. If an interface is modified or deleted externally, the
>>>>>>>>       controller
>>>>>>>>       >>> won’t notice. It won’t even recreate a deleted interface 
>>>>>>>> until
>>>>>>>>       something
>>>>>>>>       >>> changes in the configuration.
>>>>>>>>       >>>
>>>>>>>>       >>> I implemented it this way for two reasons:
>>>>>>>>       >>>
>>>>>>>>       >>>   1.
>>>>>>>>       >>>
>>>>>>>>       >>>      I thought it made sense that, once we give OVN
>>>>>>>>       responsibility for
>>>>>>>>       >>>      these interfaces, the CMS has neither a need nor a 
>>>>>>>> reason
>>>>>>>>       to touch
>>>>>>>>       >>>      them. So the controller shouldn’t need to expect changes
>>>>>>> made
>>>>>>>>       >>>      outside its control. But I agree that this reasoning
>>>>>>>>       breaks down as
>>>>>>>>       >>>      soon as a person gets involved: someone could easily 
>>>>>>>> modify
>>>>>>>>       >>>      interfaces created by OVN, either by mistake or while
>>>>>>>>       debugging. My
>>>>>>>>       >>>      assumption here was that, if the CMS or a person changes
>>>>>>>>       something,
>>>>>>>>       >>>      OVN can already break things even without my code,
>>>>>>>>       because the
>>>>>>>>       >>>      controller currently retries failed netlink transactions
>>>>>>>>       in a loop.
>>>>>>>>       >>>
>>>>>>>>       >>>   2.
>>>>>>>>       >>>
>>>>>>>>       >>>      I didn’t want to check kernel state on every controller
>>>>>>>>       iteration,
>>>>>>>>       >>>      as that seemed excessive. However,
>>>>>>>>       |neighbor_exchange_run()|already
>>>>>>>>       >>>      dumps the neighbor table for every tracked interface on
>>>>>>> every
>>>>>>>>       >>>      iteration, which is considerably more expensive than a 
>>>>>>>> few
>>>>>>>>       >>>      |RTM_GETLINK|requests. So this argument doesn’t seem to
>>>>>>>>       hold up.
>>>>>>>>       >> You make a good point here but that is something we should
>>>>>>>>       address in
>>>>>>>>       >> general anyway, we should add proper incremental processing 
>>>>>>>> for
>>>>>>>>       >> monitoring kernel state (routes, neighbors, links, etc).
>>>>>>>>       >>
>>>>>>>>       >>> There is another case the current approach doesn’t cover: an
>>>>>>>>       >>> |ovn-controller|restart (|exit --restart|or a crash), as 
>>>>>>>> Numan
>>>>>>>>       pointed
>>>>>>>>       >>> out in the review above. After startup, the internal state is
>>>>>>>>       empty,
>>>>>>>>       >>> interface creation returns |EEXIST|, and we consider the 
>>>>>>>> stack
>>>>>>>>       >>> synchronized. If, for example, the VXLAN port or local IP
>>>>>>>>       changed while
>>>>>>>>       >>> the controller was down, the device will retain the old
>>>>>>>>       values, and the
>>>>>>>>       >>> controller won’t know.
>>>>>>>>       >>>
>>>>>>>>       >>> So the question is which approach would be better:
>>>>>>>>       >>>
>>>>>>>>       >>>   1.
>>>>>>>>       >>>
>>>>>>>>       >>>      *Check on every iteration.*Send |RTM_GETLINK|requests 
>>>>>>>> for
>>>>>>>>       each VNI’s
>>>>>>>>       >>>      interfaces on every iteration. Check that each interface
>>>>>>>>       exists and
>>>>>>>>       >>>      verify its type, master, MAC address, and UP state; for
>>>>>>>>       VXLAN, also
>>>>>>>>       >>>      check the VNI, local IP, destination port, and link.
>>>>>>>>       Compare these
>>>>>>>>       >>>      against the desired configuration. Fix attributes that
>>>>>>>>       the kernel
>>>>>>>>       >>>      doesn’t allow changing on an existing device (port, 
>>>>>>>> local
>>>>>>>>       IP, VNI)
>>>>>>>>       >>>      by recreating the interface, and update everything else
>>>>>>>>       in place.
>>>>>>>>       >>>      The advantage is that this is simple and reliable: it
>>>>>>>>       doesn’t depend
>>>>>>>>       >>>      on what woke the controller up, and the same code 
>>>>>>>> handles
>>>>>>>>       restarts,
>>>>>>>>       >>>      lost events, and manual changes. The downside I’m not
>>>>>>>>       happy about is
>>>>>>>>       >>>      the amount of code needed to parse and compare 
>>>>>>>> attributes.
>>>>>>>>       >>>
>>>>>>>>       >>>   2.
>>>>>>>>       >>>
>>>>>>>>       >>>      *Rely on *|*if_notifier_cb*|*.*Check kernel state only 
>>>>>>>> when
>>>>>>>>       >>>      |host-if-monitor|reports a change to our interfaces. To
>>>>>>>>       make this
>>>>>>>>       >>>      reliable, we would need to:
>>>>>>>>       >>>
>>>>>>>>       >>>        *
>>>>>>>>       >>>
>>>>>>>>       >>>          Keep a module-level dirty flag and clear it only 
>>>>>>>> after
>>>>>>>>       >>>          successful reconciliation here in evpn devices code.
>>>>>>>>       >>>
>>>>>>>>       >>>        *
>>>>>>>>       >>>
>>>>>>>>       >>>          Handle |ENOBUFS|on the netlink socket, indicating
>>>>>>>>       lost events,
>>>>>>>>       >>>          by marking everything dirty. Currently, the callback
>>>>>>>>       simply
>>>>>>>>       >>>          returns in this case.
>>>>>>>>       >>>
>>>>>>>>       >>>        *
>>>>>>>>       >>>
>>>>>>>>       >>>          Remove the |IFF_UP|filter in |if_notifier_cb|so that
>>>>>>>>       we also
>>>>>>>>       >>>          catch interfaces going down, and add |ovnvrf*|to the
>>>>>>>>       tracked
>>>>>>>>       >>>          interfaces.
>>>>>>>>       >>>
>>>>>>>>       >>>        *
>>>>>>>>       >>>
>>>>>>>>       >>>          Always reconcile with the kernel when ovn-controller
>>>>>>>>       starts,
>>>>>>>>       >>>          since devices left over from a previous run don’t
>>>>>>>>       generate any
>>>>>>>>       >>>          events.
>>>>>>>>       >>>          The advantage is that we query the kernel only when
>>>>>>>>       something
>>>>>>>>       >>>          has actually changed. The downside is more state and
>>>>>>>>       more edge
>>>>>>>>       >>>          cases that are easy to miss. It doesn’t really seem
>>>>>>>>       worth it.
>>>>>>>>       >>>
>>>>>>>>       >>> We need the attribute parsing and comparison code in both
>>>>>>>>       approaches, at
>>>>>>>>       >>> least to handle restarts. The only difference is /when/we 
>>>>>>>> call
>>>>>>>>       it. So
>>>>>>>>       >>> I’m leaning toward the first approach: it’s simpler and more
>>>>>>>>       reliable.
>>>>>>>>       >>> But maybe you have different opinions.
>>>>>>>>       >>>
>>>>>>>>       >> This is, IIRC, the main reason why in internal discussions I
>>>>>>>>       had with
>>>>>>>>       >> Ales when trying to figure out how to integrate OVN with EVPN
>>>>>>>>       >> deployments we decided against maintaining the required host 
>>>>>>>> side
>>>>>>>>       >> devices ourselves - it's quite a lot of (arguably
>>>>>>>>       uninteresting) work.
>>>>>>>>       >> And probably with a lot of added code and the bugs that come
>>>>>>>>       with it.
>>>>>>>>       >>
>>>>>>>>       >>> What do you think?
>>>>>>>>       >>>
>>>>>>>>       >> I'm wondering now if it isn't better to just provide a helper
>>>>>>>>       utility as
>>>>>>>>       >> part of OVN utilities that the CMS can run to ensure that the
>>>>>>>>       host side
>>>>>>>>       >> interfaces are in the correct state (matching what's in the NB
>>>>>>>>       and SB
>>>>>>>>       >> database).  It doesn't have to but it could be a simple script
>>>>>>>>       I guess.
>>>>>>>>       >> And it could reconcile the host side interfaces automatically
>>>>>>>>       if needed.
>>>>>>>>       >>
>>>>>>>>       >> It would still be up to the CMS to figure out the best way to
>>>>>>>>       run the
>>>>>>>>       >> utility, e.g., triggered when specific interface changes are
>>>>>>>>       detected.
>>>>>>>>       >>
>>>>>>>>       >> Would it make sense to reconsider this as a different 
>>>>>>>> approach?
>>>>>>>>       >
>>>>>>>>       > Hi, thanks for the reply!
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi Dumitru, Alexandra and Numan,
>>>>>>> Hi Ales,
>>>>>>>
>>>>>>> Thanks for sharing your view! Please see my replies inline.
>>>>>>>>       >
>>>>>>>>       > To be honest, I’m not entirely sure the script is the right way
>>>>>>>>       to go.
>>>>>>>>       > Let me describe how I picture it, so we can check it matches
>>>>>>>>       what you
>>>>>>>>       > have in mind:
>>>>>>>>       >
>>>>>>>>       >  1. ovn-controller has a handler that validates all the 
>>>>>>>> interface
>>>>>>>>       >     attributes against the current configuration and recreates 
>>>>>>>> the
>>>>>>>>       >     interfaces if needed.
>>>>>>>>
>>>>>>>>       The way I was imagining it (maybe over-simplified) was that
>>>>>>>>       ovn-controller doesn't do any of this work.  It would be the
>>>>>>>>       responsibility of the CMS to call the script whenever it 
>>>>>>>> (re)starts
>>>>>>>>       ovn-controller (e.g., as part of the systemd scripts).
>>>>>>>>
>>>>>>>>
>>>>>>>> I also don't think that this should be OVN's responsibility
>>>>>>>> at all. Sure we have "maintain-vrf", but it shouldn't be there
>>>>>>>> IMO. The benefit vs. maintenance cost and code size doesn't
>>>>>>>> seem to favor those changes. And the discussion here
>>>>>>>> solidifies that even more for me.
>>>>>>> I understand you don't like it, but I'd like to try once more to argue
>>>>>>> why this isn't such a really bad feature:
>>>>>>>
>>>>>>> 1. The devices are part of OVN, not of the routing daemon or something
>>>>>>> else. OVN writes
>>>>>>>       into them: FDB and neighbor entries on the bridge and VXLAN 
>>>>>>> devices,
>>>>>>> route-exchange installs routes into
>>>>>>>       the VRF. Their configuration is derived purely from OVN state 
>>>>>>> (VNI,
>>>>>>>       per-VNI local IP, router port MAC, the VXLAN port derived from the
>>>>>>>       VNI). Nothing depends on something external like routing deaemon 
>>>>>>> or
>>>>>>> etc,
>>>>>>>       so switching to another routing daemon wouldn't change anything 
>>>>>>> here.
>>>>>>>
>>>>>> Well not really, the expectation is for CMS to provide them. If they 
>>>>>> don't
>>>>>> it's their fault. So I don't see them as part of OVN.
>>>>>>
>>>>> Sure, but Alexandra is trying to change this restriction in this patch 
>>>>> set.
>>>>>
>>>>>>> 2. ovn-controller knows better then CMS where and when the devices are
>>>>>>> needed.
>>>>>>>       They are needed on chassis where the switch is local, and that
>>>>>>>       changes at runtime, e.g. when a VM migrates to another hypervisor 
>>>>>>> so
>>>>>>>       ovn-controller sees that quite fast but a CMS has to track port
>>>>>>>       bindings itself and race to configure the new host in time. With
>>>>>>>       this feature, for L2 EVPN the CMS doesn't have to do anything once
>>>>>>>       FRR is running; for L3 EVPN it only has to configure the VRF in 
>>>>>>> FRR.
>>>>>>>
>>>>>> OVN does, but the same logic can be applied to FRR. Should OVN
>>>>>> start FRR after failover? If CMS already has the FRR ready on all
>>>>>> chassis where it is important why couldn't it have all the needed
>>>>>> interfaces? That's not any different, setting them up once on those
>>>>>> chassis which will make potential failover even quicker than
>>>>>> OVN catching up and adding the interfaces only after they are needed.
>>>>>>
>>>>> This is a good point I didn't think of before.  The routing control
>>>>> plane needs the interfaces in order to function properly.  And the
>>>>> routing control plane is under control of the CMS.
>>>>
>>>> My example probably wasn't the best one. Let me clarify two points:
>>>>
>>>>   1.
>>>>
>>>>      Migration may not be the best example, but what about simply
>>>>      attaching a port to a subnet? Would the CMS have to create
>>>>      interfaces for every VNI on every hypervisor? I'm not sure I
>>>>      understand what you meant.
>>>>
>>> Yes, that's the current expectation that the CMS configures interfaces
>>> for every required VNI on the required hypervisors.  The routing control
>>> plane (e.g., FRR) must be running on the required hypervisors too and
>>> must anyway be configured to handle the VNI.  That's also outside the
>>> scope of OVN and most likely configured by the CMS.
>>>
>>>>   2.
>>>>
>>>>      Regarding failover, I disagree with your argument that the CMS would
>>>>      be faster. The CMS and the controller would receive the updates at
>>>>      essentially the same time, when the old controller releases the
>>>>      chassisredirect port and the new one claims it. By that point, the
>>>>      new controller would already have recomputed the OpenFlow rules and
>>>>      routing state locally and would be ready to receive traffic. Isn't
>>>>      that the case?
>>>>
>>> I think Ales was thinking of the potential fast-failover use case - in
>>> that case FRR would have to already be running on both chassis a DGP
>>> might be scheduled on.  So the absence of the Linux interfaces on the
>>> not-yet-active chassis would make it that FRR can't immediately change
>>> the routing information it may be advertising.
>>>
>>>>>>> I hope I’ve made it clear what I mean.
>>>>>>>
>>>>>>> About the maintenance cost, I'd suggest we define once, and document (in
>>>>>>> ovn user documentation),
>>>>>>> what OVN is responsible for and then rely on that:
>>>>>>>
>>>>>>> - OVN creates and deletes the devices and makes sure they are up.
>>>>>>> - OVN owns them. If someone modifies or deletes them after OVN created
>>>>>>>      them, that's a misconfiguration.
>>>>>>> - On startup, OVN dumps the devices in its group, deletes the ones
>>>>>>>      that are no longer in the configuration, and checks the attributes
>>>>>>>      of the remaining ones once against the desired config, recreating
>>>>>>>      them on mismatch. That covers configuration changes made while
>>>>>>>      ovn-controller was down. No extra state needs to be stored: the
>>>>>>>      kernel already holds what is currently configured.
>>>>>>> - No periodic validation of attributes at runtime.
>>>>>>>
>>>>>>> The same startup cleanup would also fix stale VRFs from
>>>>>>> dynamic-routing-maintain-vrf.
>>>>>>>
>>>>>>> Does this sound like an acceptable scope?
>>>>>>>
>>>>>> That's still quite a bit of code to maintain.
>>>>>>
>>>>>>
>>>>>>> So in my patch I'd propose adding a pass on startup that dumps all
>>>>>>> the interfaces (Further down in the email, there’s a comment from me
>>>>>>> about this.) and deletes the stale ones, but doesn't validate their
>>>>>>> attributes.
>>>>>>>
>>>>>>> But this proposal still leaves one case open: if someone changes the
>>>>>>> configuration (not specifically, not changing the VNI, but rather
>>>>>>> parameters like MAC/IP)
>>>>>>>
>>>>>>>     while ovn-controller is stopped. Maybe I could store
>>>>>>> what is currently configured in the ovsdb? Then on restart we
>>>>>>> validate it, and if the user changed anything, even the MAC address,
>>>>>>> OVN recreates the interfaces, the same way it already does when the
>>>>>>> change happens while the controller is running.
>>>>>>> What do you think?
>>>>>>>
>>>>>> That broadens the scope again, now we would have to maintain
>>>>>> what was there previously.
>>>>>>
>>>>>>
>>>>> I'm on the fence when it comes to this discussion in general.  This
>>>>> argument is making me sway again into the direction of "let the CMS do
>>>>> it".  It feels like we will be adding a lot of code just to make the
>>>>> life of the CMS easier while the CMS can probably already do that in a
>>>>> simpler way (as it has more domain-specific knowledge).
>>>> I'm still struggling to understand why the CMS is considered better
>>>> placed to manage these interfaces...The CMS expresses the desired
>>>> logical configuration through NB. Why should it also need to track which
>>>> hypervisor currently hosts a particular lsp and manage the corresponding
>>>> local interfaces? To me, that seems to duplicate placement information
>>>> that OVN already maintains.
>>>>
>>> Well, we have a very simple counter example: LSPs and the OVS
>>> interfaces.  The CMS needs to create the LSP (in the NB) but also need
>>> to create the OVS interface (with correct external_ids) on the
>>> hypervisor where the workload actually runs.
>>>
>>>>>>>>       >  2. We call it on ovn-controller startup, which solves the
>>>>>>>>       problem Numan
>>>>>>>>       >     described (more on that below).
>>>>>>>>
>>>>>>>>       See above.
>>>>>>>>
>>>>>>>>       >  3. If the CMS isn’t sure the interfaces are valid, it can call
>>>>>>>>       the same
>>>>>>>>       >     entry point.
>>>>>>>>       >
>>>>>>>>       > The upside is that ovn-controller doesn’t have to check every
>>>>>>>>       attribute
>>>>>>>>       > on every iteration. But I have some doubts about it, because it
>>>>>>>>       means
>>>>>>>>       > one more knob and one more entry point. The whole idea of
>>>>>>>>       maintain-evpn
>>>>>>>>       > is that OVN owns these interfaces and the CMS doesn’t have to 
>>>>>>>> think
>>>>>>>>       > about them. A “check it if you’re not sure” knob hands that
>>>>>>>>       > responsibility back to the CMS. That said, if you think this
>>>>>>>>       approach
>>>>>>>>       > sounds good, I’m happy to add it. I will cast aside all my 
>>>>>>>> doubts
>>>>>>>>       >
>>>>>>>>
>>>>>>>>       I'm not sure it's the best thing, I was just trying to reconsider
>>>>>>>>       it as
>>>>>>>>       an alternative. :)
>>>>>>>>
>>>>>>>>       > About the problem Numan described: it also applies to the VRFs 
>>>>>>>> in
>>>>>>>>       > dynamic routing. If ovn-controller is stopped without cleanup 
>>>>>>>> and
>>>>>>>>       > someone changes the configuration in the meantime (removes the
>>>>>>>>       VNI or
>>>>>>>>       > the maintain-vrf option), then after the restart ovn-controller
>>>>>>>>       will
>>>>>>>>       > never clean up the leftover interfaces.
>>>>>>>>       >
>>>>>>>>       > I think this can be solved separately from this patch, for EVPN
>>>>>>>>       and VRF
>>>>>>>>       > at the same time. My idea is:
>>>>>>>>       >
>>>>>>>>       >   * Put all interfaces created by OVN into the same device 
>>>>>>>> group
>>>>>>>>       >     (IFLA_GROUP). It can be set right when the interface is
>>>>>>> created.
>>>>>>>>       >   * On ovn-controller startup, dump the interfaces and delete
>>>>>>>>       the ones
>>>>>>>>       >     in our group that are no longer in the current 
>>>>>>>> configuration.
>>>>>>>>       >   * Do this only once, on startup.
>>>>>>>>       >   * Interfaces created by the CMS (e.g. a VRF without 
>>>>>>>> maintain-vrf)
>>>>>>>>       >     aren’t in our group, so we won’t touch them.
>>>>>>>>       >
>>>>>>>>       > The group number should probably be configurable, in case 
>>>>>>>> groups
>>>>>>>>       are
>>>>>>>>       > already used for something else on the host.
>>>>>>>>       >
>>>>>>>>
>>>>>>>>       This is a nice idea!  I'd argue however that the group number
>>>>>>>>       should be
>>>>>>>>       mandatory if we go this way.
>>>>>>>>
>>>>>>>>
>>>>>>>> I mean is this different from a name check? Unfortunately, the kernel
>>>>>>>> doesn't allow us to filter by device group so we need to dump
>>>>>>>> everything anyway.
>>>>>>> This would only happen once, on startup, so does it really matter? I
>>>>>>> don't think filtering interfaces by group on startup is a big
>>>>>>> problem to do it in ovn-controller on startup once. And how is it
>>>>>>> different from dumping all interfaces and
>>>>>>> filtering them by name, i.e. ovnvrf*, ovnvx*, ovnbr*, ovnlo*? That
>>>>>>> would work too, but it looks kind of ugly, doesn't it? I'd still dump
>>>>>>> all the interfaces, filter them by name and then parse the VNI out of
>>>>>>>
>>>>>>> the name. I don't really like that...
>>>>>>>
>>>>>> I mean what is the difference between filtering by name and group?
>>>>>> Besides the fact that filtering by name doesn't require extra setup
>>>>>> while the group does as we don't currently do that.
>>>>>>
>>>>> I missed the lack of kernel side filtering yesterday.  That does change
>>>>> the context, I agree, today we can already by name already in
>>>>> ovn-controller, no need for the group then.
>>>>
>>>> But then, when dumping interfaces, I would have to compare each
>>>> interface name against at least four strings (ovnbr, ovnvrf, ovnvx,
>>>> ovnlo). I understand your point about introducing a group ID when we
>>>> could rely on existing information, but I still think the code would be
>>>> cleaner with a group ID.
>>>>
>>>>>>> Anyway, I’ll look forward to hearing your thoughts on how I should go
>>>>>>> about this.
>>>>>>>
>>>>>>>>       > With this approach, on ovn-controller startup we can call the
>>>>>>>>       routine
>>>>>>>>       > that syncs the attributes and cleans up the stale interfaces.
>>>>>>>>       >
>>>>>>>>       > What do you think?
>>>>>>>>       >
>>>>>>>>
>>>>>>>>       I guess as long as we document it well that CMS touching anything
>>>>>>>>       that's
>>>>>>>>       in that group results in a misconfiguration it's probably fine.
>>>>>>>>
>>>>>>>>       I'm curious what others think about this (Ales, Numan, ..) 
>>>>>>>> though.
>>>>>>>>
>>>>>>>>
>>>>>>>> As I said above I'm not sure we should maintain this in OVN.
>>>>>>>> Especially when the contract is a on the weak side, reconcile
>>>>>>>> only on restart and if anything happens in between you might
>>>>>>>> have a problem.
>>>>>>>>
>>>>>>>> However, if Dumitru and Numan are fine with this feature
>>>>>>>> I won't block it.
>>>>>>>>
>>>>>>>>
>>>>>>>>       > Thanks, Sasha
>>>>>>>>       >
>>>>>>>>
>>>>>>>>       Thanks for the discussion!
>>>>>>>>
>>>>>>>>       Dumitru
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Ales
>>>>>>>>
>> Hi Sasha, Ales, Dumitru,
>>
>> My 2 cents.
>>
>> Thanks for the discussion.  I think the answer is different for L2
>> and L3 VNIs, and that this may let us agree on a smaller scope.
>>
>> The deciding question for me is whether enabling or disabling EVPN on
>> a logical switch needs any per-VNI routing daemon configuration.
>>
>> Dumitru, you wrote that FRR "must anyway be configured to handle the
>> VNI".  For L3 VNIs that's true: each one needs "vrf X / vni N", a
>> "router bgp ... vrf X" instance and an export policy.  For L2 VNIs,
>> as far as I understand FRR, it isn't: with a one-time configuration
>> (the BGP instance, the EVPN sessions and "advertise-all-vni" under
>> "address-family l2vpn evpn"), zebra discovers a new L2 VNI from the
>> VXLAN device attached to a bridge, and BGP starts advertising its
>> Type-2 and Type-3 routes without any change to FRR.  The one
>> condition is that the route targets line up across VTEPs: the
>> default "<ASN>:<VNI>" only works with a shared ASN, so deployments
>> with per-node ASNs need a fabric-wide route-target scheme in that
>> one-time configuration.  (Please correct me here if my understanding is wrong
>> here).
>>
>> So:
>>
>> - For L2 VNIs, once FRR is set up, the kernel devices are the only
>>    per-VNI, per-chassis work left for the CMS.  That's where I think
>>    it makes sense for ovn-controller to manage them.
>>
>> - For L3 VNIs, the CMS has to configure FRR per VRF anyway, and
>>    Dumitru's fast-failover point applies: the devices have to exist
>>    on every chassis a distributed gateway port may move to, so that
>>    FRR is ready before the port moves.  There the CMS might as well
>>    create the devices too.  For what it's worth, that's what we do:
>>    our agent creates the L3 VNI devices together with the FRR
>>    configuration for each VRF.
>>
>>
>> IMO,  it makes sense for ovn-controller to manage the vni interfaces
>> for L2 VNIs.  However if it complicates the code in ovn-controller,
>> I'm fine passing on the responsibility to CMS.
>>
>> I have another suggestion -  is to take the approach along the lines
>> of the VIF plug
>> providers: ovn-controller computes the desired per-VNI devices for the
>> logical switches, and a provider (or an external agent reading that
>> state from the local OVS database) creates them.  OVN would keep the
>> placement logic and the device code would live outside the tree.
>>
>> Is this approach we can explore ?
>>
>> Thanks
>> Numan
> Hi everyone!

Hi Sasha,

> Numan, thanks for the suggestion to use a separate provider, similar to 
> VIF plug providers. Before we go down that way, I’d like to give the 
> built-in approach another look =). I think we can keep the 
> implementation fairly small if we agree on what OVN should handle. I 
> also want to explain why I think L3 fits well into the existing 
> mechanism one more time.
> 
> 1. I agree with your explanation of L2, and I’d like to keep L3 support 
> too. In the L2 case you described, once the routing daemon and route 
> targets are set up, adding a VNI doesn’t require any changes to the 
> daemon’s configuration. We just need to create the Linux interfaces on 
> the right chassis. As you said, having OVN automate this would be 
> useful. The same applies to L3, and most of the device management code 
> would be shared with L2. I checked the L3 fast failover case you 
> described: OVN already creates the necessary infrastructure on both 
> gateway chassis, primary and standby. So the CMS just needs to configure 
> and start the routing daemon on both nodes ahead of time, with nothing 
> else to do during failover. I’ll also add a test to cover the switchover 
> and check that routing works correctly afterwards.

I think a test for that would be great nevertheless (even today without
support for auto-creating EVPN interfaces).

> 2. I think we should clearly document what the controller does and 
> doesn’t handle. OVN creates, configures, and brings up its devices. When 
> the OVN configuration changes, it updates or recreates them and removes 
> anything that’s no longer needed. At startup, it picks up configuration 
> changes made while it was stopped and cleans up devices left over from 
> the previous run that are no longer needed. But it won’t keep checking 
> every device attribute at runtime, and we won’t guarantee that it 
> automatically fixes changes made outside OVN. The CMS and administrators 
> shouldn’t modify or delete these devices themselves; we’d treat that as 
> a misconfiguration. I think these are reasonable limits, and they should 
> help keep the code small and straightforward.
> 3. I’ll move all the device management code into a separate file.
> 
> What do you think? Would this approach work for you?
> 

After reading the details below, and thinking a bit more about the vif
provider alternative I have some more thoughts, please see below. :)

> Below are a few more details on what I’d like to finish in this patch, 
> including how I’d implement the behavior described in point 2. Feel free 
> to skip this part :)
> There are three things I’d like to address:
> 1. The VRF conflict
I’d like to rework this part of the patch. Right 
> now, for an L3VNI, we specify the LRP that connects the EVPN interfaces 
> to the OVN router, but we create ovnvrf<VNI> independently of the 
> router’s VRF. We then expect the CMS to set the router’s VRF to 
> ovnvrf<VNI>. That’s where the issues Numan pointed out in the review 
> come from.
My suggestion is to make the explicit VRF setting optional in 
> this mode. If no VRF is set, the controller will create one 
> automatically, using the tunnel_key of the router associated with the 
> LRP as the table ID, just as dynamic routing does today.
The evpn-device 
> code will handle creating and deleting this VRF, and route-exchange will 
> use the same VRF for route advertisements. This way, we’ll no longer tie 
> the table ID to the VNI, and the router and its EVPN interfaces will 
> share one VRF.
> 2. Handling devices at startup
I’d still like to consider saving the 
> last successfully applied device configuration in the local OVSDB. We’d 
> store the actual computed values: MAC, local VTEP IP, VNI, UDP port, 

So, essentially we'd store in the local OVSDB the "desired state" for
the EVPN required linux interfaces.  That's good.

> LRP, and so on.
At startup, the controller would compute these values 
> again from the current OVN configuration, compare them with the saved 
> values, and apply any changes. For example, if the LRP name stays the 
> same but its MAC changes, we’ll catch that and recreate the interface. 
> This would let us handle OVN configuration changes within the limits 
> described above.

But now that we have the "desired state" in OVSDB, it really feels like
a pity to half implement a controller that syncs this as Linux host
interfaces..

Briefly looking at the dummy vif provider and at the official
representor provider [0] there are some things to take into account:

The vif-plug implementation is really designed to "plug" VIFs
corresponding to LSPs.  Also, in this case there's no LSP representing
the EVPN interfaces.

I guess we could change the vif-plug implementation to handle EVPN
interface plugging - not sure how nice/ugly that'd be.  But if we do
have that covered somehow then I think it would actually be relatively
easy to have an evpn vif provider that implements a proper
reconciliation loop between the OVSDB "desired state" and host
interfaces "actual state".

The alternative to all this would still be a separate controller reading
the OVSDB "desired state".  Just to keep that logic outside of OVN.

[0]
https://github.com/ovn-org/ovn-vif/blob/main/lib/vif-plug-providers/representor/vif-plug-representor.c

> 3. Cleaning up old interfaces
I’ll use your suggestion to dump all 
> interfaces and identify the old ones by name, since the group_id 
> approach didn’t work out for you.
> It’d be nice to hear your thoughts on this.
> 
> Thanks, Sasha
> 

Regards,
Dumitru

>>
>>>>> Thanks,
>>>>> Dumitru
>>>>>
>>>>> Внимание: ВНЕШНИЙ отправитель!
>>>>>
>>>>>
>>>>> Будьте осторожны с вложениями и ссылками.
>>> Regards,
>>> Dumitru
>>>
> 

_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

Reply via email to