Thanks all for the discussion and all the ideas here. After reading the emails, I think it boils down to two proposed approaches:
1) CMS to continue to connect to the Southbound database if they need information about the physical location of the resources. That would avoid the inefficiency of having to copy data back-and-forth from the Northbound and Southbound database. I guess the downside of this approach is that CMS will have to maintain a connection with both databases (which is already the case today). If we go with this approach, it would be good to have consensus from the core OVN team where some tables in the Southbound must be kept stable with backward compatibility in case of changes. Tables such Chassis, Chassis_Private and Port_Binding at least will need that. I guess that makes part of the Southbound database to not be considered OVN internal only. 2) The second approach is what Mark has described below, having a separated database for the runtime information. One thing I like about this approach is that it keep the role and permissions of each database quite clear, as described: NB DB: CMS writes, OVN reads SB DB: OVN-internal Status DB: CMS reads, OVN writes It also seems to enable a lot of room for extra information about the runtime to be added in the future (as mentioned in Mark's email, LB health status, LSP packet counts etc...). The con is that it's yet another database that needs to be deployed and maintained and another connection for CMS to maintain if they need that sort of information. ... Is there any preference of which approach the core OVN team would prefer here ? Cheers, Lucas On Thu, Apr 13, 2023 at 6:11 PM Mark Michelson <[email protected]> wrote: > > Hi all, > > I just caught up on this discussion and wanted to complicate things > further by suggesting another idea. I think the Red Hat folks have heard > this before, but I'm not sure if it has been brought up on this list before. > > Aside from this issue, there is also this high-priority issue from Red > Hat Openstack: https://bugzilla.redhat.com/show_bug.cgi?id=2123176 . > > IMO, this all converges on the idea of introducing a third database to > OVN. We can refer to this as the "Status" DB. > > The Status DB would be a place for state information generated by > OVN/OVS to be stored. Some ideas for existing things that could go in > the Status DB would be: > * Logical port up/down state. > * Logical switch port dynamic addresses (maybe, this is more complicated) > * BFD status > * Logical port installation status and installation timestamp. > > In addition to these existing items, the Status DB would be a place for > additional items that do not exist yet, such as > * Load balancer health check status > * Logical port packet/byte counts > * Gateway port bound chassis > > With the implementation of the Status DB, it would cement a relationship > between the DBs as such: > > NB DB: CMS writes, OVN reads > SB DB: OVN-internal > Status DB: CMS reads, OVN writes > > It may be tempting to get this patch merged as-is, with the intention of > migrating this to the new DB once it gets implemented. I don't think > this is a good idea. Between this issue and the one I linked, I think > the implementation of a Status DB is a good idea, and one that should be > implemented very soon. > > Since this particular problem is already worked around by OpenStack, I > think it makes more sense to implement this feature in a way that will > be easier to maintain long-term than to get it in quickly. If we merge > this as-is, then we are on the hook for supporting this status in the NB > DB for quite a long time since we would need to take time to deprecate > it properly. If we instead treat this as the impetus to write the Status > DB, then I think this lightweight use-case would give us a good starting > point towards adding the other items we're interested in. > > What do you think? > > On 4/13/23 09:32, Lucas Martins wrote: > > Hi Han, Dumitru and Luis, > > > > Thanks for the discussion and ideas so far. My reply is inline: > > > > On Thu, Apr 13, 2023 at 10:45 AM Luis Tomas Bolivar <[email protected]> > > wrote: > >> > >> > >> > >> On Thu, Apr 13, 2023 at 9:33 AM Dumitru Ceara <[email protected]> wrote: > >>> > >>> On 4/12/23 23:07, Han Zhou wrote: > >>>> On Wed, Apr 12, 2023 at 8:01 AM <[email protected]> wrote: > >>>>> > >>>>> From: Lucas Alvares Gomes <[email protected]> > >>>>> > >>>>> In order for the CMS to know which Chassis a distributed gateway port > >>>>> is bond to, this patch updates the ovn-northd daemon to populate the > >>>>> Logical_Router_Port table with that information. > >>>>> > >>>>> To avoid changing the database schema, ovn-northd is setting a new key > >>>>> called "hosting-chassis" in the options column from the LRP table. This > >>>>> key value points to the name of the Chassis that is currently hosting > >>>>> the distributed port. > >>>>> > >>>>> Reported-at: https://bugzilla.redhat.com/show_bug.cgi?id=2107515 > >>>>> Signed-off-by: Lucas Alvares Gomes <[email protected]> > >>> > >>> Hi, Lucas, Han, > >>> > >>>> > >>>> Thanks Lucas for the patch. However, in my opinion the chassis binding > >>>> information belongs to SB and should stay there, otherwise we would make > >>>> it > >>>> consistent for LSPs and update the chassis information for them, too, > >>>> which > >>>> I think is not good in terms of clarity and extra control plane load. > >>>> We'd > >>>> better keep the separation between NB and SB clear and avoid propagating > >>>> data between them back-and-forth. > >>>> > >>> > >>> I partially agree with this but it also feels wrong that the CMS > >>> accesses the SB directly. In an ideal world (and I know that's not the > >>> case today for neutron or ovn-k8s) the CMS should not care about what's > >>> in the SB; that is internal OVN data. > >> > >> > >> Just to add some extra input in here. As Dumitru mentioned, it is not just > >> a scaling issue, but that accessing the SB has its own problems as things > >> can change there any time (it has already happened) breaking the logic on > >> the CMS about how to react to those changes. If we don't have the > >> information at the NB, that means we need 2 connections, one for the NB > >> (to be as safe as possible from the SB changes), and one for the SB to get > >> the chassis information. > >> > > > > Right. So the idea is to have the CMS to only connect to the > > Northbound database instead of maintaining a connection with both > > databases (helping scalability). I don't know what the consensus is > > but, if we agree that the Southbound database is used to store the > > internal OVN data, I think it would be in everyone's favour if CMS > > only used the Northbound database because as Luis pointed out apart > > from scalability issues, the data structure in the Southbound database > > can change overtime without any backwards compatibility and it will > > break us (it already happened). > > > >> Also, note there is already chassis information on the > >> logical_switch_ports at the NB DB, so adding that for the cr-lrps should > >> not be that different. Adding the active chassis to the HA_Chassis_group > >> also sounds good > > > > So I believe this is the option "requested-chassis" that Neutron sets > > in the LSP. The difference is that this option is set by the CMS and > > the new option "hosting-chassis" from my patch is set by northd > > instead. But, there are still similarities because it's also the CMS > > that sets the ha_chassis_group (or gateway_chassis) for a port to make > > it HA. The proposed "hosting-chassis" option is just a way for northd > > to give the CMS a feedback about which chassis from the group that > > port ended up binding to. > > > >>> > >>> > >>> I suggest a different approach if we want to go ahead and propagate such > >>> information to the NB: can't we store the "active chassis" information > >>> per Gateway_chassis/HA_Chassis_group instead? That's > >>> O(number-of-chassis) records that we need to update on chassis failover. > >>> We might even skip this for Gateway_chassis as I understand that this > >>> is the "old" way of configuring things (*). > >>> > > > > That makes sense for me as well. So in the HA_Chassis_Group we would > > have a column with the current active chassis name ? That would be > > good because we can't really rely on the "priority" order because if > > there is a fallback to another chassis, the CMS is blind to it. > > > >>> (*) Should we deprecate Gateway_chassis? > >>> > > > > I think Neutron still uses it but, with my core OVN hat on I think it > > is already time. Right now in the Northbound database we have > > HA_Chassis_Group and Gateway_Chassis doing the same thing. I believe > > that in the Southbound everything becomes a HA_Chassis_Group. So it's > > fair to get rid of the Gateway_Chassis way already. > > > >>>> For the problem mentioned in the bugzilla, it seems to me already a scale > >>>> challenge that something other than ovn-controller is connecting to OVN > >>>> SB > >>>> from every node (if I understand correctly). Moving all these connections > >>>> from SB to NB may just make it much worse, because NB DB is usually more > >>>> heavily/frequently updated by the CMS. (For small scale, this may not > >>>> matter, even if the agent connects to both NB and SB.) > >>>> > >>> > >>> An alternative to address the scale issue without changing OVN could be > >>> to use a dedicated SB relay to which all external (non-OVN) agents that > >>> need access to SB information can connect. Would that help? > >>> > > > > The problem with it is that, more often than not we actually need to > > connect to both databases (as stated above) and there's no backward > > compatibility regards the data structure in the Southbound database > > because it is supposed to be internal OVN data. That's why having the > > CMS to only connect to the Northbound is a plus. > > > > Cheers, > > Lucas > > > > > > On Thu, Apr 13, 2023 at 10:45 AM Luis Tomas Bolivar <[email protected]> > > wrote: > >> > >> > >> > >> On Thu, Apr 13, 2023 at 9:33 AM Dumitru Ceara <[email protected]> wrote: > >>> > >>> On 4/12/23 23:07, Han Zhou wrote: > >>>> On Wed, Apr 12, 2023 at 8:01 AM <[email protected]> wrote: > >>>>> > >>>>> From: Lucas Alvares Gomes <[email protected]> > >>>>> > >>>>> In order for the CMS to know which Chassis a distributed gateway port > >>>>> is bond to, this patch updates the ovn-northd daemon to populate the > >>>>> Logical_Router_Port table with that information. > >>>>> > >>>>> To avoid changing the database schema, ovn-northd is setting a new key > >>>>> called "hosting-chassis" in the options column from the LRP table. This > >>>>> key value points to the name of the Chassis that is currently hosting > >>>>> the distributed port. > >>>>> > >>>>> Reported-at: https://bugzilla.redhat.com/show_bug.cgi?id=2107515 > >>>>> Signed-off-by: Lucas Alvares Gomes <[email protected]> > >>> > >>> Hi, Lucas, Han, > >>> > >>>> > >>>> Thanks Lucas for the patch. However, in my opinion the chassis binding > >>>> information belongs to SB and should stay there, otherwise we would make > >>>> it > >>>> consistent for LSPs and update the chassis information for them, too, > >>>> which > >>>> I think is not good in terms of clarity and extra control plane load. > >>>> We'd > >>>> better keep the separation between NB and SB clear and avoid propagating > >>>> data between them back-and-forth. > >>>> > >>> > >>> I partially agree with this but it also feels wrong that the CMS > >>> accesses the SB directly. In an ideal world (and I know that's not the > >>> case today for neutron or ovn-k8s) the CMS should not care about what's > >>> in the SB; that is internal OVN data. > >> > >> > >> Just to add some extra input in here. As Dumitru mentioned, it is not just > >> a scaling issue, but that accessing the SB has its own problems as things > >> can change there any time (it has already happened) breaking the logic on > >> the CMS about how to react to those changes. If we don't have the > >> information at the NB, that means we need 2 connections, one for the NB > >> (to be as safe as possible from the SB changes), and one for the SB to get > >> the chassis information. > >> > >> Also, note there is already chassis information on the > >> logical_switch_ports at the NB DB, so adding that for the cr-lrps should > >> not be that different. Adding the active chassis to the HA_Chassis_group > >> also sounds good > >>> > >>> > >>> I suggest a different approach if we want to go ahead and propagate such > >>> information to the NB: can't we store the "active chassis" information > >>> per Gateway_chassis/HA_Chassis_group instead? That's > >>> O(number-of-chassis) records that we need to update on chassis failover. > >>> We might even skip this for Gateway_chassis as I understand that this > >>> is the "old" way of configuring things (*). > >>> > >>> (*) Should we deprecate Gateway_chassis? > >>> > >>>> For the problem mentioned in the bugzilla, it seems to me already a scale > >>>> challenge that something other than ovn-controller is connecting to OVN > >>>> SB > >>>> from every node (if I understand correctly). Moving all these connections > >>>> from SB to NB may just make it much worse, because NB DB is usually more > >>>> heavily/frequently updated by the CMS. (For small scale, this may not > >>>> matter, even if the agent connects to both NB and SB.) > >>>> > >>> > >>> An alternative to address the scale issue without changing OVN could be > >>> to use a dedicated SB relay to which all external (non-OVN) agents that > >>> need access to SB information can connect. Would that help? > >>> > >>> Regards, > >>> Dumitru > >>> > >>> _______________________________________________ > >>> dev mailing list > >>> [email protected] > >>> https://mail.openvswitch.org/mailman/listinfo/ovs-dev > >> > >> > >> > >> -- > >> LUIS TOMÁS BOLÍVAR > >> Principal Software Engineer > >> Red Hat > >> Madrid, Spain > >> [email protected] > >> > > > > _______________________________________________ > > dev mailing list > > [email protected] > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
