Hi Vladislav, (Sorry I didn't notice your update until now when reading the discussion of your OVN patch)
I think a simple solution is to update the hardcoded 60s probe in OVS to 0, for the pre-created punix mgmt controller. We could make it configurable, but it may not make much sense since it is a unix socket anyway. We have been using this 0 value downstream for years (cc Winson). @Ilya Maximets <[email protected]> do you think it is reasonable? Thanks, Han On Thu, Jun 8, 2023 at 7:38 AM Vladislav Odintsov <[email protected]> wrote: > Hi, > > I’ve sent the first part (OF probing disable) - > https://patchwork.ozlabs.org/project/ovn/patch/[email protected]/ > > Regarding configuration on the OVS side, I see it’s not working like I > wrote before, I’ve deceived myself :) > I’ve written some code in ovn-controller, which creates controller for > integration bridge with inactivity probe, it successfully creates record, > but it hits OVS controller paths restrictions: > > 2023-06-07T21:08:01.552Z|00338|bridge|ERR|bridge br-int: Not adding Unix > domain socket controller "punix:/var/run/ovn/br-int.mgmt" due to > possibility of overwriting local files. Instead, specify path in permitted > format "punix:/var/run/openvswitch/br-int.*" or connect to > "unix:/var/run/openvswitch/br-int.mgmt" (which is always available without > special configuration). > > And in my manual tests I created controller in a directory /var/run, which > in centos made a workaround how to add socket on the same path, but allowed > to bypass ovs checks (/var/run/openvswitch/… vs /run/openvswitch/…), > because of symlink /var/run/ -> /run. > > So, if you guys have any suggestion how to disable OF probes on a standard > OVS controller socket (br-int.mgmt), please give an advice. > Alternatively I’ve been thinking about creation another socket with > another path for ovs-vswitchd<->ovn-controller OF communication. > But I’m concerned about paths consistency across different distributions… > > Thoughts? > > On 6 Jun 2023, at 13:28, Vladislav Odintsov <[email protected]> wrote: > > Hi Han, > > thanks for your inputs! > Please see comments inline. > > On 5 Jun 2023, at 21:01, Han Zhou <[email protected]> wrote: > > On Mon, Jun 5, 2023 at 8:58 AM Vladislav Odintsov <[email protected] > <mailto:[email protected]>> wrote: > > > Hi Dumitru, Ilya, Han, > > do you guys have any comments here? > I’m gonna start working on this change in near future... > > Thank you. > > On 17 May 2023, at 18:34, Numan Siddique <[email protected] <mailto: > [email protected]>> wrote: > > On Wed, May 17, 2023 at 4:44 AM Vladislav Odintsov <[email protected] > <mailto:[email protected]>> > > wrote: > > > > > > On 16 May 2023, at 23:54, Numan Siddique <[email protected] <mailto: > [email protected]>> wrote: > > Hi Vladislav, > > Sorry for the late reply. > > PSB for few comments. > > > > Thanks for your reply! > My answers are inline. > > > > > On Tue, May 16, 2023 at 3:42 PM Vladislav Odintsov <[email protected] > <mailto:[email protected]> > > <mailto:[email protected]>> wrote: > > > > Hi Numan, Dumitru, Ilya, > > I guess I’ve managed to deal with some of these problems and have some > > conclusions. > > Please see inline. > > On 14 Apr 2023, at 16:26, Vladislav Odintsov <[email protected] <mailto: > [email protected]>> wrote: > > And as a follow-up, I see sometimes next error message in ovn-controller > > log, though there is a configured much higher limit in seconds for > inactivity probe value for the chassis: > > > > ovn-controller establishes 3 openflow connections to local ovs-vswitchd > > for next purposes: > > 1. openflow rules management (openflow probe interval is configured > > through ovn-openflow-probe-interval external_ids setting [1]; default — > disabled [2]); > > 2. openflow features collection (hardcoded probe interval 5s [3]); > 3. pinctrl thread (hardcoded probe interval 5s [4]). > > ovs|04426|rconn|ERR|unix:/run/openvswitch/br-int.mgmt: no response to > > inactivity probe after 8 seconds, disconnecting > > > ovs|26560|rconn|ERR|unix:/run/openvswitch/br-int.mgmt: no response to > > inactivity probe after 10 seconds, disconnecting > > > > These messages most likely related to the connection for features > > collection or pinctrl thread. So ovn-openflow-probe-interval has no matter > here. > > > > server with first log message: > # ovs-vsctl get open . external_ids:ovn-openflow-probe-interval > "30" > > server with second log message: > # ovs-vsctl get open . external_ids:ovn-openflow-probe-interval > "60" > > > > On 30 Mar 2023, at 23:19, Vladislav Odintsov <[email protected] <mailto: > [email protected]>> wrote: > > Hi all, > > I’m facing a false positive failover in the event of openflow connection > > got dropped after inactivity probe timeout due to high load of > ovn-controller handling huge amount of ovsdb updates. > > I wonder wether its a bug or I have to tune any specifics. See > > ovn-controller logs: > > > 2023-03-30T07:57:24.469Z|58440|inc_proc_eng|INFO|node: > > logical_flow_output, recompute (failed handler for input port_groups) took > 35774ms > > 2023-03-30T07:57:28.521Z|58441|lflow_cache|INFO|Detected cache inactivity > > (last active 40006 ms ago): trimming cache > > 2023-03-30T07:57:28.528Z|58442|timeval|WARN|Unreasonably long 40084ms > > poll interval (32947ms user, 6968ms system) > > 2023-03-30T07:57:28.528Z|58443|timeval|WARN|faults: 43468 minor, 0 major > ….. > 2023-03-30T07:57:48.784Z|58496|reconnect|ERR|ssl:xxxx:6642: no response > > to inactivity probe after 60.2 seconds, disconnecting > > 2023-03-30T07:57:48.784Z|58497|reconnect|INFO|ssl:xxxx:6642: connection > > dropped > > 2023-03-30T07:57:48.905Z|58498|main|INFO|OVNSB commit failed, force > > recompute next time. > > 2023-03-30T07:57:49.786Z|58499|reconnect|INFO|ssl:yyyy:6642: connecting... > 2023-03-30T07:57:49.924Z|58500|reconnect|INFO|ssl:yyyy:6642: connected > 2023-03-30T07:57:29.781Z|58501|poll_loop|INFO|wakeup due to 0-ms timeout > > at lib/stream-ssl.c:838 (74% CPU usage) > > 2023-03-30T07:57:29.836Z|58502|poll_loop|INFO|wakeup due to 0-ms timeout > > at lib/stream-ssl.c:838 (74% CPU usage) > > …. > 2023-03-30T07:57:05.976Z|58545|poll_loop|INFO|wakeup due to 0-ms timeout > > at lib/stream-ssl.c:838 (101% CPU usage) > > 2023-03-30T07:57:07.002Z|58546|memory|INFO|peak resident set size grew > > 54% in last 682257.2 seconds, from 4435696 kB to 6824820 kB > > > > 2023-03-30T07:57:07.002Z|58547|memory|INFO|idl-cells-OVN_Southbound:11491359 > idl-cells-Open_vSwitch:14416 lflow-cache-entries-cache-expr:530298 > lflow-cache-entries-cache-matches:31770 lflow-cache-size-KB:769553 > local_datapath_usage-KB:630 > > ofctrl_desired_flow_usage-KB:577052 ofctrl_installed_flow_usage-KB:409538 > > ofctrl_sb_flow_ref_usage-KB:273707 > > … > > > 2023-03-30T07:57:31.667Z|58552|rconn|WARN|unix:/run/openvswitch/br-int.mgmt: > connection dropped (Broken pipe) > > 2023-03-30T07:57:31.739Z|58553|binding|INFO|Releasing lport cr-xxx from > > this chassis (sb_readonly=0) > > 2023-03-30T07:57:31.739Z|58554|if_status|WARN|Dropped 1 log messages in > > last 1254725 seconds (most recently, 1254725 seconds ago) due to excessive > rate > > 2023-03-30T07:57:31.739Z|58555|if_status|WARN|Trying to release unknown > > interface cr-xxx > > 2023-03-30T07:57:31.743Z|58556|binding|INFO|Releasing lport cr-yyy from > > this chassis (sb_readonly=0) > > 2023-03-30T07:57:31.743Z|58557|binding|INFO|Releasing lport cr-zzz from > > this chassis (sb_readonly=0) > > … > > After some time these ports are claimed back: > > > > 2023-03-30T07:57:49.320Z|59137|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: > connecting... > > > > 2023-03-30T07:57:49.320Z|59138|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: > connected > > 2023-03-30T07:57:49.523Z|59139|poll_loop|INFO|wakeup due to 0-ms timeout > > at controller/ovn-controller.c:4296 (99% CPU usage) > > 2023-03-30T07:57:49.574Z|59140|poll_loop|INFO|wakeup due to 0-ms timeout > > at controller/ovn-controller.c:4286 (99% CPU usage) > > 2023-03-30T07:58:01.451Z|59141|poll_loop|INFO|wakeup due to [POLLIN] on > > fd 22 (xxxx:34736<->yyyy:6642) at lib/stream-ssl.c:836 (99% CPU usage) > > 2023-03-30T07:58:02.449Z|59142|binding|INFO|Claiming lport cr-xxx for > > this chassis. > > 2023-03-30T07:58:02.449Z|59143|binding|INFO|cr-xxx: Claiming > > xx:xx:xx:xx:xx x.x.x.x/xx > > 2023-03-30T07:58:02.453Z|59144|binding|INFO|Claiming lport cr-yyy for > > this chassis. > > 2023-03-30T07:58:02.453Z|59145|binding|INFO|cr-yyy: Claiming > > yy:yy:yy:yy:yy:yy y.y.y.y/yy > > … > > Previously I’ve met with similar problem but the openflow probe interval > > was 30 seconds and I set it to 60 and then to 120 seconds. > > But it seems that OVS was configured on default inactivity probe timeout > > for openflow - 60 seconds. So, the error above occured. In OVS logs it > looks like next: > > > 2023-03-30T07:57:28.632Z|61003|rconn|ERR|br-int<->unix#8: no response to > > inactivity probe after 60 seconds, disconnecting > > > > When ovn-controller looses openflow connection for flows management it > > considers OVS is dead and releases its ha-chassis-group bindings. This > logic was introduced with the patch [5]. > > > ovn-controller uses unix socket to connect to ovs-vswitchd OF socket, it > > seems useless to configure inactivity probing, right? > > In this commit message [6] Han also disables OF inactivity probing by > > default. > > > Moreover, it seems harmful to configure inactivity probing from OVS to > > ovn-controller and vise versa. > > In case ovn-controller for some reason is busy with some work and can’t > > answer to OVS in two minutes (its default probe interval is 60s [7]), so > OVS destroys OF connection with the ERR log: > > ovs|18206|rconn|ERR|br-int<->unix#17574: no response to inactivity probe > > after 60 seconds, disconnecting > > > So, when ovn-controller resumes after a big work it sees that OF > > connection is closed and triggers chassisredirect portbindings release. > > This leads to datapath downtime and unnesessary load on OVN_Southbound, > > ovn-controllers, etc. > > > To avoid such behaviour user may consider not to configure > > ovn-openflow-probe-interval AND disable probing from OVS side with the next > command: > > ovs-vsctl --inactivity-probe=0 set-controller br-int > > punix://var/run/openvswitch/br-int.mgmt > > Did you test this? It seems for the management controller it is hardcoded > to 60s: > https://github.com/openvswitch/ovs/blob/master/vswitchd/bridge.c#L3882 > > > To make this setting applied, it’s needed to reestablish OF connection. > > (for instance, restart ovn-controller) > > > My proposal: > > If ovn-controller uses only unix socket for OF connection, it should be > > safe to disable OF probing from its code: > > 1. configure controller for br-int to disable probing (ensure br-int or > > other configured bridge controller created with inactivity probe = 0); > > Do you mean the OVS side config (using ovs-vsctl)? But there is no code > change from OVN side, right? > > > Yes, I mean OVS side configuration can be done by ovn-controller (it just > needs to make an equivalent to mentioned command through ovsdb). > > > 2. delete ovn-openflow-probe-interval setting and mark in documentation > > that for OVN versions >= 20.09 it’s recommended not to configure > ovn-openflow-inactivity-probe. For versions < 20.09 default interval is 5 > seconds, so it’s reasonable to set it to higher value in the config for > loaded hypervisors; > > > I am ok with this. For the documentation, it should be ok to remove the > related description and update the NEWS. I think there is no need to > mention 20.09 or any specific version, because documentation always belongs > to the version where it is updated. > > > Okay, this makes sense. Documentation change could be backported to > relevant branches. > > > > You mean 23.09 right ? > > > No. > IIUC, the 20.03.0-2 and 20.06.0-2 versions have default interval of 5 > > seconds. > > All other versions have disabled OF probing by default. > So my idea was to mention it in docs and recommend users of those > > versions (with 5 seconds default) to disable OF manually by setting > ovs-vsctl set open . external_ids:ovn-openflow-probe-interval=0. > > > > > 3. set openflow probe interval for ovs features and pinctrl OF connection > > to 0 (disable probing); > > > I am ok with this, too. > > What do you guys think about this? > Please, correct me if I understand something wrong and described changes > > can’t be accepted. > > > > The proposal looks reasonable to me. > > There may be once concern. Let's say the physical link is fine (where > vswitchd runs bfd) but vswitchd is down on chassis 'A' for some > reason. With probing disabled, ovn-controller will never > come to know if vswitchd is down until it tries to communicate with > it. If this chassis 'A' is acting as a gateway, external traffic > from other chassis VMs will be tunnelled to this and since vswitchd is > down the traffic may not go out if no datapath flows exist to output > the traffic to the physical interface of the provider bridge. Is this > acceptable ? > At least with the probing enabled, ovn-controller will release the > chassis resident port and the next in the priority will claim it. > > IMO, this limitation is more acceptable than the issue you reported. > Let's see what others have to say about this. > > > > I’m not sure I fully understand your idea about vswitchd down. > I tried to send sigkill to ovs-vswitchd with probing disabled and > > ovn-controller recognized the failure (connection closed by peer) and > released its chassisredirect port_binding (assuming another chassis by > priority claims it): > > > > > 2023-05-17T08:37:10.134Z|00004|rconn(ovn_pinctrl0)|INFO|unix:/run/openvswitch/br-int.mgmt: > connection closed by peer > > > > 2023-05-17T08:37:10.134Z|00061|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: > connection closed by peer > > > > 2023-05-17T08:37:10.134Z|00062|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: > connection closed by peer > > 2023-05-17T08:37:10.135Z|00063|binding|INFO|Releasing lport cr-xxx from > > this chassis (sb_readonly=0) > > 2023-05-17T08:37:10.135Z|00064|if_status|WARN|Trying to release unknown > > interface cr-xxx > > 2023-05-17T08:37:10.135Z|00065|binding|INFO|Releasing lport cr-xxx from > > this chassis (sb_readonly=0) > > > Does such test look reasonable and covers your concern? If not, could you > > describe it in more details, please? > > > > I think I misunderstood earlier. Please ignore my comment. So when > ovs-vswitchd is killed, ovn-controller will detect it. > > > Thanks > Numan > > > > > > > > > > This node is a dedicated l3gateway which has is-interconn=true setting > > (interconnection case) and has quite big openflow table around 2.7M flows. > > > So, I’ve got some questions for OVN and OVS: > 1. I couldn’t find in ovn-controller code where the logic of handling > > openflow connection drop. It is a bug when after openflow connection drop > the claimed cr-ports are being released? I guess, yes. Can you please point > me to the corrent place of code to start investigating? > > When openflow connection drops, ovn-controller would wait for it to > reconnect and then trigger a recompute. It seems expected behavior that the > cr-ports are released during this time. > > 2. I see almost the same openflow amount on each l3gateway with > > interconnection role. Even l3gateways which have no allocated > cr-port-bindings have 2M+ openflow flows. Is it normal operation or it’s a > place for possible optimisation? I mean it seems that excess flows are > installed. > > > > There are a lot of mac_binding records, but the question is why these > > flows still persist even if no datapaths are allocated to the node. > > > > Does the l3gateway chassis that has no cr-port-bindings have any other > port-bindings at all? Did you observe same amount of flows if there is no > port-bindings after you restart ovn-controller? > I think there shouldn't be any flows if there is no port-bindings, but the > flows are expected if this l3gateway had port-bindings earlier and then > released them, because the incremental-processing doesn't take care of > removing local-datapaths not in use anymore. > > > You’re right. There are no so much flows after restart. Thanks. > > > > Maybe the datapath to which the mac binding records belong is in the > ovn-controllers' "local_datapath" [1] - > https://github.com/ovn-org/ovn/blob/main/controller/local_data.h#L35 > > > Thanks, I’ll check that. > > > 3. How to handle long OVS OF updates, which block inactivity probe for OF > > connection and then got dropped? From ovn-controller it’s possible to > configure inactivity probe, is it possible and should I configure from > ovs-vswitchd side? > > The probe is always two-sides. As you mentioned you can disable the probe > from ovn-controller side. From OVS side, for my understanding it is > hardcoded to 60s. (Please confirm if your ovs-vsctl command override the > setting) > > > I guess, it’s hardcoded the default value. Here ( > https://github.com/openvswitch/ovs/blob/v3.1.0/vswitchd/bridge.c#L3930-L3933) > its overridden with configured controller from ovsdb. > > Also, I see that provided command (ovs-vsctl set-controller … > inactivity_probe=0) really disables probes from ovs to ovn-controller in my > tests. > > Thanks, > Han > > 4. There is a warning on try to release unknown interface, which was > > relased right before. Is it normal situation or also should be > investigated? > > > > I think it can be ignored. Maybe @Dumitru Ceara can correct me or > > comment here > > > Thanks > Numan > > > > For points 2-4 I still haven’t found the answer yet. If someone knows the > > answer, please share it. > > > > Thanks. > > Regards, > Vladislav Odintsov > > > > Regards, > Vladislav Odintsov > > > > 1: https://github.com/ovn-org/ovn/blob/v23.03.0/controller/ofctrl.c#L421 > 2: > > > https://github.com/ovn-org/ovn/blob/v23.03.0/controller/ovn-controller.c#L108 > > 3: https://github.com/ovn-org/ovn/blob/v23.03.0/lib/features.c#L85 > 4: https://github.com/ovn-org/ovn/blob/v23.03.0/controller/pinctrl.c#L3479 > 5: > > > https://patchwork.ozlabs.org/project/openvswitch/patch/[email protected]/ > > 6: > > > https://github.com/ovn-org/ovn/commit/b8af8549396e62d6523be18e104352e334825783 > > 7: https://github.com/openvswitch/ovs/blob/v3.1.0/vswitchd/bridge.c#L3860 > > Regards, > Vladislav Odintsov > > > > > > > Regards, > Vladislav Odintsov > > _______________________________________________ > dev mailing list > [email protected] <mailto:[email protected]> > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > > > Regards, > Vladislav Odintsov > > _______________________________________________ > dev mailing list > [email protected] > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > _______________________________________________ > dev mailing list > [email protected] <mailto:[email protected]> > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > > > Regards, > Vladislav Odintsov > > _______________________________________________ > dev mailing list > [email protected] <mailto:[email protected]> > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > > > Regards, > Vladislav Odintsov > > _______________________________________________ > dev mailing list > [email protected] > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > > > Regards, > Vladislav Odintsov > > _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
