Re: [ovs-discuss] [OVN] no response to inactivity probe
Good to know, thanks! Tony > -Original Message- > From: Han Zhou > Sent: Friday, August 7, 2020 12:36 PM > To: Tony Liu > Cc: Han Zhou ; Numan Siddique ; ovs-dev > ; ovs-discuss > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe > > The raft probe is disabled if you use the latest version of OVS, e.g. > 2.13.1. > > > On Fri, Aug 7, 2020 at 12:28 PM Tony Liu <mailto:tonyliu0...@hotmail.com> > wrote: > > > Another one here, there is inactivity probe on the raft cluster > port. > > 2020-08-07T19:04:53.184Z|02735|reconnect|ERR|tcp:10.6.20.85:6644 > <http://10.6.20.85:6644> : no response to inactivity probe after 5 > seconds, disconnecting > 2020-08-07T19:04:53.184Z|02736|reconnect|INFO|tcp:10.6.20.85:6644 > <http://10.6.20.85:6644> : connection dropped > 2020-08-07T19:04:54.185Z|02737|reconnect|INFO|tcp:10.6.20.85:6644 > <http://10.6.20.85:6644> : connecting... > 2020-08-07T19:04:54.185Z|02738|reconnect|INFO|tcp:10.6.20.85:6644 > <http://10.6.20.85:6644> : connected > 2020-08-07T19:15:26.228Z|02739|reconnect|ERR|tcp:10.6.20.84:49440 > <http://10.6.20.84:49440> : no response to inactivity probe after 5 > seconds, disconnecting > 2020-08-07T19:15:26.769Z|02740|reconnect|ERR|tcp:10.6.20.84:6644 > <http://10.6.20.84:6644> : no response to inactivity probe after 5 > seconds, disconnecting > 2020-08-07T19:15:26.769Z|02741|reconnect|INFO|tcp:10.6.20.84:6644 > <http://10.6.20.84:6644> : connection dropped > 2020-08-07T19:15:27.771Z|02742|reconnect|INFO|tcp:10.6.20.84:6644 > <http://10.6.20.84:6644> : connecting... > 2020-08-07T19:15:27.771Z|02743|reconnect|INFO|tcp:10.6.20.84:6644 > <http://10.6.20.84:6644> : connected > > Which configuration is for that probe interval? > > > Thanks! > > Tony > > > -Original Message- > > From: dev mailto:ovs-dev- > boun...@openvswitch.org> > On Behalf Of Tony Liu > > Sent: Thursday, August 6, 2020 7:45 PM > > To: Han Zhou mailto:hz...@ovn.org> >; Numan > Siddique mailto:num...@ovn.org> > > > Cc: ovs-dev mailto:ovs- > d...@openvswitch.org> >; ovs-discuss> disc...@openvswitch.org <mailto:disc...@openvswitch.org> > > > Subject: Re: [ovs-dev] [ovs-discuss] [OVN] no response to > inactivity > > probe > > > > Hi Han and Numan, > > > > I'd like to have a few more clarifications. > > > > For inactivity probe: > > From ovn-controller to ovn-sb-db: ovn-remote-probe-interval > > > > From ovn-controller to ovs-vswitchd: ovn-openflow-probe-interval > > > > From ovn-controller to local ovsdb: which interval? > > > > From local ovsdb to ovn-controller: which interval? > > > > From ovs-vswitchd to ovn-controller: which interval? > > > > > > Regarding to the connection between ovn-controller and local > ovsdb- > > server, I recall that UNIX socket is lighter than TCP socket and > UNIX > > socket is recommended for local communication. > > Is that right? > > > > > > Thanks! > > > > Tony > > > > > -Original Message- > > > From: Han Zhou mailto:hz...@ovn.org> > > > > Sent: Thursday, August 6, 2020 12:42 PM > > > To: Tony Liu <mailto:tonyliu0...@hotmail.com> > > > > Cc: Han Zhou mailto:hz...@ovn.org> >; Numan > Siddique mailto:num...@ovn.org> >; ovs-dev > > > mailto:ovs-...@openvswitch.org> >; > ovs-discuss mailto:ovs- > disc...@openvswitch.org> > > > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity > probe > > > > > > > > > > > > On Thu, Aug 6, 2020 at 12:07 PM Tony Liu > mailto:tonyliu0...@hotmail.com> > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > wrote: > > > > > > > > Inline... > > > > > > > > Thanks! > > > > > > > > Tony > > > > > -Original Message- > > > > > From: Han Zhou mailto:hz...@ovn.org> > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > > > > > &
Re: [ovs-discuss] [OVN] no response to inactivity probe
Another one here, there is inactivity probe on the raft cluster port. 2020-08-07T19:04:53.184Z|02735|reconnect|ERR|tcp:10.6.20.85:6644: no response to inactivity probe after 5 seconds, disconnecting 2020-08-07T19:04:53.184Z|02736|reconnect|INFO|tcp:10.6.20.85:6644: connection dropped 2020-08-07T19:04:54.185Z|02737|reconnect|INFO|tcp:10.6.20.85:6644: connecting... 2020-08-07T19:04:54.185Z|02738|reconnect|INFO|tcp:10.6.20.85:6644: connected 2020-08-07T19:15:26.228Z|02739|reconnect|ERR|tcp:10.6.20.84:49440: no response to inactivity probe after 5 seconds, disconnecting 2020-08-07T19:15:26.769Z|02740|reconnect|ERR|tcp:10.6.20.84:6644: no response to inactivity probe after 5 seconds, disconnecting 2020-08-07T19:15:26.769Z|02741|reconnect|INFO|tcp:10.6.20.84:6644: connection dropped 2020-08-07T19:15:27.771Z|02742|reconnect|INFO|tcp:10.6.20.84:6644: connecting... 2020-08-07T19:15:27.771Z|02743|reconnect|INFO|tcp:10.6.20.84:6644: connected Which configuration is for that probe interval? Thanks! Tony > -Original Message- > From: dev On Behalf Of Tony Liu > Sent: Thursday, August 6, 2020 7:45 PM > To: Han Zhou ; Numan Siddique > Cc: ovs-dev ; ovs-discuss disc...@openvswitch.org> > Subject: Re: [ovs-dev] [ovs-discuss] [OVN] no response to inactivity > probe > > Hi Han and Numan, > > I'd like to have a few more clarifications. > > For inactivity probe: > From ovn-controller to ovn-sb-db: ovn-remote-probe-interval > > From ovn-controller to ovs-vswitchd: ovn-openflow-probe-interval > > From ovn-controller to local ovsdb: which interval? > > From local ovsdb to ovn-controller: which interval? > > From ovs-vswitchd to ovn-controller: which interval? > > > Regarding to the connection between ovn-controller and local ovsdb- > server, I recall that UNIX socket is lighter than TCP socket and UNIX > socket is recommended for local communication. > Is that right? > > > Thanks! > > Tony > > > -Original Message- > > From: Han Zhou > > Sent: Thursday, August 6, 2020 12:42 PM > > To: Tony Liu > > Cc: Han Zhou ; Numan Siddique ; ovs-dev > > ; ovs-discuss > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe > > > > > > > > On Thu, Aug 6, 2020 at 12:07 PM Tony Liu > <mailto:tonyliu0...@hotmail.com> > wrote: > > > > > > Inline... > > > > > > Thanks! > > > > > > Tony > > > > -Original Message- > > > > From: Han Zhou mailto:hz...@ovn.org> > > > > > Sent: Thursday, August 6, 2020 11:37 AM > > > > To: Tony Liu > > > <mailto:tonyliu0...@hotmail.com> > > > > > Cc: Han Zhou mailto:hz...@ovn.org> >; Numan > > > > Siddique mailto:num...@ovn.org> >; ovs-dev > > > > mailto:ovs-...@openvswitch.org> >; > > > > ovs-discuss > > > <mailto:ovs-discuss@openvswitch.org> > > > > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe > > > > > > > > > > > > > > > > On Thu, Aug 6, 2020 at 11:11 AM Tony Liu > > > <mailto:tonyliu0...@hotmail.com> <mailto:tonyliu0...@hotmail.com > > <mailto:tonyliu0...@hotmail.com> > > wrote: > > > > > > > > > > Inline... (please read with monospaced font:)) > > > > > > > > > > Thanks! > > > > > > > > > > Tony > > > > > > -Original Message- > > > > > > From: Han Zhou mailto:hz...@ovn.org> > > > > > > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > > > > > > > > Sent: Wednesday, August 5, 2020 11:48 PM > > > > > > To: Tony Liu > > > > > <mailto:tonyliu0...@hotmail.com> > > > > > > <mailto:tonyliu0...@hotmail.com > > > > > > <mailto:tonyliu0...@hotmail.com> > > > > > > > > Cc: Han Zhou mailto:hz...@ovn.org> > > > > > > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > >; Numan > > > > > > Siddique mailto:num...@ovn.org> > > > > > > <mailto:num...@ovn.org <mailto:num...@ovn.org> > >; ovs-dev > > > > > > mailto:ovs-...@openvswitch.org> > > > > > > <mailto:ovs-...@openvswitch.org > > > > > > <mailto:ovs-...@openvswitch.org> > > > > > > > >; ovs-discuss > > > > > <mailto:ovs-discuss@openvswitch.org> > > > > > > <mailto:ovs-discu
Re: [ovs-discuss] [OVN] no response to inactivity probe
Hi,
There are still some connection errors from ovn-controller.
Is that connection drop will cause flows to be deleted from vswitchd?
..
2020-08-07T03:55:22.269Z|03988|jsonrpc|WARN|tcp:127.0.0.1:6640: send error:
Broken pipe
..
2020-08-07T03:55:31.551Z|03996|reconnect|WARN|tcp:127.0.0.1:6640: connection
dropped (Broken pipe)
2020-08-07T03:55:22.268Z|03986|poll_loop|INFO|wakeup due to [POLLIN] on fd 14
(127.0.0.1:49514<->127.0.0.1:6640) at lib/stream-fd.c:157 (99% CPU usage)
2020-08-07T03:55:22.268Z|03987|poll_loop|INFO|wakeup due to [POLLIN] on fd 19
(10.6.20.91:42854<->10.6.20.84:6642) at lib/stream-fd.c:157 (99% CPU usage)
2020-08-07T03:55:22.269Z|03988|jsonrpc|WARN|tcp:127.0.0.1:6640: send error:
Broken pipe
2020-08-07T03:55:31.549Z|03989|timeval|WARN|Unreasonably long 9280ms poll
interval (9220ms user, 1ms system)
2020-08-07T03:55:31.550Z|03990|timeval|WARN|disk: 0 reads, 8 writes
2020-08-07T03:55:31.550Z|03991|timeval|WARN|context switches: 0 voluntary, 5
involuntary
2020-08-07T03:55:31.550Z|03992|coverage|INFO|Dropped 4 log messages in last 47
seconds (most recently, 9 seconds ago) due to excessive rate
2020-08-07T03:55:31.551Z|03993|coverage|INFO|Skipping details of duplicate
event coverage for hash=824dd6ab
2020-08-07T03:55:31.551Z|03994|poll_loop|INFO|wakeup due to [POLLIN] on fd 20
(<->/var/run/openvswitch/br-int.mgmt) at lib/stream-fd.c:157 (100% CPU usage)
2020-08-07T03:55:31.551Z|03995|poll_loop|INFO|wakeup due to [POLLIN] on fd 19
(10.6.20.91:42854<->10.6.20.84:6642) at lib/stream-fd.c:157 (100% CPU usage)
2020-08-07T03:55:31.551Z|03996|reconnect|WARN|tcp:127.0.0.1:6640: connection
dropped (Broken pipe)
2020-08-07T03:55:31.552Z|03997|poll_loop|INFO|wakeup due to 0-ms timeout at
controller/ovn-controller.c:2123 (100% CPU usage)
2020-08-07T03:55:40.752Z|03998|timeval|WARN|Unreasonably long 9176ms poll
interval (9118ms user, 0ms system)
2020-08-07T03:55:40.752Z|03999|timeval|WARN|context switches: 0 voluntary, 7
involuntary
2020-08-07T03:55:40.753Z|04000|poll_loop|INFO|Dropped 2 log messages in last 10
seconds (most recently, 10 seconds ago) due to excessive rate
2020-08-07T03:55:40.753Z|04001|poll_loop|INFO|wakeup due to 0-ms timeout at
lib/reconnect.c:643 (99% CPU usage)
2020-08-07T03:55:40.754Z|04002|reconnect|INFO|tcp:127.0.0.1:6640: connecting...
2020-08-07T03:55:40.771Z|04003|reconnect|INFO|tcp:127.0.0.1:6640: connected
Thanks!
Tony
> -Original Message-
> From: discuss On Behalf Of Tony
> Liu
> Sent: Thursday, August 6, 2020 8:23 PM
> To: Han Zhou ; Numan Siddique
> Cc: ovs-dev ; ovs-discuss disc...@openvswitch.org>
> Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe
>
> Interesting...
>
> with this configuration on gateway (chassis) node,
> external_ids: {ovn-bridge-mappings="physnet1:br-ex", ovn-cms-
> options=enable-chassis-as-gw, ovn-encap-ip="10.6.30.91", ovn-encap-
> type=geneve, ovn-openflow-probe-interval="30", ovn-
> remote="tcp:10.6.20.84:6642,tcp:10.6.20.85:6642,tcp:10.6.20.86:6642",
> ovn-remote-probe-interval="3", system-id="gateway-1"}
>
>
> I still see error from ovn-controller.
>
> 2020-08-07T03:17:48.186Z|02737|reconnect|ERR|tcp:127.0.0.1:6640: no
> response to inactivity probe after 8.74 seconds, disconnecting
> That tcp:127.0.0.1:6640 is the connection between ovn-controller and
> local ovsdb-server.
>
> Any settings I missed?
>
>
> Thanks!
>
> Tony
> > -Original Message-
> > From: dev On Behalf Of Tony Liu
> > Sent: Thursday, August 6, 2020 7:45 PM
> > To: Han Zhou ; Numan Siddique
> > Cc: ovs-dev ; ovs-discuss > disc...@openvswitch.org>
> > Subject: Re: [ovs-dev] [ovs-discuss] [OVN] no response to inactivity
> > probe
> >
> > Hi Han and Numan,
> >
> > I'd like to have a few more clarifications.
> >
> > For inactivity probe:
> > From ovn-controller to ovn-sb-db: ovn-remote-probe-interval
> >
> > From ovn-controller to ovs-vswitchd: ovn-openflow-probe-interval
> >
> > From ovn-controller to local ovsdb: which interval?
> >
> > From local ovsdb to ovn-controller: which interval?
> >
> > From ovs-vswitchd to ovn-controller: which interval?
> >
> >
> > Regarding to the connection between ovn-controller and local ovsdb-
> > server, I recall that UNIX socket is lighter than TCP socket and UNIX
> > socket is recommended for local communication.
> > Is that right?
> >
> >
> > Thanks!
> >
> > Tony
> >
> > > -Original Message-
> > > From: Han Zhou
> > > Sent: Thursday, Augus
Re: [ovs-discuss] [OVN] no response to inactivity probe
Interesting...
with this configuration on gateway (chassis) node,
external_ids: {ovn-bridge-mappings="physnet1:br-ex",
ovn-cms-options=enable-chassis-as-gw, ovn-encap-ip="10.6.30.91",
ovn-encap-type=geneve, ovn-openflow-probe-interval="30",
ovn-remote="tcp:10.6.20.84:6642,tcp:10.6.20.85:6642,tcp:10.6.20.86:6642",
ovn-remote-probe-interval="3", system-id="gateway-1"}
I still see error from ovn-controller.
2020-08-07T03:17:48.186Z|02737|reconnect|ERR|tcp:127.0.0.1:6640: no response to
inactivity probe after 8.74 seconds, disconnecting
That tcp:127.0.0.1:6640 is the connection between ovn-controller
and local ovsdb-server.
Any settings I missed?
Thanks!
Tony
> -Original Message-
> From: dev On Behalf Of Tony Liu
> Sent: Thursday, August 6, 2020 7:45 PM
> To: Han Zhou ; Numan Siddique
> Cc: ovs-dev ; ovs-discuss disc...@openvswitch.org>
> Subject: Re: [ovs-dev] [ovs-discuss] [OVN] no response to inactivity
> probe
>
> Hi Han and Numan,
>
> I'd like to have a few more clarifications.
>
> For inactivity probe:
> From ovn-controller to ovn-sb-db: ovn-remote-probe-interval
>
> From ovn-controller to ovs-vswitchd: ovn-openflow-probe-interval
>
> From ovn-controller to local ovsdb: which interval?
>
> From local ovsdb to ovn-controller: which interval?
>
> From ovs-vswitchd to ovn-controller: which interval?
>
>
> Regarding to the connection between ovn-controller and local ovsdb-
> server, I recall that UNIX socket is lighter than TCP socket and UNIX
> socket is recommended for local communication.
> Is that right?
>
>
> Thanks!
>
> Tony
>
> > -Original Message-----
> > From: Han Zhou
> > Sent: Thursday, August 6, 2020 12:42 PM
> > To: Tony Liu
> > Cc: Han Zhou ; Numan Siddique ; ovs-dev
> > ; ovs-discuss
> > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe
> >
> >
> >
> > On Thu, Aug 6, 2020 at 12:07 PM Tony Liu > <mailto:tonyliu0...@hotmail.com> > wrote:
> > >
> > > Inline...
> > >
> > > Thanks!
> > >
> > > Tony
> > > > -Original Message-
> > > > From: Han Zhou mailto:hz...@ovn.org> >
> > > > Sent: Thursday, August 6, 2020 11:37 AM
> > > > To: Tony Liu > > > <mailto:tonyliu0...@hotmail.com> >
> > > > Cc: Han Zhou mailto:hz...@ovn.org> >; Numan
> > > > Siddique mailto:num...@ovn.org> >; ovs-dev
> > > > mailto:ovs-...@openvswitch.org> >;
> > > > ovs-discuss > > > <mailto:ovs-discuss@openvswitch.org> >
> > > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe
> > > >
> > > >
> > > >
> > > > On Thu, Aug 6, 2020 at 11:11 AM Tony Liu > > > <mailto:tonyliu0...@hotmail.com> <mailto:tonyliu0...@hotmail.com
> > <mailto:tonyliu0...@hotmail.com> > > wrote:
> > > > >
> > > > > Inline... (please read with monospaced font:))
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Tony
> > > > > > -Original Message-
> > > > > > From: Han Zhou mailto:hz...@ovn.org>
> > > > > > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > >
> > > > > > Sent: Wednesday, August 5, 2020 11:48 PM
> > > > > > To: Tony Liu > > > > > <mailto:tonyliu0...@hotmail.com>
> > > > > > <mailto:tonyliu0...@hotmail.com
> > > > > > <mailto:tonyliu0...@hotmail.com> > >
> > > > > > Cc: Han Zhou mailto:hz...@ovn.org>
> > > > > > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > >; Numan
> > > > > > Siddique mailto:num...@ovn.org>
> > > > > > <mailto:num...@ovn.org <mailto:num...@ovn.org> > >; ovs-dev
> > > > > > mailto:ovs-...@openvswitch.org>
> > > > > > <mailto:ovs-...@openvswitch.org
> > > > > > <mailto:ovs-...@openvswitch.org>
> > > > > > > >; ovs-discuss > > > > > <mailto:ovs-discuss@openvswitch.org>
> > > > > > <mailto:ovs-discuss@openvswitch.org
> > > > > > <mailto:ovs-discuss@openvswitch.org> > >
> > > > > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity
> > > > > > probe
> > > > > >
> >
Re: [ovs-discuss] [OVN] no response to inactivity probe
Hi Han and Numan, I'd like to have a few more clarifications. For inactivity probe: >From ovn-controller to ovn-sb-db: ovn-remote-probe-interval >From ovn-controller to ovs-vswitchd: ovn-openflow-probe-interval >From ovn-controller to local ovsdb: which interval? >From local ovsdb to ovn-controller: which interval? >From ovs-vswitchd to ovn-controller: which interval? Regarding to the connection between ovn-controller and local ovsdb-server, I recall that UNIX socket is lighter than TCP socket and UNIX socket is recommended for local communication. Is that right? Thanks! Tony > -Original Message- > From: Han Zhou > Sent: Thursday, August 6, 2020 12:42 PM > To: Tony Liu > Cc: Han Zhou ; Numan Siddique ; ovs-dev > ; ovs-discuss > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe > > > > On Thu, Aug 6, 2020 at 12:07 PM Tony Liu <mailto:tonyliu0...@hotmail.com> > wrote: > > > > Inline... > > > > Thanks! > > > > Tony > > > -Original Message- > > > From: Han Zhou mailto:hz...@ovn.org> > > > > Sent: Thursday, August 6, 2020 11:37 AM > > > To: Tony Liu > > <mailto:tonyliu0...@hotmail.com> > > > > Cc: Han Zhou mailto:hz...@ovn.org> >; Numan Siddique > > > mailto:num...@ovn.org> >; ovs-dev > > > mailto:ovs-...@openvswitch.org> >; > > > ovs-discuss > > <mailto:ovs-discuss@openvswitch.org> > > > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe > > > > > > > > > > > > On Thu, Aug 6, 2020 at 11:11 AM Tony Liu > > <mailto:tonyliu0...@hotmail.com> <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > wrote: > > > > > > > > Inline... (please read with monospaced font:)) > > > > > > > > Thanks! > > > > > > > > Tony > > > > > -Original Message- > > > > > From: Han Zhou mailto:hz...@ovn.org> > > > > > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > > > > > > > Sent: Wednesday, August 5, 2020 11:48 PM > > > > > To: Tony Liu > > > > <mailto:tonyliu0...@hotmail.com> <mailto:tonyliu0...@hotmail.com > > > > > <mailto:tonyliu0...@hotmail.com> > > > > > > > Cc: Han Zhou mailto:hz...@ovn.org> > > > > > <mailto:hz...@ovn.org <mailto:hz...@ovn.org> > >; Numan Siddique > > > > > mailto:num...@ovn.org> <mailto:num...@ovn.org > > > > > <mailto:num...@ovn.org> > >; ovs-dev > > > > <mailto:ovs-...@openvswitch.org> > > > > > <mailto:ovs-...@openvswitch.org <mailto:ovs-...@openvswitch.org> > > > > > > >; ovs-discuss > > > > <mailto:ovs-discuss@openvswitch.org> > > > > > <mailto:ovs-discuss@openvswitch.org > > > > > <mailto:ovs-discuss@openvswitch.org> > > > > > > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe > > > > > > > > > > > > > > > > > > > > On Wed, Aug 5, 2020 at 9:14 PM Tony Liu > > > > <mailto:tonyliu0...@hotmail.com> <mailto:tonyliu0...@hotmail.com > > > > > <mailto:tonyliu0...@hotmail.com> > > > > > > <mailto:tonyliu0...@hotmail.com <mailto:tonyliu0...@hotmail.com> > > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > > wrote: > > > > > > > > > > > > > > > I set the connection target="ptcp:6641:10.6.20.84" for > > > > > ovn-nb- > > > db > > > > > and "ptcp:6642:10.6.20.84" for ovn-sb-db. .84 is the first > > > node > > > > > of cluster. Also ovn-openflow-probe-interval=30 on compute > > > node. > > > > > It seems helping. Not that many connect/drop/reconnect in > > > logging. > > > > > That "commit failure" is also gone. > > > > > The issue I reported in another thread "packet drop" seems > > > gone. > > > > > And launching VM starts working. > > > > > > > > > > How should I set connection table for all ovn-nb-db and > > > > > ovn- > > > sb-db > > > > > nodes in the cluster to set inactivity_probe? > > > > > One row w
Re: [ovs-discuss] [OVN] no response to inactivity probe
On Thu, Aug 6, 2020 at 12:07 PM Tony Liu wrote: > > Inline... > > Thanks! > > Tony > > -Original Message- > > From: Han Zhou > > Sent: Thursday, August 6, 2020 11:37 AM > > To: Tony Liu > > Cc: Han Zhou ; Numan Siddique ; ovs-dev > > ; ovs-discuss > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe > > > > > > > > On Thu, Aug 6, 2020 at 11:11 AM Tony Liu > <mailto:tonyliu0...@hotmail.com> > wrote: > > > > > > Inline... (please read with monospaced font:)) > > > > > > Thanks! > > > > > > Tony > > > > -Original Message- > > > > From: Han Zhou mailto:hz...@ovn.org> > > > > > Sent: Wednesday, August 5, 2020 11:48 PM > > > > To: Tony Liu > > > <mailto:tonyliu0...@hotmail.com> > > > > > Cc: Han Zhou mailto:hz...@ovn.org> >; Numan Siddique > > > > mailto:num...@ovn.org> >; ovs-dev > > > > mailto:ovs-...@openvswitch.org> >; > > > > ovs-discuss > > > <mailto:ovs-discuss@openvswitch.org> > > > > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe > > > > > > > > > > > > > > > > On Wed, Aug 5, 2020 at 9:14 PM Tony Liu > > > <mailto:tonyliu0...@hotmail.com> <mailto:tonyliu0...@hotmail.com > > <mailto:tonyliu0...@hotmail.com> > > wrote: > > > > > > > > > > > > I set the connection target="ptcp:6641:10.6.20.84" for ovn-nb- > > db > > > > and "ptcp:6642:10.6.20.84" for ovn-sb-db. .84 is the first > > node > > > > of cluster. Also ovn-openflow-probe-interval=30 on compute > > node. > > > > It seems helping. Not that many connect/drop/reconnect in > > logging. > > > > That "commit failure" is also gone. > > > > The issue I reported in another thread "packet drop" seems > > gone. > > > > And launching VM starts working. > > > > > > > > How should I set connection table for all ovn-nb-db and ovn- > > sb-db > > > > nodes in the cluster to set inactivity_probe? > > > > One row with address 0.0.0.0 seems not working. > > > > > > > > You can simply use 0.0.0.0 in the connection table, but don't > > > > specify the same connection method on the command line when starting > > > > ovsdb- server for NB/SB DB. Otherwise, these are conflicting and > > > > that's why you saw "Address already in use" error. > > > > > > Could you share a bit details how it works? > > > I thought the row in connection table only tells nbdb and sbdb the > > > probe interval. Isn't that right? Does nbdb and sbdb also create > > > socket based on target column? > > > > > > > > > In --remote option of ovsdb-server, you can specify either a connection > > method directly, or specify the db,table,column which contains the > > connection information. > > Please see manpage ovsdb-server(1). > > Here is how one of those 3 nbdb nodes invoked. > > ovsdb-server -vconsole:off -vfile:info --log-file=/var/log/kolla/openvswitch/ovn-sb-db.log --remote=punix:/var/run/ovn/ovnsb_db.sock --pidfile=/run/ovn/ovnsb_db.pid --unixctl=/var/run/ovn/ovnsb_db.ctl --remote=db:OVN_Southbound,SB_Global,connections --private-key=db:OVN_Southbound,SSL,private_key --certificate=db:OVN_Southbound,SSL,certificate --ca-cert=db:OVN_Southbound,SSL,ca_cert --ssl-protocols=db:OVN_Southbound,SSL,ssl_protocols --ssl-ciphers=db:OVN_Southbound,SSL,ssl_ciphers --remote=ptcp:6642:10.6.20.84 /var/lib/openvswitch/ovn-sb/ov sb.db > > It creates UNIX and TCP sockets, and takes configuration from DB. > Does that look ok? > Given that, what the target column should be for all nodes of the cluster? > And whatever target is set, ovsdb-server will create socket, right? > Oh... Should I do "--remote=ptcp:6642:0.0.0.0"? Then I can set the same > in connection table, and it won't cause conflict? > If --remote and connection target are the same, whoever comes in later > will be ignored, right? > In coding, does ovsdb-server create a connection object for each of > --remote and connection target, or it's one single connection object > for both of them because method:port:address is the same? I'd expect > the single object. > --remote=ptcp:6642:10.6.20.84 should be removed from the command. You already specifies --remote=db:OVN_Southbound,SB_
Re: [ovs-discuss] [OVN] no response to inactivity probe
Inline... Thanks! Tony > -Original Message- > From: Han Zhou > Sent: Thursday, August 6, 2020 11:37 AM > To: Tony Liu > Cc: Han Zhou ; Numan Siddique ; ovs-dev > ; ovs-discuss > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe > > > > On Thu, Aug 6, 2020 at 11:11 AM Tony Liu <mailto:tonyliu0...@hotmail.com> > wrote: > > > > Inline... (please read with monospaced font:)) > > > > Thanks! > > > > Tony > > > -Original Message- > > > From: Han Zhou mailto:hz...@ovn.org> > > > > Sent: Wednesday, August 5, 2020 11:48 PM > > > To: Tony Liu > > <mailto:tonyliu0...@hotmail.com> > > > > Cc: Han Zhou mailto:hz...@ovn.org> >; Numan Siddique > > > mailto:num...@ovn.org> >; ovs-dev > > > mailto:ovs-...@openvswitch.org> >; > > > ovs-discuss > > <mailto:ovs-discuss@openvswitch.org> > > > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe > > > > > > > > > > > > On Wed, Aug 5, 2020 at 9:14 PM Tony Liu > > <mailto:tonyliu0...@hotmail.com> <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > wrote: > > > > > > > > > I set the connection target="ptcp:6641:10.6.20.84" for ovn-nb- > db > > > and "ptcp:6642:10.6.20.84" for ovn-sb-db. .84 is the first > node > > > of cluster. Also ovn-openflow-probe-interval=30 on compute > node. > > > It seems helping. Not that many connect/drop/reconnect in > logging. > > > That "commit failure" is also gone. > > > The issue I reported in another thread "packet drop" seems > gone. > > > And launching VM starts working. > > > > > > How should I set connection table for all ovn-nb-db and ovn- > sb-db > > > nodes in the cluster to set inactivity_probe? > > > One row with address 0.0.0.0 seems not working. > > > > > > You can simply use 0.0.0.0 in the connection table, but don't > > > specify the same connection method on the command line when starting > > > ovsdb- server for NB/SB DB. Otherwise, these are conflicting and > > > that's why you saw "Address already in use" error. > > > > Could you share a bit details how it works? > > I thought the row in connection table only tells nbdb and sbdb the > > probe interval. Isn't that right? Does nbdb and sbdb also create > > socket based on target column? > > > > > In --remote option of ovsdb-server, you can specify either a connection > method directly, or specify the db,table,column which contains the > connection information. > Please see manpage ovsdb-server(1). Here is how one of those 3 nbdb nodes invoked. ovsdb-server -vconsole:off -vfile:info --log-file=/var/log/kolla/openvswitch/ovn-sb-db.log --remote=punix:/var/run/ovn/ovnsb_db.sock --pidfile=/run/ovn/ovnsb_db.pid --unixctl=/var/run/ovn/ovnsb_db.ctl --remote=db:OVN_Southbound,SB_Global,connections --private-key=db:OVN_Southbound,SSL,private_key --certificate=db:OVN_Southbound,SSL,certificate --ca-cert=db:OVN_Southbound,SSL,ca_cert --ssl-protocols=db:OVN_Southbound,SSL,ssl_protocols --ssl-ciphers=db:OVN_Southbound,SSL,ssl_ciphers --remote=ptcp:6642:10.6.20.84 /var/lib/openvswitch/ovn-sb/ov sb.db It creates UNIX and TCP sockets, and takes configuration from DB. Does that look ok? Given that, what the target column should be for all nodes of the cluster? And whatever target is set, ovsdb-server will create socket, right? Oh... Should I do "--remote=ptcp:6642:0.0.0.0"? Then I can set the same in connection table, and it won't cause conflict? If --remote and connection target are the same, whoever comes in later will be ignored, right? In coding, does ovsdb-server create a connection object for each of --remote and connection target, or it's one single connection object for both of them because method:port:address is the same? I'd expect the single object. > > > Is "external_ids:ovn-remote-probe-interval" in ovsdb-server on > > > compute node for ovn-controller to probe ovn-sb-db? > > > > > > OVSDB probe is bidirectional, so you need to set this value, too, if > > > you don't want too many probes handled by the SB server. (setting > > > the connection table for SB only changes the server side). > > > > In that case, how do I set probe interval for ovn-controller? > > My understanding is that, ovn-controller reads configuration from > > ovsdb-server on the local compute n
Re: [ovs-discuss] [OVN] no response to inactivity probe
On Thu, Aug 6, 2020 at 11:11 AM Tony Liu wrote:
>
> Inline... (please read with monospaced font:))
>
> Thanks!
>
> Tony
> > -Original Message-
> > From: Han Zhou
> > Sent: Wednesday, August 5, 2020 11:48 PM
> > To: Tony Liu
> > Cc: Han Zhou ; Numan Siddique ; ovs-dev
> > ; ovs-discuss
> > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe
> >
> >
> >
> > On Wed, Aug 5, 2020 at 9:14 PM Tony Liu > <mailto:tonyliu0...@hotmail.com> > wrote:
> >
> >
> > I set the connection target="ptcp:6641:10.6.20.84" for ovn-nb-db
> > and "ptcp:6642:10.6.20.84" for ovn-sb-db. .84 is the first node
> > of cluster. Also ovn-openflow-probe-interval=30 on compute node.
> > It seems helping. Not that many connect/drop/reconnect in logging.
> > That "commit failure" is also gone.
> > The issue I reported in another thread "packet drop" seems gone.
> > And launching VM starts working.
> >
> > How should I set connection table for all ovn-nb-db and ovn-sb-db
> > nodes in the cluster to set inactivity_probe?
> > One row with address 0.0.0.0 seems not working.
> >
> > You can simply use 0.0.0.0 in the connection table, but don't specify
> > the same connection method on the command line when starting ovsdb-
> > server for NB/SB DB. Otherwise, these are conflicting and that's why you
> > saw "Address already in use" error.
>
> Could you share a bit details how it works?
> I thought the row in connection table only tells nbdb and sbdb the
> probe interval. Isn't that right? Does nbdb and sbdb also create
> socket based on target column?
>
In --remote option of ovsdb-server, you can specify either a connection
method directly, or specify the db,table,column which contains the
connection information.
Please see manpage ovsdb-server(1).
> >
> > Is "external_ids:ovn-remote-probe-interval" in ovsdb-server on
> > compute node for ovn-controller to probe ovn-sb-db?
> >
> > OVSDB probe is bidirectional, so you need to set this value, too, if you
> > don't want too many probes handled by the SB server. (setting the
> > connection table for SB only changes the server side).
>
> In that case, how do I set probe interval for ovn-controller?
> My understanding is that, ovn-controller reads configuration from
> ovsdb-server on the local compute node. Isn't that right?
>
The configuration you mentioned "external_ids:ovn-remote-probe-interval" is
exactly the way to set the ovn-controller -> SB probe interval.
(SB -> ovn-controller probe is set in the connection table of SB)
You are right that ovn-controller reads configuration from the local
ovsdb-server. This setting is in local ovsdb-server.
> > Is "external_ids:ovn-openflow-probe-interval" in ovsdb-server on
> > compute node for ovn-controller to probe ovsdb-server?
> >
> > It is for the OpenFlow connection between ovn-controller and ovs-
> > vswitchd, which is part of the OpenFlow protocol.
> >
> > What's probe interval for ovsdb-server to probe ovn-controller?
> >
> > The local ovsdb connection uses unix socket, which doesn't send probe by
> > default (if I remember correctly).
>
> Here is how ovsdb-server and ovn-controller is invoked on compute node.
>
> root 41129 0.0 0.0 157556 20532 ?SJul30 1:51
/usr/sbin/ovsdb-server /var/lib/openvswitch/conf.db -vconsole:emer
-vsyslog:err -vfile:info --remote=punix:/run/openvswitch/db.sock
--remote=ptcp:6640:127.0.0.1
--remote=db:Open_vSwitch,Open_vSwitch,manager_options
--log-file=/var/log/kolla/openvswitch/ovsdb-server.log --pidfile
>
> root 63775 55.9 0.4 1477796 1224324 ? Sl Aug04 1360:55
/usr/bin/ovn-controller --pidfile=/run/ovn/ovn-controller.pid
--log-file=/var/log/kolla/openvswitch/ovn-controller.log tcp:127.0.0.1:6640
>
> Is that OK? Or UNIX socket method is recommended for ovn-controller
> to connect to ovsdb-server?
If using TCP, by default it is 5s probe interval. I think it is better to
use unix socket. (but maybe it doesn't matter that much)
>
> Here is the configuration in open_vswitch table in ovsdb-server.
>
> external_ids: {ovn-encap-ip="10.6.30.22", ovn-encap-type=geneve,
ovn-openflow-probe-interval="30", ovn-remote="tcp:10.6.20.84:6642,tcp:
10.6.20.85:6642,tcp:10.6.20.86:6642", ovn-remote-probe-interval="6",
system-id="compute-3"}
>
> ovn-controller connects to ovsdb-server and reads this configuration,
> so it knows how to
Re: [ovs-discuss] [OVN] no response to inactivity probe
Inline... (please read with monospaced font:))
Thanks!
Tony
> -Original Message-
> From: Han Zhou
> Sent: Wednesday, August 5, 2020 11:48 PM
> To: Tony Liu
> Cc: Han Zhou ; Numan Siddique ; ovs-dev
> ; ovs-discuss
> Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe
>
>
>
> On Wed, Aug 5, 2020 at 9:14 PM Tony Liu <mailto:tonyliu0...@hotmail.com> > wrote:
>
>
> I set the connection target="ptcp:6641:10.6.20.84" for ovn-nb-db
> and "ptcp:6642:10.6.20.84" for ovn-sb-db. .84 is the first node
> of cluster. Also ovn-openflow-probe-interval=30 on compute node.
> It seems helping. Not that many connect/drop/reconnect in logging.
> That "commit failure" is also gone.
> The issue I reported in another thread "packet drop" seems gone.
> And launching VM starts working.
>
> How should I set connection table for all ovn-nb-db and ovn-sb-db
> nodes in the cluster to set inactivity_probe?
> One row with address 0.0.0.0 seems not working.
>
> You can simply use 0.0.0.0 in the connection table, but don't specify
> the same connection method on the command line when starting ovsdb-
> server for NB/SB DB. Otherwise, these are conflicting and that's why you
> saw "Address already in use" error.
Could you share a bit details how it works?
I thought the row in connection table only tells nbdb and sbdb the
probe interval. Isn't that right? Does nbdb and sbdb also create
socket based on target column?
>
> Is "external_ids:ovn-remote-probe-interval" in ovsdb-server on
> compute node for ovn-controller to probe ovn-sb-db?
>
> OVSDB probe is bidirectional, so you need to set this value, too, if you
> don't want too many probes handled by the SB server. (setting the
> connection table for SB only changes the server side).
In that case, how do I set probe interval for ovn-controller?
My understanding is that, ovn-controller reads configuration from
ovsdb-server on the local compute node. Isn't that right?
> Is "external_ids:ovn-openflow-probe-interval" in ovsdb-server on
> compute node for ovn-controller to probe ovsdb-server?
>
> It is for the OpenFlow connection between ovn-controller and ovs-
> vswitchd, which is part of the OpenFlow protocol.
>
> What's probe interval for ovsdb-server to probe ovn-controller?
>
> The local ovsdb connection uses unix socket, which doesn't send probe by
> default (if I remember correctly).
Here is how ovsdb-server and ovn-controller is invoked on compute node.
root 41129 0.0 0.0 157556 20532 ?SJul30 1:51
/usr/sbin/ovsdb-server /var/lib/openvswitch/conf.db -vconsole:emer -vsyslog:err
-vfile:info --remote=punix:/run/openvswitch/db.sock
--remote=ptcp:6640:127.0.0.1
--remote=db:Open_vSwitch,Open_vSwitch,manager_options
--log-file=/var/log/kolla/openvswitch/ovsdb-server.log --pidfile
root 63775 55.9 0.4 1477796 1224324 ? Sl Aug04 1360:55
/usr/bin/ovn-controller --pidfile=/run/ovn/ovn-controller.pid
--log-file=/var/log/kolla/openvswitch/ovn-controller.log tcp:127.0.0.1:6640
Is that OK? Or UNIX socket method is recommended for ovn-controller
to connect to ovsdb-server?
Here is the configuration in open_vswitch table in ovsdb-server.
external_ids: {ovn-encap-ip="10.6.30.22", ovn-encap-type=geneve,
ovn-openflow-probe-interval="30",
ovn-remote="tcp:10.6.20.84:6642,tcp:10.6.20.85:6642,tcp:10.6.20.86:6642",
ovn-remote-probe-interval="6", system-id="compute-3"}
ovn-controller connects to ovsdb-server and reads this configuration,
so it knows how to connect to all sbdb nodes, right?
If it's TCP between ovn-controller and ovsdb-server, is that probe
interval setting will also apply to the probe from ovn-controller to
ovsdb-server?
ovn-controller connects to ovs-vswitchd by UNIX socket to program
open-flow. ovs-vswitchd and ovsdb-server are connected by UNIX too.
So, is that ovn-openflow-probe-interval for the probe from ovn-controller
to ovs-vswitchd via UNIX?
As a summary for the probe setting,
+--+ driver configuration
| ovn-driver |
+--+
^|
|v
+--+ inactivity_probe in table "Connection"
| ovn-nb-db |
+--+
^|
|v
+--+ options:northd_probe_interval in table "NB_Global"
| ovn-northd | in nbdb.
+--+
^|
|v
+--+ inactivity_probe in table "Connection"
| ovn-sb-db |
+--+
^|
|v
++ in table "Open_vSwitch" in ovsdb-server
|ovn-control
Re: [ovs-discuss] [OVN] no response to inactivity probe
On Wed, Aug 5, 2020 at 9:14 PM Tony Liu wrote: > I set the connection target="ptcp:6641:10.6.20.84" for ovn-nb-db > and "ptcp:6642:10.6.20.84" for ovn-sb-db. .84 is the first node > of cluster. Also ovn-openflow-probe-interval=30 on compute node. > It seems helping. Not that many connect/drop/reconnect in logging. > That "commit failure" is also gone. > The issue I reported in another thread "packet drop" seems gone. > And launching VM starts working. > > How should I set connection table for all ovn-nb-db and ovn-sb-db > nodes in the cluster to set inactivity_probe? > One row with address 0.0.0.0 seems not working. > You can simply use 0.0.0.0 in the connection table, but don't specify the same connection method on the command line when starting ovsdb-server for NB/SB DB. Otherwise, these are conflicting and that's why you saw "Address already in use" error. > Is "external_ids:ovn-remote-probe-interval" in ovsdb-server on > compute node for ovn-controller to probe ovn-sb-db? > > OVSDB probe is bidirectional, so you need to set this value, too, if you don't want too many probes handled by the SB server. (setting the connection table for SB only changes the server side). > Is "external_ids:ovn-openflow-probe-interval" in ovsdb-server on > compute node for ovn-controller to probe ovsdb-server? > > It is for the OpenFlow connection between ovn-controller and ovs-vswitchd, which is part of the OpenFlow protocol. > What's probe interval for ovsdb-server to probe ovn-controller? > > The local ovsdb connection uses unix socket, which doesn't send probe by default (if I remember correctly). For ovn-controller, since it is implemented with incremental-processing, even if there are probes from openflow or local ovsdb, it doesn't matter. If there is no configuration change, ovn-controller simply replies the probe and there is no extra cost. > Thanks! > > Tony > > -Original Message- > > From: discuss On Behalf Of Tony > > Liu > > Sent: Wednesday, August 5, 2020 4:29 PM > > To: Han Zhou > > Cc: ovs-dev ; ovs-discuss > disc...@openvswitch.org> > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe > > > > Hi Han, > > > > After setting connection target="ptcp:6642:0.0.0.0" for ovn-sb-db, I see > > this error. > > > > 2020-08- > > 05T23:01:26.819Z|06799|ovsdb_jsonrpc_server|ERR|ptcp:6642:0.0.0.0: > > listen failed: Address already in use Anything I am missing > > here? > > > > > > Thanks! > > > > Tony > > > -Original Message- > > > From: Han Zhou > > > Sent: Tuesday, August 4, 2020 4:44 PM > > > To: Tony Liu > > > Cc: Numan Siddique ; Han Zhou ; ovs- > > > discuss ; ovs-dev > > > > > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe > > > > > > > > > > > > On Tue, Aug 4, 2020 at 2:50 PM Tony Liu > > <mailto:tonyliu0...@hotmail.com> > wrote: > > > > > > > > > Hi, > > > > > > Since I have 3 OVN DB nodes, should I add 3 rows in connection > > table > > > for the inactivity_probe? Or put 3 addresses into one row? > > > > > > "set-connection" set one row only, and there is no > "add-connection". > > > How should I add 3 rows into the table connection? > > > > > > > > > > > > > > > You only need to set one row. Try this command: > > > > > > ovn-nbctl -- --id=@conn_uuid create Connection > > > target="ptcp\:6641\:0.0.0.0" inactivity_probe=0 -- set NB_Global . > > > connections=@conn_uuid > > > > > > > > > > > > Thanks! > > > > > > Tony > > > > > > > -Original Message- > > > > From: Numan Siddique mailto:num...@ovn.org> > > > > > Sent: Tuesday, August 4, 2020 12:36 AM > > > > To: Tony Liu > > <mailto:tonyliu0...@hotmail.com> > > > > > Cc: ovs-discuss mailto:ovs- > > > disc...@openvswitch.org> >; ovs-dev > > > d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe > > > > > > > > > > > > > > > > On Tue, Aug 4, 2020 at 9:12 AM Tony Liu > > <mailto:tonyliu0...@hotmail.com> > > > > <mailto:tonyliu0...
Re: [ovs-discuss] [OVN] no response to inactivity probe
I set the connection target="ptcp:6641:10.6.20.84" for ovn-nb-db and "ptcp:6642:10.6.20.84" for ovn-sb-db. .84 is the first node of cluster. Also ovn-openflow-probe-interval=30 on compute node. It seems helping. Not that many connect/drop/reconnect in logging. That "commit failure" is also gone. The issue I reported in another thread "packet drop" seems gone. And launching VM starts working. How should I set connection table for all ovn-nb-db and ovn-sb-db nodes in the cluster to set inactivity_probe? One row with address 0.0.0.0 seems not working. Is "external_ids:ovn-remote-probe-interval" in ovsdb-server on compute node for ovn-controller to probe ovn-sb-db? Is "external_ids:ovn-openflow-probe-interval" in ovsdb-server on compute node for ovn-controller to probe ovsdb-server? What's probe interval for ovsdb-server to probe ovn-controller? Thanks! Tony > -Original Message- > From: discuss On Behalf Of Tony > Liu > Sent: Wednesday, August 5, 2020 4:29 PM > To: Han Zhou > Cc: ovs-dev ; ovs-discuss disc...@openvswitch.org> > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe > > Hi Han, > > After setting connection target="ptcp:6642:0.0.0.0" for ovn-sb-db, I see > this error. > > 2020-08- > 05T23:01:26.819Z|06799|ovsdb_jsonrpc_server|ERR|ptcp:6642:0.0.0.0: > listen failed: Address already in use Anything I am missing > here? > > > Thanks! > > Tony > > -Original Message- > > From: Han Zhou > > Sent: Tuesday, August 4, 2020 4:44 PM > > To: Tony Liu > > Cc: Numan Siddique ; Han Zhou ; ovs- > > discuss ; ovs-dev > > > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe > > > > > > > > On Tue, Aug 4, 2020 at 2:50 PM Tony Liu > <mailto:tonyliu0...@hotmail.com> > wrote: > > > > > > Hi, > > > > Since I have 3 OVN DB nodes, should I add 3 rows in connection > table > > for the inactivity_probe? Or put 3 addresses into one row? > > > > "set-connection" set one row only, and there is no "add-connection". > > How should I add 3 rows into the table connection? > > > > > > > > > > You only need to set one row. Try this command: > > > > ovn-nbctl -- --id=@conn_uuid create Connection > > target="ptcp\:6641\:0.0.0.0" inactivity_probe=0 -- set NB_Global . > > connections=@conn_uuid > > > > > > > > Thanks! > > > > Tony > > > > > -Original Message- > > > From: Numan Siddique mailto:num...@ovn.org> > > > > Sent: Tuesday, August 4, 2020 12:36 AM > > > To: Tony Liu > <mailto:tonyliu0...@hotmail.com> > > > > Cc: ovs-discuss mailto:ovs- > > disc...@openvswitch.org> >; ovs-dev > > d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe > > > > > > > > > > > > On Tue, Aug 4, 2020 at 9:12 AM Tony Liu > <mailto:tonyliu0...@hotmail.com> > > > <mailto:tonyliu0...@hotmail.com > > <mailto:tonyliu0...@hotmail.com> > > wrote: > > > > > > > > > In my deployment, on each Neutron server, there are 13 > > Neutron > > > server processes. > > > I see 12 of them (monitor, maintenance, RPC, API) connect > > to both > > > ovn-nb-db > > > and ovn-sb-db. With 3 Neutron server nodes, that's 36 OVSDB > > clients. > > > Is so many clients OK? > > > > > > Any suggestions how to figure out which side doesn't > > respond the > > > probe, > > > if it's bi-directional? I don't see any activities from > > logging, > > > other than > > > connect/drop and reconnect... > > > > > > BTW, please let me know if this is not the right place to > > discuss > > > Neutron OVN > > > ML2 driver. > > > > > > > > > Thanks! > > > > > > Tony > > > > > > > -Original Message- > > > > From: dev mailto:ovs- > > dev-boun...@openvswitch.org> <mailto:ovs-dev- <mailto:ovs-dev-> > > > boun...@openvswitch.org <mailto:boun...@openvswitch.org> > > On > > Behalf Of Tony Liu > &g
Re: [ovs-discuss] [OVN] no response to inactivity probe
Hi Han, After setting connection target="ptcp:6642:0.0.0.0" for ovn-sb-db, I see this error. 2020-08-05T23:01:26.819Z|06799|ovsdb_jsonrpc_server|ERR|ptcp:6642:0.0.0.0: listen failed: Address already in use Anything I am missing here? Thanks! Tony > -Original Message- > From: Han Zhou > Sent: Tuesday, August 4, 2020 4:44 PM > To: Tony Liu > Cc: Numan Siddique ; Han Zhou ; ovs- > discuss ; ovs-dev > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe > > > > On Tue, Aug 4, 2020 at 2:50 PM Tony Liu <mailto:tonyliu0...@hotmail.com> > wrote: > > > Hi, > > Since I have 3 OVN DB nodes, should I add 3 rows in connection > table > for the inactivity_probe? Or put 3 addresses into one row? > > "set-connection" set one row only, and there is no "add-connection". > How should I add 3 rows into the table connection? > > > > > You only need to set one row. Try this command: > > ovn-nbctl -- --id=@conn_uuid create Connection > target="ptcp\:6641\:0.0.0.0" inactivity_probe=0 -- set NB_Global . > connections=@conn_uuid > > > > Thanks! > > Tony > > > -Original Message- > > From: Numan Siddique mailto:num...@ovn.org> > > > Sent: Tuesday, August 4, 2020 12:36 AM > > To: Tony Liu <mailto:tonyliu0...@hotmail.com> > > > Cc: ovs-discuss mailto:ovs- > disc...@openvswitch.org> >; ovs-dev> d...@openvswitch.org <mailto:d...@openvswitch.org> > > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe > > > > > > > > On Tue, Aug 4, 2020 at 9:12 AM Tony Liu <mailto:tonyliu0...@hotmail.com> > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > wrote: > > > > > > In my deployment, on each Neutron server, there are 13 > Neutron > > server processes. > > I see 12 of them (monitor, maintenance, RPC, API) connect > to both > > ovn-nb-db > > and ovn-sb-db. With 3 Neutron server nodes, that's 36 OVSDB > clients. > > Is so many clients OK? > > > > Any suggestions how to figure out which side doesn't > respond the > > probe, > > if it's bi-directional? I don't see any activities from > logging, > > other than > > connect/drop and reconnect... > > > > BTW, please let me know if this is not the right place to > discuss > > Neutron OVN > > ML2 driver. > > > > > > Thanks! > > > > Tony > > > > > -Original Message- > > > From: dev mailto:ovs- > dev-boun...@openvswitch.org> <mailto:ovs-dev- <mailto:ovs-dev-> > > boun...@openvswitch.org <mailto:boun...@openvswitch.org> > > On > Behalf Of Tony Liu > > > Sent: Monday, August 3, 2020 7:45 PM > > > To: ovs-discuss mailto:ovs- > disc...@openvswitch.org> <mailto:ovs- <mailto:ovs-> > > disc...@openvswitch.org <mailto:disc...@openvswitch.org> > >; > ovs-dev> > d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > Subject: [ovs-dev] [OVN] no response to inactivity probe > > > > > > Hi, > > > > > > Neutron OVN ML2 driver was disconnected by ovn-nb-db. > There are > > many > > > error messages from ovn-nb-db leader. > > > > > > 2020-08- > 04T02:31:39.751Z|03138|reconnect|ERR|tcp:10.6.20.81:58620 > <http://10.6.20.81:58620> > > <http://10.6.20.81:58620> : no > > > response to inactivity probe after 5 seconds, > disconnecting > > > 2020-08- > 04T02:31:42.484Z|03139|reconnect|ERR|tcp:10.6.20.81:58300 > <http://10.6.20.81:58300> > > <http://10.6.20.81:58300> : no > > > response to inactivity probe after 5 seconds, > disconnecting > > > 2020-08- > 04T02:31:49.858Z|03140|reconnect|ERR|tcp:10.6.20.81:59582 > <http://10.6.20.81:59582> > > <http://10.6.20.81:59582> : no > > > respo
Re: [ovs-discuss] [OVN] no response to inactivity probe
In that case, I can use set-connection to set one row. Thanks! Tony > -Original Message- > From: Han Zhou > Sent: Tuesday, August 4, 2020 4:44 PM > To: Tony Liu > Cc: Numan Siddique ; Han Zhou ; ovs- > discuss ; ovs-dev > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe > > > > On Tue, Aug 4, 2020 at 2:50 PM Tony Liu <mailto:tonyliu0...@hotmail.com> > wrote: > > > Hi, > > Since I have 3 OVN DB nodes, should I add 3 rows in connection > table > for the inactivity_probe? Or put 3 addresses into one row? > > "set-connection" set one row only, and there is no "add-connection". > How should I add 3 rows into the table connection? > > > > > You only need to set one row. Try this command: > > ovn-nbctl -- --id=@conn_uuid create Connection > target="ptcp\:6641\:0.0.0.0" inactivity_probe=0 -- set NB_Global . > connections=@conn_uuid > > > > Thanks! > > Tony > > > -Original Message- > > From: Numan Siddique mailto:num...@ovn.org> > > > Sent: Tuesday, August 4, 2020 12:36 AM > > To: Tony Liu <mailto:tonyliu0...@hotmail.com> > > > Cc: ovs-discuss mailto:ovs- > disc...@openvswitch.org> >; ovs-dev> d...@openvswitch.org <mailto:d...@openvswitch.org> > > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe > > > > > > > > On Tue, Aug 4, 2020 at 9:12 AM Tony Liu <mailto:tonyliu0...@hotmail.com> > > <mailto:tonyliu0...@hotmail.com > <mailto:tonyliu0...@hotmail.com> > > wrote: > > > > > > In my deployment, on each Neutron server, there are 13 > Neutron > > server processes. > > I see 12 of them (monitor, maintenance, RPC, API) connect > to both > > ovn-nb-db > > and ovn-sb-db. With 3 Neutron server nodes, that's 36 OVSDB > clients. > > Is so many clients OK? > > > > Any suggestions how to figure out which side doesn't > respond the > > probe, > > if it's bi-directional? I don't see any activities from > logging, > > other than > > connect/drop and reconnect... > > > > BTW, please let me know if this is not the right place to > discuss > > Neutron OVN > > ML2 driver. > > > > > > Thanks! > > > > Tony > > > > > -Original Message- > > > From: dev mailto:ovs- > dev-boun...@openvswitch.org> <mailto:ovs-dev- <mailto:ovs-dev-> > > boun...@openvswitch.org <mailto:boun...@openvswitch.org> > > On > Behalf Of Tony Liu > > > Sent: Monday, August 3, 2020 7:45 PM > > > To: ovs-discuss mailto:ovs- > disc...@openvswitch.org> <mailto:ovs- <mailto:ovs-> > > disc...@openvswitch.org <mailto:disc...@openvswitch.org> > >; > ovs-dev> > d...@openvswitch.org <mailto:d...@openvswitch.org> > <mailto:d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > > Subject: [ovs-dev] [OVN] no response to inactivity probe > > > > > > Hi, > > > > > > Neutron OVN ML2 driver was disconnected by ovn-nb-db. > There are > > many > > > error messages from ovn-nb-db leader. > > > > > > 2020-08- > 04T02:31:39.751Z|03138|reconnect|ERR|tcp:10.6.20.81:58620 > <http://10.6.20.81:58620> > > <http://10.6.20.81:58620> : no > > > response to inactivity probe after 5 seconds, > disconnecting > > > 2020-08- > 04T02:31:42.484Z|03139|reconnect|ERR|tcp:10.6.20.81:58300 > <http://10.6.20.81:58300> > > <http://10.6.20.81:58300> : no > > > response to inactivity probe after 5 seconds, > disconnecting > > > 2020-08- > 04T02:31:49.858Z|03140|reconnect|ERR|tcp:10.6.20.81:59582 > <http://10.6.20.81:59582> > > <http://10.6.20.81:59582> : no > > > response to inactivity probe after 5 seconds, > disconnecting > > > 2020-08- > 04T02:31:53.057Z|03141|reconnect|ERR|tcp:10.6.20.83:42626 > <http://10.6.20.83:42626> >
Re: [ovs-discuss] [OVN] no response to inactivity probe
On Tue, Aug 4, 2020 at 2:50 PM Tony Liu wrote: > Hi, > > Since I have 3 OVN DB nodes, should I add 3 rows in connection table > for the inactivity_probe? Or put 3 addresses into one row? > > "set-connection" set one row only, and there is no "add-connection". > How should I add 3 rows into the table connection? > > You only need to set one row. Try this command: ovn-nbctl -- --id=@conn_uuid create Connection target="ptcp\:6641\:0.0.0.0" inactivity_probe=0 -- set NB_Global . connections=@conn_uuid > Thanks! > > Tony > > > -Original Message- > > From: Numan Siddique > > Sent: Tuesday, August 4, 2020 12:36 AM > > To: Tony Liu > > Cc: ovs-discuss ; ovs-dev > d...@openvswitch.org> > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe > > > > > > > > On Tue, Aug 4, 2020 at 9:12 AM Tony Liu > <mailto:tonyliu0...@hotmail.com> > wrote: > > > > > > In my deployment, on each Neutron server, there are 13 Neutron > > server processes. > > I see 12 of them (monitor, maintenance, RPC, API) connect to both > > ovn-nb-db > > and ovn-sb-db. With 3 Neutron server nodes, that's 36 OVSDB > clients. > > Is so many clients OK? > > > > Any suggestions how to figure out which side doesn't respond the > > probe, > > if it's bi-directional? I don't see any activities from logging, > > other than > > connect/drop and reconnect... > > > > BTW, please let me know if this is not the right place to discuss > > Neutron OVN > > ML2 driver. > > > > > > Thanks! > > > > Tony > > > > > -Original Message- > > > From: dev mailto:ovs-dev- > > boun...@openvswitch.org> > On Behalf Of Tony Liu > > > Sent: Monday, August 3, 2020 7:45 PM > > > To: ovs-discuss mailto:ovs- > > disc...@openvswitch.org> >; ovs-dev > > d...@openvswitch.org <mailto:d...@openvswitch.org> > > > > Subject: [ovs-dev] [OVN] no response to inactivity probe > > > > > > Hi, > > > > > > Neutron OVN ML2 driver was disconnected by ovn-nb-db. There are > > many > > > error messages from ovn-nb-db leader. > > > > > > 2020-08-04T02:31:39.751Z|03138|reconnect|ERR|tcp: > 10.6.20.81:58620 > > <http://10.6.20.81:58620> : no > > > response to inactivity probe after 5 seconds, disconnecting > > > 2020-08-04T02:31:42.484Z|03139|reconnect|ERR|tcp: > 10.6.20.81:58300 > > <http://10.6.20.81:58300> : no > > > response to inactivity probe after 5 seconds, disconnecting > > > 2020-08-04T02:31:49.858Z|03140|reconnect|ERR|tcp: > 10.6.20.81:59582 > > <http://10.6.20.81:59582> : no > > > response to inactivity probe after 5 seconds, disconnecting > > > 2020-08-04T02:31:53.057Z|03141|reconnect|ERR|tcp: > 10.6.20.83:42626 > > <http://10.6.20.83:42626> : no > > > response to inactivity probe after 5 seconds, disconnecting > > > 2020-08-04T02:31:53.058Z|03142|reconnect|ERR|tcp: > 10.6.20.82:45412 > > <http://10.6.20.82:45412> : no > > > response to inactivity probe after 5 seconds, disconnecting > > > 2020-08-04T02:31:54.067Z|03143|reconnect|ERR|tcp: > 10.6.20.81:59416 > > <http://10.6.20.81:59416> : no > > > response to inactivity probe after 5 seconds, disconnecting > > > 2020-08-04T02:31:54.809Z|03144|reconnect|ERR|tcp: > 10.6.20.81:60004 > > <http://10.6.20.81:60004> : no > > > response to inactivity probe after 5 seconds, disconnecting > > > > > > > > Could anyone share a bit details how this inactivity probe works? > > > > > > > > The inactivity probe is sent by both the server and clients > > independently. > > Meaning ovsdb-server will send an inactivity probe every 'x' configured > > seconds to all its connected clients and if it doesn't get a reply from > > the client within some time, it disconnects the connection. > > > > The inactivity probe from the server side can be configured. Run "ovn- > > nbctl list connection" > > and you will see inactivity_probe column. You can set this column to > > desired value like - ovn-nbctl set connection . inactivity_probe=3 > > (for 30 seconds) > > > > The same thing for SB o
Re: [ovs-discuss] [OVN] no response to inactivity probe
Hi, Since I have 3 OVN DB nodes, should I add 3 rows in connection table for the inactivity_probe? Or put 3 addresses into one row? "set-connection" set one row only, and there is no "add-connection". How should I add 3 rows into the table connection? Thanks! Tony > -Original Message- > From: Numan Siddique > Sent: Tuesday, August 4, 2020 12:36 AM > To: Tony Liu > Cc: ovs-discuss ; ovs-dev d...@openvswitch.org> > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe > > > > On Tue, Aug 4, 2020 at 9:12 AM Tony Liu <mailto:tonyliu0...@hotmail.com> > wrote: > > > In my deployment, on each Neutron server, there are 13 Neutron > server processes. > I see 12 of them (monitor, maintenance, RPC, API) connect to both > ovn-nb-db > and ovn-sb-db. With 3 Neutron server nodes, that's 36 OVSDB clients. > Is so many clients OK? > > Any suggestions how to figure out which side doesn't respond the > probe, > if it's bi-directional? I don't see any activities from logging, > other than > connect/drop and reconnect... > > BTW, please let me know if this is not the right place to discuss > Neutron OVN > ML2 driver. > > > Thanks! > > Tony > > > -Original Message- > > From: dev mailto:ovs-dev- > boun...@openvswitch.org> > On Behalf Of Tony Liu > > Sent: Monday, August 3, 2020 7:45 PM > > To: ovs-discuss mailto:ovs- > disc...@openvswitch.org> >; ovs-dev> d...@openvswitch.org <mailto:d...@openvswitch.org> > > > Subject: [ovs-dev] [OVN] no response to inactivity probe > > > > Hi, > > > > Neutron OVN ML2 driver was disconnected by ovn-nb-db. There are > many > > error messages from ovn-nb-db leader. > > > > 2020-08-04T02:31:39.751Z|03138|reconnect|ERR|tcp:10.6.20.81:58620 > <http://10.6.20.81:58620> : no > > response to inactivity probe after 5 seconds, disconnecting > > 2020-08-04T02:31:42.484Z|03139|reconnect|ERR|tcp:10.6.20.81:58300 > <http://10.6.20.81:58300> : no > > response to inactivity probe after 5 seconds, disconnecting > > 2020-08-04T02:31:49.858Z|03140|reconnect|ERR|tcp:10.6.20.81:59582 > <http://10.6.20.81:59582> : no > > response to inactivity probe after 5 seconds, disconnecting > > 2020-08-04T02:31:53.057Z|03141|reconnect|ERR|tcp:10.6.20.83:42626 > <http://10.6.20.83:42626> : no > > response to inactivity probe after 5 seconds, disconnecting > > 2020-08-04T02:31:53.058Z|03142|reconnect|ERR|tcp:10.6.20.82:45412 > <http://10.6.20.82:45412> : no > > response to inactivity probe after 5 seconds, disconnecting > > 2020-08-04T02:31:54.067Z|03143|reconnect|ERR|tcp:10.6.20.81:59416 > <http://10.6.20.81:59416> : no > > response to inactivity probe after 5 seconds, disconnecting > > 2020-08-04T02:31:54.809Z|03144|reconnect|ERR|tcp:10.6.20.81:60004 > <http://10.6.20.81:60004> : no > > response to inactivity probe after 5 seconds, disconnecting > > > > > Could anyone share a bit details how this inactivity probe works? > > > > The inactivity probe is sent by both the server and clients > independently. > Meaning ovsdb-server will send an inactivity probe every 'x' configured > seconds to all its connected clients and if it doesn't get a reply from > the client within some time, it disconnects the connection. > > The inactivity probe from the server side can be configured. Run "ovn- > nbctl list connection" > and you will see inactivity_probe column. You can set this column to > desired value like - ovn-nbctl set connection . inactivity_probe=3 > (for 30 seconds) > > The same thing for SB ovsdb-server. > > Similarly each client (ovn-northd, ovn-controller, neutron server) sends > inactivity probe every 'y' seconds and if the client doesn't get any > reply from ovsdb-server it will disconnect the connection and reconnect > again. > > For ovn-northd you can configured this as - ovn-nbctl set NB_Global . > options:northd_probe_interval=3 > > For ovn-controllers - ovs-vsctl set open . external_ids:ovn-remote- > probe-interval=3 > > There is also a probe interval for openflow connection from ovn- > controller to ovs-vswitchd which you can configure as ovs-vsctl set > open . external_ids:ovn-openflow-probe-interval=30 (this is in seconds) > > > Regarding the neutron server I think it is set to 60 seconds. Plea
Re: [ovs-discuss] [OVN] no response to inactivity probe
On Tue, Aug 4, 2020 at 9:12 AM Tony Liu wrote: > In my deployment, on each Neutron server, there are 13 Neutron server > processes. > I see 12 of them (monitor, maintenance, RPC, API) connect to both ovn-nb-db > and ovn-sb-db. With 3 Neutron server nodes, that's 36 OVSDB clients. > Is so many clients OK? > > Any suggestions how to figure out which side doesn't respond the probe, > if it's bi-directional? I don't see any activities from logging, other than > connect/drop and reconnect... > > BTW, please let me know if this is not the right place to discuss Neutron > OVN > ML2 driver. > > > Thanks! > > Tony > > > -Original Message- > > From: dev On Behalf Of Tony Liu > > Sent: Monday, August 3, 2020 7:45 PM > > To: ovs-discuss ; ovs-dev > d...@openvswitch.org> > > Subject: [ovs-dev] [OVN] no response to inactivity probe > > > > Hi, > > > > Neutron OVN ML2 driver was disconnected by ovn-nb-db. There are many > > error messages from ovn-nb-db leader. > > > > 2020-08-04T02:31:39.751Z|03138|reconnect|ERR|tcp:10.6.20.81:58620: no > > response to inactivity probe after 5 seconds, disconnecting > > 2020-08-04T02:31:42.484Z|03139|reconnect|ERR|tcp:10.6.20.81:58300: no > > response to inactivity probe after 5 seconds, disconnecting > > 2020-08-04T02:31:49.858Z|03140|reconnect|ERR|tcp:10.6.20.81:59582: no > > response to inactivity probe after 5 seconds, disconnecting > > 2020-08-04T02:31:53.057Z|03141|reconnect|ERR|tcp:10.6.20.83:42626: no > > response to inactivity probe after 5 seconds, disconnecting > > 2020-08-04T02:31:53.058Z|03142|reconnect|ERR|tcp:10.6.20.82:45412: no > > response to inactivity probe after 5 seconds, disconnecting > > 2020-08-04T02:31:54.067Z|03143|reconnect|ERR|tcp:10.6.20.81:59416: no > > response to inactivity probe after 5 seconds, disconnecting > > 2020-08-04T02:31:54.809Z|03144|reconnect|ERR|tcp:10.6.20.81:60004: no > > response to inactivity probe after 5 seconds, disconnecting > > > > Could anyone share a bit details how this inactivity probe works? > The inactivity probe is sent by both the server and clients independently. Meaning ovsdb-server will send an inactivity probe every 'x' configured seconds to all its connected clients and if it doesn't get a reply from the client within some time, it disconnects the connection. The inactivity probe from the server side can be configured. Run "ovn-nbctl list connection" and you will see inactivity_probe column. You can set this column to desired value like - ovn-nbctl set connection . inactivity_probe=3 (for 30 seconds) The same thing for SB ovsdb-server. Similarly each client (ovn-northd, ovn-controller, neutron server) sends inactivity probe every 'y' seconds and if the client doesn't get any reply from ovsdb-server it will disconnect the connection and reconnect again. For ovn-northd you can configured this as - ovn-nbctl set NB_Global . options:northd_probe_interval=3 For ovn-controllers - ovs-vsctl set open . external_ids:ovn-remote-probe-interval=3 There is also a probe interval for openflow connection from ovn-controller to ovs-vswitchd which you can configure as ovs-vsctl set open . external_ids:ovn-openflow-probe-interval=30 (this is in seconds) Regarding the neutron server I think it is set to 60 seconds. Please see this - https://github.com/openstack/neutron/blob/master/neutron/conf/plugins/ml2/drivers/ovn/ovn_conf.py#L80 >From the logs you shared, it looks like ovsdb-server is not getting the probe reply from neutron server after 5 seconds and hence it is disconnecting. Not sure what's happening though. You can try increasing the inactivity probe interval on the ovsdb-server side with the first command I shared. Note: If "ovn-nbctl list connection" returns empty, you need to create a connection row like - ovn-nbctl set-connection ptcp:6641: Thanks Numan > From OVN ML2 driver log, I see it connected to the leader, then the > > connection was closed by leader after 5 or 6 seconds. Is this probe one- > > way or two-ways? > > Both sides are not busy, not taking much CPU cycles. Not sure how this > > could happen. Any thoughts? > > > > > > Thanks! > > > > Tony > > > > > > > > ___ > > dev mailing list > > d...@openvswitch.org > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > ___ > discuss mailing list > disc...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > > ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] [OVN] no response to inactivity probe
In my deployment, on each Neutron server, there are 13 Neutron server processes. I see 12 of them (monitor, maintenance, RPC, API) connect to both ovn-nb-db and ovn-sb-db. With 3 Neutron server nodes, that's 36 OVSDB clients. Is so many clients OK? Any suggestions how to figure out which side doesn't respond the probe, if it's bi-directional? I don't see any activities from logging, other than connect/drop and reconnect... BTW, please let me know if this is not the right place to discuss Neutron OVN ML2 driver. Thanks! Tony > -Original Message- > From: dev On Behalf Of Tony Liu > Sent: Monday, August 3, 2020 7:45 PM > To: ovs-discuss ; ovs-dev d...@openvswitch.org> > Subject: [ovs-dev] [OVN] no response to inactivity probe > > Hi, > > Neutron OVN ML2 driver was disconnected by ovn-nb-db. There are many > error messages from ovn-nb-db leader. > > 2020-08-04T02:31:39.751Z|03138|reconnect|ERR|tcp:10.6.20.81:58620: no > response to inactivity probe after 5 seconds, disconnecting > 2020-08-04T02:31:42.484Z|03139|reconnect|ERR|tcp:10.6.20.81:58300: no > response to inactivity probe after 5 seconds, disconnecting > 2020-08-04T02:31:49.858Z|03140|reconnect|ERR|tcp:10.6.20.81:59582: no > response to inactivity probe after 5 seconds, disconnecting > 2020-08-04T02:31:53.057Z|03141|reconnect|ERR|tcp:10.6.20.83:42626: no > response to inactivity probe after 5 seconds, disconnecting > 2020-08-04T02:31:53.058Z|03142|reconnect|ERR|tcp:10.6.20.82:45412: no > response to inactivity probe after 5 seconds, disconnecting > 2020-08-04T02:31:54.067Z|03143|reconnect|ERR|tcp:10.6.20.81:59416: no > response to inactivity probe after 5 seconds, disconnecting > 2020-08-04T02:31:54.809Z|03144|reconnect|ERR|tcp:10.6.20.81:60004: no > response to inactivity probe after 5 seconds, disconnecting > > Could anyone share a bit details how this inactivity probe works? > From OVN ML2 driver log, I see it connected to the leader, then the > connection was closed by leader after 5 or 6 seconds. Is this probe one- > way or two-ways? > Both sides are not busy, not taking much CPU cycles. Not sure how this > could happen. Any thoughts? > > > Thanks! > > Tony > > > > ___ > dev mailing list > d...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-dev ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
[ovs-discuss] [OVN] no response to inactivity probe
Hi, Neutron OVN ML2 driver was disconnected by ovn-nb-db. There are many error messages from ovn-nb-db leader. 2020-08-04T02:31:39.751Z|03138|reconnect|ERR|tcp:10.6.20.81:58620: no response to inactivity probe after 5 seconds, disconnecting 2020-08-04T02:31:42.484Z|03139|reconnect|ERR|tcp:10.6.20.81:58300: no response to inactivity probe after 5 seconds, disconnecting 2020-08-04T02:31:49.858Z|03140|reconnect|ERR|tcp:10.6.20.81:59582: no response to inactivity probe after 5 seconds, disconnecting 2020-08-04T02:31:53.057Z|03141|reconnect|ERR|tcp:10.6.20.83:42626: no response to inactivity probe after 5 seconds, disconnecting 2020-08-04T02:31:53.058Z|03142|reconnect|ERR|tcp:10.6.20.82:45412: no response to inactivity probe after 5 seconds, disconnecting 2020-08-04T02:31:54.067Z|03143|reconnect|ERR|tcp:10.6.20.81:59416: no response to inactivity probe after 5 seconds, disconnecting 2020-08-04T02:31:54.809Z|03144|reconnect|ERR|tcp:10.6.20.81:60004: no response to inactivity probe after 5 seconds, disconnecting Could anyone share a bit details how this inactivity probe works? >From OVN ML2 driver log, I see it connected to the leader, then the connection was closed by leader after 5 or 6 seconds. Is this probe one-way or two-ways? Both sides are not busy, not taking much CPU cycles. Not sure how this could happen. Any thoughts? Thanks! Tony ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
