Hi Numan, Thanks for the input. I did a fresh install again and used an older OVS 2.6 package.(not using the TOT). Things are working now.
Pranab On 2 February 2017 at 19:19, Numan Siddique <[email protected]> wrote: > > > On Thu, Feb 2, 2017 at 6:05 PM, pranab boruah <[email protected] > > wrote: > >> Numan, >> Sorry for making this thread too long. >> Kind of deadlock scenario here: >> The process listening on port 6641 is ovsdb-server(not NB db). >> > > This is strange. The default port should be 6640 I suppose if not > specified. > > > > >> Details: >> Start OVS: >> # /usr/share/openvswitch/scripts/ovs-ctl start --system-id="random" >> # netstat -putna | grep 6641 >> tcp 0 0 192.168.10.10:6641 0.0.0.0:* >> LISTEN 5118/ovsdb-server >> # ps -afx | grep ovsdb >> 5118 ? S< 0:00 \_ ovsdb-server /etc/openvswitch/conf.db >> -vconsole:emer -vsyslog:err -vfile:info >> --remote=punix:/var/run/openvswitch/db.sock >> --private-key=db:Open_vSwitch,SSL,private_key >> --certificate=db:Open_vSwitch,SSL,certificate >> --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --no-chdir >> --log-file=/var/log/openvswitch/ovsdb-server.log >> --pidfile=/var/run/openvswitch/ovsdb-server.pid --detach --monitor >> >> Start NB db >> # /usr/share/openvswitch/scripts/ovn-ctl restart_northd >> --db-sb-create-insecure-remote=yes --db-nb-create-insecure-remote=yes >> >> # netstat -putna | grep 6641 >> tcp 0 0 192.168.10.10:6641 0.0.0.0:* >> LISTEN 5118/ovsdb-server >> >> # ps -afx | grep ovsdb >> 5118 ? S< 0:00 \_ ovsdb-server /etc/openvswitch/conf.db >> -vconsole:emer -vsyslog:err -vfile:info >> --remote=punix:/var/run/openvswitch/db.sock >> --private-key=db:Open_vSwitch,SSL,private_key >> --certificate=db:Open_vSwitch,SSL,certificate >> --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --no-chdir >> --log-file=/var/log/openvswitch/ovsdb-server.log >> --pidfile=/var/run/openvswitch/ovsdb-server.pid --detach --monitor >> >> 5215 ? S 0:00 \_ ovsdb-server --detach --monitor >> -vconsole:off --log-file=/var/log/openvswitch/ovsdb-server-nb.log >> --remote=punix:/var/run/openvswitch/ovnnb_db.sock >> --pidfile=/var/run/openvswitch/ovnnb_db.pid >> --remote=db:OVN_Northbound,NB_Global,connections --unixctl=ovnnb_db.ctl >> --private-key=db:OVN_Northbound,SSL,private_key >> --certificate=db:OVN_Northbound,SSL,certificate >> --ca-cert=db:OVN_Northbound,SSL,ca_cert --remote=ptcp:6641:192.168.10.10 >> /etc/openvswitch/ovnnb_db.db >> >> 5223 ? S 0:00 \_ ovsdb-server --detach --monitor >> -vconsole:off --log-file=/var/log/openvswitch/ovsdb-server-sb.log >> --remote=punix:/var/run/openvswitch/ovnsb_db.sock >> --pidfile=/var/run/openvswitch/ovnsb_db.pid >> --remote=db:OVN_Southbound,SB_Global,connections --unixctl=ovnsb_db.ctl >> --private-key=db:OVN_Southbound,SSL,private_key >> --certificate=db:OVN_Southbound,SSL,certificate >> --ca-cert=db:OVN_Southbound,SSL,ca_cert --remote=ptcp:6642:192.168.10.10 >> /etc/openvswitch/ovnsb_db.db >> >> Northbound DB logs show the same error. >> >> Well I am not supposed to kill the ovsdb-server process. Am I ? >> >> Also when I kill the ovsdb-server process, the error goes away. But my >> doubt is we need the ovsdb-server process on the Controller node. Right? >> > > OVN as such doesn't expect ovs-vswitchd (and ovsdb-server for ovs db) > > running in controller node. May be you can stop the ovs-vswitchd service > if you don't need it in the controller. > Also you can configure the NB and SB db ports to some other values if you > wish so and update the new values in appropriate locations. > Glad that you figured out the issue. > > Numan > > > >> Thanks for your patience. >> >> Pranab >> >> >> >> >> On 2 February 2017 at 17:45, Numan Siddique <[email protected]> wrote: >> >>> In the ovsdb logs I see >>> >>> 2017-02-02T11:07:12.366Z|00020|socket_util|ERR|6641:192.168.10.10: >>> bind: Address already in use >>> 2017-02-02T11:07:14.869Z|00021|socket_util|ERR|6641:192.168.10.10: >>> bind: Address already in use >>> 2017-02-02T11:07:16.182Z|00022|socket_util|ERR|6641:192.168.10.10: >>> bind: Address already in use >>> 2017-02-02T11:07:16.183Z|00023|socket_util|ERR|6641:192.168.10.10: >>> bind: Address already in use >>> >>> Probably you can use netstat and see which processes are listening on >>> port 6641, kill those processes and restart ovn-northd service and see if >>> it works. >>> >>> Thanks >>> Numan >>> >>> >>> On Thu, Feb 2, 2017 at 5:26 PM, pranab boruah < >>> [email protected]> wrote: >>> >>>> NB db is not listening on the port 6641: >>>> # ovn-nbctl --db=tcp:192.168.10.10:6641 show >>>> 2017-02-02T11:35:10Z|00001|ovsdb_idl|ERR|tcp:192.168.10.10:6641: >>>> requested schema not found >>>> ovn-nbctl: tcp:192.168.10.10:6641: database connection failed (No such >>>> file or directory) >>>> >>>> # ovn-sbctl --db=tcp:192.168.10.10:6642 show >>>> Chassis "e8968c35-11db-4b3b-ab24-c20866ac3dc8" >>>> hostname: "compute-447" >>>> Encap geneve >>>> ip: "172.16.10.11" >>>> options: {csum="true"} >>>> Encap vxlan >>>> ip: "172.16.10.11" >>>> options: {csum="true"} >>>> Chassis "9f20e7bd-5e16-4273-8ce8-96d0e0a12047" >>>> hostname: "compute-397" >>>> Encap geneve >>>> ip: "172.16.10.12" >>>> options: {csum="true"} >>>> Encap vxlan >>>> ip: "172.16.10.12" >>>> options: {csum="true"} >>>> >>>> # ovsdb-client dump unix:/var/run/openvswitch/ovnnb_db.sock Connection >>>> Connection table >>>> _uuid external_ids inactivity_probe >>>> is_connected max_backoff other_config status >>>> target >>>> ------------------------------------ ------------ ---------------- >>>> ------------ ----------- ------------ --------------- >>>> ------------------------- >>>> c110ffff-5ba2-4125-8a51-3c3db583f267 {} [] >>>> false [] {} {state=BACKOFF} "ptcp: >>>> 192.168.10.10:6641" >>>> >>>> # ovsdb-client dump unix:/var/run/openvswitch/ovnsb_db.sock Connection >>>> Connection table >>>> _uuid external_ids inactivity_probe >>>> is_connected max_backoff other_config read_only status >>>> target >>>> ------------------------------------ ------------ ---------------- >>>> ------------ ----------- ------------ --------- --------------- >>>> ------------------------- >>>> 1beb9cba-cc01-4981-8a67-a8b265326df9 {} [] >>>> false [] {} false {state=BACKOFF} "ptcp: >>>> 192.168.10.10:6642" >>>> >>>> No change after restarting Controller node and restating Northd as >>>> mentioned here: >>>> https://github.com/openvswitch/ovs/commit/84d0ca5d00fe01b291 >>>> 63236d48fa0f9105687149 >>>> >>>> Pranab >>>> >>>> >>>> >>>> >>>> >>>> On 2 February 2017 at 17:00, Numan Siddique <[email protected]> >>>> wrote: >>>> >>>>> >>>>> >>>>> On Thu, Feb 2, 2017 at 4:44 PM, pranab boruah < >>>>> [email protected]> wrote: >>>>> >>>>>> Numan, >>>>>> >>>>>> Thanks, but, the above commands didn't changed anything. Checked the >>>>>> logs and found this: >>>>>> >>>>>> >>>>> >>>>> Could you run the below two commands and see how it goes >>>>> >>>>> - "ovn-nbctl --db=tcp:192.168.10.10:6641 show" -> See if this >>>>> command is successful or not. If not then ovsdb-server is either not >>>>> listening on this port or it may be possible that you need to add iptables >>>>> rule to allow this port. >>>>> >>>>> - >>>>> "ovn- >>>>> s >>>>> bctl --db=tcp:192.168.10.10:664 >>>>> 2 >>>>> show" >>>>> - >>>>> >>>>> >>>>> On the controller node where you are running the ovsdb-server's run >>>>> - "sudo ovsdb-client dump unix:/var/run/openvswitch/ovnnb_db.sock >>>>> Connection" >>>>> - "sudo ovsdb-client dump unix:/var/run/openvswitch/ovnsb_db.sock >>>>> Connection" >>>>> >>>>> If the Connection table is empty, it means ovsdb-server is not >>>>> listening on the tcp ports. >>>>> >>>>> Thanks >>>>> Numan >>>>> >>>>> >>>>> LOG of NB database: >>>>>> >>>>>> 2017-02-02T11:07:12.366Z|00020|socket_util|ERR|6641:192.168.10.10: >>>>>> bind: Address already in use >>>>>> 2017-02-02T11:07:14.869Z|00021|socket_util|ERR|6641:192.168.10.10: >>>>>> bind: Address already in use >>>>>> 2017-02-02T11:07:16.182Z|00022|socket_util|ERR|6641:192.168.10.10: >>>>>> bind: Address already in use >>>>>> 2017-02-02T11:07:16.183Z|00023|socket_util|ERR|6641:192.168.10.10: >>>>>> bind: Address already in use >>>>>> 2017-02-02T11:07:16.184Z|00024|socket_util|ERR|6641:192.168.10.10: >>>>>> bind: Address already in use >>>>>> 2017-02-02T11:09:55.403Z|00025|ovsdb_jsonrpc_server|INFO|ptcp:6642: >>>>>> 192.168.10.10: remote deconfigured >>>>>> 2017-02-02T11:09:55.403Z|00026|socket_util|ERR|6641:192.168.10.10: >>>>>> bind: Address already in use >>>>>> 2017-02-02T11:09:55.403Z|00027|ovsdb_jsonrpc_server|ERR|Dropped 10 >>>>>> log messages in last 176 seconds (most recently, 159 seconds ago) due to >>>>>> excessive rate >>>>>> 2017-02-02T11:09:55.403Z|00028|ovsdb_jsonrpc_server|ERR|ptcp:6641: >>>>>> 192.168.10.10: listen failed: Address already in use >>>>>> 2017-02-02T11:09:55.403Z|00029|socket_util|ERR|6641:192.168.10.10: >>>>>> bind: Address already in use >>>>>> 2017-02-02T11:09:55.403Z|00030|ovsdb_jsonrpc_server|ERR|ptcp:6641: >>>>>> 192.168.10.10: listen failed: Address already in use >>>>>> 2017-02-02T11:09:57.460Z|00031|socket_util|ERR|6641:192.168.10.10: >>>>>> bind: Address already in use >>>>>> 2017-02-02T11:09:57.460Z|00032|ovsdb_jsonrpc_server|ERR|ptcp:6641: >>>>>> 192.168.10.10: listen failed: Address already in use >>>>>> 2017-02-02T11:09:57.460Z|00033|socket_util|ERR|6641:192.168.10.10: >>>>>> bind: Address already in use >>>>>> 2017-02-02T11:09:59.961Z|00034|socket_util|ERR|6641:192.168.10.10: >>>>>> bind: Address already in use >>>>>> 2017-02-02T11:10:02.464Z|00035|socket_util|ERR|6641:192.168.10.10: >>>>>> bind: Address already in use >>>>>> 2017-02-02T11:10:04.967Z|00036|socket_util|ERR|6641:192.168.10.10: >>>>>> bind: Address already in use >>>>>> 2017-02-02T11:10:07.469Z|00037|socket_util|ERR|6641:192.168.10.10: >>>>>> bind: Address already in use >>>>>> 2017-02-02T11:10:09.972Z|00038|socket_util|ERR|6641:192.168.10.10: >>>>>> bind: Address already in use >>>>>> 2017-02-02T11:10:12.475Z|00039|socket_util|ERR|6641:192.168.10.10: >>>>>> bind: Address already in use >>>>>> 2017-02-02T11:10:14.978Z|00040|socket_util|ERR|6641:192.168.10.10: >>>>>> bind: Address already in use >>>>>> 2017-02-02T11:10:17.481Z|00041|socket_util|ERR|6641:192.168.10.10: >>>>>> bind: Address already in use >>>>>> >>>>>> -Pranab >>>>>> >>>>>> >>>>>> On 2 February 2017 at 16:36, Numan Siddique <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Feb 2, 2017 at 4:00 PM, pranab boruah < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Numan, >>>>>>>> Details as requested: >>>>>>>> >>>>>>>> ___________ >>>>>>>> ml2_conf.ini:| >>>>>>>> ``````````````````` >>>>>>>> ovn_nb_connection = tcp:192.168.10.10:6641 >>>>>>>> ovn_sb_connection = tcp:192.168.10.10:6642 >>>>>>>> ovn_l3_mode = True >>>>>>>> ovn_l3_scheduler = chance >>>>>>>> ovn_l3_admin_net_cidr = 169.254.128.0/30 >>>>>>>> ovn_native_dhcp = True >>>>>>>> >>>>>>>> Note: 192.168.10.10 is the management ip of controller. Firewall, >>>>>>>> NetworkManager and SElinux are disabled on all nodes. >>>>>>>> ____________________________ >>>>>>>> output of "ps -aef | grep ovsdb" : | >>>>>>>> ````````````````````````````````````````````````` >>>>>>>> root 21220 21219 0 Feb01 ? 00:00:44 ovsdb-server >>>>>>>> /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info >>>>>>>> --remote=punix:/var/run/openvswitch/db.sock >>>>>>>> --private-key=db:Open_vSwitch,SSL,private_key >>>>>>>> --certificate=db:Open_vSwitch,SSL,certificate >>>>>>>> --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --no-chdir >>>>>>>> --log-file=/var/log/openvswitch/ovsdb-server.log >>>>>>>> --pidfile=/var/run/openvswitch/ovsdb-server.pid --detach --monitor >>>>>>>> >>>>>>>> root 21324 21323 0 Feb01 ? 00:00:03 ovsdb-server >>>>>>>> --detach --monitor -vconsole:off >>>>>>>> --log-file=/var/log/openvswitch/ovsdb-server-nb.log >>>>>>>> --remote=punix:/var/run/openvswitch/ovnnb_db.sock >>>>>>>> --pidfile=/var/run/openvswitch/ovnnb_db.pid >>>>>>>> --remote=db:OVN_Northbound,NB_Global,connections >>>>>>>> --unixctl=ovnnb_db.ctl --private-key=db:OVN_Northbound,SSL,private_key >>>>>>>> --certificate=db:OVN_Northbound,SSL,certificate >>>>>>>> --ca-cert=db:OVN_Northbound,SSL,ca_cert >>>>>>>> /etc/openvswitch/ovnnb_db.db >>>>>>>> >>>>>>>> root 21332 21331 0 Feb01 ? 00:00:03 ovsdb-server >>>>>>>> --detach --monitor -vconsole:off >>>>>>>> --log-file=/var/log/openvswitch/ovsdb-server-sb.log >>>>>>>> --remote=punix:/var/run/openvswitch/ovnsb_db.sock >>>>>>>> --pidfile=/var/run/openvswitch/ovnsb_db.pid >>>>>>>> --remote=db:OVN_Southbound,SB_Global,connections >>>>>>>> --unixctl=ovnsb_db.ctl --private-key=db:OVN_Southbound,SSL,private_key >>>>>>>> --certificate=db:OVN_Southbound,SSL,certificate >>>>>>>> --ca-cert=db:OVN_Southbound,SSL,ca_cert >>>>>>>> /etc/openvswitch/ovnsb_db.db >>>>>>>> >>>>>>>> Note: NB and SB databases are supposed to be listening on a tcp >>>>>>>> socket. But it isn't. >>>>>>>> >>>>>>>> >>>>>>> Running the below commands should solve your problem >>>>>>> - ovn-nbctl set-connection ptcp:6641:192.168.10.10 >>>>>>> >>>>>>> - ovn- >>>>>>> s >>>>>>> bctl set-connection ptcp:664 >>>>>>> 2 >>>>>>> :192.168.10.10 >>>>>>> >>>>>>> L >>>>>>> et me know if you still face problems. >>>>>>> >>>>>>> The reason for this is this commit - >>>>>>> https://github.com/openvswitch/ovs/commit/84d0ca5d00fe01b291 >>>>>>> 63236d48fa0f9105687149 >>>>>>> >>>>>>> P >>>>>>> lease have a look at it. >>>>>>> >>>>>>> Thanks >>>>>>> Numan >>>>>>> >>>>>>> >>>>>>> >>>>>>>> _________________________ >>>>>>>> output of logs | >>>>>>>> ``````````````````````````````````````````` >>>>>>>> Neutron server log : >>>>>>>> - http://paste.openstack.org/show/597320/ >>>>>>>> ovs-nothd log: >>>>>>>> - http://paste.openstack.org/show/597322/ >>>>>>>> ovs-vswitchd log on Controller: >>>>>>>> - http://paste.openstack.org/show/597324/ >>>>>>>> _________________________________________ >>>>>>>> # ovn-nbctl and ovn-sbctl show no output. | >>>>>>>> ```````````````````````````````````````````````````````````` >>>>>>>> ``````````` >>>>>>>> ________________________________________ >>>>>>>> OVN RPMS installation and download details: | >>>>>>>> ```````````````````````````````````````````````````````````` >>>>>>>> `````````` >>>>>>>> On controller: >>>>>>>> # git clone https://github.com/openvswitch/ovs.git >>>>>>>> # ./boot.sh >>>>>>>> # ./configure >>>>>>>> # make rpm-fedora RPMBUILD_OPT="--without check" >>>>>>>> # make rpm-fedora-kmod >>>>>>>> >>>>>>>> ALL nodes: >>>>>>>> >>>>>>>> # rpm -i openvswitch-kmod-2.6.90-1.el7.centos.x86_64.rpm# rpm -U >>>>>>>> openvswitch-2.6.90-1.el7.centos.x86_64.rpm >>>>>>>> >>>>>>>> Only Controller: >>>>>>>> >>>>>>>> # rpm -i openvswitch-ovn-common-*.x86_64.rpm# rpm -i >>>>>>>> openvswitch-ovn-central-*.x86_64.rpm >>>>>>>> >>>>>>>> Only Compute: >>>>>>>> >>>>>>>> # rpm -i openvswitch-ovn-common-*.x86_64.rpm# rpm -i >>>>>>>> openvswitch-ovn-host-*.x86_64.rpm >>>>>>>> >>>>>>>> # lsmod | grep open >>>>>>>> - shows all the required modules >>>>>>>> >>>>>>>> Let me know if you need anything else. >>>>>>>> >>>>>>>> Thanks. >>>>>>>> pjb >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 2 February 2017 at 15:23, Numan Siddique <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Adding openstack-dev ML with [networking-ovn] tag. >>>>>>>>> >>>>>>>>> Please see comments inline >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Feb 2, 2017 at 3:01 PM, pranab boruah < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi guys, >>>>>>>>>> We have a OpenStack multi-node lab. Everything was working, until >>>>>>>>>> I tried integrating with OVN. >>>>>>>>>> Followed the official guide for integration: >>>>>>>>>> >>>>>>>>>> http://docs.openstack.org/developer/networking-ovn/install.html >>>>>>>>>> >>>>>>>>>> Lab details: >>>>>>>>>> 1 Controller, 2 computes >>>>>>>>>> >>>>>>>>>> CentOS-7.3, OpenStack Newton, separate network for mgmt and >>>>>>>>>> tunnel. >>>>>>>>>> >>>>>>>>>> Error details : >>>>>>>>>> Neutron Server log shows : >>>>>>>>>> >>>>>>>>>> ERROR networking_ovn.ovsdb.impl_idl_ovn [-] OVS database >>>>>>>>>> connection to OVN_Northbound failed with error: '{u'error': u'unknown >>>>>>>>>> database', u'details': u'get_schema request specifies unknown >>>>>>>>>> database >>>>>>>>>> OVN_Northbound', u'syntax': u'["OVN_Northbound"]'}'. Verify that the >>>>>>>>>> OVS >>>>>>>>>> and OVN services are available and that the 'ovn_nb_connection' and >>>>>>>>>> 'ovn_sb_connection' configuration options are correct. >>>>>>>>>> >>>>>>>>>> Stuck in this for more than 2 week. Starting to hate my life. >>>>>>>>>> Please help. Before I tried installing in physical machines, >>>>>>>>>> followed the >>>>>>>>>> same guide and installed in VMs. Everything was working then. >>>>>>>>>> >>>>>>>>>> Solutions I tried: >>>>>>>>>> 1) Completely fresh installing everything. >>>>>>>>>> 2) Rebuilding ovs >>>>>>>>>> 3) Check SB and NB configuration options in plugin.ini are >>>>>>>>>> exactly correct. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> Could you please share >>>>>>>>> - the values of ovn_nb_connection and ovn_sb_connection defined >>>>>>>>> in ml2_conf.ini. >>>>>>>>> - the output of "ps -aef | grep ovsdb" on the node where you are >>>>>>>>> running the ovn-northd server. >>>>>>>>> - If you could share the output of neutron-server logs in some >>>>>>>>> paste link, it would be helpful. >>>>>>>>> >>>>>>>>> Are you able to run "ovn-nbctl show" ? I presume you have >>>>>>>>> installed the appropriate ovn RPMS ? If not so, please let me know >>>>>>>>> how you >>>>>>>>> have installed it. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> Numan >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> Let me know if additional details are required. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -pjb >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> discuss mailing list >>>>>>>>>> [email protected] >>>>>>>>>> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
