Numan, Sorry for making this thread too long. Kind of deadlock scenario here: The process listening on port 6641 is ovsdb-server(not NB db). Details: Start OVS: # /usr/share/openvswitch/scripts/ovs-ctl start --system-id="random" # netstat -putna | grep 6641 tcp 0 0 192.168.10.10:6641 0.0.0.0:* LISTEN 5118/ovsdb-server # ps -afx | grep ovsdb 5118 ? S< 0:00 \_ ovsdb-server /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info --remote=punix:/var/run/openvswitch/db.sock --private-key=db:Open_vSwitch,SSL,private_key --certificate=db:Open_vSwitch,SSL,certificate --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --no-chdir --log-file=/var/log/openvswitch/ovsdb-server.log --pidfile=/var/run/openvswitch/ovsdb-server.pid --detach --monitor
Start NB db # /usr/share/openvswitch/scripts/ovn-ctl restart_northd --db-sb-create-insecure-remote=yes --db-nb-create-insecure-remote=yes # netstat -putna | grep 6641 tcp 0 0 192.168.10.10:6641 0.0.0.0:* LISTEN 5118/ovsdb-server # ps -afx | grep ovsdb 5118 ? S< 0:00 \_ ovsdb-server /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info --remote=punix:/var/run/openvswitch/db.sock --private-key=db:Open_vSwitch,SSL,private_key --certificate=db:Open_vSwitch,SSL,certificate --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --no-chdir --log-file=/var/log/openvswitch/ovsdb-server.log --pidfile=/var/run/openvswitch/ovsdb-server.pid --detach --monitor 5215 ? S 0:00 \_ ovsdb-server --detach --monitor -vconsole:off --log-file=/var/log/openvswitch/ovsdb-server-nb.log --remote=punix:/var/run/openvswitch/ovnnb_db.sock --pidfile=/var/run/openvswitch/ovnnb_db.pid --remote=db:OVN_Northbound,NB_Global,connections --unixctl=ovnnb_db.ctl --private-key=db:OVN_Northbound,SSL,private_key --certificate=db:OVN_Northbound,SSL,certificate --ca-cert=db:OVN_Northbound,SSL,ca_cert --remote=ptcp:6641:192.168.10.10 /etc/openvswitch/ovnnb_db.db 5223 ? S 0:00 \_ ovsdb-server --detach --monitor -vconsole:off --log-file=/var/log/openvswitch/ovsdb-server-sb.log --remote=punix:/var/run/openvswitch/ovnsb_db.sock --pidfile=/var/run/openvswitch/ovnsb_db.pid --remote=db:OVN_Southbound,SB_Global,connections --unixctl=ovnsb_db.ctl --private-key=db:OVN_Southbound,SSL,private_key --certificate=db:OVN_Southbound,SSL,certificate --ca-cert=db:OVN_Southbound,SSL,ca_cert --remote=ptcp:6642:192.168.10.10 /etc/openvswitch/ovnsb_db.db Northbound DB logs show the same error. Well I am not supposed to kill the ovsdb-server process. Am I ? Also when I kill the ovsdb-server process, the error goes away. But my doubt is we need the ovsdb-server process on the Controller node. Right? Thanks for your patience. Pranab On 2 February 2017 at 17:45, Numan Siddique <[email protected]> wrote: > In the ovsdb logs I see > > 2017-02-02T11:07:12.366Z|00020|socket_util|ERR|6641:192.168.10.10: bind: > Address already in use > 2017-02-02T11:07:14.869Z|00021|socket_util|ERR|6641:192.168.10.10: bind: > Address already in use > 2017-02-02T11:07:16.182Z|00022|socket_util|ERR|6641:192.168.10.10: bind: > Address already in use > 2017-02-02T11:07:16.183Z|00023|socket_util|ERR|6641:192.168.10.10: bind: > Address already in use > > Probably you can use netstat and see which processes are listening on port > 6641, kill those processes and restart ovn-northd service and see if it > works. > > Thanks > Numan > > > On Thu, Feb 2, 2017 at 5:26 PM, pranab boruah <[email protected] > > wrote: > >> NB db is not listening on the port 6641: >> # ovn-nbctl --db=tcp:192.168.10.10:6641 show >> 2017-02-02T11:35:10Z|00001|ovsdb_idl|ERR|tcp:192.168.10.10:6641: >> requested schema not found >> ovn-nbctl: tcp:192.168.10.10:6641: database connection failed (No such >> file or directory) >> >> # ovn-sbctl --db=tcp:192.168.10.10:6642 show >> Chassis "e8968c35-11db-4b3b-ab24-c20866ac3dc8" >> hostname: "compute-447" >> Encap geneve >> ip: "172.16.10.11" >> options: {csum="true"} >> Encap vxlan >> ip: "172.16.10.11" >> options: {csum="true"} >> Chassis "9f20e7bd-5e16-4273-8ce8-96d0e0a12047" >> hostname: "compute-397" >> Encap geneve >> ip: "172.16.10.12" >> options: {csum="true"} >> Encap vxlan >> ip: "172.16.10.12" >> options: {csum="true"} >> >> # ovsdb-client dump unix:/var/run/openvswitch/ovnnb_db.sock Connection >> Connection table >> _uuid external_ids inactivity_probe >> is_connected max_backoff other_config status >> target >> ------------------------------------ ------------ ---------------- >> ------------ ----------- ------------ --------------- >> ------------------------- >> c110ffff-5ba2-4125-8a51-3c3db583f267 {} [] >> false [] {} {state=BACKOFF} "ptcp: >> 192.168.10.10:6641" >> >> # ovsdb-client dump unix:/var/run/openvswitch/ovnsb_db.sock Connection >> Connection table >> _uuid external_ids inactivity_probe >> is_connected max_backoff other_config read_only status >> target >> ------------------------------------ ------------ ---------------- >> ------------ ----------- ------------ --------- --------------- >> ------------------------- >> 1beb9cba-cc01-4981-8a67-a8b265326df9 {} [] >> false [] {} false {state=BACKOFF} "ptcp: >> 192.168.10.10:6642" >> >> No change after restarting Controller node and restating Northd as >> mentioned here: >> https://github.com/openvswitch/ovs/commit/84d0ca5d00fe01b291 >> 63236d48fa0f9105687149 >> >> Pranab >> >> >> >> >> >> On 2 February 2017 at 17:00, Numan Siddique <[email protected]> wrote: >> >>> >>> >>> On Thu, Feb 2, 2017 at 4:44 PM, pranab boruah < >>> [email protected]> wrote: >>> >>>> Numan, >>>> >>>> Thanks, but, the above commands didn't changed anything. Checked the >>>> logs and found this: >>>> >>>> >>> >>> Could you run the below two commands and see how it goes >>> >>> - "ovn-nbctl --db=tcp:192.168.10.10:6641 show" -> See if this command >>> is successful or not. If not then ovsdb-server is either not listening on >>> this port or it may be possible that you need to add iptables rule to allow >>> this port. >>> >>> - >>> "ovn- >>> s >>> bctl --db=tcp:192.168.10.10:664 >>> 2 >>> show" >>> - >>> >>> >>> On the controller node where you are running the ovsdb-server's run >>> - "sudo ovsdb-client dump unix:/var/run/openvswitch/ovnnb_db.sock >>> Connection" >>> - "sudo ovsdb-client dump unix:/var/run/openvswitch/ovnsb_db.sock >>> Connection" >>> >>> If the Connection table is empty, it means ovsdb-server is not listening >>> on the tcp ports. >>> >>> Thanks >>> Numan >>> >>> >>> LOG of NB database: >>>> >>>> 2017-02-02T11:07:12.366Z|00020|socket_util|ERR|6641:192.168.10.10: >>>> bind: Address already in use >>>> 2017-02-02T11:07:14.869Z|00021|socket_util|ERR|6641:192.168.10.10: >>>> bind: Address already in use >>>> 2017-02-02T11:07:16.182Z|00022|socket_util|ERR|6641:192.168.10.10: >>>> bind: Address already in use >>>> 2017-02-02T11:07:16.183Z|00023|socket_util|ERR|6641:192.168.10.10: >>>> bind: Address already in use >>>> 2017-02-02T11:07:16.184Z|00024|socket_util|ERR|6641:192.168.10.10: >>>> bind: Address already in use >>>> 2017-02-02T11:09:55.403Z|00025|ovsdb_jsonrpc_server|INFO|ptcp:6642: >>>> 192.168.10.10: remote deconfigured >>>> 2017-02-02T11:09:55.403Z|00026|socket_util|ERR|6641:192.168.10.10: >>>> bind: Address already in use >>>> 2017-02-02T11:09:55.403Z|00027|ovsdb_jsonrpc_server|ERR|Dropped 10 log >>>> messages in last 176 seconds (most recently, 159 seconds ago) due to >>>> excessive rate >>>> 2017-02-02T11:09:55.403Z|00028|ovsdb_jsonrpc_server|ERR|ptcp:6641: >>>> 192.168.10.10: listen failed: Address already in use >>>> 2017-02-02T11:09:55.403Z|00029|socket_util|ERR|6641:192.168.10.10: >>>> bind: Address already in use >>>> 2017-02-02T11:09:55.403Z|00030|ovsdb_jsonrpc_server|ERR|ptcp:6641: >>>> 192.168.10.10: listen failed: Address already in use >>>> 2017-02-02T11:09:57.460Z|00031|socket_util|ERR|6641:192.168.10.10: >>>> bind: Address already in use >>>> 2017-02-02T11:09:57.460Z|00032|ovsdb_jsonrpc_server|ERR|ptcp:6641: >>>> 192.168.10.10: listen failed: Address already in use >>>> 2017-02-02T11:09:57.460Z|00033|socket_util|ERR|6641:192.168.10.10: >>>> bind: Address already in use >>>> 2017-02-02T11:09:59.961Z|00034|socket_util|ERR|6641:192.168.10.10: >>>> bind: Address already in use >>>> 2017-02-02T11:10:02.464Z|00035|socket_util|ERR|6641:192.168.10.10: >>>> bind: Address already in use >>>> 2017-02-02T11:10:04.967Z|00036|socket_util|ERR|6641:192.168.10.10: >>>> bind: Address already in use >>>> 2017-02-02T11:10:07.469Z|00037|socket_util|ERR|6641:192.168.10.10: >>>> bind: Address already in use >>>> 2017-02-02T11:10:09.972Z|00038|socket_util|ERR|6641:192.168.10.10: >>>> bind: Address already in use >>>> 2017-02-02T11:10:12.475Z|00039|socket_util|ERR|6641:192.168.10.10: >>>> bind: Address already in use >>>> 2017-02-02T11:10:14.978Z|00040|socket_util|ERR|6641:192.168.10.10: >>>> bind: Address already in use >>>> 2017-02-02T11:10:17.481Z|00041|socket_util|ERR|6641:192.168.10.10: >>>> bind: Address already in use >>>> >>>> -Pranab >>>> >>>> >>>> On 2 February 2017 at 16:36, Numan Siddique <[email protected]> >>>> wrote: >>>> >>>>> >>>>> >>>>> On Thu, Feb 2, 2017 at 4:00 PM, pranab boruah < >>>>> [email protected]> wrote: >>>>> >>>>>> Numan, >>>>>> Details as requested: >>>>>> >>>>>> ___________ >>>>>> ml2_conf.ini:| >>>>>> ``````````````````` >>>>>> ovn_nb_connection = tcp:192.168.10.10:6641 >>>>>> ovn_sb_connection = tcp:192.168.10.10:6642 >>>>>> ovn_l3_mode = True >>>>>> ovn_l3_scheduler = chance >>>>>> ovn_l3_admin_net_cidr = 169.254.128.0/30 >>>>>> ovn_native_dhcp = True >>>>>> >>>>>> Note: 192.168.10.10 is the management ip of controller. Firewall, >>>>>> NetworkManager and SElinux are disabled on all nodes. >>>>>> ____________________________ >>>>>> output of "ps -aef | grep ovsdb" : | >>>>>> ````````````````````````````````````````````````` >>>>>> root 21220 21219 0 Feb01 ? 00:00:44 ovsdb-server >>>>>> /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info >>>>>> --remote=punix:/var/run/openvswitch/db.sock >>>>>> --private-key=db:Open_vSwitch,SSL,private_key >>>>>> --certificate=db:Open_vSwitch,SSL,certificate >>>>>> --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --no-chdir >>>>>> --log-file=/var/log/openvswitch/ovsdb-server.log >>>>>> --pidfile=/var/run/openvswitch/ovsdb-server.pid --detach --monitor >>>>>> >>>>>> root 21324 21323 0 Feb01 ? 00:00:03 ovsdb-server --detach >>>>>> --monitor -vconsole:off >>>>>> --log-file=/var/log/openvswitch/ovsdb-server-nb.log >>>>>> --remote=punix:/var/run/openvswitch/ovnnb_db.sock >>>>>> --pidfile=/var/run/openvswitch/ovnnb_db.pid >>>>>> --remote=db:OVN_Northbound,NB_Global,connections >>>>>> --unixctl=ovnnb_db.ctl --private-key=db:OVN_Northbound,SSL,private_key >>>>>> --certificate=db:OVN_Northbound,SSL,certificate >>>>>> --ca-cert=db:OVN_Northbound,SSL,ca_cert /etc/openvswitch/ovnnb_db.db >>>>>> >>>>>> root 21332 21331 0 Feb01 ? 00:00:03 ovsdb-server --detach >>>>>> --monitor -vconsole:off >>>>>> --log-file=/var/log/openvswitch/ovsdb-server-sb.log >>>>>> --remote=punix:/var/run/openvswitch/ovnsb_db.sock >>>>>> --pidfile=/var/run/openvswitch/ovnsb_db.pid >>>>>> --remote=db:OVN_Southbound,SB_Global,connections >>>>>> --unixctl=ovnsb_db.ctl --private-key=db:OVN_Southbound,SSL,private_key >>>>>> --certificate=db:OVN_Southbound,SSL,certificate >>>>>> --ca-cert=db:OVN_Southbound,SSL,ca_cert /etc/openvswitch/ovnsb_db.db >>>>>> >>>>>> Note: NB and SB databases are supposed to be listening on a tcp >>>>>> socket. But it isn't. >>>>>> >>>>>> >>>>> Running the below commands should solve your problem >>>>> - ovn-nbctl set-connection ptcp:6641:192.168.10.10 >>>>> >>>>> - ovn- >>>>> s >>>>> bctl set-connection ptcp:664 >>>>> 2 >>>>> :192.168.10.10 >>>>> >>>>> L >>>>> et me know if you still face problems. >>>>> >>>>> The reason for this is this commit - >>>>> https://github.com/openvswitch/ovs/commit/84d0ca5d00fe01b291 >>>>> 63236d48fa0f9105687149 >>>>> >>>>> P >>>>> lease have a look at it. >>>>> >>>>> Thanks >>>>> Numan >>>>> >>>>> >>>>> >>>>>> _________________________ >>>>>> output of logs | >>>>>> ``````````````````````````````````````````` >>>>>> Neutron server log : >>>>>> - http://paste.openstack.org/show/597320/ >>>>>> ovs-nothd log: >>>>>> - http://paste.openstack.org/show/597322/ >>>>>> ovs-vswitchd log on Controller: >>>>>> - http://paste.openstack.org/show/597324/ >>>>>> _________________________________________ >>>>>> # ovn-nbctl and ovn-sbctl show no output. | >>>>>> ```````````````````````````````````````````````````````````` >>>>>> ``````````` >>>>>> ________________________________________ >>>>>> OVN RPMS installation and download details: | >>>>>> ```````````````````````````````````````````````````````````` >>>>>> `````````` >>>>>> On controller: >>>>>> # git clone https://github.com/openvswitch/ovs.git >>>>>> # ./boot.sh >>>>>> # ./configure >>>>>> # make rpm-fedora RPMBUILD_OPT="--without check" >>>>>> # make rpm-fedora-kmod >>>>>> >>>>>> ALL nodes: >>>>>> >>>>>> # rpm -i openvswitch-kmod-2.6.90-1.el7.centos.x86_64.rpm# rpm -U >>>>>> openvswitch-2.6.90-1.el7.centos.x86_64.rpm >>>>>> >>>>>> Only Controller: >>>>>> >>>>>> # rpm -i openvswitch-ovn-common-*.x86_64.rpm# rpm -i >>>>>> openvswitch-ovn-central-*.x86_64.rpm >>>>>> >>>>>> Only Compute: >>>>>> >>>>>> # rpm -i openvswitch-ovn-common-*.x86_64.rpm# rpm -i >>>>>> openvswitch-ovn-host-*.x86_64.rpm >>>>>> >>>>>> # lsmod | grep open >>>>>> - shows all the required modules >>>>>> >>>>>> Let me know if you need anything else. >>>>>> >>>>>> Thanks. >>>>>> pjb >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On 2 February 2017 at 15:23, Numan Siddique <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Adding openstack-dev ML with [networking-ovn] tag. >>>>>>> >>>>>>> Please see comments inline >>>>>>> >>>>>>> >>>>>>> On Thu, Feb 2, 2017 at 3:01 PM, pranab boruah < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi guys, >>>>>>>> We have a OpenStack multi-node lab. Everything was working, until I >>>>>>>> tried integrating with OVN. >>>>>>>> Followed the official guide for integration: >>>>>>>> >>>>>>>> http://docs.openstack.org/developer/networking-ovn/install.html >>>>>>>> >>>>>>>> Lab details: >>>>>>>> 1 Controller, 2 computes >>>>>>>> >>>>>>>> CentOS-7.3, OpenStack Newton, separate network for mgmt and tunnel. >>>>>>>> >>>>>>>> Error details : >>>>>>>> Neutron Server log shows : >>>>>>>> >>>>>>>> ERROR networking_ovn.ovsdb.impl_idl_ovn [-] OVS database >>>>>>>> connection to OVN_Northbound failed with error: '{u'error': u'unknown >>>>>>>> database', u'details': u'get_schema request specifies unknown database >>>>>>>> OVN_Northbound', u'syntax': u'["OVN_Northbound"]'}'. Verify that the >>>>>>>> OVS >>>>>>>> and OVN services are available and that the 'ovn_nb_connection' and >>>>>>>> 'ovn_sb_connection' configuration options are correct. >>>>>>>> >>>>>>>> Stuck in this for more than 2 week. Starting to hate my life. >>>>>>>> Please help. Before I tried installing in physical machines, followed >>>>>>>> the >>>>>>>> same guide and installed in VMs. Everything was working then. >>>>>>>> >>>>>>>> Solutions I tried: >>>>>>>> 1) Completely fresh installing everything. >>>>>>>> 2) Rebuilding ovs >>>>>>>> 3) Check SB and NB configuration options in plugin.ini are exactly >>>>>>>> correct. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> Could you please share >>>>>>> - the values of ovn_nb_connection and ovn_sb_connection defined in >>>>>>> ml2_conf.ini. >>>>>>> - the output of "ps -aef | grep ovsdb" on the node where you are >>>>>>> running the ovn-northd server. >>>>>>> - If you could share the output of neutron-server logs in some >>>>>>> paste link, it would be helpful. >>>>>>> >>>>>>> Are you able to run "ovn-nbctl show" ? I presume you have installed >>>>>>> the appropriate ovn RPMS ? If not so, please let me know how you have >>>>>>> installed it. >>>>>>> >>>>>>> Thanks >>>>>>> Numan >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Let me know if additional details are required. >>>>>>>> >>>>>>>> >>>>>>>> -pjb >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> discuss mailing list >>>>>>>> [email protected] >>>>>>>> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
