Hi Anil, I added some log printout for the VM and it seems the OVS dies 3.5 minutes after it is started, can you please bring manually 1 VM and verify why is this happening?
sudo cat /var/log/openvswitch/ovs-vswitchd.log 2017-10-13T07:05:17.896Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log 2017-10-13T07:05:17.948Z|00002|ovs_numa|INFO|Discovered 2 CPU cores on NUMA node 0 2017-10-13T07:05:17.948Z|00003|ovs_numa|INFO|Discovered 1 NUMA nodes and 2 CPU cores 2017-10-13T07:05:17.948Z|00004|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting... 2017-10-13T07:05:17.948Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected 2017-10-13T07:05:17.950Z|00006|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.5.2 2017-10-13T07:08:39.345Z|00007|memory|INFO|8268 kB peak resident set size after 201.4 seconds 2017-10-13T07:08:39.347Z|00002|daemon_unix(monitor)|INFO|pid 1647 died, exit status 0, exiting 2017-10-13T07:09:08.300Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log 2017-10-13T07:09:08.301Z|00002|ovs_numa|INFO|Discovered 2 CPU cores on NUMA node 0 2017-10-13T07:09:08.301Z|00003|ovs_numa|INFO|Discovered 1 NUMA nodes and 2 CPU cores 2017-10-13T07:09:08.301Z|00004|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting... 2017-10-13T07:09:08.302Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected 2017-10-13T07:09:08.304Z|00006|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.5.2 2017-10-13T07:09:09.089Z|00002|daemon_unix(monitor)|INFO|pid 4938 died, exit status 0, exiting /var/log/openvswitch/ovsdb-server.log 017-10-13T07:05:17.703Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovsdb-server.log 2017-10-13T07:05:17.709Z|00002|ovsdb_server|INFO|ovsdb-server (Open vSwitch) 2.5.2 2017-10-13T07:05:27.715Z|00003|memory|INFO|3456 kB peak resident set size after 10.0 seconds 2017-10-13T07:05:27.715Z|00004|memory|INFO|cells:16 monitors:1 sessions:1 2017-10-13T07:08:39.349Z|00002|daemon_unix(monitor)|INFO|pid 1594 died, exit status 0, exiting 2017-10-13T07:09:08.285Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovsdb-server.log 2017-10-13T07:09:08.288Z|00002|ovsdb_server|INFO|ovsdb-server (Open vSwitch) 2.5.2 2017-10-13T07:09:09.191Z|00002|daemon_unix(monitor)|INFO|pid 4928 died, exit status 0, exiting FYI The test starts at 07:10:19 and by then the OVS is not available. Thanks/Luis > On Oct 12, 2017, at 8:48 PM, Anil Belur <[email protected]> wrote: > > > > On Fri, Oct 13, 2017 at 6:32 AM, Luis Gomez <[email protected]> wrote: > FYI, since yesterday OF csit jobs are failing: > > https://jenkins.opendaylight.org/releng/view/Sanity-Jobs/job/openflowplugin-csit-1node-sanity-only-oxygen/ > > Looking at the robot log, it seems like OVS is not installed in > mininet-ovs-25 VM: > > *** Creating network > *** Adding controller > *** Adding hosts: > h1 h2 h3 h4 > *** Adding switches: > ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No > such file or directory) > ovs-vsctl exited with code 1 > *** Error connecting to ovs-db with ovs-vsctl > Make sure that Open vSwitch is installed, that ovsdb-server is running, and > that > "ovs-vsctl show" works correctly. > You may wish to try "service openvswitch-switch start". > [jenkins@releng-09818-14-1-mininet-ovs-25-0 ~]> > > > Has anything changed lately in this VM? > > > > Luis, Nothing has changed recently on the VM images, from the logs in [1.] > shows OVS install is going through while these images were updates during > September. > > [1.] > https://logs.opendaylight.org/releng/jenkins092/builder-packer-merge-ubuntu-16.04-mininet-ovs-2.5/6/ > > > Thanks, > Anil > public_cloud: ---> Install OpenVSwitch 2.5.0 > public_cloud: Get:1 > http://security.ubuntu.com/ubuntu > xenial-security InRelease [102 kB] > public_cloud: Hit:2 > http://us.archive.ubuntu.com/ubuntu > xenial InRelease > public_cloud: Hit:3 > http://rackspace.clouds.archive.ubuntu.com/ubuntu > xenial InRelease > public_cloud: Get:4 > http://rackspace.clouds.archive.ubuntu.com/ubuntu > xenial-updates InRelease [102 kB] > public_cloud: Get:5 > http://rackspace.clouds.archive.ubuntu.com/ubuntu > xenial-backports InRelease [102 kB] > public_cloud: Ign:6 > http://apt.puppetlabs.com > xenial InRelease > private_cloud: mv -f > thrift/parse/.deps/thrift_thrift_bootstrap-t_typedef.Tpo > thrift/parse/.deps/thrift_thrift_bootstrap-t_typedef.Po > private_cloud: g++ -DHAVE_CONFIG_H -I. -I../../.. > -I../../../lib/cpp/src/thrift -I../../../lib/c_glib/src/thrift -Wall > -Wextra -pedantic -g -O2 -std=c++11 -MT > thrift/parse/thrift_thrift_bootstrap-parse.o -MD -MP -MF > thrift/parse/.deps/thrift_thrift_bootstrap-parse.Tpo -c -o > thrift/parse/thrift_thrift_bootstrap-parse.o `test -f 'thrift/parse/ > parse.cc' || echo './'`thrift/parse/parse.cc > > public_cloud: Hit:7 > http://apt.puppetlabs.com > xenial Release > public_cloud: Get:8 > http://rackspace.clouds.archive.ubuntu.com/ubuntu > xenial-updates/main amd64 Packages [632 kB] > public_cloud: Get:9 > http://rackspace.clouds.archive.ubuntu.com/ubuntu > xenial-updates/universe amd64 Packages [528 kB] > private_cloud: mv -f thrift/parse/.deps/thrift_thrift_bootstrap-parse.Tpo > thrift/parse/.deps/thrift_thrift_bootstrap-parse.Po > private_cloud: g++ -DHAVE_CONFIG_H -I. -I../../.. > -I../../../lib/cpp/src/thrift -I../../../lib/c_glib/src/thrift -Wall > -Wextra -pedantic -g -O2 -std=c++11 -MT > thrift/generate/thrift_thrift_bootstrap-t_cpp_generator.o -MD -MP -MF > thrift/generate/.deps/thrift_thrift_bootstrap-t_cpp_generator.Tpo -c -o > thrift/generate/thrift_thrift_bootstrap-t_cpp_generator.o `test -f > 'thrift/generate/t_cpp_generator.cc' || echo > './'`thrift/generate/t_cpp_generator.cc > public_cloud: Fetched 1,466 kB in 1s (1,391 kB/s) > public_cloud: Reading package lists... > public_cloud: + apt-get install openvswitch-switch openvswitch-vtep > public_cloud: Reading package lists... > public_cloud: Building dependency tree... > public_cloud: Reading state information... > public_cloud: The following packages were automatically installed and are > no longer required: > public_cloud: fonts-lato javascript-common libjs-jquery libruby2.3 rake > ruby > public_cloud: ruby-did-you-mean ruby-json ruby-minitest ruby-net-telnet > ruby-power-assert > public_cloud: ruby-test-unit ruby2.3 rubygems-integration virt-what > public_cloud: Use 'apt autoremove' to remove them. > public_cloud: The following additional packages will be installed: > public_cloud: openvswitch-common python-openvswitch > public_cloud: The following NEW packages will be installed: > public_cloud: openvswitch-common openvswitch-switch openvswitch-vtep > python-openvswitch > public_cloud: 0 upgraded, 4 newly installed, 0 to remove and 1 not > upgraded. > public_cloud: Need to get 2,066 kB of archives. > public_cloud: After this operation, 9,001 kB of additional disk space > will be used. > public_cloud: Get:1 > http://rackspace.clouds.archive.ubuntu.com/ubuntu > xenial-updates/main amd64 openvswitch-common amd64 2.5.2-0ubuntu0.16.04.1 > [657 kB] > public_cloud: Get:2 > http://rackspace.clouds.archive.ubuntu.com/ubuntu > xenial-updates/main amd64 openvswitch-switch amd64 2.5.2-0ubuntu0.16.04.1 > [1,152 kB] > public_cloud: Get:3 > http://rackspace.clouds.archive.ubuntu.com/ubuntu > xenial-updates/main amd64 python-openvswitch all 2.5.2-0ubuntu0.16.04.1 > [66.4 kB] > public_cloud: Get:4 > http://rackspace.clouds.archive.ubuntu.com/ubuntu > xenial-updates/universe amd64 openvswitch-vtep amd64 2.5.2-0ubuntu0.16.04.1 > [191 kB] > public_cloud: Fetched 2,066 kB in 7s (276 kB/s) > public_cloud: Selecting previously unselected package openvswitch-common. > public_cloud: (Reading database ... 129088 files and directories > currently installed.) > public_cloud: Preparing to unpack > .../openvswitch-common_2.5.2-0ubuntu0.16.04.1_amd64.deb ... > public_cloud: Unpacking openvswitch-common (2.5.2-0ubuntu0.16.04.1) ... > public_cloud: Selecting previously unselected package openvswitch-switch. > public_cloud: Preparing to unpack > .../openvswitch-switch_2.5.2-0ubuntu0.16.04.1_amd64.deb ... > public_cloud: Unpacking openvswitch-switch (2.5.2-0ubuntu0.16.04.1) ... > public_cloud: Selecting previously unselected package python-openvswitch. > public_cloud: Preparing to unpack > .../python-openvswitch_2.5.2-0ubuntu0.16.04.1_all.deb ... > public_cloud: Unpacking python-openvswitch (2.5.2-0ubuntu0.16.04.1) ... > public_cloud: Selecting previously unselected package openvswitch-vtep. > public_cloud: Preparing to unpack > .../openvswitch-vtep_2.5.2-0ubuntu0.16.04.1_amd64.deb ... > public_cloud: Unpacking openvswitch-vtep (2.5.2-0ubuntu0.16.04.1) ... > public_cloud: Processing triggers for man-db (2.7.5-1) ... > public_cloud: Processing triggers for systemd (229-4ubuntu19) ... > public_cloud: Processing triggers for ureadahead (0.100.0-19) ... > public_cloud: Setting up openvswitch-common (2.5.2-0ubuntu0.16.04.1) ... > public_cloud: Setting up openvswitch-switch (2.5.2-0ubuntu0.16.04.1) ... > public_cloud: update-alternatives: using > /usr/lib/openvswitch-switch/ovs-vswitchd to provide /usr/sbin/ovs-vswitchd > (ovs-vswitchd) in auto mode > public_cloud: openvswitch-nonetwork.service is a disabled or a static > unit, not starting it. > public_cloud: Setting up python-openvswitch (2.5.2-0ubuntu0.16.04.1) ... > public_cloud: Setting up openvswitch-vtep (2.5.2-0ubuntu0.16.04.1) ... > public_cloud: Processing triggers for systemd (229-4ubuntu19) ... > public_cloud: Processing triggers for ureadahead (0.100.0-19) ... > _______________________________________________ openflowplugin-dev mailing list [email protected] https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev
