Cool news Robb, Just a little bit of info from me checking last night.
Around 12:35 am, I logged into the NSS boxes and the P&T asa's to see if all was well and I was able to ping and ssh to both asa's and ping out from the asa's (internally and externally). Failover between the two asa's was still intact. There were no errors when querying the status of the openvswitch process. I would like to add the NSS boxes to Zabbix, more specifically, add a <f*ill in zabbix term here*> to monitor the openvswitch process. The only other anomaly is that it doesn't seem to fail until the morning, (noticeable by the Zabbix alert stating that one of the asa's is down. I would like to get the zabbix monitoring implemented first thing tomorrow morning, so John Powell might have to get involved. Thanks, --Carl On Tue, Jul 12, 2016 at 12:32 PM, Robb Cavalluzzi < robert.cavallu...@weather.com> wrote: > Greetings, > We have been troubleshooting an issue where all traffic to the bridge > on the ovs suddenly stops. Upon further examination it appears that the > ovsdb-server service died in an inconsistent state. It has happened twice > in the past 24 hours and occurs around 12 hour intervals. Don't have more > data yet, as it had been running fine previously for some time. > > [root@ash-dc2-c240-nss-1 ~]# /etc/init.d/openvswitch stop > Killing ovs-vswitchd (27053) [ OK ] > Killing ovs-vswitchd (27053) with SIGKILL [ OK ] > Killing ovsdb-server (27043) [ OK ] > [root@ash-dc2-c240-nss-1 ~]# /etc/init.d/openvswitch status > ovsdb-server is not running > Pidfile for ovs-vswitchd (/var/run/openvswitch/ovs-vswitchd.pid) is stale > [root@ash-dc2-c240-nss-1 ~]# cat /var/run/openvswitch/ovs-vswitchd.pid > 27053 > [root@ash-dc2-c240-nss-1 ~]# ps axwu | grep -i 27053 > root 35930 0.0 0.0 103312 936 pts/2 S+ 11:37 0:00 grep -i > 27053 > [root@ash-dc2-c240-nss-1 ~]# rm /var/run/openvswitch/ovs-vswitchd.pid > rm: remove regular file `/var/run/openvswitch/ovs-vswitchd.pid'? y > [root@ash-dc2-c240-nss-1 ~]# /etc/init.d/openvswitch start > Starting ovsdb-server [ OK ] > Configuring Open vSwitch system IDs [ OK ] > Starting ovs-vswitchd [ OK ] > Enabling remote OVSDB managers [ OK ] > [root@ash-dc2-c240-nss-1 ~]# ps axwu | grep -i vswitchd > root 35994 0.0 0.0 41404 936 ? S<s 11:37 0:00 > ovs-vswitchd: monitoring pid 35995 (healthy) > > after removing the stale pid and restarting services network traffic > resumes as normal. > > Version: > ovs-vswitchd (Open vSwitch) 2.4.0 > Compiled Oct 9 2015 01:52:33 > > Kernel" > Linux version 2.6.32-642.1.1.el6.x86_64 (mockbu...@worker1.bsys.centos.org) > (gcc version 4.4.7 20120313 (Red Hat 4.4.7-17) (GCC) ) #1 SMP Tue May 31 > 21:57:07 UTC 2016 > > contents of conf.db: > Attached > > ovs-dpctl show: > system@ovs-system: > lookups: hit:187640502 missed:913077 lost:642854 > flows: 46 > masks: hit:388152846 total:3 hit/pkt:2.06 > port 0: ovs-system (internal) > port 1: br1 (internal) > port 2: bond0 > port 3: vnet0 > port 4: vnet1 > port 5: vnet2 > port 6: vnet3 > port 7: vnet4 > port 8: vnet5 > port 9: vnet6 > port 10: vnet7 > port 11: vnet8 > port 12: vnet9 > port 13: vnet10 > port 14: vnet11 > port 15: vnet12 > port 16: vnet13 > port 17: vnet14 > > ovs-ofctl show br1: > OFPT_FEATURES_REPLY (xid=0x2): dpid:000074a2e6d3c627 > n_tables:254, n_buffers:256 > capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP > actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src > mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst > 1(bond0): addr:74:a2:e6:d3:c6:27 > config: 0 > state: 0 > speed: 0 Mbps now, 0 Mbps max > 18(vnet11): addr:fe:54:00:62:8b:c8 > config: 0 > state: 0 > current: 10MB-FD COPPER > speed: 10 Mbps now, 0 Mbps max > 19(vnet12): addr:fe:54:00:a7:91:cd > config: 0 > state: 0 > current: 10MB-FD COPPER > speed: 10 Mbps now, 0 Mbps max > 20(vnet13): addr:fe:54:00:0e:3f:3b > config: 0 > state: 0 > current: 10MB-FD COPPER > speed: 10 Mbps now, 0 Mbps max > 21(vnet14): addr:fe:54:00:28:e2:c9 > config: 0 > state: 0 > current: 10MB-FD COPPER > speed: 10 Mbps now, 0 Mbps max > 22(vnet0): addr:fe:54:00:4d:15:dc > config: 0 > state: 0 > current: 10MB-FD COPPER > speed: 10 Mbps now, 0 Mbps max > 23(vnet1): addr:fe:54:00:4c:fc:e8 > config: 0 > state: 0 > current: 10MB-FD COPPER > speed: 10 Mbps now, 0 Mbps max > 24(vnet2): addr:fe:54:00:4a:a2:a5 > config: 0 > state: 0 > current: 10MB-FD COPPER > speed: 10 Mbps now, 0 Mbps max > 25(vnet3): addr:fe:54:00:06:1e:28 > config: 0 > state: 0 > current: 10MB-FD COPPER > speed: 10 Mbps now, 0 Mbps max > 26(vnet4): addr:fe:54:00:03:70:b8 > config: 0 > state: 0 > current: 10MB-FD COPPER > speed: 10 Mbps now, 0 Mbps max > 27(vnet5): addr:fe:54:00:4a:4f:ae > config: 0 > state: 0 > current: 10MB-FD COPPER > speed: 10 Mbps now, 0 Mbps max > 28(vnet6): addr:fe:54:00:0a:f6:14 > config: 0 > state: 0 > current: 10MB-FD COPPER > speed: 10 Mbps now, 0 Mbps max > 29(vnet7): addr:fe:54:00:05:7b:ef > config: 0 > state: 0 > current: 10MB-FD COPPER > speed: 10 Mbps now, 0 Mbps max > 30(vnet8): addr:fe:54:00:77:90:fb > config: 0 > state: 0 > current: 10MB-FD COPPER > speed: 10 Mbps now, 0 Mbps max > 31(vnet9): addr:fe:54:00:9d:d6:76 > config: 0 > state: 0 > current: 10MB-FD COPPER > speed: 10 Mbps now, 0 Mbps max > 32(vnet10): addr:fe:54:00:3d:a9:49 > config: 0 > state: 0 > current: 10MB-FD COPPER > speed: 10 Mbps now, 0 Mbps max > LOCAL(br1): addr:74:a2:e6:d3:c6:27 > config: PORT_DOWN > state: LINK_DOWN > speed: 0 Mbps now, 0 Mbps max > OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0 > > Current fix: > kill the pid file manually and restart the service. We'd turned on debug > logging for the ovs and I can send a bugreport I ran, it's 42MB. > > > Thank you, > > > -- > *Robert **Cavalluzzi *|Senior Systems Engineer > *w:* 770-226-2589 *e:* robert.cavallu...@weather.com > <http://weather.com/apps> <http://weather.com/apps> > <http://weather.com/apps> <http://weather.com/apps> > <http://weather.com/apps> <http://weather.com/apps> > -- *Carl **Wilkerson *| Network Security Engineer *w:* 770-226-2070 *e:* carl.wilker...@weather.com <http://weather.com/apps> <http://weather.com/apps> <http://weather.com/apps> <http://weather.com/apps> <http://weather.com/apps> <http://weather.com/apps>
_______________________________________________ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss
