On Tue, Nov 08, 2016 at 06:27:34PM -0500, George Mihaiescu wrote: :Hi Jonathan, : :The openvswitch-agent is out of sync on compute 4, try restarting it.
Nope. Restarted the agent on 4 and when that failed also on 1 they still can't talk. I had previously migrated 4 to a new host to test that theory. Also if I start 5 (also on new hypervisor from all others) 4 an d5 can talk to eachother but neither 4 nor 5 can talk to 1 -Jon : : : :> On Nov 8, 2016, at 17:43, Jonathan Proulx <j...@csail.mit.edu> wrote: :> :> :> I have an odd issue that seems to just be affecting one private :> network for one tenant, though I saw a similar thing on a different :> project network recently which I 'fixed' by rebooting the hypervisor. :> Since this has now (maybe) happened twice I figure I should try to :> understand what it is. :> :> Given the following four VMs on 4 different hypervisors :> :> vm1 on Hypervisor1 :> vm2 on Hypervisor2 :> vm3 on Hypervisor3 :> ------------------- :> vm4 on Hypervisor4 :> :> :> vm1 -> vm3 talk fine among themselves but none to 4 :> :> examining ping traffic transiting from vm1-vm4 I can see arp requests :> and responses at vm4 and GRE encapsulated ARP responses on :> Hypervisor1's physical interface. :> :> They look the same to me (same ecap id) coming in as the working vms :> traffic, but they never make it to the qvo device which is before :> iptables sec_group rules are applied at the tap device. :> :> attempting to tare down and recreate this resuls in the same first 3 :> work last one doesn't split (possibly becuase scheduler puts them in :> the same place? haven't checked) :> :> ovs-vsctl -- set Bridge br-int mirrors=@m -- --id=@snooper2 get Port snooper2 -- --id=@gre-801e0347 get Port gre-801e0347 -- --id=@m create Mirror name=mymirror select-dst-port=@gre-801e0347 select-src-port=@gre-801e0347 output-port=@snooper2 :> :> tcpdump -i snooper2 :> :> Only sees ARP requests but no response, what's broken if I can see GRE :> encap ARP responses on physical interface but not on gre-<hex> :> interface? And why is it not broken for all tunnels endpoints? :> :> Oddly if I boot a 5th VM on a 5th hypervisor it can talk to 4 but not 1-3 ... :> :> hypervisors are Ubuntu 14.04 running Mitaka from cloud archive w/ :> xenial-lts kernels (4.4.0) :> :> -Jon :> :> -- :> :> _______________________________________________ :> OpenStack-operators mailing list :> OpenStack-operators@lists.openstack.org :> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators _______________________________________________ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators