Public bug reported:
For some reason, some compute nodes are missing ovs flows of various
tenants (not all of them), resulting in vm isolation (no dhcp/metadata
on boot). A particular tenant "A" might have issues with node "B" whilst
tenant "B" might have problems with node "A" and not "B". All of the
affected tenant VMs running in an affected node are isolated.
In those compute nodes, the ovs-vswitchd process has crashed previously:
ubuntu@niagara:~$ ps -ef | grep vswitchd
root 1959 1 0 Feb12 ? 00:41:15 ovs-vswitchd: monitoring pid
62005 (4 crashes: pid 59408 died, killed (Segmentation fault), core dumped)
After restarting the openvswitch-switch service, ps shows that the missing
flows are being created by neutron (e.g. sudo neutron-rootwrap
/etc/neutron/rootwrap.conf ovs-ofctl mod-flows br-tun
table=21,dl_vlan=11,actions=strip_vlan,set_tunnel:4,output:5,6,2,4,7,3).
---
ApportVersion: 2.14.1-0ubuntu3.5
Architecture: amd64
DistroRelease: Ubuntu 14.04
Package: neutron-common 1:2014.1.3-0ubuntu1
PackageArchitecture: all
ProcEnviron:
TERM=xterm
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcVersionSignature: User Name 3.13.0-45.74-generic 3.13.11-ckt13
Tags: trusty uec-images
Uname: Linux 3.13.0-45-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm audio cdrom dialout dip floppy libvirtd netdev plugdev sudo
video
_MarkForUpload: True
modified.conffile..etc.neutron.api.paste.ini: [deleted]
modified.conffile..etc.neutron.fwaas.driver.ini: [deleted]
modified.conffile..etc.neutron.l3.agent.ini: [deleted]
modified.conffile..etc.neutron.neutron.conf: [deleted]
modified.conffile..etc.neutron.policy.json: [deleted]
modified.conffile..etc.neutron.rootwrap.conf: [deleted]
modified.conffile..etc.neutron.rootwrap.d.debug.filters: [deleted]
modified.conffile..etc.neutron.rootwrap.d.iptables.firewall.filters: [deleted]
modified.conffile..etc.neutron.rootwrap.d.l3.filters: [deleted]
modified.conffile..etc.neutron.rootwrap.d.vpnaas.filters: [deleted]
modified.conffile..etc.neutron.vpn.agent.ini: [deleted]
modified.conffile..etc.sudoers.d.neutron.sudoers: [deleted]
** Affects: neutron
Importance: Undecided
Status: New
** Tags: cts
** Tags added: apport-collected trusty uec-images
** Description changed:
For some reason, some compute nodes are missing ovs flows of various
tenants (not all of them), resulting in vm isolation (no dhcp/metadata
on boot). A particular tenant "A" might have issues with node "B" whilst
tenant "B" might have problems with node "A" and not "B". All of the
affected tenant VMs running in an affected node are isolated.
In those compute nodes, the ovs-vswitchd process has crashed previously:
ubuntu@niagara:~$ ps -ef | grep vswitchd
root 1959 1 0 Feb12 ? 00:41:15 ovs-vswitchd: monitoring pid
62005 (4 crashes: pid 59408 died, killed (Segmentation fault), core dumped)
- After restarting the openvswitch-switch service, ps shows that the
- missing flows are being created by neutron (e.g. sudo neutron-rootwrap
- /etc/neutron/rootwrap.conf ovs-ofctl mod-flows br-tun
- table=21,dl_vlan=11,actions=strip_vlan,set_tunnel:4,output:5,6,2,4,7,3).
+ After restarting the openvswitch-switch service, ps shows that the missing
flows are being created by neutron (e.g. sudo neutron-rootwrap
/etc/neutron/rootwrap.conf ovs-ofctl mod-flows br-tun
table=21,dl_vlan=11,actions=strip_vlan,set_tunnel:4,output:5,6,2,4,7,3).
+ ---
+ ApportVersion: 2.14.1-0ubuntu3.5
+ Architecture: amd64
+ DistroRelease: Ubuntu 14.04
+ Package: neutron-common 1:2014.1.3-0ubuntu1
+ PackageArchitecture: all
+ ProcEnviron:
+ TERM=xterm
+ PATH=(custom, no user)
+ XDG_RUNTIME_DIR=<set>
+ LANG=en_US.UTF-8
+ SHELL=/bin/bash
+ ProcVersionSignature: User Name 3.13.0-45.74-generic 3.13.11-ckt13
+ Tags: trusty uec-images
+ Uname: Linux 3.13.0-45-generic x86_64
+ UpgradeStatus: No upgrade log present (probably fresh install)
+ UserGroups: adm audio cdrom dialout dip floppy libvirtd netdev plugdev sudo
video
+ _MarkForUpload: True
+ modified.conffile..etc.neutron.api.paste.ini: [deleted]
+ modified.conffile..etc.neutron.fwaas.driver.ini: [deleted]
+ modified.conffile..etc.neutron.l3.agent.ini: [deleted]
+ modified.conffile..etc.neutron.neutron.conf: [deleted]
+ modified.conffile..etc.neutron.policy.json: [deleted]
+ modified.conffile..etc.neutron.rootwrap.conf: [deleted]
+ modified.conffile..etc.neutron.rootwrap.d.debug.filters: [deleted]
+ modified.conffile..etc.neutron.rootwrap.d.iptables.firewall.filters: [deleted]
+ modified.conffile..etc.neutron.rootwrap.d.l3.filters: [deleted]
+ modified.conffile..etc.neutron.rootwrap.d.vpnaas.filters: [deleted]
+ modified.conffile..etc.neutron.vpn.agent.ini: [deleted]
+ modified.conffile..etc.sudoers.d.neutron.sudoers: [deleted]
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1431958
Title:
Missing ovs flows results in vm isolation for particular tenants
Status in OpenStack Neutron (virtual network service):
New
Bug description:
For some reason, some compute nodes are missing ovs flows of various
tenants (not all of them), resulting in vm isolation (no dhcp/metadata
on boot). A particular tenant "A" might have issues with node "B"
whilst tenant "B" might have problems with node "A" and not "B". All
of the affected tenant VMs running in an affected node are isolated.
In those compute nodes, the ovs-vswitchd process has crashed previously:
ubuntu@niagara:~$ ps -ef | grep vswitchd
root 1959 1 0 Feb12 ? 00:41:15 ovs-vswitchd: monitoring pid
62005 (4 crashes: pid 59408 died, killed (Segmentation fault), core dumped)
After restarting the openvswitch-switch service, ps shows that the missing
flows are being created by neutron (e.g. sudo neutron-rootwrap
/etc/neutron/rootwrap.conf ovs-ofctl mod-flows br-tun
table=21,dl_vlan=11,actions=strip_vlan,set_tunnel:4,output:5,6,2,4,7,3).
---
ApportVersion: 2.14.1-0ubuntu3.5
Architecture: amd64
DistroRelease: Ubuntu 14.04
Package: neutron-common 1:2014.1.3-0ubuntu1
PackageArchitecture: all
ProcEnviron:
TERM=xterm
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcVersionSignature: User Name 3.13.0-45.74-generic 3.13.11-ckt13
Tags: trusty uec-images
Uname: Linux 3.13.0-45-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm audio cdrom dialout dip floppy libvirtd netdev plugdev sudo
video
_MarkForUpload: True
modified.conffile..etc.neutron.api.paste.ini: [deleted]
modified.conffile..etc.neutron.fwaas.driver.ini: [deleted]
modified.conffile..etc.neutron.l3.agent.ini: [deleted]
modified.conffile..etc.neutron.neutron.conf: [deleted]
modified.conffile..etc.neutron.policy.json: [deleted]
modified.conffile..etc.neutron.rootwrap.conf: [deleted]
modified.conffile..etc.neutron.rootwrap.d.debug.filters: [deleted]
modified.conffile..etc.neutron.rootwrap.d.iptables.firewall.filters: [deleted]
modified.conffile..etc.neutron.rootwrap.d.l3.filters: [deleted]
modified.conffile..etc.neutron.rootwrap.d.vpnaas.filters: [deleted]
modified.conffile..etc.neutron.vpn.agent.ini: [deleted]
modified.conffile..etc.sudoers.d.neutron.sudoers: [deleted]
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1431958/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
