Re: [Openstack-operators] sporadic missing vxlan-tunnel-port assignment
Hi Fabian, We are running Pike. We have DVR with centralized SNAT on L3-HA routers. We also have L2 population enabled which I think is responsible for the issue. We can reproduce the issue almost every time we are spawning 100 VMs at once. Usually 3-6 VMs have no connectivity to services running on network nodes like DHCP, DNS so they don't have network configured. As I mentioned in the bug we also observing issues with L3-HA routers provisioned using Heat. After debugging I'm almost sure there is a race condition which is triggered by massive network resources provisioning. When we provision routers manually we always have a working router. On 19.03.2018 15:37, Fabian Zimmermann wrote: > Hi Piotr, > > yes, we still debugging this. > > Im currently trying to reproduce the issue in our dev-env, but without > luck so far. > > Are you able to reproduce the issue? > > We are running on ocata and currently trying to upgrade to pike > followed by queens as fast as possible. > > What kind of neutron implementation do you use? (f.e. DVR with > network-nodes for SNAT)? > > IRC: #openstack -> devfaz > > Thanks a lot, > > Fabian > > Am 18.03.2018 um 22:37 schrieb Piotr Misiak: >> Hi Fabian, >> >> >> We also encounter issues with OVS table 22. >> >> Please take a look at this bug: >> https://bugs.launchpad.net/neutron/+bug/1754695 >> >> We are still debugging this. >> >> Maybe you hit the same bug? >> >> >> >> On 07.03.2018 08:56, Fabian Zimmermann wrote: >>> Hi, >>> >>> sorry, it is table 22. >>> >>> Fabian >>> >>> >>> ___ >>> OpenStack-operators mailing list >>> OpenStack-operators@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators >> >> ___ >> OpenStack-operators mailing list >> OpenStack-operators@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators > ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] sporadic missing vxlan-tunnel-port assignment
Hi Fabian, We also encounter issues with OVS table 22. Please take a look at this bug: https://bugs.launchpad.net/neutron/+bug/1754695 We are still debugging this. Maybe you hit the same bug? On 07.03.2018 08:56, Fabian Zimmermann wrote: > Hi, > > sorry, it is table 22. > > Fabian > > > ___ > OpenStack-operators mailing list > OpenStack-operators@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
[Openstack-operators] sporadic missing vxlan-tunnel-port assignment
Hi,
we currently see sporadic communication problems.
After some research we found out, that this is caused by missing
tunnel-port assignments in table 21 of openvswitch.
Today we had the issue again and here the logs of the add_fdb_entries
calls at the affected system:
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent
method add_fdb_entries called with arguments (object at 0x7f7264509810>,) {u'fdb_entries':
{u'cd2baf3d-427c-41be-be56-7cbb8176067f': {u'segment_id': 96, u'ports':
{u'10.78.23.12': [[u'00:00:00:00:00:00', u'0.0.0.0'],
[u'fa:16:3e:d0:a0:77', u'192.168.0.2']]}, u'network_type': u'vxlan'}}}
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent
method add_fdb_entries called with arguments (object at 0x7f7264509bd0>,) {u'fdb_entries':
{u'cd2baf3d-427c-41be-be56-7cbb8176067f': {u'segment_id': 96, u'ports':
{u'10.78.23.11': [[u'00:00:00:00:00:00', u'0.0.0.0'],
[u'fa:16:3e:29:0c:d5', u'192.168.0.3']]}, u'network_type': u'vxlan'}}}
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent
method add_fdb_entries called with arguments (object at 0x7f7264509d90>,) {u'fdb_entries':
{u'cd2baf3d-427c-41be-be56-7cbb8176067f': {u'segment_id': 96, u'ports':
{u'10.78.12.101': []}, u'network_type': u'vxlan'}}}
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent
method add_fdb_entries called with arguments (object at 0x7f7264627190>,) {u'fdb_entries':
{u'cd2baf3d-427c-41be-be56-7cbb8176067f': {u'segment_id': 96, u'ports':
{u'10.79.20.102': [[u'00:00:00:00:00:00', u'0.0.0.0']]},
u'network_type': u'vxlan'}}}
The missing tunnel-port is the connection to 10.78.12.101, it looks like
the empty array/dict may cause this issue.
Any hints how to further debug the situation?
What may cause an empty dict in add_fdb_entries?
Thanks a lot,
Fabian Zimmermann
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
