Hi folks, I see that bug 8186 is marked as a DUP of a resolved bug (8628) which was marked resolved because a config knob was added to modify the heartbeat timer.
But, we are still having trouble it seems. If you tweak the timers, does it help? What version of ODL is this? Can we reproduce the problems in a more sanitized way? Meaning, can we just run a script to connect/disconnect OVS in a loop to hit the issues? We don't really have a suite that does this, but it would be fairly simple to implement. I can try some tests manually myself, but I'll wait for a response to the above first. Thanks, JamO On 06/11/2017 04:07 PM, Nikolas Hermanns wrote: > Hey, > > AL, Daniel shall we sit together in the lunch break today? > > > just to give a idea of our problem. The scenario looks like this: > > Shared Openstack/ODL controller 4 CPU cores. 3-4 compute nodes each having 1 > OVS dpnode. So we are far away from 400 > switches. And in the time CPU load from Openstack services is rather low but > from ODL is very high. This is the bug we wrote > to ODL: > > https://bugs.opendaylight.org/show_bug.cgi?id=8186 > > And this is our corresponding jira issue in OPNFV: > > https://jira.opnfv.org/browse/SDNVPN-144 > > BR Nikolas > > > On 11.06.2017 11:33, Tim Irnich wrote: >> >> Luis, I have not looked at the test you link below but what we have observed >> is that vSwitch connects and disconnects cause >> a high load on ODL, and that the processing of heartbeat messages from >> other, already connected vSwitches can get delayed >> due to this, causing a cascade effect. >> >> >> >> /Tim >> >> >> >> *From:* Luis Gomez [mailto:[email protected]] >> *Sent:* Friday, June 09, 2017 17:07 >> *To:* Tim Irnich <[email protected]> >> *Cc:* MORTON, ALFRED C (AL) <[email protected]>; Daniel Farrell >> <[email protected]>; >> [email protected]; >> [email protected]; Nikolas Hermanns >> <[email protected]> >> *Subject:* Re: [openflowplugin-dev] [integration-dev] Many OVS >> connects/disconnects causing high load, disconnects, failure >> >> >> >> I guess we are talking about OpenFlow connections to OVS switches here. If >> so what is is the high load scenario the >> controller is in this test? For single controller (4 CPUs) we support up to >> 400 switches loaded with 10K flows in Carbon: >> >> >> >> https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-carbon/plot/Switch%20Scalability/ >> >> >> >> Soon I will bring a test to see if this number holds in a cluster. >> >> >> >> BR/Luis >> >> >> >> >> >> >> >> On Jun 9, 2017, at 5:05 AM, Tim Irnich <[email protected] >> <mailto:[email protected]>> wrote: >> >> >> >> Al I think you’re hitting the nail on the head here. We were thinking >> the same, giving heartbeat messages priority over >> other message processing should prevent this cascading effect we have >> seen. Not sure if the current framework allows >> this though… >> >> >> >> Regards, Tim >> >> >> >> *From:* MORTON, ALFRED C (AL) [mailto:[email protected]] >> *Sent:* Friday, June 09, 2017 14:02 >> *To:* Daniel Farrell <[email protected] <mailto:[email protected]>>; >> [email protected] >> <mailto:[email protected]>; >> [email protected] >> <mailto:[email protected]> >> *Cc:* Tim Irnich <[email protected] >> <mailto:[email protected]>>; Nikolas Hermanns >> <[email protected] <mailto:[email protected]>> >> *Subject:* RE: [integration-dev] Many OVS connects/disconnects causing >> high load, disconnects, failure >> >> >> >> Hi Daniel, >> >> (I’m not impersonating Luis or Jamo, but measuring lost >> >> southbound packets has been one of my pet projects, as you know...) >> >> >> >> If we add the Latte golang tool that Nikkos contributed >> >> to the OPNFV Cperf project, we should be able to >> >> correlate ODL load levels (cbench) with OVS message loss ratios >> >> (and the OVS disconnects, I’m assuming that heartbeats >> >> have the same priority as PACKETINs for ODL processing, >> >> and maybe prioritization is part of the solution...) >> >> >> >> Definitely worth discussing further in Beijing next week, >> >> Al >> >> >> >> >> >> *From:* [email protected] >> <mailto:[email protected]> >> [mailto:[email protected]] *On >> Behalf Of *Daniel Farrell >> *Sent:* Friday, June 09, 2017 6:51 AM >> *To:* [email protected] >> <mailto:[email protected]>; >> [email protected] >> <mailto:[email protected]> >> *Cc:* Tim Irnich; Nikolas Hermanns >> *Subject:* [integration-dev] Many OVS connects/disconnects causing high >> load, disconnects, failure >> >> >> >> Hey Integration/Test, openflowplugin, >> >> >> >> OPNFV vswitch perf folks are reporting ODL problems caused by lots of >> OVS disconnects. See the description below. >> >> @Luis, Jamo - What's the most relevant ODL test? >> >> >> >> @Others - Can we fix this? >> >> >> >> Thanks, >> >> Daniel Farrell >> >> On Fri, Jun 9, 2017 at 6:19 AM Nikolas Hermanns >> <[email protected] <mailto:[email protected]>> >> wrote: >> >> Hey Daniel, >> >> >> >> I hope you will come to the opnfv summit next week :-D. I would like >> to discuss with you a new addition to vsperf >> may be. We have an issue that through lots of connects and >> disconnects of ovs, odl is going into to high load and >> through that the heart beats from ovs do not reach odl anymore. Then >> even more switches do disconnect and finally >> the whole cluster does not have networking anymore. >> >> There are some workarounds for that but basically we would like to >> setup a test cases testing the amount of >> switches odl can easily handle. Not sure yet something like that. >> >> >> >> Can we have a small chat about it next week. >> >> >> >> Reach out to me: >> >> +491729607904 <tel:+49%20172%209607904> (whatsapp + sms) >> >> [email protected] <mailto:[email protected]> >> (sometimes faster + hangouts) >> >> Or just this mail address. >> >> >> >> BR Nikolas >> >> _______________________________________________ >> openflowplugin-dev mailing list >> [email protected] >> <mailto:[email protected]> >> https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev >> >> >> > > > > _______________________________________________ > integration-dev mailing list > [email protected] > https://lists.opendaylight.org/mailman/listinfo/integration-dev > _______________________________________________ openflowplugin-dev mailing list [email protected] https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev
