Yeah, the +8 failures in FRS are due to some failure to reconcile oper flow with config flow (alien flow ID), I was able to reproduce the issue locally but in my case it fails in not exactly same flows. What is interesting is that if I try the flows that fail in the suite manually one by one, they show fine in operational (no alien ID) which makes me think there is some sort of race condition or interference when we install many flows very fast like this test does.
> On Oct 4, 2016, at 5:00 AM, Miroslav Macko <[email protected]> > wrote: > > Hi guys, > > Luis. We agree with you. > > 1) Current issues in FRS flow service suite using OVS 2.0 are due to FRS > being slower than FRM. > Yes the FRS is slower then FRM. > > 2) Many flow templates we use in OVS 2.0 do not work in recent versions like > OVS 2.5. We tried to repair this is in the past but we never got the > resources/time to do it, I hope this release we can do it :) > Great. > > But what we wanted to point out is: > > - that both, FRM and also FRS is working with OVS vesrsion 2.0. > - and both FRM and FRS is failing, with the same errors, with OVS version > 2.3+. I would like to emphasize, that also FRM is failing. > > So this is the part what we would like to solve. And that is why we taught, > it is running on different OVS versions. > > Results form local tests: > FRM with OVS version 2.0 - 1 fail > FRM with OVS version 2.4 - 112 fail - logs in the attachment > Flows_Additional_TCs-frm-OVS24 > FRS with OVS version 2.0 - 8 fail - logs in the attachment > Flows_Additional_TCs-frs-OVS20 > FRS with OVS version 2.4 - 120 fail > > @Jamo thank you for checking the virtual machine for both jobs. > > @Luis is there something about the jobs that would make ovs have a different > version please? > > If it is running with the same OVS version, than it is really strange. > > Thanks a lot. > Miro > > Od: Luis Gomez <[email protected] <mailto:[email protected]>> > Odoslané: 4. októbra 2016 11:30 > Komu: Andrej Leitner > Kópia: Jamo Luhrsen; Miroslav Macko; [email protected] > <mailto:[email protected]>; > [email protected] > <mailto:[email protected]> > Predmet: Re: [integration-dev] [openflowplugin-dev] Failing robot tests > openflowplugin-csit-1node-flow-services > > I think we are talking about 2 different things: > > 1) Current issues in FRS flow service suite using OVS 2.0 are due to FRS > being slower than FRM. > 2) Many flow templates we use in OVS 2.0 do not work in recent versions like > OVS 2.5. We tried to repair this is in the past but we never got the > resources/time to do it, I hope this release we can do it :) > > BR/Luis > > >> On Oct 4, 2016, at 2:16 AM, Andrej Leitner <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hi guys, >> we know that FRS is currently slower than FRM, but a week ago we refactored >> with Miroslav the robot test locally to sleep more than 1 or 3 seconds >> just because we wanted to eliminate the failures caused by perfromance >> issues. We tried to run tests for FRM and FRS with OVS versions >> 2.0/2.3/2.4/2.5 >> and we see both are failing on same flows with particular ovs versions - >> example from todays testing: >> >> flow: odl/test/csit/variables/xmls/f18.xml >> log:set DEBUG >> org.opendaylight.openflowplugin.impl.services.SalFlowServiceImpl >> >> feature:install odl-openflowplugin-flow-services-rest >> OVS 2.0 >> 2016-10-04 10:46:58,499 | DEBUG | entLoopGroup-7-2 | SalFlowServiceImpl >> | 209 - org.opendaylight.openflowplugin.impl - 0.4.0.SNAPSHOT | >> Flow add with id=141 finished without error >> >> OVS 2.5 >> 2016-10-04 10:50:17,148 | DEBUG | entLoopGroup-7-4 | SalFlowServiceImpl >> | 209 - org.opendaylight.openflowplugin.impl - 0.4.0.SNAPSHOT | >> Flow add failed for flow=AddFlowInput [_cookie=FlowCookie [_value=18], >> _cookieMask=FlowCookie [_value=255], _flowName=FooXf18, _flowRef=FlowRef >> [_value=KeyedInstanceIdentifier{targetType=interface >> ... >> _strict=false, augmentation=[]], errors=Device reported error type BADMATCH >> code BADWILDCARDS >> >> >> feature:install odl-restconf; feature:install >> odl-openflowplugin-app-config-pusher; feature:install >> odl-openflowplugin-app-topology; feature:install >> odl-openflowplugin-app-forwardingrules-sync; >> OVS 2.0 >> 2016-10-04 10:55:02,697 | DEBUG | entLoopGroup-9-5 | SalFlowServiceImpl >> | 281 - org.opendaylight.openflowplugin.impl - 0.4.0.SNAPSHOT | >> Flow add with id=141 finished without error >> >> OVS 2.5 >> 2016-10-04 10:54:08,438 | DEBUG | entLoopGroup-9-3 | SalFlowServiceImpl >> | 281 - org.opendaylight.openflowplugin.impl - 0.4.0.SNAPSHOT | >> Flow add failed for flow=AddFlowInput [_cookie=FlowCookie [_value=18], >> _cookieMask=FlowCookie [_value=255], _flowName=FooXf18, _flowRef=FlowRef >> [_value=KeyedInstanceIdentifier{targetType=interface >> ... >> _priority=18, _tableId=2, _installHw=false, _strict=false, augmentation=[]], >> errors=Device reported error type BADMATCH code BADWILDCARDS >> >> >> Are you sure the problem is not in different ovs versions? >> >> From: Luis Gomez <[email protected] <mailto:[email protected]>> >> Sent: Monday, October 3, 2016 10:55 PM >> To: Jamo Luhrsen; Miroslav Macko >> Cc: [email protected] >> <mailto:[email protected]>; >> [email protected] >> <mailto:[email protected]> >> Subject: Re: [integration-dev] [openflowplugin-dev] Failing robot tests >> openflowplugin-csit-1node-flow-services >> >> So looking at the FRS test in detail, the reason for the multiple failures >> is not the OVS version but the FRS feature requiring more time to install >> flows + update flow stats in oper DS. >> >> In general it is bad practice to use sleep in tests but in this case it was >> useful to flag the slowness of FRS, in particular: >> >> - Flows_OF13 suite is failing because we give 1 sec between programming the >> flow and see the flow in OVS, with FRS we need more time and that is why it >> is fails. >> - Stats_Manager_extended suite is failing because we give 3 secs between >> programming flows and see them in operational DS, with FRS we need more time >> and thats is why it fails. >> - The Groups_Meters_OF13 is also failing because CPqD switch cannot connect >> to controller, this could be a CPqD issue, with the old FRM is working but >> it takes long time to connect. >> >> Finally you can observe the differences in performance in these jobs: >> >> https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-scale-stats-collection-daily-frs-only-boron/plot/ >> >> <https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-scale-stats-collection-daily-frs-only-boron/plot/> >> https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-scale-stats-collection-daily-only-boron/plot/ >> >> <https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-scale-stats-collection-daily-only-boron/plot/> >> >> The plot titles are not very good but I can explain: >> >> - Config Performance measures time for updating all flow stats in oper DS >> after adding and after deleting. >> - Performance rate measures rate and time for getting flows in config DS. >> >> In both cases the FRM is superior than FRS. >> >> BR/Luis >> >> >>> On Oct 3, 2016, at 9:20 AM, Jamo Luhrsen <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> Miro, >>> >>> both of the jobs you pointed to are using the same VM to run OVS. >>> "ubuntu-trusty-mininet-2c-2g" >>> >>> is there something about the jobs that would make ovs have a different >>> version? >>> >>> >>> As for using ovs 2.5, I recall hearing the plan would be to get all of the >>> openflowplugin csit >>> running 2.5+. I think it will create some issues that will need to be >>> debugged and fixed in our >>> CSIT. >>> >>> JamO >>> >>> >>> >>> On 10/03/2016 06:33 AM, Miroslav Macko wrote: >>>> Hi guys, >>>> >>>> >>>> Could you please check this two robot tests [1] and [2]? >>>> >>>> >>>> [1] is not failing >>>> >>>> [2] is failing - some flows are rejected by device ("match inconsistent"). >>>> >>>> >>>> It looks like that [1] is running on the OVS version 2.0 and [2] is >>>> probably running on some higher version of the OVS. >>>> >>>> >>>> When we tried to use OVS 2.4 for [1], it was also failing with "match >>>> inconsistent". >>>> >>>> When we tried to use OVS 2.0 for [2], it is also not failing. >>>> >>>> >>>> >>>> We have discussed it with Luis already, and he would like to use OVS >>>> version 2.5.0 for all tests. >>>> >>>> >>>> So it will be needed to update failing flows xml input files >>>> (/test/csit/variables/xmls). >>>> >>>> >>>> Thank you. >>>> >>>> Miro >>>> >>>> >>>> >>>> [1] >>>> https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-flow-services-only-carbon/ >>>> >>>> <https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-flow-services-only-carbon/> >>>> >>>> >>>> [pastedImage.png] >>>> >>>> >>>> [2] >>>> https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-flow-services-frs-only-carbon/ >>>> >>>> <https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-flow-services-frs-only-carbon/> >>>> >>>> >>>> >>>> [pastedImage.png] >>>> >>>> >>>> >>>> >>>> MiroslavMacko >>>> >>>> Software Developer >>>> >>>> >>>> Sídlo / Mlynské Nivy 56 / 821 05 Bratislava / Slovakia >>>> R&D centrum / Janka Kráľa 9 / 974 01 Banská Bystrica / Slovakia >>>> / [email protected] <mailto:[email protected]> >>>> reception: +421 2 206 65 114 / www.pantheon.sk <http://www.pantheon.sk/> >>>> >>>> logo >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> openflowplugin-dev mailing list >>>> [email protected] >>>> <mailto:[email protected]> >>>> https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev >>>> <https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev> >>>> >>> _______________________________________________ >>> integration-dev mailing list >>> [email protected] >>> <mailto:[email protected]> >>> https://lists.opendaylight.org/mailman/listinfo/integration-dev >>> <https://lists.opendaylight.org/mailman/listinfo/integration-dev> >> AndrejLeitner >> Software Developer >> >> Sídlo / Mlynské Nivy 56 / 821 05 Bratislava / Slovakia >> R&D centrum / Janka Kráľa 9 / 974 01 Banská Bystrica / Slovakia >> / [email protected] <mailto:[email protected]> >> reception: +421 2 206 65 114 / www.pantheon.sk <http://www.pantheon.sk/> >> >> > > MiroslavMacko > Software Developer > > Sídlo / Mlynské Nivy 56 / 821 05 Bratislava / Slovakia > R&D centrum / Janka Kráľa 9 / 974 01 Banská Bystrica / Slovakia > / [email protected] <mailto:[email protected]> > reception: +421 2 206 65 114 / www.pantheon.sk <http://www.pantheon.sk/> > > > <Flows_Additional_TCs-frm-OVS24.zip><Flows_Additional_TCs-frs-OVS20.zip>
_______________________________________________ openflowplugin-dev mailing list [email protected] https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev
