On 03/29/2017 06:13 AM, Diego Jesus Granados Lopez wrote: > Jamo, > > Thanks for the explanation. Still, this lack of branching in integration/test > repo is something very hard to understand to > me, especially when that fact is translating into the need of adding > branching intelligence into the testcases. When > branches are needed, it is version control where they belong. It's not only > that: in my experience, when you require from > testcases to be valid for different software tracks, at the end you're > required that the testcases are verified on all > those tracks, a requirement that is really hard on individuals contributing > new tests, because of how hard it is to change > tracks during development. As I said, I experienced a similar situation in my > organization, and this level of demand > translated into the dysfunction of people actively avoiding to contribute > tests (because of not having the time to verify > them in all tracks).
I understand these concerns. The conclusion before was that branches are not needed (except for a few corner cases). Either SFC is one of these corner cases or this is a problem starting to show up more often and we should think about branches again. My gut feeling it's the former. :( > > Also in the spirit of trying to improve things, , it was me who contributed > four of those testcases which are failing in > Boron (they support a correction which is meant only for Carbon), having no > clue that they would be run vs all active > tracks. It would be really nice if this "not trivial to anticipate" behavior > was easier to stumble upon in documentation > (at least I couldn't find anything about it; can totally be my fault), or at > least if this expectation was an explicit > comment in code reviews for people starting to contribute tests as I am > myself, at least when that branching-support logic > is clearly not present in the new/modified tests. This isn't the full answer, but I just put this patch: https://git.opendaylight.org/gerrit/54059 that should add a verify job to run when sfc does csit patches. Currently that job will be run with Boron. If that was there before, we probably would have noticed the failures there even as you are validating things on carbon in the sandbox. > I'm sure there is a background that lead to this decision of not branching > the integration/test repo, and I'd really > appreciate if you could point me to that info so I can understand if there > was a reasoning that I'm not getting Hopefully my last email to Ursicio made some sense. let me know if not. > Back to the original point, it is true that we have eight testcases that we > didn't notice were failing in Boron. As my > colleague Ursicio said, we'll be working to fix them, but in the meanwhile, > is it ok that we simply acknowledge the > failure as not blocking in the spreadsheet Colin shared? (I'll be doing it > right away) this is totally fine. This is actually more than some projects provide when we ask them to analyze their test failures. We all now fully understand what's going on, why the failures are there and that they are clearly not blocking anything. Please know that your contributions are really appreciated and the timely analysis and work is awesome. Thanks a ton. JamO > Best regards, Diego > > -----Original Message----- From: Jamo Luhrsen [mailto:[email protected]] > Sent: martes, 28 de marzo de 2017 23:24 To: > Colin Dixon <[email protected]>; Diego Jesus Granados Lopez > <[email protected]>; Ursicio Javier > Martin <[email protected]>; [email protected] > Subject: Re: [sfc-dev] SFC Boron-SR3 test > failure checkoff > > first thing is we should be noticing these things well before release people > are noticing them as we try to get an SR out > the door. I have a hunch this started happening before 2017 started. > > secondly, there is some confusion here: > > our test code comes from the integration/test repo and we only have a master > branch. There is no such thing as a boron > test branch. The git snippets you gave are from integration/test so it makes > sense that you only see master. > > our jobs have the branch of the *controller* in their name. So, for the job > we are looking at here > (sfc-csit-3node-rest-basic-all-boron), it is downloading a distribution .zip > made from the boron branch. It's not building > it, just downloading from nexus. > > SFC csit has dealt with this in the past where CSIT needs to be smart about > knowing which ODL branch it's testing against. > Other projects deal with this as well. It looks like SFC is now only assuming > master in it's suites: > > https://github.com/opendaylight/integration-test/blob/master/csit/suites/sfc/SFC_Basic/010__sfc_service_functions.robot#L116-L118 > > here's a patch for sfc csit that came in a few months ago playing in that > same area. > > https://git.opendaylight.org/gerrit/#/c/45819 > > integration-dev can help with reviews of patches that come in to get this > cleaned up. > > hope it helps, JamO > > > On 03/28/2017 06:27 AM, Colin Dixon wrote: >> Thanks for looking into this. We've heard other people say the same things >> too. >> >> --Colin >> >> On Tue, Mar 28, 2017 at 8:57 AM Diego Jesus Granados Lopez >> <[email protected] >> <mailto:[email protected]>> wrote: >> >> I'm adding more evidence. From one of the failed tasks [1]: >> >> GIT_URL ssh://[email protected]:29418/integration/test >> <http://[email protected]:29418/integration/test> >> GIT_BRANCH origin/master GIT_COMMIT >> 9f3c006279ca73a063f923e64a0c74aebf809a9d >> >> At execution logs: >> >> 04:52:41 > git fetch --tags --progress >> ssh://[email protected]:29418/integration/test >> <http://[email protected]:29418/integration/test> master >> 04:52:41 > git rev-parse FETCH_HEAD^{commit} >> # timeout=10 04:52:41 Checking out Revision >> 9f3c006279ca73a063f923e64a0c74aebf809a9d (origin/master) 04:52:41 > git >> config core.sparsecheckout # timeout=10 04:52:41 > git checkout -f >> 9f3c006279ca73a063f923e64a0c74aebf809a9d >> >> That commit being used (9f3c) is the carbon tip (not in boron). So the wrong >> version of integration/test is being used >> for running the test. >> >> I'm not sure of which project is in charge of that script / task >> configuration in order to forward this info to them. >> Is it releng/builder? >> >> [1] https://jenkins.opendaylight.org/releng/job/sfc-csit-3node-rest-basic- >> all-boron/334 >> >> -----Original Message----- From: Ursicio Javier Martin Sent: martes, 28 de >> marzo de 2017 12:55 To: Diego Jesus Granados >> Lopez <[email protected] >> <mailto:[email protected]>>; Jamo Luhrsen >> <[email protected] <mailto:[email protected]>>; >> [email protected] >> <mailto:[email protected]> Cc: Colin Dixon >> <[email protected] <mailto:[email protected]>> Subject: >> RE: [sfc-dev] SFC Boron-SR3 test failure checkoff >> >> Hi Jamo, >> >> After looking at the execution logs I agree with Diego, for some reason >> tests devoted to Carbon are being executed in >> Boron scope. >> >> Maybe a Releng issue ... >> >> BR, >> >> Ursicio >> >> -----Original Message----- From: [email protected] >> <mailto:[email protected]> >> [mailto:[email protected] >> <mailto:[email protected]>] On Behalf Of Diego >> Jesus Granados Lopez Sent: martes, 28 de marzo de 2017 10:29 To: Jamo >> Luhrsen <[email protected] >> <mailto:[email protected]>>; [email protected] >> <mailto:[email protected]> Cc: Colin Dixon >> <[email protected] <mailto:[email protected]>> Subject: Re: [sfc-dev] >> SFC Boron-SR3 test failure checkoff >> >> Hi, >> >> I checked the failing job and most (not sure if all, but probably) of the >> failing tests shouldn't be executing in boron >> checks. E.g. there are 4 tests consistently failing that shouldn't be >> executing in Boron (Service Path validation: >> added by me) they were merged to carbon only one month ago. >> >> Any chance the Jenkins job is misconfigured and using master branch instead >> of boron-something for running that jobs? I >> looked at the job config and it contains a "GERRIT_BRANCH" parameter set to >> "stable/boron", but as I said if it was >> being used, tests added to carbon only shouldn't be being run >> >> BR, Diego >> >> -----Original Message----- From: [email protected] >> <mailto:[email protected]> >> [mailto:[email protected] >> <mailto:[email protected]>] On Behalf Of Jamo >> Luhrsen Sent: lunes, 27 de marzo de 2017 22:57 To: >> [email protected] >> <mailto:[email protected]> Cc: Colin Dixon >> <[email protected] <mailto:[email protected]>> Subject: >> Re: [sfc-dev] SFC Boron-SR3 test failure checkoff >> >> SFC, >> >> I took a look at one of your failing jobs from the tracking sheet and it >> does have seemingly consistent failures now. >> But, the same job back in the SR2 timeframe was passing 100%. Maybe there is >> a regression? >> >> job from SR2-ish: >> >> >> https://logs.opendaylight.org/releng/jenkins092/sfc-csit-3node-rest-ba >> sic-all-boron/212/archives/log.html.gz >> >> >> Thanks, JamO >> >>> tl;dr Your project has test failures in the Boron-SR3 distribution tests. >>> Please look at them and mark them as OKAY in >>> the spreadsheet if they're not blocking or start a conversation with us >>> _now_ if they are. >>> >>> >>> Full details: The SFC project has at least one test failure in the >>> distribution test from the proposed Boron-SR3 >>> release here: >>> https://wiki.opendaylight.org/view/Simultaneous_Release:Boron_Release_ >>> Plan#Boron_SR3_Download >>> >>> To find the test failures, go to the distribution test report link at the >>> page above or directly here: >>> https://jenkins.opendaylight.org/releng/view/distribution/job/integrat >>> ion-distribution-test-boron/173/ >>> >>> Then look for yellow or red balls after tests that start with your >>> project's short name and click on the number to go >>> to the actual run that produced the failures. >>> >>> Please check to see if the failures are a blocking issue for releasing >>> Boron-SR3 and if so, start the discussion and >>> if not, update the spreadsheet here with OK in front of the issues: >>> https://docs.google.com/spreadsheets/d/1zImtd764e-hOgJAxoJKl85fxHCPu2a >>> gLfqsBtf13zQY/edit#gid=1776065453 >>> >>> Ideally, also follow up here or on this thread: >>> https://lists.opendaylight.org/pipermail/release/2017-March/009712.htm l >>> >>> Cheers, --Colin >> _______________________________________________ sfc-dev mailing list >> [email protected] >> <mailto:[email protected]> >> https://lists.opendaylight.org/mailman/listinfo/sfc-dev >> _______________________________________________ sfc-dev mailing list >> [email protected] >> <mailto:[email protected]> >> https://lists.opendaylight.org/mailman/listinfo/sfc-dev >> _______________________________________________ sfc-dev mailing list [email protected] https://lists.opendaylight.org/mailman/listinfo/sfc-dev
