Hi Jamo, Here you have the proposal in integration/test to overcome this Boron issue:
https://git.opendaylight.org/gerrit/#/c/54034/ BR, Ursicio -----Original Message----- From: Diego Jesus Granados Lopez Sent: miércoles, 29 de marzo de 2017 15:13 To: Jamo Luhrsen <[email protected]>; Colin Dixon <[email protected]>; Ursicio Javier Martin <[email protected]>; [email protected] Subject: RE: [sfc-dev] SFC Boron-SR3 test failure checkoff Jamo, Thanks for the explanation. Still, this lack of branching in integration/test repo is something very hard to understand to me, especially when that fact is translating into the need of adding branching intelligence into the testcases. When branches are needed, it is version control where they belong. It's not only that: in my experience, when you require from testcases to be valid for different software tracks, at the end you're required that the testcases are verified on all those tracks, a requirement that is really hard on individuals contributing new tests, because of how hard it is to change tracks during development. As I said, I experienced a similar situation in my organization, and this level of demand translated into the dysfunction of people actively avoiding to contribute tests (because of not having the time to verify them in all tracks). Also in the spirit of trying to improve things, , it was me who contributed four of those testcases which are failing in Boron (they support a correction which is meant only for Carbon), having no clue that they would be run vs all active tracks. It would be really nice if this "not trivial to anticipate" behavior was easier to stumble upon in documentation (at least I couldn't find anything about it; can totally be my fault), or at least if this expectation was an explicit comment in code reviews for people starting to contribute tests as I am myself, at least when that branching-support logic is clearly not present in the new/modified tests. I'm sure there is a background that lead to this decision of not branching the integration/test repo, and I'd really appreciate if you could point me to that info so I can understand if there was a reasoning that I'm not getting Back to the original point, it is true that we have eight testcases that we didn't notice were failing in Boron. As my colleague Ursicio said, we'll be working to fix them, but in the meanwhile, is it ok that we simply acknowledge the failure as not blocking in the spreadsheet Colin shared? (I'll be doing it right away) Best regards, Diego -----Original Message----- From: Jamo Luhrsen [mailto:[email protected]] Sent: martes, 28 de marzo de 2017 23:24 To: Colin Dixon <[email protected]>; Diego Jesus Granados Lopez <[email protected]>; Ursicio Javier Martin <[email protected]>; [email protected] Subject: Re: [sfc-dev] SFC Boron-SR3 test failure checkoff first thing is we should be noticing these things well before release people are noticing them as we try to get an SR out the door. I have a hunch this started happening before 2017 started. secondly, there is some confusion here: our test code comes from the integration/test repo and we only have a master branch. There is no such thing as a boron test branch. The git snippets you gave are from integration/test so it makes sense that you only see master. our jobs have the branch of the *controller* in their name. So, for the job we are looking at here (sfc-csit-3node-rest-basic-all-boron), it is downloading a distribution .zip made from the boron branch. It's not building it, just downloading from nexus. SFC csit has dealt with this in the past where CSIT needs to be smart about knowing which ODL branch it's testing against. Other projects deal with this as well. It looks like SFC is now only assuming master in it's suites: https://github.com/opendaylight/integration-test/blob/master/csit/suites/sfc/SFC_Basic/010__sfc_service_functions.robot#L116-L118 here's a patch for sfc csit that came in a few months ago playing in that same area. https://git.opendaylight.org/gerrit/#/c/45819 integration-dev can help with reviews of patches that come in to get this cleaned up. hope it helps, JamO On 03/28/2017 06:27 AM, Colin Dixon wrote: > Thanks for looking into this. We've heard other people say the same things > too. > > --Colin > > On Tue, Mar 28, 2017 at 8:57 AM Diego Jesus Granados Lopez > <[email protected] > <mailto:[email protected]>> wrote: > > I'm adding more evidence. From one of the failed tasks [1]: > > GIT_URL ssh://[email protected]:29418/integration/test > <http://[email protected]:29418/integration/test> > GIT_BRANCH origin/master > GIT_COMMIT 9f3c006279ca73a063f923e64a0c74aebf809a9d > > At execution logs: > > 04:52:41 > git fetch --tags --progress > ssh://[email protected]:29418/integration/test > <http://[email protected]:29418/integration/test> master > 04:52:41 > git rev-parse FETCH_HEAD^{commit} # timeout=10 > 04:52:41 Checking out Revision 9f3c006279ca73a063f923e64a0c74aebf809a9d > (origin/master) > 04:52:41 > git config core.sparsecheckout # timeout=10 > 04:52:41 > git checkout -f > 9f3c006279ca73a063f923e64a0c74aebf809a9d > > That commit being used (9f3c) is the carbon tip (not in boron). So the > wrong version of integration/test is being used > for running the test. > > I'm not sure of which project is in charge of that script / task > configuration in order to forward this info to them. Is > it releng/builder? > > [1] > https://jenkins.opendaylight.org/releng/job/sfc-csit-3node-rest-basic- > all-boron/334 > > -----Original Message----- > From: Ursicio Javier Martin > Sent: martes, 28 de marzo de 2017 12:55 > To: Diego Jesus Granados Lopez <[email protected] > <mailto:[email protected]>>; Jamo Luhrsen > <[email protected] <mailto:[email protected]>>; > [email protected] <mailto:[email protected]> > Cc: Colin Dixon <[email protected] <mailto:[email protected]>> > Subject: RE: [sfc-dev] SFC Boron-SR3 test failure checkoff > > Hi Jamo, > > After looking at the execution logs I agree with Diego, for some reason > tests devoted to Carbon are being executed in > Boron scope. > > Maybe a Releng issue ... > > BR, > > Ursicio > > -----Original Message----- > From: [email protected] > <mailto:[email protected]> > [mailto:[email protected] > <mailto:[email protected]>] On Behalf Of Diego Jesus > Granados Lopez > Sent: martes, 28 de marzo de 2017 10:29 > To: Jamo Luhrsen <[email protected] <mailto:[email protected]>>; > [email protected] > <mailto:[email protected]> > Cc: Colin Dixon <[email protected] <mailto:[email protected]>> > Subject: Re: [sfc-dev] SFC Boron-SR3 test failure checkoff > > Hi, > > I checked the failing job and most (not sure if all, but probably) of the > failing tests shouldn't be executing in boron > checks. E.g. there are 4 tests consistently failing that shouldn't be > executing in Boron (Service Path validation: added > by me) they were merged to carbon only one month ago. > > Any chance the Jenkins job is misconfigured and using master branch > instead of boron-something for running that jobs? I > looked at the job config and it contains a "GERRIT_BRANCH" parameter set > to "stable/boron", but as I said if it was being > used, tests added to carbon only shouldn't be being run > > BR, > Diego > > -----Original Message----- > From: [email protected] > <mailto:[email protected]> > [mailto:[email protected] > <mailto:[email protected]>] On Behalf Of Jamo Luhrsen > Sent: lunes, 27 de marzo de 2017 22:57 > To: [email protected] <mailto:[email protected]> > Cc: Colin Dixon <[email protected] <mailto:[email protected]>> > Subject: Re: [sfc-dev] SFC Boron-SR3 test failure checkoff > > SFC, > > I took a look at one of your failing jobs from the tracking sheet and it > does have seemingly consistent failures now. > But, the same job back in the SR2 timeframe was passing 100%. Maybe there > is a regression? > > job from SR2-ish: > > > https://logs.opendaylight.org/releng/jenkins092/sfc-csit-3node-rest-ba > sic-all-boron/212/archives/log.html.gz > > > Thanks, > JamO > > > tl;dr > > Your project has test failures in the Boron-SR3 distribution tests. > > Please look at them and mark them as OKAY in the spreadsheet if > > they're not blocking or start a conversation with us _now_ if they are. > > > > > > Full details: > > The SFC project has at least one test failure in the distribution test > > from the proposed Boron-SR3 release here: > > https://wiki.opendaylight.org/view/Simultaneous_Release:Boron_Release_ > > Plan#Boron_SR3_Download > > > > To find the test failures, go to the distribution test report link at > > the page above or directly here: > > https://jenkins.opendaylight.org/releng/view/distribution/job/integrat > > ion-distribution-test-boron/173/ > > > > Then look for yellow or red balls after tests that start with your > > project's short name and click on the number to go to the actual run > > that produced the failures. > > > > Please check to see if the failures are a blocking issue for releasing > > Boron-SR3 and if so, start the discussion and if not, update the > > spreadsheet here with OK in front of the issues: > > https://docs.google.com/spreadsheets/d/1zImtd764e-hOgJAxoJKl85fxHCPu2a > > gLfqsBtf13zQY/edit#gid=1776065453 > > > > Ideally, also follow up here or on this thread: > > https://lists.opendaylight.org/pipermail/release/2017-March/009712.htm > > l > > > > Cheers, > > --Colin > _______________________________________________ > sfc-dev mailing list > [email protected] <mailto:[email protected]> > https://lists.opendaylight.org/mailman/listinfo/sfc-dev > _______________________________________________ > sfc-dev mailing list > [email protected] <mailto:[email protected]> > https://lists.opendaylight.org/mailman/listinfo/sfc-dev > _______________________________________________ sfc-dev mailing list [email protected] https://lists.opendaylight.org/mailman/listinfo/sfc-dev
