>From an IRC chat we devised a plan: 11:54 < vmassol> imo ensuring that all fialing tests are false positives is enough 11:54 <@cjdelisle> How long do you expect that to take? 11:54 < vmassol> dunno 36 failing tests, maybe 1 day max 11:54 < vmassol> you already did a few 11:54 < vmassol> so maybe 1/2 day from now on max 11:55 <@cjdelisle> There are 16 selenium 1 tests and 16 webstandards tests. 11:55 < vmassol> btw did hudson rebuilt the tests with the rollback? 11:55 <@cjdelisle> http://hudson.xwiki.org/job/xwiki-enterprise-tests/575/org.xwiki.enterprise$xwiki-enterprise-test-ui/#showFailuresLink 11:56 <@cjdelisle> Those 3 failing tests I can make pass repeatably as long as I run them in isolation. The tests themselves appear to be buggy. 11:56 <@cjdelisle> *I can make them fail repeatably by running all tests. 11:56 <@cjdelisle> I am repoting that on jira 11:57 < vmassol> cjdelisle: if you can reproduce then that's cool 11:57 < vmassol> I can work on the SectionTest one if you tell me how to rperoduce 11:57 <@cjdelisle> So are you proposing a fork of the tree today, stabilization tomorrow and release on monday? 11:58 <@cjdelisle> *today and tomorrow 11:58 < vmassol> could be a good idea
The plan is a hybrid of #1 and #2: A. Postpone the release day to Monday. B. Lower the requirement from fixing tests to knowing that the tests are not failing because of a bug. C. Branch the repository now so that new commits do not affect the stabilization effort. As of build #575 of xwiki-enterprise-tests we have 48 test failures. I am adding the list of failing tests to http://dev.xwiki.org/xwiki/bin/view/Community/ReleasePlans now. In order for this proposal to be acceptable: I need, by the end of today, a name next to each of these tests of a developer willing to fix the test, prove that it is not failing from a bug, or connect the failure to a known bug which we are willing to ship with M1. Thanks Caleb On 04/28/2011 11:16 AM, Vincent Massol wrote: > > On Apr 28, 2011, at 4:51 PM, Caleb James DeLisle wrote: > >> >> >> On 04/28/2011 10:25 AM, Vincent Massol wrote: >>> >>> On Apr 28, 2011, at 4:24 PM, Vincent Massol wrote: >>> >>>> >>>> On Apr 28, 2011, at 3:40 PM, Caleb James DeLisle wrote: >>>> >>>>> I offered to take over the release since Jerome was unable to do it and >>>>> right now we have 38 test >>>>> failures. >>>>> 16 selenium1 tests, >>>>> 6 selenium2/ui-tests >>>>> 16 webstandards tests. >>>>> >>>>> We have 3 options: >>>>> 1. We can release now and accept bugs in the release. >>>>> 2. We can have volunteers to take over tests and get them passing for >>>>> tomorrow so tomorrow I can >>>>> release. >>>>> 3. We can opt to postpone the release. If I work on these alone I expect >>>>> it to take about 1 week as >>>>> long as nobody commits code which introduces further regressions. >>>>> >>>>> I don't think postponing the release is wise at this point and #2 is >>>>> contingent upon volunteers >>>>> claiming ownership of specific tests so I think #1 is the lesser of the >>>>> evils. >>>>> >>>>> Do I hear any objections/volunteers? >>>> >>>> I fear 1 will make a precedent meaning that we'll consider it in the >>>> future too to do that, meaning that tests will have less and less values. >>>> Right now they don't seem to have any value since apparently nobody cares >>>> about them since we keep having issues when doing releases whereas they >>>> should just all run all the time. Normally when someone commits something >>>> and it fails the tests, that person must fix the code/test and not wait >>>> till the release happens. So we need to fix this somehow. Ideas anyone? >> >> If I am the release manager then I will propose a lock period when nobody >> commits anything except >> stabilization code for a week. > > Actually there's a better way which we have proposed during the git > discussion and explained on > http://nvie.com/posts/a-successful-git-branching-model/ : > > You always perform a release on a branch and this becomes the stabilization > branch of the release. Nobody commits on that branch except for helping in > stabilizing the branch. > > Now generally, I'd also be in favor of rolling back commits too whenever a > commit breaks a functional test for more than 1 day. > > Thanks > -Vincent > >> During that time I will sort out what changes broke tests and if the >> committer is unwilling to fix them then revert their patches. If we need >> quality then we need a >> policy with teeth. >> >> Caleb >> >>>> >>>> So I'm fine with 1 but only provided it's decided in conjunction with a >>>> plan to fix the tests as otherwise we'll just have a doubling of failing >>>> tests for the next 3.1M2 release.... >>> >>> Actually what would be good is also to verify manually that the tests are >>> not real issues because if they are we shouldn't release or if we do >>> release then it's because we've considered the bugs to be non blocking and >>> they'll need to be detailed in the release notes as regressions. >>> >>> Thanks >>> -Vincent >>> >>>> >>>> More precisely we need to answer: >>>> * Who's going to fix the currently failing tests? >>>> * What strategy to put in place so that failing tests don't creep in for >>>> more than, say, 1 full day? >>>> >>>> Thanks >>>> -Vincent > _______________________________________________ > devs mailing list > [email protected] > http://lists.xwiki.org/mailman/listinfo/devs > _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
