On 06/02/2011 12:35 AM, Vincent Massol wrote: > > On Jun 1, 2011, at 6:30 PM, Marius Dumitru Florea wrote: > >> On 05/31/2011 05:54 PM, Vincent Massol wrote: >>> Hi devs, >>> >>> Here's the situation: >>> >>> * We've been having a hard time releasing on time and the main issue is >>> test stability. We lag by at least a week and we even release with failing >>> tests, causing regressions. >>> * It's not the role of the release manager to fix tests before releasing >>> * It's not normal that some people spend time fixing issues caused by >>> others and that others continue to work on the next thing they are working >>> on. Everyone needs to help. >>> >>> Here's what I propose as a drastic and temporary measure till we get better: >>> >>> 1) It's forbidden to commit anything till all tests are passing (unit AND >>> functional tests), unless what is committed is about fixing tests. On the >>> exceptional case when a committer absolutely needs to commit even though >>> tests are failing he needs to ask the permission explicitly. >>> 2) When tests are failing, everyone should stop what they're doing and help >>> stabilize again. We synchronize on IRC. >> >>> 3) Flickering tests can be marked as @Ignore and a jira issue created to >>> stabilize the build. >> >> Does this apply to Selenium 1 tests? >> >>> 4) Release Manager creates a release branch 1 week before the release to >>> let everyone stabilize the build >> >> I think (1) and (2) are hard to achieve because tests take too long to >> run. CI is at least 2h behind the latest commit. What are we supposed to >> do while CI runs the tests? I think we should fix the real problem now: >> add more agents and review the tests to see why they take so long to >> run. I'm willing to spend time to make the WYSIWYG tests run faster. >
> BTW on this topic XWiki SAS has purchased some new build server today so that > we can have several new agents on it (it should be able to run 4 agents). > I'll let you know once it's set up. That's great news! Should we set up a Selenium Grid ( http://selenium-grid.seleniumhq.org/ )? Thanks, Marius > > Thanks > -Vincent > >> >> So +0. >> >> Thanks, >> Marius >> >>> >>> On a long term we need to work on improving our CI so that functional tests >>> are built faster. One idea is: more agents and functional tests spread on >>> several agents. >>> >>> Here's my +1 to apply this now for master (3.2-SNAPSHOT leading to 3.2M1), >>> which means not committing anything more till we have all functional tests >>> passing. >>> >>> Thanks >>> -Vincent > > _______________________________________________ > devs mailing list > [email protected] > http://lists.xwiki.org/mailman/listinfo/devs _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
