I would agree if the jenkins we have would be stable; i have the feeling that a lot of build failures on that instance are simply because of some problems of the build server itself. I tried to chased too many times such issues. For example, right now, every day we get build errors reported in some modules while at the next day they pass - without any changes at all.
So I think the first step is to have a stable build env But I'm fine with any direction as long as we fix known bugs before a release Carsten 2013/6/4 Jeff Young <[email protected]> > I used to agree as well, but my opinion is now more nuanced. I've > experienced projects where a test keeps failing day after day, and after a > while developers stop looking at the test results with the same level of > discipline. > > Perhaps Sling is small enough (and the developers are pro-active enough) > that this isn't an issue. But it certainly is on some other, larger, more > disperse projects (such as CQ). In those, moving a failing test to an > issue (which can be assigned to an individual) can produce better results > than everyone simply getting used to the build being red. > > Cheers, > Jeff. > > > > -----Original Message----- > > From: Carsten Ziegeler [mailto:[email protected]] > > Sent: 03 June 2013 07:01 > > To: [email protected] > > Subject: Re: Disabling flaky tests > > > > I agree as well, especially for the error handling as this is partially > not > > a problem of the test but really a bug in Sling - we have an issue for > > that, it just needs to be done :) > > > > Carsten > > > > > > 2013/6/3 Felix Meschberger <[email protected]> > > > > > I agree here: Disabling the test and having an issue keeps the build > green > > > but bears the danger of forgetting about it ... > > > > > > Regards > > > Felix > > > > > > Am 02.06.2013 um 16:04 schrieb Eric Norman: > > > > > > > Personally, I'm not a big fan of hiding flaky/failing tests since it > > > tends > > > > to remove some of the motivation to stabilize/fix them in a timely > > > manner. > > > > > > > > That's my 2 cents. > > > > > > > > Regards, > > > > Eric > > > > > > > > On Fri, May 31, 2013 at 12:14 PM, Robert Munteanu < > [email protected] > > > >wrote: > > > > > > > >> Hi, > > > >> > > > >> It seems that the ErrorHandlingTest fails sporadically when run > inside a > > > >> full maven build. I've tried locating the root cause for a couple of > > > >> hours but failed. For this test, and for future flaky/failing > tests, I > > > >> suggest that we > > > >> > > > >> 1. Create an issue for the failing test > > > >> 2. Disable the test and mark it with the issue key > > > >> 3. Re-enable the test when it is stable/passing ( which may be > > > >> considerably later than step 2) > > > >> 4. Close the issue after the test is re-enabled > > > >> > > > >> This has the advantage of keeping the build green and making it > easier > > > >> to find regressions since a failing or unstable build will actually > mean > > > >> something. > > > >> > > > >> What do you think? > > > >> > > > >> Robert > > > >> > > > >> > > > > > > > > > > > > -- > > Carsten Ziegeler > > [email protected] > -- Carsten Ziegeler [email protected]
