Re: Annotation for "run this test, but don't fail build if it fails" ?

2012-05-13 Thread Dawid Weiss
> jenkins "run all tests continuously" build target that only runs the > @AwaitsFix group, and overrides haltonfailure when calling the junit > macro. (which would save us an extra jenkins run) Yep, doable. > decreased the likelyhood of failure, but didn't completely fix the > problem -- a test m

Re: Annotation for "run this test, but don't fail build if it fails" ?

2012-05-12 Thread Chris Hostetter
: This is doable by enabling/disabling test groups. A new build plan : would need to be created that would do: : : ant -Dtests.haltonfailure=false -Dtests.awaitsfix=true : -Dtests.unstable=true test right ...that's an idea that came up the other day when i was talking to simon at revolution an

Re: Annotation for "run this test, but don't fail build if it fails" ?

2012-05-09 Thread Dawid Weiss
> That was really the main question i had, as someone not very familiar with > the internals of JUnit, is wether it was possible for our test runner to > make the ultimate decision about the success/fail status of the entire > run based on the annotations of the tests that fail/succed There are tw

Re: Annotation for "run this test, but don't fail build if it fails" ?

2012-05-07 Thread Chris Hostetter
: as a build artifact). Yet another problem is that jenkins wouldn't : _fail_ on such pseudo-failures because the set of JUnit statuses is : not extensible (it'd be something like FAILED+IGNORE) so we'd need to That was really the main question i had, as someone not very familiar with the intern

Re: Annotation for "run this test, but don't fail build if it fails" ?

2012-05-06 Thread Yonik Seeley
On Sun, May 6, 2012 at 3:38 PM, Dawid Weiss wrote: > I also admit I've never seen anything like > this -- a suite of tests with an allowed failure ratio over time and a > threshold that would trigger a warning... Not so much an "allowed" failure rate... more of "it fails sometimes and no one has

Re: Annotation for "run this test, but don't fail build if it fails" ?

2012-05-06 Thread Dawid Weiss
> If the tests are run many times a day, it would be great to get a > daily report of the percent of time the tests pass.  Then if it goes > from 5% to 50%, we can go uh-oh... Yeah, well... but this is beyond the runner as it aggregates over time -- it looks like a jenkins plugin that would analyz

Re: Annotation for "run this test, but don't fail build if it fails" ?

2012-05-06 Thread Yonik Seeley
On Sun, May 6, 2012 at 2:39 PM, Dawid Weiss wrote: > Any ideas? Hoss -- how do you envision "monitoring" of these tests? Manually? If the tests are run many times a day, it would be great to get a daily report of the percent of time the tests pass. Then if it goes from 5% to 50%, we can go uh-oh

Re: Annotation for "run this test, but don't fail build if it fails" ?

2012-05-06 Thread Dawid Weiss
So, I started thinking about it -- I can implement something that will report failures (much like we do right now) it's quite tricky to fit it into the reporting system and continuous integration system. Here's why -- if a test doesn't fail then its output (sysout/syserrs) are not currently printed

Re: Annotation for "run this test, but don't fail build if it fails" ?

2012-05-05 Thread Dawid Weiss
One other way to do it that is already implemented is to run full tests without failing on failures and only touch some marker file to fail at the end. Ant test-help gives a hint on how to run tests this way currently. Dont know how it'd play with others - speak up. On May 5, 2012 11:25 AM, "Mikhai

Re: Annotation for "run this test, but don't fail build if it fails" ?

2012-05-05 Thread Mikhail Khludnev
It's a pretty useful technique especially from CI perspective. I use it via JUnit's assumptions. Failed assumption is shown as an ignored test. On Sat, May 5, 2012 at 2:29 AM, Chris Hostetter wrote: > > Dawid: > > With the new test runner you created, would it be possible to setup an > annotation

Re: Annotation for "run this test, but don't fail build if it fails" ?

2012-05-05 Thread Dawid Weiss
Not a problem. Ill be at work on monday. Can you file an issue an assign it to me please? Im currently on mobile only. Dawid On May 5, 2012 5:08 AM, "Yonik Seeley" wrote: > On Fri, May 4, 2012 at 6:29 PM, Chris Hostetter > wrote: > > > > Dawid: > > > > With the new test runner you created, would

Re: Annotation for "run this test, but don't fail build if it fails" ?

2012-05-04 Thread Yonik Seeley
On Fri, May 4, 2012 at 6:29 PM, Chris Hostetter wrote: > > Dawid: > > With the new test runner you created, would it be possible to setup an > annotation that we could use instead to indicate that a test should in fact > be run, and if it fails, include the failure info in the build report, but >

Annotation for "run this test, but don't fail build if it fails" ?

2012-05-04 Thread Chris Hostetter
Dawid: With the new test runner you created, would it be possible to setup an annotation that we could use instead to indicate that a test should in fact be run, and if it fails, include the failure info in the build report, but do not fail the build? I'm thinking in particular about some o