On Tue, Sep 18, 2012 at 12:45 AM, Dawid Weiss <[email protected]> wrote:
> I think we can even integrate hossman's suggestion and generate a > stability report like weekly or something. > > I will take a look at this this week but it is definitely something that > will require everyone's consensus. > What would they add in addition to the test histories you can see on jenkins? > Dawid > > Sent from mobile phone. > On Sep 17, 2012 2:42 PM, "Michael McCandless" <[email protected]> > wrote: > >> I agree that a test that frequently fails, and does not get fixed, is >> nearly pointless: everybody ignores it so it's as if the test didn't >> exist. And so it should be disabled. >> >> I say *nearly* because the failures are in fact useful to devs who do >> have the itch/time to debug/fix them. >> >> So I think we need some middle ground here, where the tests keep >> failing but only those that are interested in the failures see the >> notifications. We need to switch from a "push" model (any failure is >> broadcast to everybody) to a "pull" model (those devs that want to >> debug the failures go and check the logs), for such tests. >> >> When someone wants to make sure their change didn't break something >> (Erick's original use case) then these tests should not run. >> >> I like Dawid's idea (a separate test plan that Jenkins runs with these >> "difficult" tests, and it wouldn't email dev on failure). >> >> Mike McCandless >> >> http://blog.mikemccandless.com >> >> On Mon, Sep 17, 2012 at 7:58 AM, Robert Muir <[email protected]> wrote: >> > On Sun, Sep 16, 2012 at 11:10 PM, Mark Miller <[email protected]> >> wrote: >> >> I get value from this test - if it was disabled, I'd probably >> re-enable it. >> >> would be great if it didn't fail so much, but the type of fail tells me >> >> something. >> > >> > That means the assert in question isnt important at all. I'll remove it. >> > >> > Again my problem is the idea that having a failing build is "ok" >> > because certain types of failures "don't matter". If they dont matter >> > they should be removed. >> > >> > It causes a ton of noise when people are lazy about tests in this way, >> > and it wastes a ton of peoples time. R >> > >> > Remember every time one of these tests fails it sends an email, that I >> > must read (we don't yet have a way to put in the subject header its a >> > SOLR test fail versus a LUCENE one, or i'd filter the solr ones and >> > not be complaining as much). >> > >> > -- >> > lucidworks.com >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: [email protected] >> > For additional commands, e-mail: [email protected] >> > >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> -- Chris Male | Open Source Search Developer | elasticsearch | www.e<http://www.dutchworks.nl> lasticsearch.com
